« first day (1 day earlier)   

12:17 AM
@uhoh you tell me
@glad to chat with you offline after knowing you the master of questions on SO
and SE
nobody qute asks a question like you
How is life in the country of semiconductors and typhoons ?
You know we put a satellite on the south pole of the moon
My name is Ashwin by the way and you are David right ? :-)
@uhoh more than talking python just good to talk with you !!
about life in general
@AndrejKesely Could you please leave this room ?
This is a private room sorry about that
@uhoh If our intruder does not leave we will create a new room one in which only the two of us will be present
i dont know how to remove him
i dont think we can remove him
i made a mistake by creating this room with public access
@uhoh he has left the room
room mode changed to Gallery: anyone may enter, but only approved users can talk
 
 
1 hour later…
2:05 AM
@uhoh you are here !
 
2:23 AM
@uhoh that was quick . you came you saw and you left.
 
 
2 hours later…
3:58 AM
@uhoh I will be in and out of the room too
 
I can now write here. I'm not very conversational or a chat-er, can you describe the vectorization situation you'd like to post a question about?
 
 
2 hours later…
6:25 AM
@uhoh got it. Let;'s get straight to the question then
I have this quadruple for loop
for i in range(self.x_ll_subset, self.x_ur_subset):
for j in range(self.y_ur_subset, self.y_ll_subset):
# Computing the contribution to each other grid point (masked area).
# x1 and y1 refer to x' and y' in the Oertel and Schemm (2021) equations.
for x1 in range(self.x_ll, self.x_ur):
for y1 in range(self.y_ur, self.y_ll):
outer_point = [i,j]
inner_point = [x1,y1]
if inner_point != outer_point:

# Compute x-x', y-y', and r^2...
xdiff = (i-x1)*self.dx[y1,x1].magnitude
ydiff = (j-y1)*self.dy[y1,x1].magnitude
The IF statement prevents the vectorization of the for loops
My thought was to eliminate the IF statement completely and then perhaps achieve a 2d vectorization.
My own thought process was converting this into a 4d array would be too much. Keep it a 2d array and then convert the inner 2 loops into a vectorization stateement
Does that look as a possible question - keep it as a 2d array and eliminate the 2 innter loops versus getting rid of all the 4 loops and doing a full vectorization ?
By the way never formatted code on a chat room
` for i in range(self.x_ll_subset, self.x_ur_subset):
for j in range(self.y_ur_subset, self.y_ll_subset):
# Computing the contribution to each other grid point (masked area).
# x1 and y1 refer to x' and y' in the Oertel and Schemm (2021) equations.
for x1 in range(self.x_ll, self.x_ur):
for y1 in range(self.y_ur, self.y_ll):
outer_point = [i,j]
inner_point = [x1,y1]
if inner_point != outer_point:

# Compute x-x', y-y', and r^2...
xdiff = (i-x1)*self.dx[y1,x1].magnitude
ydiff = (j-y1)*self.dy[y1,x1].magnitude
 
 
1 hour later…
7:56 AM
OK I have copied what you've put here to my computer, and I'll take a look and let you know what occurs to me. Thanks!
 
@uhoh thanks
I can show you how i did the vectorization too
iindex = np.zeros((x,y))
jindex = np.zeros((x,y))
for i in range(x_ll_subset, x_ur_subset):

for j in range(y_ur_subset, y_ll_subset):

iindex[:,:] = i
jindex[:,:] = j
xdiff = (iindex[x_ll_subset:x_ur_subset,y_ur_subset:y_ll_subset]-xindex[y_ur_subset:y_ll_subset,x_ll_subset:x_ur_subset])*dx[y_ur_subset:y_ll_subset,x_ll_subset:x_ur_subset]
ydiff = (jindex[x_ll_subset:x_ur_subset,y_ur_subset:y_ll_subset]-yindex[y_ur_subset:y_ll_subset,x_ll_subset:x_ur_subset])*dy[y_ur_subset:y_ll_subset,x_ll_subset:x_ur_subset]
In this I have elimianted 2 of the 4 loops and also elimniated the IF statement
 
well I'm not a "vectorizer" I've always found a way to get what I need done using straight NumPy and can speed it up with @jit (numba.pydata.org)
 
Mine is straight numpy too
my only question is whether a 2d vectorization is more realistic than a 4d vectorization
 
There's nothing wrong with a 4D array in numpy, and there are ways to implement element by element conditional operations by tricking numpy using either masked arrays or booleans
 
ok what i will do then is post a question on SO or code review asking whether 2d is better than 4d or vice versa !
thanks your help
 
8:02 AM
what does "realistic" mean and why don't you just try both and get the answer yourself? What is the specific problem - the specific question to which you are unable to get an answer?
"Which is better" questions will be insta-closed.
nobody knows what your "better" means.
 
exactly that is what i was hoping to clarify as a question
 
and without a specific example it's a hypothetical opinion quesiton
 
ok let me formulate the question with some realistic data and post a question on SO thanks for your help @uhoh
 
you need to write a short piece of script that RUNs and then time it.
 
what is your expereince with 4d arrays ?
 
8:04 AM
Code Review is excellent! I love that site! But you must submit running code that works and does what you want
I use them all the time!
 
my thought was i wasnt sure where my question would fit
SO or code review
because my code works
there is nothing to answer except which way is better
 
NumPy is written in Fortran, and they are stored in memory as a single long array. The shape of your python array (how many dimensions etc) does not actually exist in memory. Python creates that illusion for you
 
are there any python discussion forums you use for opinionated questions ?
 
I let Stack Exchange teach me not to ask opinionated questions
Let me ask you, what does your "better" mean exactly?
 
a) which takes less memory
b) is readable
to me it sounds like a code review question than SO which is about i am stuck and please help me
or perhaps even software engineering SE
 
8:08 AM
memory has several issues. 1) Does it exceed your computer's RAM so it must use virtual memory (actually swapping memory to hard disk) 2) have you written it so that the L2 cache on-chip memory has to constantly swap with the RAM?
3) Have you tried @jit?
 
@uhoh i cannot jit because i am using a product and i cannot force my clients to install it
 
Now that is a very important point to put within the first sentence of your question!!!
 
aah ok apologies then
 
That totally changes the answer, it's a constraint that people will need to know about
 
completely
so where do i fit in ? So or code review ?
 
8:10 AM
Could the customer's data sets get so big that they won't all fit in RAM at the same time?
SO is for very very specific questions, you definitely have a nice Code Review question.
 
yes verry likely. ERA5 which is the European forecast model uses high resolution data and that could be gigs
that is what i thought code review would be best
 
Great, now you know what your first two sentences are!
Yes code review is it.
Is the data numerical or mixed with text and other things?
 
purely numerical we are in meteorology
cyclone data
like the one that hit Libya last week
 
OK so does your short program that you're using as an example require realistic data, or can you run it on random numbers?
 
defnitely realistic data
and that is the challenge. for SO question most use random numbers
i cannot !
 
8:13 AM
If an answer author can generate a big database locally using random numbers, they don't have to go somewhere to download a data file. Can you provide a link to realistic data?
 
the code sample will download meteorological data
see we dont have python meteorological groups to discuss these sort of questions
that is why we have to come to SO
 
Great! OK then be sure to let the readers know the volume of data that will be downloaded so they can decide if/when they can do it. It would be great if you offered two alternaties, a big set and a small set
 
is 50 mb a lot ?
 
no
 
then i thank you for your help !
and i wish you a good day
 
8:16 AM
go ahead and ping this person also when you post - they deal with satellite data and I'm pretty sure are decent with Python earthscience.stackexchange.com/users/6/gerrit
just say that uhoh said they might have something to offer
 
oh yea i know gerrit very well
but ESSE gets low traction with python questions
ultimately i have to formulate this for code review
 
ping gerrit with a link to your Code Review question and say uhoh thought he might find it interesrting
or I'll do it when I get your ping
I'll do it so it doesn't feel weird
 
first let me write it out in a way that Code review will find it reasonable
 
yep, have fun!
 
have you used green's function in python ?
my code is doing that
one way to write it would be to say i have a meteorological version of green''s function
can you guide me on which optimization is better
 
8:21 AM
sadly I could never wrap my head around Green's functions, I drank too much beer when I was in school
 
aah ok
 
Well, in THAT case you may ALSO have a question for scicomp.stackexchange.com
 
i thought about that one too
between scicomp and code review i need to sit down and post a reasonable question
 
Ask separate questions. Put a link to each question in the other question. In SciComp ask about algorithm alternatives, in Code Review ask about speed and memory optimization.
 
irs hard i will tell you why because you need to blank out the meteorological context
 
8:23 AM
But if you don't think those two are separable, then you'll have to choose. I'd say ask separate questions and learn from both
 
if there were specific python meteorological groups then a lot easier
 
Where do you need to "blank out the meteorological context"? I disagree!
Answer authors in both sites will want to know EXACTLY what you're trying to do!
 
my code is about 20 lines ok ?
in that i have to explain the context i.e. green''s function
and it's relevance to meteorology
 
I think 100 lines or more is still okay
 
then ask a question on 2d arrays vs 4d arrays
 
8:26 AM
In both sites the readers are programmers!
 
isnt there a limit on how many lines of code you can ask in a question ?
i think its 20 lines or so
 
I've never, ever seen a complaint about length.
 
its been raised on GIS
and SO too on meta
 
The issue of 2D vs 4D is not the number of dimensions, it's the ORDER in which you acces the data.
 
and whether it will scale or not
 
8:28 AM
In other words, an array with a shape of (10000) and one that's (10, 10, 10, 10) are both stored in memory as a single row 10000 long.
The (10, 10, 10, 10) array just has a few special instructions telling the FORTRAN how to jump around in it.
 
yes
 
You want to make sure your inner loop is looping over the shortest distance, i.e. the last dimension
 
i have already eliminated 2 loops from a quadruple for loop
non trivial
 
But the number of dimensions by itself is irrelevant.
 
the next question should i eliminate the outer for loops too
 
8:30 AM
If you loop over the longer distances in the first dimension, you may slow down because the CPU has to keep swapping data from on-chip L2 cache memory and the computer's main RAM.
The first comment will ask you if you have tried it both ways and how the performance differs.
Everyone will wonder why you haven't just done that test by yourself.
 
yes i understand but i have not written the second part
is it worth it ?
i dont know
 
now even I am getting irritated.
 
ok then we close the conversation
thanks for your help :-)
 
Just do it!!!!!!
 
all right @uhoh bye
 
8:32 AM
"Which is better, I'm not sure which one to try" questions will get downvoted and shut down and you'll get a bad reputation.
OK have fun, ping me when you've posted something
 
@uhoh i spent 2 weeks optimizing a non triivial quadriple for loop
2 weeks !!!
 
add that information into your question also, you can gain some sympathy and people will go easier on you
bye for now
 
bye
 
 
5 hours later…
1:54 PM
locking this room by room owner request
 
Ryan M has frozen this room.
 

« first day (1 day earlier)