last day (53 days later) » 

8:06 PM
5
Q: Why does my loop require more memory on each iteration?

IseaI am trying to reduce the memory requirements of my python 3 code. Right now each iteration of the for loop requires more memory than the last one. I wrote a small piece of code that has the same behaviour as my project: import numpy as np from multiprocessing import Pool from itertools import...

 
You should guard those main operations with if __name__ == '__main__': at least.
 
@snakecharmerb results[0][0] is just a float.
@Ilja Yes, good point. I do this in my real code but I figured for this example it wasn't required. I'll insert it (but it doesn't change the behaviour).
 
What do you btw exactly mean by "the threads (processes) each have a memory usage equal to the total memory used by my code"? You're not mixing forking's COW behaviour with actual mem usage perhaps? Which somewhat means "only copy the memory it requires to execute f()".
Been running the test script with larger and larger for loop iteration counts, but my memory consumption is pretty steady. Fluctuates quite a bit, but doesn't rise.
 
@Ilja I mean that, for example, if my code has 3% memory usage when it calls simulation(), each thread (as shown on htop) will start with 3% real memory (not virtual). Then on the next iteration my code will use more than 3% memory per thread because it also seems to copy the results and results_total array when creating new threads. The code doesn't seem to implement COW behaviour (if I understand it correctly) because each thread seems to take up actual physical memory the moment it starts execution.
@PadraicCunningham I check htop's RES column.
 
@Isea after your edit I can observe the behaviour you describe.
Still at it, one thing came to mind: example code is example code, but in this case the whole multiprocess mapping simulation step could be replaced with np.random.random((steps, 2)) * y[i - 1]. Have you considered vectorization of the actual problem more thoroughly?
 
8:06 PM
@Ilja Great to hear you're working on it! My real code involves constructing a matrix with the random values I generate and then calling np.linalg.eigh() on this matrix. Since the matrix needs to be generated for each set of random values I cannot use the solution you proposed.
 
I've managed to reduce the memory consumption a tad bit, but not find the reason of the whole phenomenon.
For example using the zip(range(steps), repeat(y)) with chunked pooling will create largish lists of (step, y) tuples. It's a bit easier on the memory to make a partial function from f like partial(f, y=y) and use that in the mapping
 
Hello, thank you for taking so much time to help me :)
Interesting, any optimisations are welcome. I run 32 threads in parallel so even just a small saving can have a big impact
 
btw when using multiprocessing they're full blown processes
but it does get around the GIL in python
 
exactly
it will create a partially applied version of the function where given arguments are bound
 
8:19 PM
yes, i need the the processes to run in parallel, my understanding is that the threading library doesn't allow execution on multiple CPU cores
so that's why i am using multiprocessing
 
sadly that is the case
 
I can explain the workings of my real code a bit more closely so that you can get an idea of what optimisation problems i'm facing, if you are interested
 
sure
might not be able to help much, but will give it a try
 
so as mentioned i'm using np.linalg.eigh() on matrices/arrays of dimension 8x8, they return 8 vectors of length 8 and 8 floats (eigenvectors and eigenvalues if you are familiar with that)
i need to save all these vectors and floats for 2^24 steps or more, per iteration
 
I know the terms, I've forgotten the rest :(
 
8:23 PM
then i need to iterate, my next iteration's matrix will vary based on the results of the previous iteration
so the results need to be kept in memory or stored to disk
 
I'd vote storing to disk slash memory with mmap
numpy has support for that
 
i tried saving the arrays with np.save(), it uses pickle
after that i would delete the array using del, but it did not clean memory even if i explicity called gc.collect()
 
memmapping will allow the OS to keep what it can in memory, map to disk what it can't (a simplified explanation)
and it happens transparently :)
 
ah i see, that sounds very useful
is it like swap memory, but file based?
 
somewhat yeah
it has some fancy stuff like sharing, protection and so on, but with numpy you don't need to know that
 
8:29 PM
it sounds like it could help, let me just check what my disk space allowance is on the computer that executes my code
it seems to be 8 TiB so that should be no problem. i have a question about the implementation though
if i use memmap will i have to use that to store my arrays to disk and then treat them as files, or do i just declare my arrays as "memmap arrays" and then they will automatically write the data to disk when memory is low?
 
You'd treat a memmap as numpy array that has file backing, so if the backing file is opened in suitable mode it should be able to "swap" back if mem is low
 
ah that makes sense!
 
With some edge cases that I don't remember right away
 
hello
 
hi
 
8:40 PM
hello!
 
did you make any progress?I don't have a PC to help you :-)
 
ilja had some suggestions, that might circumvent the problem
but nothing that fixes the issue outright
@ilja i will try turning total_results into a memmap backed array now
 
not quite circumvent, just reduce bits and pieces of mem usage
adding that total_results handling thingie btw was what started producing the mem usage growing
 
that's odd, because on my system it displayed the behaviour even before i added total_results. i just added it to better showcase that i need to store all my results
 
the 1. revision was pretty stable, even when I added simulation iterations
 
8:48 PM
i'm running the code with total_results commented right now and it increases memory consumption between iterations
 
That one seems a bit easier on the memory, though my measurement method is just watching top while it's running
 
mine is watching htop ;)
 
actually htop here too :P
so notable things: I preallocate the memmap for all results beforehand and just slice in data
so system doesn't have to realloc so much
the del total_results in the end is unnecessary
 
yeah i see
is there a reason for using with Pool...
instead of my implementation?
 
minor premature optimization, but felt that building and tearing down the processes for all iterations is bad style
there are valid reasons to do that though
freeing up resources held by the processes for example, if they should accumulate during their lifetime
but all in all almost always if you find yourself writing something.close(), you prolly should be using it as a context manager
 
9:05 PM
alright :)
i've been running your code, it is indeed less of a memory hog :D
 
or the spikes are too fast to spot with htop :P
 
:) i will adapt my code to use your improvements and then test it on the remote machine
shouldn't take long i think!
 
Btw when you talked about those 8x8 random matrices and having a lot of them
I'd still try and think if plain numpy would be able to create an array of matrices and apply the eigh function over
 
i'm open to the suggestion but wouldn't that just increase memory consumption because then you'd have all 8x8 matrices existing at the same time rather than only at the time when eigh() is doing its computation on the specific matrix
 
it would
 
9:20 PM
this is actually my first time posting on stack exchange and i have to say it's really a great website :) many helpful people!
 
people me included can be kind of a doushebags too...
there are very protective people, who defend the quality of the content
which is a good thing, but can be harsh
 
yeah, i spend a lot of time writing my question because i saw bad phrasings get shot down
*spent
 
it's also important to show that you've tried, since SO's not a coding service :P
and then there's that minimal working example and ...
lots of criteria
 
yeah, i'm a bit in over my head with this python project so i was worried my python code was just too obviously flawed to receive help
 
a middle ground between vectorization and running one by one
also: 0,66s vs. 150s+
 
9:27 PM
i get the code you wrote but i don't understand why it would be faster
 
because numpy is backed with very well optimized C
 
aren't you just using fsteps as a counter same way I did?
 
and true threading, if built with suitable options
 
i see
 
so usually vectorization is always a good thing
but running that eigh over those matrices cannot be vectorized, at least I don't know how
and yeah it's still being used as a kind of a counter
with the exception that this time we do the "chunking" manually, by splitting steps to smaller values of steps that get passed to f
and then f performs so many steps
 
9:30 PM
shouldn't that be equivalent to the chunksize parameter that i use?
 
instead of creating a huge list of integers [0, 1, 2, 3, 4, ..., 1048575], chunking that up for sending to worker processes as tasks and then funning f over each integer
that's the difference
 
ah i see!
 
the chunking parameter to pool says: "split this list to chunks of N, send those chunks to worker processes that then work on those chunks item at a time"
 
shouldn't i be able to implement that anyway, the eigh() function is called within f() so i can use your logic to call the worker processes
 
yep
just remember to then do N amount of work in f instead of just one matrix
and then the result of the map will be a list of lists of results
 
9:35 PM
what i'd have to do is just put a while loop in f(). then it would execute each chunk in serial
but if i understand it correctly it would still speed things up because f is called less often?
 
yes
...I think
and you can just for i in range(steps): inside f too
collect the results and return
 
right now i have a slightly related problem, my f returns a numpy array of shape (8, 5). then pool.map collects all the results as a list, so i have a list of numpy arrays
so i'm trying to convert it to a numpy array but right now that involves copying the entire list of arrays into a numpy array one entry at a time
so it's actually very memory inefficient and it stops me from using memmap on results directly because they need to be recast
 
hmm hmm hmm
 
(so in the example code, use np.random.random(()) as the return statement of f and you get an idea of what the problem is)
 
how small are your memory constraints btw
 
9:45 PM
256GB
 
I do use a 2 dimensional return value from f in the latest version :P
 
and 2TB of disk data
 
used np.vstack to collect the results of the pool.map
 
i didn't notice, let me check that
 
9:47 PM
(i could also request more nodes which would increase the available memory but python seems unable to make use of several nodes at the same time and i can't find any documentation on how to make it behave otherwise)
 
iirc multiprocessing can do that too with a bit of configuration, but perhaps we can do without
an array of 2**20 matrices of 8x8 is ~256MiB
that's not a lot of memory
I'd sacrifice that in a hart beat to save CPU time
 
in reality the processes is up at 2**27, but i decided to decrease it in the code example to not make people go out of memory if they tried executing the code :D
 
so 32G, that's a bit more then :P
 
yes :)
 
and sorry for going through stuff like that, just making sure and trying different angles
 
9:50 PM
they're also complex valued matrices so each entry is actually 2 floats
 
yay
 
so 8*8*2 is the actual dimension of the matrix i think
so then it's 64GB...
and please don't apologise. i never expected anyone to help me this much and i really appreciate you going through all the approaches
 
but you have 256G to burn, at least iterating through them 1 by 1 is a lot of heat from the CPU
and functions can be costly in this case I think
 
you mean serial execution instead of parallel?
 
also python reserves space for function frames from the heap. It does keep a kind of a free list, so I think in this mapping case it shouldn't reserve more and more
 
9:53 PM
that would extend the computation time beyond 5 days, which is the maximum allowed by the system
i see
 
no I mean that at the moment the original code was quite "serial"
it does split the work to 8 processes, but
it does a lot of function calling then
 
i see what you mean
 
to be honest I'm not that good with this kind of stuff, been a learning experience for me too
my colleague would be much better, but he might be sleeping already :P
 
you seem to know more than me :)
this is actually my first real coding project
 
heh, why start with small fish, eh? :D
 
9:57 PM
yeah it's my bachelor thesis (i'm a physics student) :D
 
a forever a student here too...
work life kind of swept me away from univ.
 
i want to do a masters in numerical physics, so i figured it would be a good idea to try a project in that before i apply to the masters degree, but the problem is i don't really have the experience for coding on this level
never too late go back :)!
 
indeed
 
*to go
 
but hmm hmm
so all in all: if you need very very large matrices and arrays, consider memmaps, avoid too much function calling, since that can get expensive too, ...
numpy is C, it is fast
do as much as you can with just numpy operations
in as big chunks as you can
 
10:00 PM
they're all very good tips, i'll probably implement them tomorrow as i'm getting quite tired
 
don't be afraid to burn memory, exchanging memory for CPU can save a lot of time
tired here too, 0:00
did not btw find info how those memmap objects behave if shared between processes
worst case scenario: the memory buffer is still copied
 
could i contact you again tomorrow perhaps?
 
yeah
 
i'll try implementing your changes and then we can see if that makes the program efficient enough, if not perhaps we could try to find some more optimisations
ok how would i go about that?
 
ah I meant that if you share a memmap object between those pool processes it might still do some nasty copying of memory buffers
ideally it won't, but ... did not find that info
 
10:04 PM
yeah, i will try it and see what happens
that's what i was worried about earlier too, even if i can store the total_results to disk then it might still copy the contents to RAM once it starts pool
 
yeah that might happen, but on the other hand it'd require that those pool processes actually use it, I think
COW again :)
(on linux etc.)
 
i would agree with you but this entire problem started with the fact that it seems to copy the entire environment to RAM straight away instead of using COW
i really didn't expect that behaviour either..
 
this time COW would apply just to the memmapped region
 
i see, i'll try it tomorrow and see what happens :)
 
Also not 100% sure about at least the example codes copying the entire process, there was clear increasing memory consumption, but that might be the python processes mucking about with numpy arrays, allocating memory, freeing, allocating, freeing and then the poor OS tries to keep up
I observed it to have a clear upwards trend with a lot of fluctuation
 
10:09 PM
i saw the threads all starting with the same memory as the parent process had just before the parallel processes began
 
ah that is actually a bit of a lie :P
 
so say python was consuming 3%, then right when the threads start they'd all have 3% memory usage
so that's why i figured it was copying everything in RAM
 
they all display the same mem usage, but actually it is the same memory
mostly
 
yeah, but isn't htop's RES column showing physical memory?
VIRT should show what you describe
 
it is, but but
 
again I don't remember and know all the facts, but I'm fairly certain that that RES can "lie" a bit when forking
 
wouldn't surprise me if it lied :)
i will go to bed now, if i want to contact you tomorrow will you be in this chat?
 
ah managed to make an example
>>> lots_of_stuff = np.ones((2**21,8,8,2),dtype='float32')
>>> p = Process(target=forever)
>>> p.start()
where forever is just a function looping forever
now, before launching I created that large array
saw my memory consumption raise that 1G
spawned the process
both have about the same RES, 1G, but
still the overall memory consumption actually didn't raise
 
hmh alright
 
COW :)
 
10:18 PM
:D well that's good
if we assume that my code doesn't copy everything then the extra memory allocated would be purely from results_total so that would make the problem more manageable
 
that and lots of other allocate / free cycles
but yeah
but it will copy everything you modify in those pool processes
so global variables and the like
if modified and used in both
 
yeah i don't actually modify any data
 
so the reason all processes display about the same RES is
 
i just have a lot of variables that are used to create the matrices and then i receive the output which is what i'm interested in
 
they share most of their code and data
and when the kernel goes looking process at a time how much stuff it has...
well they all have "the same amount", except it's mostly the same stuff :P
 
10:24 PM
oh ok, so then RES shows the real memory usage of the thread, but it just happens to also be the same memory that the parent process uses
yes :)
so RES only makes sense when looking at non related programs
 
simplified yes
 
one last question
>>> a = np.random.random((2,2))
>>> b = [a,a,a,a,a,a,a]
how do i turn b into an array without using something like c = np.array([b[i] for i in range(7)])
(i want to avoid that construction because it doubles memory requirements)
i looked at your code but your solution doesn't seem applicable
 
hmm at least with that can't you just np.array(b)
also
with that example, np.random.random((8,2,2))
 
actually, it seems np.vstack(b) does the trick!
(i don't understand why but hey :D)
 
and np.array(b) works too
ah you meant that combine them
 
10:33 PM
produces slightly different output though
 
from an array of array of arrays to an array of array
even better than vstack: reshape
>>> a = np.random.random((2, 2, 2))
>>> a
array([[[ 0.61215233, 0.35275043],
[ 0.4108367 , 0.46875469]],

[[ 0.01878168, 0.85095863],
[ 0.05358895, 0.37022605]]])
>>> a.reshape((4,2))
array([[ 0.61215233, 0.35275043],
[ 0.4108367 , 0.46875469],
[ 0.01878168, 0.85095863],
[ 0.05358895, 0.37022605]])
 
yeah but the difference is that the b i showed you is a list of arrays, not an array of arrays
so not all numpy functions work on that type
 
ahm yeah
 
but let me check if reshape works on it
 
nope
go with the stack :P
 
10:36 PM
i will do it like this:
np.vstack(b)
and then call reshape
 
are you btw using ipython?
 
no, just cpython
 
IPython is a really nice wrapper
especially for developing and testing numpy, scipy etc. stuff
 
it's just an advanced python shell though right?
 
ipython --pylab command will import most of the important stuff
yep
 
10:38 PM
i'll give it a try :)
 
with goodies like %timeit
%timeit some_command
easy to compare how long stuff takes
 
nice :)
 
but perhaps it's time to sleep :P
 
yeah definitely i've been coding for 12 hours now
so thank you for all your help and sleep well!
 

  last day (53 days later) »