« first day (1379 days earlier)      last day (1852 days later) » 

8:26 AM
@AndrasDeak I run out of RAM and am unsure why. I have a 5.6GB binary file, with 8 columns of double precision values, and about 88M rows. My PC has 16GB RAM, with 13GB free. When I try to read this data into Python, I run out of memory before I reach the end of the dataset. I read the file using the following:
def read_Residual_data(file_in, start, no_rec=-1):
    with open(file_in, 'rb') as file_in:
        # File has 8 columns: [N T R Theta Phi Br Btheta Bphi]
        # N: orbit number, T: decimal days since 1st Jan 1999
        # R, Theta, Phi: satellite location in planetocentric
        # Br, Btheta, Bphi: residual magnetic field, i.e. measurements - model
        #
        # Move to starting position
        file_in.seek(start * 8 * 8)

        k = 1
        rows = []
        if no_rec == -1:
            while True:
When loading a 5.6GB dataset, why do I end up more than doubling it, making me run out of RAM, when trying to load it in Python?
 
have you tried oppening a smaller file and debigging it?
 
When running it with num_rec=int(1e6), thus reading 1M out of 88M rows, it goes fine
 
but how much memory it needs, where is it going
 
Hm, running it with 10M rows it takes 80% of my RAM to just load the data; that's 12GB, so it is approximately 20 times bigger than expected
Whoops, disregard that
25% RAM, thus ~3.5GB, for 10M rows
still 8 times bigger than expected
 
8:58 AM
I dont know wnough about python to spot what could be doing that
maybe the try catch?
 
@AnderBiguri that basically reads 8 bytes (= one row), appends it to my output variable (rows), then reads the next line, appends it, and so forth, until I hit an error, which in my case only happens on end of file...
 
but maybe the try catch is saving memory inc ase it fails, or something like that, under the hood
 
Ah, belly deep problems, could be
I'll just stick with reading maximum 10M rows at a time, it doesn't matter too much anyway
I just hoped that cutting down my 18GB set to 5GB of useful data would enable me to do processing in a single go, but oh well, it works in chunks as well
 
Just for clarification: "8 columns of double precision values" vs. "basically reads 8 bytes (= one row)" ... 1 byte = 1 double??
 
@AnderBiguri no
 
9:04 AM
@HansHirse oh sorry, no, 8 blobs of 8 bytes, so 64 in total. I meant to say it reads the 8 values on each row
 
@AndrasDeak fair, it was just a wild guess
 
List is not a numpy array, has overhead, but shouldn't be that much. I'll take a look from laptop
Maybe some leaking variable
 
@AndrasDeak thanks
 
9:35 AM
OK, so lists are too wasteful here
>>> import numpy as np
>>> arr = np.random.rand(10000, 8)  # 10000x8 doubles
>>> lstlst = arr.tolist()  # list of 8-length lists
>>> lsttup = [tuple(row) for row in lstlst]  # list of 8-length tuples
>>> lstempty = [[] for row in lstlst]  # 10000 empty lists
>>> from pympler.asizeof import asizeof
>>> asizeof(arr)  # size in bytes
640112
>>> asizeof(lstlst)
3280064
>>> asizeof(lsttup)
3127624
>>> asizeof(lstempty)
727624
The problem is the enclosing list. So I'd suggest reading in batches, and converting each batch to a numpy array and collecting those.
ah no, the problem is the 10k lists inside
>>> lstnone = [None for row in lstlst]  # 10000 Nones in a list
>>> asizeof(lstnone)
87640
So you might get away with converting each row to an array...but that has an overhead of its own. You'll probably have to convert in batches.
 
Interesting; so I would have to read in batches (say 1M rows), then store that in a preallocated numpy array, and read until I finished?
Is this interesting enough for a SO Q/A?
 
not preallocated, you can convert on the fly
@Adriaan there are probably dupes around
something like arr_list.append(np.array(rows)); rows = [] with the appropriate checks for batches
and at the end res = np.concatenate(arr_list); return res or something like that
if the data is in a sane binary format there's maybe even a way to read it directly into numpy, I don't know, I never use binary
this is basically a self-answered "yeah, I had to cut it up in batches"
 
9:51 AM
So reading and processing in batches isn't too different from reading in batches, then processing the whole set, as the processing is per orbit anyway, and thus in batches
 
yeah, then that would be natural
 
The only thing now is that I don't have a fixed number of observations per orbit, to I read 1M rows, chop off the last orbit (which is probably part), then process, and read from the chopped orbit for the next iteration; making me read somewhere between 0 and 4k rows twice, each iteration. That'd be solved by first reading and storing the full set.
But that's too little improvement I think.
4% memory usage, 400% CPU usage. Nice.
 
10:12 AM
As long as it works :)
 
It certainly does! Köszönöm!
When the boss comes back from "working" on the Maledives tomorrow I can finally show him some results
 
poor boss :P
 
His first lecture already started with that: "The ETH operates an observatory on the Maledives, and for some reason it always needs repairing around February/March. This takes 10 days."
 
and with repairing he means: lift the old instrument from its pedestal, place the new one, and be done in 30 mins :P
 
10:18 AM
better to be safe than sorry!
 
He sent me pictures of his hotel; white beaches and palm trees, the poor sod
 
Must be a struggle...
Far from his beloved snowy hills
 
But didn't he miss then the burning (snow)man festival? ;-)
 
On the picture of the observatory itself you just see a small hut on a flat grass field with a few trees, since it's on the airport grounds. Said one of my colleagues: Oh, looks Dutch; flat and green
 
Needs more sea and tulips
 
10:22 AM
And less tourists... There's a campaign going on to try and stop tourists from going like a bull through a chinashop in the flower beds, but they just don't get it. They ignore signs, and when someone tells them to not trample the flowers they simply react with "Oh, we were just cycling about, we didn't know it was not allowed to rampage through your livelihood and destroy everything"
 
@AnderBiguri so...the only impact of exception handling is a slight runtime overhead in the except branch. If the exception is exceptional you won't see it. If you're doing it wrong and raise in the majority of your cases then you might notice.
@Adriaan ugh, that should even go without saying
 
@AndrasDeak try telling the tourists that, especially the Asians
One more thing @AndrasDeak I copied the boss's code and adapted for my purposes, but b = np.concatenate((collect[mask,5], collect[mask,6], collect[mask,7])) is the same as b = collect[mask,5:7] right?
 
if the shapes match then yes
it would be possible that columns that converted to rows if it's multidimensional...
oh, also, you need a copy, the latter will be a reference
 
if the shapes didn't match, collect could not exist in the first place right?
 
so collect[mask, 5:8].copy()
note the 8 in the index
@Adriaan I'd have to think about it and I just had lunch :P
 
10:26 AM
Oh yes, I keep forgetting that.
 
this is easier to test in practice with an example
 
I'm about to head off to lunch.
 
and 3d arrays can be indexed with 2 (leading) indices, which might make it non-trivial
you know the size and shape of the input, so you can test
if it works for an example then it'll work for the real one as well
 
I'll make plots after lunch, then if I have time left I'll toy around with this stuff
@AndrasDeak ah no, simple 2D array; each row is a measurement and contains the orbit number, satellite location (time, XYZ), and measurement in XYZ
 
OK, then each slice is a 1d array, so concatenate will give you a 1d array
 
10:29 AM
ah, so it stacks the columns, rather than making a 3-column array?
 
so what your boss has is probably collect[mask, 5:8].ravel('F')
@Adriaan I would think so, yes
 
cewl
 
alternatively, collect[mask, 5:8].T.ravel() or something like that, but that won't necessarily give you a copy
 
so much options, so much headaches
 
@AndrasDeak depends on what you are running inside the try it can be quite slower, but yeah, that was my understanding too
 
10:30 AM
ah, but .ravel guarantees a contiguous array, so it will copy in this case
 
I'll check after lunch, thanks
 
@AnderBiguri but than that's not the try/except, is it now? :|
@Adriaan have a good one
 
ravel was using 50% of my time, and I am doing GPU reconstruction
avoid ravel
@AndrasDeak yeah yeah. I never use trys anyway
 
.T.reshape(-1).copy() isn't any better I think
has to do the same work
 
10:45 AM
Hey, silly question
Anyone knows how the "green" tags in the readme are called?
 
yus
thanks man
 
no prob
 
In fact shields.io was what I was looking for, but you sent a nice demo
 
10:54 AM
lol
 
11:53 AM
fig, ax = plt.subplots(1,3,sharey=True,figsize=(16, 6), dpi= 80, )
ax[0].plot(Br_res);
This works in my jupyter notebook, but when trying the same outside the notebook (in a regular .py file) I get the error AttributeError: 'numpy.ndarray' object has no attribute 'plot'
But in the notebook type(ax) == numpy.ndarray as well...
 
12:12 PM
ax must be shape (1,3), so ax[0] is a 1darray of axes objects
use subplots(ncols=3) instead, that should give you a 1d array
 
Right. But then why does this run on the Jupyter book?
 
Hmm, I missed that detail. Perhaps inline matplotlib behaves differently.
no, in ipython I also get a 1d array
perhaps you have an older version where squeezing singleton dimensions was not the default?
 
Not a clue; I think I installed everything fresh last month
 
check help(plt.subplots) interactively
the new versions have squeeze=True as a default in the signature
hmm, signature hasn't been changed since 2010
what's import matplotlib; matplotlib.__version__?
 
12:32 PM
3.0.2
but I installed it only once on my system, and both the .py files as well as the notebook should be using that
 
12:47 PM
did you check the version from the .py file just to make sure?
 
that was the .py file
 
OK, then I dunno
print ax.shape just to be sure, but it's probably (1,3)
can't repro on 3.0.2
 
1:26 PM
mm writting question: Should I add the fact that the fish I am using was featured in the BBC documentary in the desrciption of the sample in the paper??
Its correct and good information about what I am scanning
but also seems a bit like boasting, not sure how necessary it is
 
How about a footnote, or just a reference?
 
definetly there will be acknoledgments
 
@Adriaan nice, I was waiting for it
 
(Currently in the questions part)
 
1:30 PM
it has less resolution than what I hoped for :( I want thinking of interstellar, but somewhere inbetween
 
Next step: element zero mass effect drive cores and visit!
 
It looks a bit like an EU RC propaganda :P
 
For sure, especially the first question, after which the commissioner just put feathers in his own behind
Good material to watch whilst calculating more results nonetheless
 
suresure
 
1:53 PM
@AnderBiguri can we share this picture / is there an "official" source?
 
soon there will an official source. I rather you dont share it too much for now, but its not a big problem (otherwise I would not have put it here)
IF you wat a bit I will soon link to the preprint of the paper
 
yup that's great:)
 
2:23 PM
It certaionly looks more spectacular than the black hole picture they released today.
 
hahaha
rand(Gaussian)
 
@flawr prime example of porn for American cops that was
 
2:41 PM
Really digging the new Rammstein
 
@Adriaan hehe:)
 
@Adriaan awww... we like pictures of flaming donuts :'(
 
@gnovice splendid controversy around the globe, especially in Germany
 
I like it, everyone gets upset about everything all the time. I like when people dont give a f**k
 
@Adriaan I can imagine. And that's some serious production value for the video, too.
 
2:51 PM
Now we get edit suggestions that add both three back-tick markup and four space indenting for the same code. LOL! stackoverflow.com/review/suggested-edits/22713339
 
@CrisLuengo fence all the blocks!
 
@CrisLuengo wait here while i add in a <pre>...
 
@beaker and <code>?
 
@Adriaan mark ALL the ups!
 
@Adriaan woot
 
3:30 PM
@Adriaan add \begin{lstlisting} \end{lstlisting} just to be sure
 
@beaker Apparently that will lead to a big mess
 
3:46 PM
@CrisLuengo I am... Professor Chaos!
 
5:02 PM
@CrisLuengo blech
 
 
4 hours later…
9:22 PM
M87 Black Hole Size Comparison: xkcd.com/2135
 
9:42 PM
@beaker That thing is big...
 
@CrisLuengo In the press conference they mentioned that it's 1.5 light days across
 
oof
 
10:23 PM
@CrisLuengo I included both. Thanks! I knew there had to be a dupe but I couldn't find it — Luis Mendo 22 mins ago
@LuisMendo Often it's easier to answer the very simple questions than to look for duplicates, eh?
A little bit more and I'll be able to dupe-hammer myself.
Thanks!
 
10:50 PM
@CrisLuengo Indeed! :-)
 

« first day (1379 days earlier)      last day (1852 days later) »