« first day (3857 days earlier)      last day (1074 days later) » 

2:46 AM
hello guys
 
 
4 hours later…
6:59 AM
@grrigore Yes.
 
 
3 hours later…
10:04 AM
This was the most starred message last year on this day:
Hmm. Actually, nothing interesting was said on this day last year.
 
 
2 hours later…
11:55 AM
Guys guys guys guys guys guys
This
 
Is Sparta?
 
wait i'm on mobile and uploading a photo is not as easy as in desktop
user image
3
This is Kruger!!! :D
 
Tim
12:17 PM
niceeeeeeeeeeeeeeee
what's your address? my girlfriend is on her way to steal him
 
xD
She's going to come all the way to spain and then fight my sister?
 
 
4 hours later…
4:48 PM
@Mehdi I got a really big tsv file that I need to modify. python's pandas seem too laggy. got any suggestion?
 
 
3 hours later…
7:46 PM
@grrigore I assume you're working on a computer? or a cluster? for the former, you could consider using docs.dask.org/en/latest/install.html, if it's still not enough, I'd chunk the entry data and process them sequentially then aggregate the computed parts later on in the process
 
7:59 PM
I ve used pyspark
computer btw
 
in that case, I'd stick with pyspark and do all trannsformations with it, don't convert back to pandas
 
yes
atm I need to pull tweets I found github.com/DocNow/hydrator this one
and I need to extract some ids from a large file
 
once you're done with all your operations, write the results in the file system with spark, then reload them and convert them to pandas .to_csv or something
 
yes
thanks
 
keep in mind that most spark operations are lazy, as long as you don't run actions such as "count, collect, show, toPandas, write" etc...
 
8:10 PM
yea count is soo long
while filter is almost instant..
 
yes, because filter is not an action, it will only be applied once you run an action e.g count / write
 
8:47 PM
interesting
 

« first day (3857 days earlier)      last day (1074 days later) »