« first day (4068 days earlier)      last day (863 days later) » 

1:11 AM
probably a candidate for this one stackoverflow.com/questions/70194636/…
 
 
11 hours later…
 
2 hours later…
2:24 PM
Umm... is it me or is ^^^ somewhat confusing...
 
When has the pandas API not been confusing?
I guess it makes sense to me that joining on an index would be faster than an arbitrary column, but I don't know what pandas does internally to support that
Actually, does "id" have implicit support as a reference to the index? Now I'm confused :/
 
Oh, I gave up with that a year or so ago. Everything I tried was slower than pandas itself on simple tasks and I thought the project was going to die
Plus the lacking API
 
2:45 PM
dask isnt exactly for speed by itself, it should be slower than pandas. i think dask's selling point was that it could deal with data that wouldnt fit in memory, thus ruling out pandas
unless theres some multicore stuff, which i'll admit i didn't explore dask much, it should be slower on single core vs pandas
so if speed on simple tasks is your use case, dask isnt the tool for you
 
Why should it be slower?
pandas is a first-pass, so there's plenty of learning on what has gone wrong on that. Other than spawning additional processes, I would have thought that dask could have built on the failings
Your OS doesn't run slow when you don't need to use swap, for example
 
cause dask isnt holding everything in memory. youre simply comparing apples to oranges i believe.
in my limited understanding, its like how you'd expect hadoop to be slower than just about everything that does work in memory
i do want to check whether dask does work "on disk" or "pyspark" like.
to phrase it differently, if data fits in memory, why dask*
 
How can I always know a priori that it will?
My program might pull a file from S3 with no understanding of how big it will be
 
and therein lies the rub
 
But dask should be able to fall back to an in-memory model, surely?
 
2:56 PM
I dont know whether it does. i was under the impression it doesnt. whether it "could/should" seems like a fair point to make
 
If my timings (albeit a while back) were correct, it doesn't. So if you switch to dask, you're getting a lot of overhead for everything
 
3:08 PM
thinking about it a bit further, isnt it the same problem though
how do you really determine the operations would be doable in memory until you either did them or calculated or estimated based on heuristics, with even then a chance of getting it wrong
and i imagine that even said heuristics would be at least some what time consuming (perhaps O(n) or so on to figure out lengths and stuff)
so yes, absolutely , if you switch to dask, you get an overhead no matter what
its why you shouldnt be using dask if your work can be done with just pandas
docs.dask.org/en/stable/best-practices.html#start-small they raise the part about alternatives very early on
> Parallelism brings extra complexity and overhead. Sometimes it’s necessary for larger problems, but often it’s not. Before adding a parallel computing system like Dask to your workload you may want to first try some alternatives:
 
Anyway - the only reason I was looking at dask was I happened across stackoverflow.com/questions/70235174/…
 
> In many workloads it is common to use Dask to read in a large amount of data, reduce it down, and then iterate on a much smaller amount of data. For this latter stage on smaller data it may make sense to stop using Dask, and start using normal Python again.
I am taking that to mean that they give pandas their blessing for small datasets
and okay. dask works off of disk, but has an option where you can explicitly tell it to "persist" in RAM. docs.dask.org/en/stable/…
that should clear everything up. so yeah, dask isnt guessing for you, and working off of disk by default
 
3:38 PM
I feel like I've been told off :P
I might re-visit dask but atm I don't think it does what it says on tin for me
 
I so want this mug: i.stack.imgur.com/HCMxc.jpg :p
(seems quite apt. given the attitude of some people regarding covid these days...)
 
 
5 hours later…
9:08 PM
@JonClements lol
@roganjosh ^polars :P
 
@AnttiHaapala interesting. Would be curious about what this benchmark result be on other non-included db implementation
 
@NordineLotfi scusi?
 
@AnttiHaapala I meant that for the image you posted above ^^ since I don't see some other db in that list, so got curious about their possible result in that benchmark
 
db? that is not a db, it is a df library benchmark
 
oh
I saw DuckDB in there and got confused
 
9:25 PM
@AnttiHaapala I've seen the benchmarks and raised it at work to see if anyone has used it. It's on my list to play with, I'm just worried the API isn't going to cover all our common pandas work
At the moment I'm trying to slog through the official tutorial for Rust. I assume polars has apply for arbitrary python lambda? (I'm on my phone, I'll just check later :P )
 
 
1 hour later…
10:40 PM
@roganjosh yea I've not used it either other than just quickly checked, but... I'm doing serious data engineering and one of the biggest pain points I've got with pandas is that it is so contrary to import this when it comes to datatype conversions, and since this is arrow-native, then it will nicely map 1 on 1 to native parquet libraries...
 
10:58 PM
Thanks for pointing it out... I'm definitely going to have to play with that... it's got some nice native functionality in it that I really wish pandas had...
 
11:30 PM
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
<none>                               <none>              9dfa1a1b6acd        7 months ago        1.6GB
Ummm... wonder what the heck that docker image is... :p
 
"rabid dog" or whatever the hell docker chooses for releases. I swear it looks like a virus every time I open Docker
This is just....
 
At 1.6gb it's not a particularly discreet virus... but then... Windows gets away with it :p
@roganjosh wow... that poor thing is in emotional turmoil! :p
 
objective_kilby will sort it out. Emotionally cold, but they speak the truth (what the hell are all these things?)
 
shrugs
 

« first day (4068 days earlier)      last day (863 days later) »