« first day (3364 days earlier)      last day (1599 days later) » 

12:03 AM
HaPpY nEw YeAr!
 
 
1 hour later…
1:10 AM
Happy New Year to All!
 
Sam
1:30 AM
@AndrasDeak Thanks Andras for the reply. Those are definitely better ways to simplify my code. Currently I have to take a step back to figure out what's causing my values to explode, but I'll probably be back with more basic questions when I have to speed up my code to be able to make 2000 frames of three 200x500 plots. :)
 
1:53 AM
Happy new year
 
Here is an updated demo of the arithmetic evaluator (I'm not going to call it PyRithmetic), with some javascript bits to display buttons for easy entry of some of the non-ASCII characters. ptmcg.pythonanywhere.com/arithrepl Enter 'code' to see the Python code. Thanks to everyone for your testing efforts!
 
 
3 hours later…
4:56 AM
Happy new year all!
 
5:49 AM
@roganjosh yeah mate , is there any other way to do that in more dynamic way?
Happy new year all gentle men here........
 
Happy new year all!
 
6:18 AM
Hello flask framework mate's , any guiding steps on how should I go further or guide me a correct way from scratch for this Q stackoverflow.com/questions/59539950/… ........sorry if this makes no sense , would explain in detail
 
Happy new year everyone !
 
 
4 hours later…
10:00 AM
,happy new year
 
Happy New Year
What to do with this highly-upvoted 2010 question How does polymorphism work in Python? It has many views but the question body is not general and contains a wrong premise: OP thinks myDog.__class__ is animal does what isinstance(myDog, animal). Also, OP isn't asking "How does polymorphism work in Python?" but "How to test if an object is an instance of specified class or its subclasses?"
Also as to the ongoing SEO wars for Python doc sites: when I google for python isinstance, the top hits are: 1) w3schools.com 2) python-reference.readthedocs.io 3) programiz.com 4) docs.python.org 5) pynative.com 6) journaldev.com 7) SO 8) appdividend.com 9,10) geeksforgeeks.org
 
10:43 AM
@SebastianNielsen is there any hidden significance to your miXeD-cAsE...?
Wondering if this Q&A to "Which Exception for notifying that subclass should implement a method?" (2010) are misleading? OP wants a Base class which implements f() but not g(), but Base.f() explicitly calls g(). But when Base.g() is called, OP wants an exception raised (so Base is an abstract base class). Is it thus good or bad design for Base.f() to explicitly call g()? ...
... I suppose it depends on what f(), g() really turn out to be and why f() has to call g(), why can't it have a sensible default action. For example it might make sense if g() was __repr__(). But not __str__(), whose default behavior should be to just display something unique-ish, not throw exception.
What do you all think?
 
 
2 hours later…
1:08 PM
 
Happy new year!
 
 
2 hours later…
3:09 PM
Are dict items rehashed & reinserted when a dict is resized (automatically)?
 
 
3 hours later…
closed ^^
 
happy new year everyone
quick question, let's say you have this code. For me, you can only pass the variable when you instantiate your class. Let's say I'm rewriting my class, is there a way to pass your variable once the instantiation has been already done? I'm unsure as it is dunder method but I maybe wrong. Still asking, out of curiosity. Thanks.
 
6:07 PM
@PM2Ring did you see how Shanghai did the fireworks
 
@AndyK nothing stops a non-dunder method from setting self.miaou if that's what you're asking
 
@AndyK if you're evil you can even call the __init__ explicitly 2nd time
 
6:19 PM
Hi
 
Hello
 
6:36 PM
cbg
 
6:55 PM
@AndrasDeak not sure what do you mean by that Andras
 
what i was asking is that. If I want to use len, I can do that
print(len(comptage([1,2,3,4,5])))
or do the following
compt = comptage([1,2,3,4,5])
print(len(compt))
is there a way to do this instead?
compt = comptage()
print(len(compt([1,2,3,4,5])))
 
So you want compt to be callable? If so, implement a __call__ method
 
7:10 PM
@AnttiHaapala Ok. That is impressive.
 
@Aran-Fey aha !!!
 
@AndyK Why not just do compt.miaou = [1,2,3,4,5] and then do print(len(compt)) ?
 
asked the question to a French pythonista and all he could say was "pas faisable" (not doable)
@PM2Ring I've done that already, I just want to that , once the class has been instantiated
happy new year btw @PM2Ring
even though as Gramsci said and I paraphrasing it, I'm not fond of new year
 
Oh yeah. Happy New Year, everybody. :)
 
@Aran-Fey that's it ! yeah ! I need you to me be my translator... actually, I will be my own translator
 
7:27 PM
@AndyK In that case, I think we may have a communication problem. ;) You can change the value of the miaou attribute at any time, and len will return the current length of that attribute (via the __len__ method).
As Aran-Fey said, you can make class instances callable, but please use that feature carefully. Callable instances should do something that seems natural for that kind of object, don't use it for weird or obscure tricks. Readability counts!
 
fair enough @PM2Ring, readability counts indeed
but I'm trying new things, that's why I'm asking this question
but I'm hearing your advice , so if I'm tempted to use call in the near future, I'll be careful about it
 
8:16 PM
Hi bonjour
are you allowed to link questions onto here
 
if they're older than 48 hours
 
 
2 hours later…
10:23 PM
If I have a question but my code works should I post on SO or code review?
My question is kinda both
 
Unless the question is "how can I improve this code?", post on SO
 
I want to improve it but also add functionality
 
that might be too broad for SO then
hard to say
 
I will ask on code review
I kinda already solved it just now
didn't know numpy and pandas worked so well together
 
pandas is mostly built on top of numpy for the time being
 
10:28 PM
you say that like that won't always be the case?
 
there is (was?) the pandas2 project partly aimed at decoupling from numpy dev.pandas.io/pandas2/goals.html
the corresponding repo was last committed to 2 years ago which might mean anything, I don't really use nor follow pandas
 
10:48 PM
Seems like a dead project
Which makes sense, why re-invent the wheel by writing a whole new version of numpy
 
11:09 PM
6
Q: multithreading for data from dataframe pandas

goodXI'm struggling to use multithreading for calculating relatedness between list of customers who have different shopping items on their baskets. So I have a pandas data frame consists of 1,000 customers, which means that I have to calculate the relatedness 1 million times and this takes too long to...

Is DASK really the answer to multithreaded pandas?
 
@Vader When I last tried dask, I found a lot of overhead
I have seen it suggested in the past that spark, even on a single computer, was better for this kind of thing, but I wouldn't be able to dig out the reference even with my best efforts, I think it came in some discussion a couple of years back
 
I thought spark was for petabytes worth of data and dask is for a less, My csv are only about 150mb
 
My experience of dask was such that I wouldn't consider it for any application where the df fits into memory. But that was a couple of years back, so take it with a grain of salt. But threads are only going to be useful here for I/O, I think, so you'd be better on concentrating on how you vectorize what you're doing (since you can load 150MB really simply in pandas). What exactly is it you're trying to do?
 
multithread some work on the df that can be parallelized
I need to do some work on every row of the df and the outcome is independent from surrounding rows so I think it makes sense to multithread it
 
11:24 PM
That doesn't mean that it can't be vectorized
We need some idea of what "some work" is in an MCVE. If you're calling functions on every row of the df, threading in python is just not going to help with CPU work
 
Yes it will
I will split up the df into smaller dfs, one for each thread, do the work across all thread then at the end concat
 
Are you going to substantiate that with some indication of how it gets around the GIL?
 
Can you simplify that question?
 
The global interpreter lock means that only one thread is going to be doing CPU work at one time
 
But I know that multi-threading is a thing in python. How are other people doing it?
 
11:30 PM
multiprocessing would a possible option, but my suggestion would be to approach this with caution. When you throw in the overhead of spawning extra processes, plus the concat at the end, you're driving roughshod over things that pandas itself is optimised for
I don't think you'll find many examples of pandas Q/A where a good answer is employing either threading or multiprocessing. One example where I've needed threads was bombarding an API with pair-to-pair matrix requests for real-road distances between points. They're (generally) good for I/O tasks. But even then, I pulled the data out of pandas, did the requests, then put the results back into the df
 
How can you say that pandas is optimized when it only uses 1 core?
 
It doesn't if you leverage numpy, which is not bound by the GIL. This is why I asked for an MCVE
 
I suggest multiprocessing with 10 cores computing while True: pass
 
Oh
 
That should work all the cores nicely
 
11:37 PM
It's a $500 space heater
 
Is there any way to see how many RPM it can do without having a counter? @AndrasDeak
@roganjosh I didn't know about that numpy stuff
The thing I want to multi-thread is regex
 
If your csv is 150 MB are you sure you've located the bottleneck?
 
There is no real bottleneck
it is all reasonable fast
 
@Vader The rough idea being that you can drop component parts of your calculation into numpy to get around the GIL and into C code. But it's tough to talk about anything other than abstracts atm.
 
I am not looking for someone to do the work for me, just some insight from people that have done this sorta stuff in the past
this is not an MVCE but it is the function I want to multi-thread
The regex is probably the slowest part
the rest is python conditionals
@roganjosh I know you said numpy stuff get multi-threaded but what about basic python math?
 
11:45 PM
That's more than "minimal". I'm also getting deja vu on this code base :) My suggestion would be to try break the problem down to its component parts as step one. Even that might be interesting in understanding the problem (much like how many people solve their own problems just by making an MCVE). Then ask about the individual, intractable parts
 
Right, I know it is not minimal
I was just saying that there is no way this function could be multithreaded
This function is called on every row of the df
 
Never say never
 
instead of core0 doing rows 0-600k why not do:
core0: 0-150k
core1: 150k-300k
core2: 300k-450k
core3: 450k-600k
That is my idea
 
You run match.group two times on each search
 
That would require multi processing. You should be clear about the terminology here; threads and processes are different things
 
11:48 PM
I figured I could save RAM? @theGtknerd
I may be slightly confused about the terminalogy of multi-threading, multi-processing, pooling
But does my idea make more sense now @roganjosh
 
I understood what you were thinking from the start :) How much experience do you have with numpy?
 
I use np.vectorize when I can
I know that helps, not sure how it does, but it does
 
Don't. It's nothing but a for loop in disguise
 
From what I've been told
so much conflicting advice
 
Then you've been misinformed. It's a convenience that basically just wraps a regular for loop. Look at the "Notes" of the docs
 
11:53 PM
hmm, that's dissapointing
so it's not bad. But it's also no better than a for loop
 
It's not necessarily conflicting information. Numpy did a job on itself by creating a np.vectorize method, which is not what people are generally referring to when they say "you should try vectorize this"
 
so what does it mean?
Also instead of doing python math should I do np.add() etc when possible??
 
I've skimmed this but it looks on the right track of what we actually mean
 
I have some reading to do it seems
thanks for the help @roganjosh
I will use numpy as much as possible
 
@Vader no
numpy-likes already implement addition efficiently
 

« first day (3364 days earlier)      last day (1599 days later) »