« first day (4813 days earlier)      last day (359 days later) » 

04:18
@smci Because I thought it would be easy to implement. Didn't expect these weird shenanigans...
The JSON thing is just background information. The core of the problem is "I have a data structure filled with numpy datatypes {np.int32: int, ...} and a numpy number, but for some reason type(numpy_number) isn't present in my data structure"
 
1 hour later…
05:47
not reproducible (code doesn't correspond to error); has an high-quality explanation of how to read stack traces that is unfortunately completely inappropriate as an answer here. stackoverflow.com/questions/6997138
caused by typo (separate num and numb variables giving the wrong results; the question about IndentationError is incidental as it's a natural result of attempting to eval the wrongly-created string, and the question is really about how to create the string) stackoverflow.com/questions/63332514
Actually, changed my mind. It should be duped to whatever canonical for stripping trailing whitespace.
 
3 hours later…
09:08
@Aran-Fey Integers in JSON are a bit funny because JavaScript doesn't actually have an integer type. Using integers in JSON with more than 53 bits of precision isn't portable, but it won't cause problems in your use case, since the Python JSON module can handle arbitrary precision integers.
As a practical matter, Javascript integers are limited to about 2^53 (there are no integers; just IEEE floats). But the JSON spec is quite clear that JSON numbers are unlimited size. — Nelson Dec 6, 2013 at 18:24
09:26
@Aran-Fey just for completeness, I found the behaviour I was thinking of. Having read it again, it might not apply but it could be an edge case to look for
@roganjosh as said, just define a threshold. I mean they for sure have stats on that. I would just start with a default value like 50 lines. And then see how often did the heuristic finish and so you could slowly find a sensible value which matches most use cases
 
2 hours later…
11:59
@Aran-Fey the only explanation I can think of is two numpies. Basically what Miyagi suggested. Kill the doppelganger... unless numpy was built from source and broken somehow. Ufunc eq does have a loop for int32 and str, at least it works on all other machines.
@Aran-Fey this version only works with two numpies. Unless you're conflating np.int32(0) with np.int32([0]) (latter is an array, but eq should still work).
 
6 hours later…
18:21
This article is driving me nuts. Someone has recently posted a question about it - am I missing something about numpy.vectorize() here (method 8)? They keep saying you can test it in a code lab but I can't find it
Isn't np.vectorize just a fancy for-loop wrapper?
Yeah, and the docs still say so in the notes
I thought I missed something. But it looks like to test, I'd have to pull all of their code blocks together and generate fake data, probably to find it unreproducible but never know what they actually ran properly. This can't be right
There was a part of me thinking "huh, maybe I'm wrong and numpy now has a go at compiling your vectorized function with numba or just defaults back and I hadn't heard of it" but I think that article is BS
Meh, seeing how well plain list-comp and zip perform I'd guess the usecase is actually pretty useless for that comparison.
The real reason I wanted all the code in one place is that, in the back of my mind, they've probably propagated a pre-filtered DF forwards among their tests. Otherwise I can't see the motivation for the weird numbers
18:36
Dunno, it seems like the code is all over the place. For example, their "map implementation" also casually uses whole-column operations; I'm guessing the label they put on these cases isn't descriptive for what actually does the heavy lifting.
Ok, thanks for giving it a look over. I was pretty damn confident to make the assertion it should be ignored but I appreciate the second pair of eyes
18:54
All I want for Xmas is for people to stop writing yam blog posts
As long as ads revenue are a thing, people these days will keep on doing that
it's even worse now where people just generate a blog post with just a short description and title but do not verify if it's correct or not and copy paste it. I found more than 10 different blogpost that most definitely, 100% were entirely generated by chatgpt without any verification
even the about page of their so called author had the typical "I am sorry but as a large language model I'm..."
@roganjosh No worries, GenAI is here to do the writing!
be careful what you wish for. Man, I wish I paid more attention to the films. They tried to teach me
some even rewrite older blogpost using LLM for better SEO while completely disregarding meaning and context of the older blogpost. It become a semantic mess and they don't even care
@NordineLotfi The flipside of that coin is clearly marked GenAI-answers including stories about "their" childhood/life/upbringing/tragedies/…. It's pretty clear that thing isn't fit for autopilot yet...
19:14
I took a crack at it: https://gist.github.com/secemp9/16a65c98d8c15be858ed2276015047f1 seems like Miyagi is right in saying list comp is faster, but only for a specific version pair of pandas and numpy (both on 3.8.10): Python: 3.8.10 numpy: 1.24.4 pandas: 2.0.3
{'loop': 21.9099904, 'apply': 4.764312799999999, 'itertuples': 0.4603243999999975, 'list_comprehension': 0.20237750000000077, 'numpy_vectorize': 0.14856430000000032, 'direct_vectorize': 0.008104400000000567}

Here vectorize is faster than lisp_comp, but the fastest is still direct_vectorize (at least for current dataset I tried it on)
But for numpy:1.21.2, pandas: 1.3.2 (also on 3.8.10):
{'apply': 18.156991676999994,
'itertuples': 1.5214916549999913,
'list_comprehension': 0.6677151359999982,
'numpy_vectorize': 0.719260606000006,
'direct_vectorize': 0.010720880999997462}
list_comp is somehow faster than vectorize then? weird
Good lord, you're a sucker for punishment
I was just curious :)
For string operations, list comps should beat pandas general approach. That's been that way for a while. This is just numeric
For datetimes, I'm less sure
yeah, I didn't try to benchmark everything, just the dataframe (since this was mentioned on the blogpost you linked)
You're half (or more) way to answering it so here - the question. I got bogged down in the article and then got bored
19:18
Thank you, will try to answer :o
I might have a go in polars. I've not actually used UDFs yet so this might be a decent test case
hmm, on second thought, it's hard for me to conceptualize what they want. If they could at least show an MRE for the "SQL row oriented python class from a row based calculation" then it would be easier to answer
here it's as if they aren't actually asking a question, beside the part where they say "So how to avoid with many columns explicitly writing out: df['colA'], df['colB'], df['colC'] etc"
19:36
I think you're probably right on a re-read. Worse, they probably could do their calc in SQL faster than in python anyway
I have never wrapped a df, so initialising a class needing to know the column names makes little sense
 
3 hours later…
22:42
I have class with a variety of methods, and I now want that a particular subset of these methods to raise an error if the instance's self.is_open variable is set to False. How would you go about implementing this? i.e., how could I wrap all my methods in a function that would check the value of self.is_open?
22:59
Either a function decorator, or simply start each method with a function call like self._ensure_not_open()
23:25
hm, yeah, decorator seems the way to go. ty!
the method call is probably more obvious to someone reading the code (less indirection)

« first day (4813 days earlier)      last day (359 days later) »