« first day (5142 days earlier)      last day (25 days later) » 

01:38
@roganjosh (It's only kind of superficially similar to yours, and I'll add an answer to it on a low-priory basis, if I find it's not a dupe and shouldn't be closed: read_html infer_types arg was deprecated aeons ago in pandas time. And that user bounced way back in 2018, FWIW) ...
...Your question was quite different: you asked three-in-one questions and it was (initially, until you edited it) unclear if you wanted to just ask why dtype inference engine works like that/ find a workaround/ raise an enhance request on pandas. Anyway I'll have to retest both their question and yours on current versions).
 
3 hours later…
04:44
@roganjosh I rechecked your question about dtype inference from 0.x, the behavior now in 2.x with df4.convert_dtypes().info() is now to infer numeric_missing as Int64, a nullable int64 dtype.
But it still isn't MCVE. You should use df4.info(), then you'd see that 'string_boolean' column was inferred as object (i.e. string), not 'bool' or even int. Don't say "this doesn't work", say "I want this to infer string_boolean as bool or else string or else object, and numeric_missing as float (or Int_64)" (with pandas 2.x)
04:55
and as for wanting 'string_boolean' to infer correctly bool type and values, not that since it's type 'object', Python infers any non-Falsey string value as boolean True, so even "FALSE" and "TRUE" will both infer True:
>>> df4['string_boolean'].astype(bool)
0    True
1    True
2    True
But you can get it by CamelCasing "TRUE"/"FALSE" -> "True"/"False":
>>> df4['string_boolean'].str.capitalize()
0    False
1     True
2    False
Name: string_boolean, dtype: object
>>> df4['string_boolean'].str.capitalize().astype(bool)
0    True
1    True
2    True
Name: string_boolean, dtype: bool
>>> df4['string_boolean'].str.capitalize().astype(str)
0    False
1     True
2    False
Name: string_boolean, dtype: object
If you want to reopen it, I'll type up that 2.x behavior as an answer.
(Hey I'm not an apologist for pandas type inference by any means, but they know it's a weak spot and they're trying to fix some things about it. I'm curious how that all compares to polars)
05:19
Actually, that still doesn't work in 2.x:
>>> df4['string_boolean'].str.capitalize().astype(str).astype(bool)
0    True
1    True
2    True
Name: string_boolean, dtype: bool
but here's a workaround:
>>> df4['string_boolean'].isin(['TRUE','True'])
0    False
1     True
2    False
Name: string_boolean, dtype: bool
Aha, here's working 2.x code:
>>> df4['string_boolean'].str.capitalize().convert_dtypes()
0    False
1     True
2    False
 
3 hours later…
08:37
@smci The point was not really to ask for an answer since it's already well gone, just like the question you were mentioning. The point was that pandas had a lot of fringe behaviour with dtypes back in that era
A lot of work went into stabilising the behaviour. In any case, there already is a comprehensive answer to my question
Hmm, I think this goes beyond dataframe libraries and must be something to do with the properties of CSV in general. You can run it in polars
import polars as pl

data = [['string_boolean', 'numeric', 'numeric_missing'],
        ['FALSE', 23, 50],
        ['TRUE', 19, 12],
        ['FALSE', 4.8, '']]

df = pl.DataFrame(data[1:], schema=data[0])
print(df.dtypes)
print(df.head())

df.write_csv("test_types.csv")

df = pl.read_csv("test_types.csv")
print(df.dtypes)
print(df.head())
That comes back with:
[String, Float64, String] and [Boolean, Float64, String]
If you include an empty string in the string_boolean column it actually returns a string type instead. What the yam? I wonder if this is just an accepted convenience based on how Excel would save a CSV and that most people want this type inference
@roganjosh No, it's not that pandas didn't have a lot of fringe behaviour with dtypes back in that era, it still does and they seem to be blundering from one kludge to another, arbitrarily adding and dropping positional args and types... they still haven't gotten categoricals and strings working as first-class dtypes... their version whatsnew and errata feel like a confessional... I thought at least they wouldn't flip the big switch to 2.0 until they'd at least gotten those right...
...if v2.0 is still this clunky, few are going to stick around for v3.0... LinkedIn is non-stop people blogging about polars... I was even casually speaking to an acquaintance the other day, he started telling me about how much better polars syntax is than pandas, IHO...
08:53
Well, the code I showed is polars so this behaviour has been carried over
...I've raised bugs on pandas for over a decade... strength is fading...
Separately on the syntax, polars is a lot more aligned to pyspark
"Confessions of a dtype abuser: ... I followed it down a dark alley and coerced it to within an inch of its life... but that didn't stick... so I hacked a workaround... except that doesn't work on 'object'... well maybe in the next release... then again maybe not..."
@roganjosh I was telling you to stop using 'FALSE'/'TRUE' and write 'False'/'True'... when camelcase attacks...
That's not really the point... I was asking out of curiosity and the fact that exactly the same point came up 5 years later in polars is also a curiosity of mine
The CSV parser is geared up to do... things. Mysterious things that aren't easily accessible in either pandas or polars API without the CSV round trip
09:16
@roganjosh What if someone got an LLM to write a parody version 2.3 of pandas, merrily performing mayhem with dtypes and related positional args... not updating docpages, reopening old closed bugs... then actually released that. Would pandas users even notice they were being trolled...?
Probably not. They'd be more likely to explode and then melt
@roganjosh "Do please continue..."
reset_index()
Do you think we'll meet pandas v5.0 in the afterlife... what happens if the doors of perception break on their hinges and fall over...
TIL even with its huge API, it's hard to build full sentences with pandas methods. I've got "cut" on the table, but there is no "run" to go with it. "pivot" is also on the table but that's a tough one to work in
09:28
@roganjosh That sounded Lovecraftian...
D'ya know at the end of 'Raiders of the Lost Ark' ("Dietrich's head collapses in on itself, Toht's face melts off his skull, and Belloq's head explodes")? Was this an early csv parser?
Oh for God's sake, I've been beaten to the punch :'(
@roganjosh "It was a dark and stormy night... it was dawning on him that he could .cut() but not .run()... not with this API, at least..."
@smci Iä Iä, CSV parser fthagn!
@VLAZ "Looks good, send us a pull request..." 🤪
"In the Cursed Earth where mutants dwell. There is no law, just an inconsistent jumble of CSV parser, dtype hierarchy and functions to kludge around until things almost begin to hang together..." with apologies to Anthrax
09:54
@smci This is a great summary of NLTK, also explains why I hated doing nlp. Like the post is so spot on fills my heart with joy
I think open source software should be very careful to make its limitations clear. It’s a disservice to provide something that’s much less useful than you imply. It’s like offering your friend a lift and then not showing up. It’s totally fine to not do something — so long as you never suggested you were going to do it. There are ways to do worse than nothing.

AMEN!
@VLAZ I should finally at some point read Lovecraft
10:13
@Hakaishin Sure but the gap between how loosely-associated groups of people should do stuff vs what exactly does/doesn't get done is where 99.9% of things actually exist. Look how much hilarity I had parsing a string/'object' column containing a boolean. In v2.0 for G*d's sake.
@smci v2.0 of what?
Right, I finally managed to track it down in polars. This is very deliberate behaviour that runs through the CSV parser
It's case-insensitive so it wouldn't matter how you stored those string values as a string, as long as it's just "true" or "false" and nothing else in the column, you're going to get a bool back
@Hakaishin You might be disappointed. And not because it's bad, but because the popular culture has drastically shifted what Lovecraft is about. I've literally talked to people who think that "popculture Lovecraft" is Lovecraft. "popculture Lovecraft" is all about Cthulhu and tentacles everywhere. Also, neatly ordering elder deities into little boxes, and categorising all of them.
Meanwhile, real Lovecraft mentions Cthulhu in few works, but doesn't write about it much. And portrays elder deities in a confusing and conflicting manner because they are supposed to be impossible to comprehend. A lot of the stories are not even related to deities at all - in one, a person slowly succumbs to madness from living in his grandfather's home. In another a colour infects the land and things that live there.
I mean, there are also some downsides to real Lovecraft's writing. Racism being a big one. But more writing-related thing would be that the prose gets repetitive after a while. If read a few stories in one sitting, they all start to blend together. They are mostly "Yo, it was so scary you guys. I mean very scary. But also I'm insane and can't express how scary it is." Sort of gets old after a while. But do read some Lovecraft, nonetheless. It's still in popculture for a reason.
10:36
@VLAZ I don't know much about popular HPL, but as you said it's popular for a reason so I think reading a few stories could be fun
@VLAZ for sure
Now that I think about it, Lovecraft has gotten the popculture treatment of...the Bible. Popculture angels are beautiful people with wings. Rather than, say, wheels within wheels with eyes. Or being that use their six wings to hide their true form from humans. Similarly, artist's depictions of Lovecraftian deities are...same-y. Tentacled and/or fish-like. While most tend to defy comprehension. Here is the google image search for Yog-Sothot:
You can see they mostly revolve around the same theme. And that theme isn't the glowing balls that Yog-Sothot regularly manifests as.
10:53
@Hakaishin v2.0 of pandas, which is what roganjosh and I have been discussing
11:39
@VLAZ I think this makes a lot of sense, because the extrapolation of two or our very powerful features, eyes and hands is many eyes and many even more versatile hands xD
 
4 hours later…
15:57
Wow, I spent solid 2 hours going super deep into non determinism of sklearns PCA just to find out at the end that I was using once version 1.3.2 and once 1.5.0. Still weird to change the output of the function, given that I call it with the exact same arguments
 
2 hours later…
17:41
While looking more into the polars dtype parsing, I'm really curious about the difference in these two regex patterns. I'm utterly hopeless with regex but I wonder if someone with experience might be able to easily answer what's different about what they match?
It comes down to the very end of the pattern but I'm not sure what is significant about it. I'm curious as to whether FLOAT_RE_DECIMAL goes through to become an actual Decimal data type and what would trigger that
The best I've got to is that FLOAT_RE throws in a wildcard match but I'm not sure what would not fall through FLOAT_RE_DECIMAL in the first place to make the distinction :(
18:05
It's not helped by the fact that they only test for float parsing but it is definitely being used
nm, both patterns return DataType::Float64 anyway. 🦆
@roganjosh decimal dot vs decimal comma
2.3 or 2,3 (German or Hungarian locale e.g.)
Is that accounting for regional differences between decimals using . and ,? I wasn't too sure about whether my interpretation there was correct
Cool, thank you for the confirmation. I was trying to see whether , meant something more
 
6 hours later…
23:58
@Hakaishin How did you manage to pick up different versions on different runs? Wrong rundir? envvars? venv? path? image?

« first day (5142 days earlier)      last day (25 days later) »