« first day (4604 days earlier)      last day (360 days later) » 

4:44 AM
@CeliusStingher Hey I asked you to narrow down the issue and post us an Minimal Reproducible Example, not just write about it without us being able to reproduce it. Here's your MRE:
pd.to_datetime(['04 May 2022','01 Apr 2022','02 April 2022'], infer_datetime_format=True, errors='raise')
# DatetimeIndex(['2022-05-04', '2022-04-01', '2022-04-02'], dtype='datetime64[ns]', freq=None)

pd.to_datetime(['04 May 2022','01 Apr 2022','02 April 2022'], infer_datetime_format=True, errors='coerce')
# DatetimeIndex(['2022-05-04', 'NaT', '2022-04-02'], dtype='datetime64[ns]', freq=None)
So as you found, the inconsistency is pandas inferring with errors='coerce' infers the full month-name (%B) for 'May' hence it gets 'Apr' wrong and gives NaT ...whereas inferring with errors='raise' infers the abbreviated month-name (%b) and gets things right. But, I'd expect infer_datetime_format to be glitchy, I'd pass it an explicit datetime format. Anyway infer_datetime_format option is deprecated in pandas 2.0.
The above was with pandas 1.4.x. Can you confirm it's fixed on pandas 2.0?
 
 
2 hours later…
7:28 AM
infer_datetime_format is just plain evil. At least in one of the older versions of pandas it would happily swap from MM-DD to DD-MM half way through a column when its expectations were subverted (classic movie plot material). I don't know whether it still does that, but I wouldn't take the chance
 
7:42 AM
stares at screen and mumbles Oh my god. It is... adapting... evolving... gets eaten by a panda
 
@roganjosh MM-DD-YYYY is evil in and of itself
 
It's all about the polars now, MM! Ironically, the only time I've used pandas in the last 2 weeks is to dump data containing lists in columns to CSV because it will cast those lists to strings and make it valid, but polars will have none of that. I want it on the record that it wasn't me that put lists into the columns in the first place
 
8:06 AM
I'm sure the polar power bears that be won't judge you too harshly for it.
 
 
4 hours later…
11:55 AM
Hi
Anyone worked with llama_index?
 
12:15 PM
This and 318 issues already. I don't suppose I can ignore the existence of LLMs any more. The pace of this is crazy
 
 
1 hour later…
1:45 PM
Is black and isort used a lot in the industry? I'm kind of hesitant of using 3rd party tools, for secu rity reasons.
 
The main reason to use Python is the amount of 3rd party tools
If you worry about security, really, most likely vulnerability comes from what you write and not 400 milion downloads library/tool
 
Python is open source...
Linux is opensource. There's a reason to be skeptical about some things but if this comes from mgmt then it's usually a bull argument. black is used widely
 
Yeah, come to think of it, what would actually "1st party" be in case of Python?
 
I was thinking of non python standard libraries, as 3rd party
You are right, when they are widely adopted the odds of vulnerabilities goes down a lot.
 
python standard libraries don't have any more security than other ones, other than how many people are using them.
 
2:00 PM
Open source is awesome. The more I think of Windows, the more I appreciate it :P
@matszwecja i trust it more because I assume there are more experienced devs and higher standards to code quality. I could be wrong though. Can anyone contribute to Python?
 
Yes, just like any other python library contained within it
 
interesting, I didn't know that.
 
well, they do have a review process and lots of tests
 
There needs to be a board of people that review stuff, but there's nothing to say that they are infallible. Or less fallible than a library owner
 
@matszwecja 1st party is the code you write, 2nd party is the standard lib, and third party is anything you have to explicitly install with e.g. pip
 
2:04 PM
fair enough
@roganjosh Just a note, we're referring to the most common CPython implementation. There are implementations that are not open source.
 
@Riya safety is the industry standard for 3rd party vulnerabilities in python projects. they have a free tier as well
 
Fair shout. I tend not to think of the other implementations
 
 
1 hour later…
3:33 PM
I know hash collisions are extremely unlikely, but it's not ideal to implement equality like this, right?
def __eq__(self, other): return hash(self) == hash(other)
 
hash is not a cryptographic hash, so I'm pretty sure the chance of collisions are pretty high in that case
It just drops out an int, right?
It's covered here
 
thought as mcuh, just wanted a sanity check on that, because I was told "it's common to see this"
 
If I'm honest - I don't think I've ever seen hash() in the wild
 
surely to implemt __hash__ for a custom type it's commen though?
 
<shrug>
It probably does some stuff in the internals, but I don't recall it being used to check for anything in the libraries I've dug into. This is the point at which someone points out an obvious example of it, but I can't call one to mind
Wait, wouldn't it be the underlying implementation of a dict? PM has lots of posts about this. But a dict doesn't really care about false positives because it can bucket anything that has a collision
 
3:50 PM
sure but it uses __eq__ to check for collisions, how else would it detect them
 
@PeterT Yes, that's definitely not ideal. With the default hash function, the odds of collision aren't extremely unlikely. But on a modern 64 bit system they are pretty low: the odds of a "birthday paradox" collision are roughly 1 in 2**32. en.wikipedia.org/wiki/Birthday_attack
Modern CPython uses SipHash, which is a pretty good hash, and definitely superior to the original (a variant of FNV). It's certainly not crypto grade, but it has pretty good collision resistance, considering that it's only a 64 bit hash.
 
@PeterT it does detect them, and it buckets collided (?) objects and then goes through an O(N) search through that bucket. You just hope there's only 1 item in the bucket
That's different than checking for an absolute equality of object values, isn't it?
 
@roganjosh right, but I'm saying that if __eq__ is implemented in terms of hash it will prevent collisions from being detected
 
Yes, it would. I don't think hash() is meant for that. I meandered from the initial point
 
It's ok to use a simple hash to determine that 2 objects aren't equal. But it's not a Good Idea to use a simple hash to see if they are equal, unless you don't care about false positives. However, there are alternatives if you have big objects and you want to use a hash-based identity check. The hashlib module provides Blake2, which is cryptographic, can produce large hashes, and is blazingly fast.
A custom object is free to implement whatever __hash__ method it wants, and if you call hash() on such an object your custom hash code will be used rather than the default SipHash.
 
 
6 hours later…
10:26 PM
hey @roganjosh, I got around to turn the import-cycle-script into a proper project. took a little longer than I thought, and a little more code too. But if you still feel like adding better visualization, I think the internals are "stable" now.
 
Love the name!
 
@roganjosh Absolutely, I'd avoid inferring, I'd always iterate over and try-catch a list of possible format-strings. But have you checked the new behavior under 2.0? They did implement Allow format='mixed' in to_datetime #50972, although their example suggests it's only mixing cases where the format can unambiguously be guessed ("12 Jan 2000", "2000-01-13"), not DD-MM... and MM-DD...
Could be amusing to try dating invoices/transactions Feb 29 (of a non-leap-year) or Mar 32. I wonder how those are legally interpreted.
 
10:55 PM
@PeterT The 'right' answer in your context depends on how many objects you have, and what the real-world cost of a False-Positive match/misidentification is. For everyday images or social-media posts, it's probably fine; for medical or financial records, probably not. Do you want to use a non-standard hash to reduce that probability?
 
11:11 PM
In this example of a factory. github.com/faif/python-patterns/blob/master/patterns/creational/… Shouldn't class GreekLocalizer: be class GreekLocalizer(Localizer):? Since the typehint for the dict is Dict[str, Type[Localizer]] which says it should be a subclass of Localizer?
 
Subclassing isn't necessary because it's a typing.Protocol
 
11:51 PM
@Aran-Fey Thanks, I was pondering names for ~three days before I finally found this one that I liked. Naming things is hard work.
 

« first day (4604 days earlier)      last day (360 days later) »