« first day (4812 days earlier)      last day (361 days later) » 

01:38
14
Q: Why are some numpy datatypes JSON serializable and others not?

iFreilichtNumpy has a lot of different basic types, all of which are listed here. I've tracked down an issue in my program to float32s not being JSON-serializable, so I've started testing all datatypes from the list above: >>> import numpy as np >>> from json import dumps >>> dumps(np.bool(True)) 'true' >>>

@Aran-Fey What exactly is the user trying to do? Serialize a single numpy type? Serialize an object containing lots of thing, including some numpy types? (Serialize an ML model? If so, people on kaggle use joblib)
We need better more positive canonicals for "How do I serialize...?" with actual constructive advice, instead of just "Why can't I serialize type X?". Some users want exactness and don't care about memory-efficiency. Some care about crossplatform exactness, but most just want to serialize/deserialize on their own platform(/OS). etc.
Is the serialized object trusted, or do they care about importing malicious pickles? etc. etc.
02:04
@FelipeVieroGoulart Always post the full line with the error messages, so you know which part of the code caused it. Presumably in your case `response = openai.chat.completions.create(model=assistant_id, ...) is failing. You can find this discussed at community.openai.com/t/model-id-is-not-working-for-me/304779
@Aran-Fey How are you getting a comparison error? What is the traceback?
 
7 hours later…
08:47
@smci Just a single number. They aren't doing it on purpose though. I have a function that takes an int as input and they accidentally passed in a numpy int32. It's not that the user wants to serialize it, that's something my library does internally and I want it to gracefully handle numpy data types
@Peilonrayz I don't remember exactly, but it was something along the lines of "ufunc eq has no loop for types int32 and str"
 
1 hour later…
09:56
I'm torn. After years of rolling my own IPC I finally learned about multiprocessing.connection. And that happens precisely on the day I need async IPC.
10:12
Does my Celery container need the same env vars as my Django container?
I mean in Azure. My guess is that it does, but not that experienced with either Celery or Django.
 
2 hours later…
11:48
@smci I was just asking about that in the canonicals room actually
 
2 hours later…
14:04
@Aran-Fey This works better if you use isinstance: gist.github.com/secemp9/f8d7f18750b8d2cb65e3e49b2961580f
@Aran-Fey fyi, I can reproduce the error on Python 3.8 and numpy version 1.23.5
(sorry for double ping, forgot)
14:30
I really wish Pycharm would get "if foo: x = 5, ..... some lines, if foo: y = x*3" code, I feel like it would be easy to go trough a function and check if there is another condition with the same variable and check the values defined there. I always get the warning on the second condition x is maybe not defined, which it either is defined or the statement is never hit and the warning is unnecessary
foo might have changed inbetween
14:46
Working with asyncio streams and I don't get how the sync-write + async-drain works. If I write(message) and then drain, how does drain know I'm only interested in message being written? It doesn't, right? So how long can that thing block me from proceeding?
I'm tempted to throw a Lock at write+drain but that seems like such a waste...
15:09
@matszwecja if there are like 5 lines just check if it is being changed
What if there is some function call to a 100-line function?
Also, the rules shouldn't change based on how many lines you've put there
 
1 hour later…
16:16
@matszwecja I mean sure check ALL lines between the two ifs, but as a heuristic to make it faster I would do it only for x lines, like 50. And don't let perfect be the enemy of good. Better to solve 99% of peoples warnings than none
@matszwecja I don't see how that is relevant
 
4 hours later…
20:00
@Hakaishin where would the heuristic stop. Part of your assumption here is that you have 100% guaranteed that the two conditions are mutually exclusive. In many cases you're probably right, but you'd appreciate the warning in the one time you're not. That said, it would be good for you to be able to disable it on case-by-case basis in particular places once you've seen it once, but I don't use pycharm so I don't know
@NordineLotfi Interesting. I'm starting to suspect that there were some shenanigans with a virtualenv going on, and the code was running with a different numpy version than they reported
@MisterMiyagi I don't think it does. If I understand correctly, drain() could theoretically block forever if you have other tasks writing to the stream
I really struggle to understand the IT policy at this place. This is the company that asked me to tell them my password over the phone to their help centre, then reset it to, basically, "password" when I refused. At the same time, they force a restart of my browser literally once a week and now it looks like they wiped everything to sign me out of everything. le sigh
20:21
Was that on purpose? I've heard of people's data getting lost due to botched browser updates
Although... not recently
I don't know if the sign-out was on purpose but the forced restarts are because I keep getting notifications
I don't understand what any of the infra departments are. The last ticket I raised to get early access to the new company-wide system got "escalated" 5 times for the final response to be "ask someone on your team that knows". Eventually I figured it out and, by their own guide, the first thing to do is click "forgotten password" to actually set it initially (wut?). I did that and it told me I didn't have permission to reset my password...
Permission to reset your own password? Weird concept
I could probably ring up the central resource and tell them over the phone the password I want... they have 400k employees so this all seems utterly bonkers
 
1 hour later…
21:40
@Aran-Fey Sorry, but not enough information. a) Your library serializes, must use json? Or are you looking for recommendations for alternative packages? json seems to be incompatible with "want it to gracefully handle numpy data types". Why not just coerce numpy types back to Python builtins? b) Is that just spec creep, nice-to-have? What's the damage if you simply don't? c) This has many existing dupes, did you check them? ...
...The last thing we want is more narrow point-questions, instead of "How to serialize objects w mixed Python, numpy, pandas types?" which is the great white canonical noone has ever written.
@Aran-Fey Honest to God that reminds me of many experiences of feature-creep for no good reason. What's your reason not to simply tell the user "No we can't (serialize non-builtin types), you need to catch that exception and handle serialization yourself"? Are they paying you for adding this?
By the time you get the code to communicate that to the user, you could have coerced it
In the back of my head, np.ndarry([0]) (or other ints) could behave as scalars but this is years ago
I think it comes as a side effect of the broadcasting rules?
@KarlKnechtel Right but I'm saying to people in this room: that topic is a disaster area (™), the short answer is "No you can't (with json), so don't", the long answer is a (fairly prescriptive) canonical which none of us have the stomach to write and will only attract complaints. I don't see that freely sprinkling numpy types into base Python objects, then expecting the result to behave nicely with base Python libraries, is a good way to go.
For comparison:
Pickle vs cPickle vs json vs ...? (2010) Depends do we care more about interoperability, security, human readability, performance (output size, time to serialize, time to load)...

« first day (4812 days earlier)      last day (361 days later) »