« first day (4588 days earlier)      last day (363 days later) » 

3:28 AM
I'm proud to announce that I am the 8th recipient of the Stellar Answer badge: stackoverflow.com/help/badges/11221/stellar-answer
10
 
4:10 AM
@cs95 congratulation :)
 
 
1 hour later…
5:11 AM
@cs95 Nice
 
5:27 AM
Hello, does someone have a knowledge about Python asyncio? I want to measure time spent blocking, and need some help with that.
 
5:52 AM
Like this?
start = time.monotonic()
await something
duration = time.monotonic() - start
 
The problem is that I want to measure time that current coroutine spends doing something, without waiting part.
I prepared a question with that.
https://stackoverflow.com/questions/76201333/how-can-time-spent-in-asynchronous-generators-be-measured
I was hoping that maybe someone here might want to discus it with me.
 
thank you
 
What do you mean without waiting?
 
I mean that I dont want to include time spent waiting for other coroutines. Just the time that given coroutine was actively executed.
 
That's... not gonna be easy
 
 
3 hours later…
9:21 AM
Turns out I still don't understand how asyncio works under the hood. I thought you could execute a coroutine by simply looping over the iterator returned from __await__(), but no
async def main():
    coro = asyncio.sleep(5)
    for value in coro.__await__():
        print(value)

asyncio.run(main())

# Output
# <Future pending>
# RuntimeError: await wasn't used with future
 
Well, certain things still expect to talk to the event loop during that iteration.
 
I guess in hindsight it makes sense, iterators don't allow the mainloop to send input back into the coroutine, so the iterator has to return a container (a Future) where the mainloop can put the value that the coroutine is waiting for
Does that make sense? I think it does. Idunno
That raises the question how Future.__await__ works, I suppose
 
9:37 AM
Afraid you'll have to look at some C code for that.
 
I'll just mark it down as "magic"
async def main():
    coro = asyncio.sleep(5)
    for future in coro.__await__():
        future.set_result('dummy data')
That works
 
 
2 hours later…
11:45 AM
Does anyone know of a good tool for analysing circular imports? I now have ~30 modules that I'm trying to make into a package and the explosions are quite something to track down individually
Spyder has allowed this repo to become a thing but the imports are utterly broken :'(
 
12:19 PM
Well, I do know it's not pycycle
(esc) joshpilkington@Joshs-MacBook-Pro esc-laep % pycycle --here
Parsing of file failed: /Users/joshpilkington/Desktop/git/laep/esc-laep/src/merge_performance_test.py
There were errors during the operation, perhaps you are trying to parse python 3 project, with python 2 version of the script? (or vice versa)
Project successfully transformed to AST, checking imports for cycles..
No worries, no cycles here!
If you think some cycle was missed, please open an Issue on Github.
Finished.
Beep boop, LGTM
 
 
2 hours later…
2:02 PM
@cs95 wow, 4 out of 8 answers in that list are python related, what a ratio
@roganjosh I think I could write a script that does that, but I don't know how to do visualization
 
If your hypothetical script generates a graph you could maybe just plop it in graphviz?
 
that's what I was going to start with, I'll have to try and find out how to make a node red if it has a cycle
 
Meet my new profile pic! His name is NAT, he was the mascot for the company where my grandfather worked, National Screw and Manufacturing (no jokes, please). NAT was the logo character printed on product boxes, marketing swag, etc. Employees would receive a little desktop NAT about 4" tall, made of brass and steel parts and a wooden head. The story is that my grandfather made the first one (he worked in the machine shop) as a toy for my brother, and they caught on in the company.
4
 
2:19 PM
aw, that's heartwarming. thanks for sharing
 
nice :o
 
@Arne At the moment I think what screws is it the fact that the modules can't initialise. I would imagine such a program needs to do the analysis entirely offline
 
wait, are you telling me you got dynamic imports?
 
They're not dynamic but there are circular imports everywhere. I'm entirely fresh to this project so I can't figure out where they're importing each other
So far I think people have been running this on a per-module basis in spyder and it's done all the heavy lifting to fix the imports. Now I've tried to make a package, it's completely blowing up
Don't build packages in spyder, kids :/
@Arne what's the basis of what you think you could do here? My understanding would still be that you need the python interpreter running?
 
no, just build the AST of each file, same as pycycle was doing. that doesn't run the code, so no runtime errors possible
 
2:35 PM
I might give it another go in that case. I found part of what was crashing was %% magic methods in the middle of a module
 
> %% magic methods
what? is that something spyder-specific
 
2:51 PM
Yeah, spyder and jupyter. It's from ipython
Just to spice things up, you see. It's too easy if all the modules in the package could be syntactically valid too
Even with that fixed, pycycle doesn't seem to be working. I'll just keep going manually
 
> %conda¶
>
> Run the conda package manager within the current kernel.
I'm reeling already, spicy stuff for sure
"people coming from other programming languages feel that python can be a bit of a foot-gun with its dynamic typing and script-like nature, and say that it's missing important safety nets. well, we felt that it was missing multi-barrels."
because if your foot has to go, the floor should go with it
4
 
Spyder really just shouldn't be doing this yam. I'm glad I dropped it years ago but I know how it allowed, nay encouraged, this mess
 
3:37 PM
hmm, the fact that __init__.py files are executed during imports makes this a bit more complicated. any imports that happen in them propagate to modules importing any of their children
@roganjosh can't say what exactly was messed up in your project at hand, but if there are imports in __init__.py files you might be able to simplify your problem by getting rid of all of those
 
3:51 PM
That's what I spent the day putting in __init__.py :P
 
Is it common to combine multiprocess and multithread in the same application or am I asking for trouble?
 
The problem is there's a config god-object that actually imports a load of modules that it itself is imported into. Honestly, I'm tempted to go get a kitten so it can cry and I can hold it up to the screen
In any case, I think I've whittled it down to line 28 of a 30 line __init__.py blowing up now so I'm nearly there. Sorry for my rants, I was just curious if there was a tool at hand to run over the package
 
4:11 PM
@hello It's not impossible, I could see a semi-plausible scenario where this might end up happening. But it's not common.
If I wanted multiple processes, and one or more of them needing to do something asynchronous, I would break them up a little more overtly, and not have them run inside some multiprocessing basket. Or use something like supervisor or uvicorn to manage multiple processes that would communicate using some network connection.
But I should really ask, what are you trying to accomplish by doing this?
 
Ha! Makes sense.
I'm probably overthinking my current situation by even imagining this setup. I have a noisy log producer that is sending messages to rabbitmq and I'm using Python (pika and Pydantic) to parse, validate and insert into Postgres. I'm at around 4k messages/sec, but looking to up that to at least double.

My thinking is to start a specified number of processes that can consume the stream of messages from rabbitmq, wait until around 1000 msgs in their list, then each process can send to a pool of threads to parse, validate and send to a Postgres connection pool.
 
4k messages/sec is anything but trivial
If you split that across processes I think you might struggle. Presumably you want to go for bulk uploads
 
Currently doing bulk uploads as well at 10k
 
From a single process?
 
Yeah
 
4:27 PM
Interesting. I'm pretty surprised that a pydantic pipeline can run at that speed. I think I'll sit this one out because I can't get that kind of responsiveness out of the C++ servers I hammer, let alone python ones
 
I'm doing 0.0452580451965332 sec for 10k messages to Postgres using COPY, which I believe is decent enough.
 
I don't think you can beat COPY. Is that with setting indexes too?
 
I have no indexes at the moment I'm fine without...for now
 
Oooo, that could be a spanner in the works
 
:(
 
4:36 PM
That's a lot of data to be pumping into your db. More than I've ever handled. I obviously don't know your application but just plonking data without an index will become a problem for retrieval
 
After adding an index on the two columns(out of 7) that I use on most of my WHEREs:
The average of 10k insert is now: 0.10083197442515082 sec
 
That's still pretty decent
 
I think so. I don't think my bottleneck will happen there....at least not at this point.
 
What is the bottleneck, then? It seems you have this pretty much covered in a single process doing the batching for you
 
4:51 PM
@roganjosh At this point, I am clueless, really. Pydantic (v2 using Rust) seems fast enough (0.0000193119 sec), and my string parsing seems decent (0.14925919532 sec), but for whatever reason, I can't seem to break 4k messages/sec from RabbitMQ.
Not even when I increase the number of processes.
 
4k requests/sec is insane. Do you not have engineers on your side to talk about K8s or something? It's not really a python task at this point
 
I really do not want to involve K8s. I really dont.
 
Spoken as someone that, I assume, hasn't used multiprocessing before in Python :P
 
LOL
Currently using ProcessPoolExecutor but no luck with speed gain
 
You won't win. I don't think you can have a handy processor pool just hanging around. The overhead of launching the pool will probably kill you every time with your batching
I'm happy to be wrong, but I think it's outside of python's capability here
 
5:02 PM
So, is K8s not just going to run the python script a few more times?
In separate pods (not even sure I know what that means, I hear it often)
 
That, or another language to take this weight of requests. I did try to back out earlier and then kept going. Maybe someone knows a trick I don't, but multiprocessing is grossly inefficient with it copying over the whole namespace
 
Sigh!
 
There's still hope yet! I guess it's just out of my domain now. I think I'd be surprised if it can rise to the challenge, though
Best of luck :)
 
I'll need all I can get.
 
0,00025s per message isn't really something I would try in an interpreted language. You are not just going to send and receive, means to serialize and de-serialize the data, you also need to interpret that data to work with it. Without claiming to be an expert on the field, but going with K8s seems reasonable, because go-lang´s compiler is crazy fast, faster than C and therefor faster as python could ever be.
 
5:16 PM
You don't need K8s to use golang and I'm not sure why the compiler is mentioned
 
I just assumed that K8s makes use of it.
 
Perhaps I'm missing something here, if a process is not faster when I use multiprocess in python, how's K8s going to make it faster, potentially?
 
Load balancing
 
I have that in front of the RabbitMQ cluster at the moment.
3 nodes
 
But the bottleneck is consuming that queue?
 
5:23 PM
That is the only reasonable conclusion I could come to.
 
In theory you could multiprocess consumption of the queue. That might be something to try. I forgot about the rabbitMQ intermediary
 
That is what I'm doing right now. Multiple processes consuming the same queue 300 messages at a time (prefetch_count)
Each with independent connection and channel
 
@hello Python's multiprocessing module waste a lot of resources when creating the process, and Python creating/deleting object constantly make this even slower (which most interpreted language have to do to work generally)
Your best bet is either a compiled language (C, Go, etc) or you use existing project that can handle such heavy load/work better for parallelism compared to Python (eg: K8)
@Thingamabobs to be honest, most difference between XYZ languages and C/Assembly is that you can have better "default" speedup in those so called "faster language", than you'd get in C or Assembly, since you'd have to define/clearly make it faster or implement different optimization yourself
@roganjosh I think the fact that the project is written in a so called fast language like Golang will usually automatically make this faster than if it was in say, Python. That still depends though and can still be slower, so that's just an assumption (which I also make sometimes)
 
5:42 PM
I genuinely have no idea what point you're addressing
 
what I was trying to say was, that this was probably just an assumption
but I might be reading too much into it
 
K8s can run any language. It doesn't need to be Go or anything else. I'm not sure that this speculation from both posts adds anything helpful
If you can help with the lateral scaling, please do so. But that is the open question here
 
yeah, I get what you mean
 
 
4 hours later…
9:58 PM
you're probably done fixing it manually by now, but it was decent fun to work out. still a bunch of caveats that could be sussed out, but the core is there
- namespaces aren't common, but my guess it that they aren't handled well
- if your distribution name and source-folder name are not equal, it's going to trip
- if imports are guarded with `if`s for typing or python versions, or dynamic imports in functions - all are treated equally as top-level imports
- `__init__.py`s can import their sub-packages without creating a cycle, don't know how to handle that without accidentally missing actual issues
oh, and the plot is way to small so for any project that isn't tiny it looks like a mess. I'm giving up trying to make the plot look good though.
 
10:32 PM
Folks, I see Python libraries which provide multiple classes. 4 to 6 classes for external consumption, and a few internal helpers. The library consists of a single .py file, and the file with all these classes becomes quite large. Let's say the .py file is 3kloc, give or take.
Is it possible to split the files into different classes but keep them as one package?
The calling code would have one import statements. It would be transparent to the calling code whether there is a single .py file, or multiple .py files .
 
Are these libraries that someone else maintains? You should check with them first before embarking on such a restructuring effort. (And yes, they can be broken up into multiple files, and still imported as a single package.)
 
10:49 PM
I'm asking so that I can put together my own libraries from scratch.
My question doesn't come from restructuring existing libraries.
Here's a more general request. Could anyone point me to a "blueprint" for a good file structure for a Python package?
 
Golang definitely performed much better. Using a single worker each (on Macbook Pro) to grab messages from the queue and doing nothing to it, just remove from the queue:
Python: 12358 msg/sec
Go: 28030 msg/sec
 

« first day (4588 days earlier)      last day (363 days later) »