last day (14 days later) » 

2:37 PM
1
Q: Preforking a Multithreaded Python application

the_drowI have a Python program that is already multithreaded and I'd like to replace some of the threads with processes in order to reduce context switching and utilize gevent for async I/O. The main process is I/O bound so I'd like to use gevent in order to be able to handle a lot of concurrent I/O. We...

 
If your workers truly are CPU-bound, threads are exactly not the way to scale. Because of the GIL. So your overall architecture looks perverted (or better inverted ;)) to me - you could use threads for your Receiver, because they'll block. Being async via gevent resource footprint, but shouldn't affect performance that much. But then your workers should be standard python lib's multiprocessing-based. And you could pre-allocate a few dozen of them, because again if you were 100% CPU-bound, it makes no sense having more processes than Cores. Add some blocking IO to it, so add some process.
 
I am aware of the GIL but not afraid of it as much as other programmers do. For what reason would you not use gevent for I/O only tasks? The autoscaling feature is a feature that I really need since our workload varies. Creating and destroying processes is much more expensive. Should I not be concerned about it? Also having as many processes as many cores that I have means that I can only process a few requests at the time for each server which is pretty expensive.
Also why should one choose blocking I/O over nonblocking I/O?
 
well, you contradict yourself here: if you are CPU-bound in your workers, you must be afraid of the GIL. Because it effectively serializes execution. Which is the exact reason for the introduction of the multiprocessing-module. Regarding blocking/threaded IO vs. non-blocking: I'm not saying you shouldn't use gevent. But AFAIK async processes are no magical cure for anything, all they do is reduce the footprint of your process resource-wise (less memory) because the OS manages the IO dispatch instead of you having threads sitting on it. Relevant in high-load scenarios. No idea if you have one
To elaborate some more: if your task is truly 100% CPU-bound, how do you imagine you can serve more requests than your number of cores? It's physically impossible. As few things are a 100% CPU bound, you can have a few more processes doing the work, but creating and re-creating them every now and then is not a burden your OS and overall app will suffer from. But if you want to serve more requests than you have cores, you need a cluster of machines. Again with processes doing the heavy lifting. In C/C++/Java, you could instead use threads. But there, too, apply the limits of core #.
 
My tasks are mostly CPU bound. If I trade context switches for parallelism I can process more events concurrently. Yes it will be a bit slower but won't it be quicker (if I define a sensible threshold of maximum threads in the worker's pool) then blocking the response until a worker is available?
 
Well, nothing is 100% CPU bound. Otherwise, the task couldn't get parameters, nor influence the world through output. But it's a fact that a Python process with multiple threads can only run on one core. So - you have to launch a pool of processes to utilise several cores. And as your tasks need IO, yes, you can use threads. But what is your reasoning of doing that in your workers when your receiver is supposedly better of using async IO for the exact same thing - dispatching IO?
just trying this out, as Stackoverflow suggested it :)
 
3:03 PM
Hi
 
hi. never tried this :)
 
So, how would you design such a program?
 
I have designed similar programs (at least from the outset)
and there is no way aronud using multiprocessing
there are various ways to use it, but something like a worker pool should fit the bill I'd say
it will spawn the processes, but not for each request. instead, a pool. AFAIK you can configure it to only serve so many requests with one process, as a safety measure against degradation.
what you do inside these processes is of course up to you.
but you need them for core utilization. inside the worker, you can use threads. but why do you prefer them over gevent for this?
there is an argument for max tasks per child.
 
Because I am crunching data and only fetching some data from other sources at times
I only plan to use threads inside my workers
 
where do you fetch the data from?
 
3:11 PM
in the main process which uses gevent
hold on. going to a 10 minutes meeting
 
no sweat. try catching me later, I'll be around for a while I guess.
 
3:22 PM
brb
I meant b
as in back :P
 
haha
ok.
 
I'm pulling the requests in the main process from SQS
 
so - do your workers only crunch, or do they perform IO (beyond the obvious communication with the receiver)
SQS?
 
Sometimes the workers will fetch additional data from databases or cache
Amazon's queuing service
 
ah, that thing. ok.
 
3:24 PM
Horrible piece of shit but it works
 
so the database or cache requests, how frequent are they in comparison to your crunching ?
 
Right now very but we'll be reducing them soon
 
hm. it's hard to really give a final advice to this beyond what i already said. You must use multiprocessing - otherwise, you won't use your CPU resources. You can use threading in your workers to allow for blocking IO for thes db/cache requests.
 
But using threads won't allow me to process more events concurrently if I use gevent for the I/O?
What if I use time.sleep(0) after each operation? (Nasty but it will release the GIL)
 
doesn't change a dime.
you can use gevent, or threading to the same effect of accepting & buffering requests.
but this will not change the amount of concurrent processing possible - that's always 1 python bytecode per python process.
now if you use libraries that release the GIL (such as Numpy) and spend a lot of time in them (and I mean a lot), then you might not suffer from GIL congestion that much. But that's a very unlikely scenario.
I had an actual app similar to what you describe here
instead of SQS, I used RabbitMQ
 
3:33 PM
Leave your answer and I'll accept it. I don't like it but it's the correct one.
 
postgres for storage
I started usiing threads in hope of achieving what you want - but to no avail.
the boost I got from using multiprocessing was amazing, all of a sudden all cores maxed out. so - yes, my advice stands :)
will do so.
 
What happens if I use more processes than cores?
 
that depends on how much IO they really use. If it's a lot, not much. If they are truly CPU-bound, they will grind the system quite hard. It depends on a few factors what happens then, but I have actually configured my system to leave a spare core or two for the general system.
 
4:20 PM
 

last day (14 days later) »