« first day (905 days earlier)      last day (4044 days later) » 

3:59 AM
Hi Folks, I'm writing a python producer-consumer variant using multithreading and want to udnerstand the following point:-
the consumer needs some time for processing of the data after it does a get using Queue.get(). But, I dont want the producer to put anything in the queue until the data received from Queue.get() has been processed.
so, the default function calls in the Queue module dont seem to provide such a functionality. Is there any way I can achieve this?
 
4:33 AM
I had an idea on this. I was thinking of using a semaphore/lock to achieve this functionality. I would acquire the lock before the queue get operation and release it only after I'm done with processing of the data received from Queue.get(). Can I please ask you to correct this if I'm wrong. Thanks
 
 
1 hour later…
5:47 AM
hello guys, anyone working on celery or any kind of parallel processing framework here? I have a question regarding celery and its usecase
 
6:05 AM
just ask
 
 
1 hour later…
7:25 AM
mental note, should close chat before going on weekend
 
:P
I am proud of myself for this answer: stackoverflow.com/questions/15873350/… I actually figured out what the OP wanted.
 
nice answer ;)
 
Wow - I'm still not entirely sure what that question was.... but ummm, good answer, maybe?
 
i think inbar got it right, i would guess that the OP doesn't want as many nested for loops as his code shows.
 
7:41 AM
Yup - it's certainly a simplified version of the OP's code, but I'm not convinced it's what the OP really wants - probably wrong, but got a horrible feeling it's an XY problem
 
meh, that is what his code does - if it is not what the OP wants, he will probably make another question, or comment.
 
hey guys
 
I can return the multithread exec function
I use global var
after .join, can read var
oh , I know how to do
 
8:11 AM
Congratulations.
 
did he just answer himself in a way that i have no real idea what he was saying?
interesting
 
8:36 AM
Yeah.
 
reading several blogs on status vs exceptions... it feels like religion haha
 
morning everbody
 
hey colin
 
8:58 AM
Morning. Just seen a 2 year old question get bumped because someone went and edited out the "thanks" from the end. What a waste of time! I can only presume the person is after a badge or something.
 
"live and let live"
:)
 
still dont get the SO policy of not having the thanks
shrug
 
Well at least he brought a completely dead question out of the ground to attract close votes.
 
you even commented on it 2 years ago haha
 
Crikey, I hadn't spotted that
 
9:05 AM
your getting old (or depraved of caffeine)
 
cough deprived... depraved is something else
 
well.....
 
But you're probably right on both counts
 
you could always get a fleshlight
 
say it again: "live and let live" ... secondly there comes new threads into the SO queue, so it didn't really matter whether a x years old thread comes out of his tomb for several minutes or not
 
9:06 AM
Dude, save it for the teenagers
 
hi all can you help me with a question?
I'vo got an answer but I don't know very well how to adjust to my code..
1
Q: Python BadYieldError: yielded unknown object HTTPError('HTTP 599: Connection closed',)

sharkbaitI want to know why in this function: @tornado.gen.engine def check_status_changes(netid, sensid): como_url = "".join(['http://131.114.52:44444/ztc?netid=', str(netid), \ '&sensid=', str(sensid), '&start=-5s&end=-1s']) http_client = AsyncHTTPClient() response = yield ...

please
 
sorry, I don't work with tornado
 
I never even heard of it, can't help you mate
 
9:22 AM
@sharkbait: I've never used tornado and don't particularly want to either. I did a little research for your other question and it looked like the problem was that you were raising an exception instead of coding around the remote server not being there.
 
ok I thinked was a python problem sorry ;)
thank you anyway
 
I recommend you to wait until the last thread poster will maybe answer your question
ahm, no, that's a framework specific problem, or better "question" you have
 
yes I think so thank you
i'll wait
it's a dead or life matter haha
 
@JonClements you are actually trying to answer that horrible question? stackoverflow.com/questions/15875546/…
Just close it as too localized.
 
No - not trying to answer it - just trying to make a point before I do vote to close
 
9:28 AM
someone slap me... i can't even get a binary tree to work today
damn mondays
errr binary search*
 
slap
 
Now, I am off to eat some steak in honor of a work buddy who is leaving to another job! (the perks of working in a startup) :)
 
have fun :)
 
Ummm - steak hasn't been mentioned in a while ;)
 
9:38 AM
@Bas: bloody mondays
 
nods
 
Oh yeah - it is actually Monday - always have to double-think when @ColinO'Coal declares a day ;)
 
hehe
 
9:53 AM
Morning.
@mgilson not here? There is another Fortran-to-Python question for him.
 
Heya @MartijnPieters
 
Heya
@Jon: There is a MoinMoin ACL question, surely you are by now the expert to answer that? :-P
 
No - but that reminds me I should probably put some form of ACL on our main page ;)
 
Every time I see a 'what is the best IDE/Web Framework/GUI toolkit for Python' question on Google+ / Twitter / LinkedIn I twitch to hit the close link..
 
LOL
CLOSE ALL THE QUESTIONS!
 
10:05 AM
hehehe
 
Knew I had it somewhere...
 
Zac
hello
 
heya @Zac - welcome
 
Zac
does anyone here know ActionScript 3, or where I can get advice on actionscript 3 for creating an IRC bot in flash ?
I know this is the python channel
but thought you might be able to help anyway
 
I don't, but why would you want to create an IRC bot in flash anyway?
 
Zac
10:11 AM
I made an IRC bot in python but it just isn't working out as planned
well long story
I broadcast my desktop when playing computer games
i use a program called xsplit to do so
it has a nice drag and drop interface for adding flash media to the screen
 
Okay...
 
Zac
so I thought I'd like to write an IRC chat overlay for my broadcast
I currently have a bot written in python,but rather complicatedly it dumps the IRC chat to a text file that is then read by a title text plugin that displays the text on the broadcast
but it is a pain to format it and constantly read-writting to a file doesn't seem that good of a way to do it
so I thought I'd cut out the middle man
but
there are no good IRC libraries for AC3
AS3 ratger
rather
 
Keep the Python bot, but work out some form of IPC between it and AS3 ?
 
Zac
IPC ?
 
inter-process communication
 
Zac
10:15 AM
I'll google that
 
Might be possible to just have AS3 listen on a socket, and just constantly print, and have your IRC bot put to that socket
 
Zac
that sounds promising
 
Don't do AS3 but seems feasible, but details are up to you I'm afraid ;)
 
Zac
I
I'll look up sockets for python and do some reading
That'll keep me buzy today
 
Well - best of luck - out of curiosity - are you using a framework at all for your IRC bot?
 
GOD
10:19 AM
x = [] # 0.0537 usec per loop
x = list() # 0.171 usec per loop
x = [0] # 0.166 usec per loop
what you guys think
 
Hello, good folks
 
@GOD about what?
heya @pcalcao
 
GOD
@pcalcao hello
@JonClements about which one is fast, first wins !
@JonClements but some random guy tolled me to use list(), not sure why .
 
Well, [] is built into the grammar - list requires a name lookup and function call - so [] is always going to win
 
    >>> f = lambda: []
    >>> g = lambda: list()
    >>> h = lambda: [0]
    >>> import dis
    >>> dis.dis(f)
      1           0 BUILD_LIST               0
                  3 RETURN_VALUE
    >>> dis.dis(g)
      1           0 LOAD_GLOBAL              0 (list)
                  3 CALL_FUNCTION            0
                  6 RETURN_VALUE
    >>> dis.dis(h)
      1           0 LOAD_CONST               1 (0)
                  3 BUILD_LIST               1
                  6 RETURN_VALUE
 
GOD
10:24 AM
@JonClements thanks
 
jesus... i need to get slapped hard.... (max-min)/2 != the middle
 
GOD
@MattH cool
 
10:37 AM
Hey, just revising an old issue of mine and seem to be having an issue with pip clearing out my python path when it installs a package. Anyone any ideas how to fix this?stackoverflow.com/questions/14436196/…
 
GOD
@ChristopherHackett i use this script to install pip, peak.telecommunity.com/dist/ez_setup.py always work fine for me .
 
You on windows?
 
GOD
@ChristopherHackett yup on windows,
 
11:00 AM
So... who'll be the first to suggest changing operating system?
 
GOD
@pcalcao haha
 
@ChristopherHackett on a more serious note, I don't use windows at work or home, so I'm not of much help. Even so, I would suggest you start using Virtualenv, if you aren't already, perhaps it'll help.
If you really want to go crazy (I would), use Vagrant with a *nix VM running, it's quite doable for most projects, but this is just a very high level suggestion, take it with a grain of salt since I don't know what your needs are.
 
back.
 
11:16 AM
@pcalcao You
 
@MattH Seems like it, no one else was stepping up. We need to maintain some degree of order on the Internet. No reference to windows on a programming website/forum/wtv shall go without a corresponding "Windowz sux use <linux_distro> ffs!!".
 
11:43 AM
Should I start suggest that people using linux switch to OS X because I don't know Linux as well?
 
When i'm running uwsgi server with uwsgi --emperor /vassals/path -b 32768
Then getting error invalid request block size: 21327 (max 4096)...skip
While RPC from client getting that error.
 
I was exaggerating with the OS change suggestion, I think it showed. Still, if the problem has anything to do with pip and conflicting libraries/versions, starting to use Virtualenv can help.
 
12:08 PM
Hello people. Is there anybody awesome with graphs here in the room?
 
@Laci: What kind of awesome graphs are you after?
 
Yen's K-shortest path implementation with Igraph or Networkx
@MattH stackoverflow.com/questions/15878204/… I've posted details in this question. I hope at least the question it self I got it right this time :-P
 
12:24 PM
@Laci: The formatting of your code in your question is... sub optimal
 
@Laci is your problem in understanding why you should remove nodes and restore them after?
Also, you have some problems in your iterations
this: for i in range(0,len(a)-1):
should be: for i in range(0,len(a)):
because the upper bound of range is exclusive, so your current implementation won't even enter the second loop, since at first you'll only have one entry in "A", the shortest path.
 
@pcalcao yeah. Actaully both of those. I just can't get it right.
 
range(len(seq)) is an anti-pattern: enumerate(seq) almost always better
 
@MattH I believe you are very right. I've spent most of my time trying networkx implementations, different kind of DFS and BFS and there really isn't anything else out there on wide web
 
In the same way, in your upper iteration, you want: "for xk in range(1,k+1):"
Until you fix these limits, you won't get any results whatsoever.
That being said, keep in mind that I'm not a graph expert and I didn't even know of this algorithm until you mentioned it, but I can try and explain my understanding of the remove/restore mechanism.
 
12:34 PM
@pcalcao Thank you. I really appreciate these. I get back to work right away
 
@Laci: I'm a little confused. Do you want k shortest paths or graph searching?
 
@MattH shortest paths actually. There are actually tons of ways to do it beginning with the most simple: all_simple_paths with networkx but they all work for small graphs
my graph is very big. Every implementation looks like infinite and I really want at least 10-15 shortest paths like shortest, second shortest, third shortest and so on
Now I get back with pcalcao's corrections and at least make it look like a script. It is indeed embarassing but I'm very at my limits of understanding and perception
 
Ok... if the problem is also in the size of the Graph, you'll want to make sure that the algorithm that you're using to find the shortest path is as effective as possible.
 
@Laci: what problem are you trying to solve? Do you have a computer science graduate around to talk to about your problem?
 
Since I don't know the graph library you're using, I'm assuming you have an efficient implementation of Dijkstra in there, because it'll be needed.
 
12:39 PM
Why oh why do people insist on using XML when they aren't prepared to generate it or process it with XML tools? Why not use something simple like plain text or CSV?
0
Q: Bash Shell Script uses Sed to create and insert multiple lines after a particular line in an existing file

GlenPetersonThis code seems to work, but I'd like someone with shell-fu to review it. Is the temp file a good idea? How would I do without? Is sed the right tool? Any other general advice for improving my shell script Script/Code to Review: # Grab max field lengths from each .hbm.xml file and # put the...

 
@MattH no I don't. I would've asked it already. This somekind of self improving stuff
@pcalcao Yes that is very efficient. Moths of use proved it
@pcalcao Igraph usually is very good and fast. C based.
 
@Laci the algorithm isn't trivial, so I understand you might be struggling a bit with it. Do you understand what it does, in broader terms? If you don't, you need to understand it before you can implement it.
 
@GarethRees I suspect when the right answer to a codereview question is to not use the programming language you started with, you've probably gone offtopic for codereview :)
 
@pcalcao It is very well explained on Wikipedia also with a nice gif. I do watch it. Maybe it will get clearel meanwhile. I'm mostly confused with the delete-replace edges. I see it, I know it's necessary but can't really explain my self why
derails of details thing
 
@Wooble I always feel that code review is 50% "what's wrong with your way of doing it" and 50% "here's the right way to do it" and sometimes the right way is radically different from the wrong way. It would be unfair to leave the OP with the idea that all they needed to do was tidy up their code a little bit. See for example:
3
Q: Feedback on a small Python parser

waitinforatrainI'm working on a feature for the HamlPy (Haml for Django) project: About Haml For those who don't know, Haml is an indentation-based markup language which compiles to HTML: %ul#atheletes - for athelete in athelete_list %li.athelete{'id': 'athelete_{{ athelete.pk }}'}= athelete.name...

 
12:48 PM
@pcalcao I even wonder if the algorithm will actually behave in the end as expected. I mean would it be fast in a 2000 nodes graph. So much dissapointment so far with other ways. But have to find out.
 
@GarethRees Yeah, your answer was awesome. I just question the sanity of someone who says "I want to generate Java code from XML. I know, I'll use sed!"
 
It's worse than that: it's "I want to generate Java code from my database schema. I know, I'll write out parts of the schema description in (ill-formed) XML and then process the XML in shell+sed"
 
@Laci The algorithm works by forcing variations on the previously calculated paths. At first, you only have the best (shortest) path. Until you have the number of paths you need, you basically create variations on the graph to force Dijkstra to find another (less optimal), path.
You do this by taking the shortest path, and removing one edge from that path, and running the path-finding algorithm. This will guarantee that you'll get a new path.
After you do that, you have to restore the edge you removed, otherwise you would end up with an empty(ish) graph, and would be excluding valid paths.
 
Wow I actually start to get it. I really appreciate it!
 
No problem, glad to be of help.
 
1:05 PM
@pcalcao Well done, I couldn't figure out what @Laci wanted help with.
 
1:17 PM
Do we edit out superfluous tags?
0
Q: Python extract values from JSON

MatthewI'm looking to extract sets of values from a JSON and write them to a file. The format of the JSON is as follows: "interactions": [ { "type": "free", "input": [ [ 1, 4594, 119218, 0, [71, 46], [2295, 1492], [71, 46], [2295, 1492], 16017, 52079...

 
I do - I'd get rid of and
 
Why is there even a tag for ?
 
No idea
 
for loopy people?
 
I'd also be tempted to ditch as well
 
1:18 PM
(that means crazy in brittish english right?)
 
@BasJansen yes it does ;)
 
@BasJansen: That's loopy
 
so lets keep it :)
for the loopy people
 
^^
great
 
I'm clearly feeling grumpy today, I've just had a look for some facile tags and discovered that they all exist. I will now try to forget that they exist.
 
1:25 PM
@MattH Yepp. But I still struggle with python implementation. Must take on paper first :P The edge delete-add edge part is definitely rubbish as I do it
 
Today I learned that, if you use a function decorator to implement memoization, you'll hit the max recursion limit in half the time.
 
@Kevin: you'll hit the max recursion limit in half the time - something you do regularly?
 
Not for production-quality code, no. But when I'm doing dumb experiments, yes.
Ex. Today I wrote a recursive pow implementation and asked it to find pow(2,1000).
 
what would you lot recommend, if you slightly modify a standard function (like bisect_left) for your own purposes would you keep the name so that it's origins are recognizable like my_bisect_left or something else?
 
@Kevin Shouldn't that only take about 10 recursive calls?
 
1:41 PM
yes, but it also uses a recursive mult method that I implemented
 
(((((((((2**2*2)**2*2)**2*2)**2*2)**2)**2*2)**2)**2)**2)
 
anybody on twitter search? i am writing a simple script to read and collect hashtags where the text contains a particular hashtag which is a search criteria
in every hourly run
now i was wondering what if i get repeated tweets in next hours run .. how could i verify it ... i am just maintaining the count of the hashtags
it is kind of a find the most popular type of script
 
@BasJansen: Depends on the extent of the modification. Occasionally, I monkey patch.
 
modified input params and content to handle nD arrays
 
@Kevin If your implementation has anything at all to gain from memoization, it's only natural that you hit the max-recursion limits faster, since, well, you will be avoiding duplicate calculations. Memoization by itself isn't exactly meant to prevent maxing out your recursion limits. It can have that effect in certain circumstances, but it's not guaranteed.
 
1:44 PM
instead of only 1D
 
@BasJansen I'd probably create a new module for multidimensional bisect support
 
Memoization can prevent recursion limit errors if you're iteratively passing larger and larger arguments to your method. Ex. for i in range(1000): print fibonacci(i)
 
i should
it just feels such a tiny mod to call it 'new'
 
Remember, Namespaces are one honking great idea -- let's do more of those!
 
It can prevent them, yes, in some problems, where you have optimal sub-structure of the problems. And it won't so much prevent them as basically delay them.
More functional languages that rely on recursive methods to do... well, mostly anything, usually support constructs that are actually non-stack consuming, like tail-call optimization.
 
1:52 PM
fibonacci(1000) takes only 23 recursive calls, if you use a sensible recursion:
1
A: Matrix Exponentiation Algorithm for large values of N

Gareth ReesUse these recurrences: F2n−1 = Fn2 + Fn−12 F2n = (2Fn−1 + Fn) Fn together with memoization. For example, in Python you could use the @memoized decorator from the Python Decorator Library like this: @memoized def fibonacci_modulo(n, m): """Compute the nth Fibonacci number modulo m....

The values needed are 1, 2, 3, 4, 6, 7, 8, 14, 15, 16, 30, 31, 32, 61, 62, 63, 124, 125, 249, 250, 499, 500, 1000
 
Shame that the recurrences link is dead, I'd like to read more about it.
Well, not dead, but the anchor doesn't point to the proper subsection of the page.
 
@GarethRees very nice, those recurrences are very elegant.
 
Some editor made a mess of that section of the article, and then another editor deleted it because it was a mess. Oh Wikipedia, we love you but sometimes you are so rubbish.
 
I like that most of the examples now are "serious" algorithms you might use in actual code. GCD, binary search, etc. (Yes, I am aware that the fibonacci sequence has some practical applications, but those aren't usually covered in introductory articles)
 
2:09 PM
I can't seem to find a good link to replace it with. The identities I'm using are (28) and (58) from this MathWorld article but that's not as good a link (it doesn't have a derivation)
 
The Fibonacci number page talks about a "Matrix form". Is that it?
 
Ah yes, that'll do
Answer updated
One nice thing about using memoization is that you can see which recursive calls were needed
e.g. sorted(k[0] for k in fibonacci_modulo.cache.keys())
 
Ah, didn't know that, that can come in handy.
 
2:28 PM
That is kinda cool ;)
 
2:53 PM
quick question
lets say i have the following zeros matrix
data=np.zeros(25,dtype='float32, float32')
index = 0
for i in data:
i[0] = index*1.0 # i[0] contains 0..24
i[1] = index*random.random() # i[1] contains a random value between 0.0 and 1.0
index+=1
print array[len(array.nonzero())]
should then return the value for i[24] if i understand it correctly
yet it gives (1.0, 0.1168169304728508)
anyone got any idea?
(not sure why my code formatting got so shoddy while pasting)
 
What is array in your code?
 
array = data
array is the name i use in a function
you could stick print data[len(data.nonzero())] directly below the matrix creation and it gives the same effect
 
Have you looked at the result of data.nonzero()?
 
(array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24]),)
is the result for that
 
And what's its len?
 
2:58 PM
oh fudge
 
You see it now!
 
the array at start was kinda... a give away
 
need a new fast way to see how many elements are not 0 in my array then.. off to read some more docs
 
3:03 PM
off to read the .py file for that (to see if it iterates the array or what)
 
3:17 PM
@GarethRees Fascinating. I modded the decorator to count the calls per argument and tried it without the memoization. With memoize, there are a total of 95 calls and the highest cache-hit count is 2. Without memoize there are 738k calls and the highest per-argument call counts are in the region of 131k.
 
4:07 PM
@mgilson yeah - callable yo-yo'd a bit...
 
:)
Ooo ... I just realized it's lunch time.
be back in a bit.
 
enjoy :)
 
 
1 hour later…
5:31 PM
Hi Folks, I'm writing a python producer-consumer variant using multithreading and want to udnerstand the following point:-
the consumer needs some time for processing of the data after it does a get using Queue.get(). But, I dont want the producer to put anything in the queue until the data received from Queue.get() has been processed.
so, the default function calls in the Queue module dont seem to provide such a functionality. Is there any way I can achieve this?
 
Why shouldn't the producer keep on producing?
I have no idea what your expected use case is, just curious. Is it supposed to be a synchronous communication?
 
The actual problem is:
I have input data coming in from a database (which is connected to a portal). but, I can process only n requests in parallel at the max at any given time (which I have to do in parallel). I want to accept the next input data only I'm done processing with at least 1 request , so that another thread can start processing the next input data.
so, I had thought of this implementation:
the input data will be taken from the database (using polling) by a producer thread and will be put in a queue (of max n elements). Then, the main consumer thread will take out each of the n elements and create sub-threads and wait for any one to complete so that it can free the queue by 1 element. This will let the producer to put in the next element in the queue which would again be picked by the main consumer, to be sent to another sub-thread
 
Does your producer need to get a response from the consumer or the threads that handle the "jobs"?
 
not required from the producer's point of view. I had thought that I'll let the producer wait until the main consumer thread has finished processing 1 sub-thread. So, I want to end that wait from the sub-thread of the consumer as soon it has done its job, so that producer's wait ends and he can write 1 data to the queue, which will again be picked by the main consumer thread
PS: I really do not know whether I should use a queue for this use case or not.
 
Well... there are several options for that, including using a MessagingQueue like ZeroMQ or Rabbit or whatever, that's been build for this kind of things.
 
5:45 PM
zmq or a messaging queue would be a heavy duty stuff
I cant use any third party modules actually
 
If it's more of an academic thing, or a very low load application, you can use a simple queue indeed
I would cut out the consumer, if possible
spawn a pool of threads/processes that wait until there's something to retrieve from the queue
 
the environment does not support using any other thing than pure python or its batteries-included modules
 
How about, don't impose any size limit on the consumer's queue. The producer may add as many elements as it likes, and it's the consumer's responsibility to not remove more than N at once.
 
when the produces pushes something in the queue, one of the workers will get it, and do whatever he needs to do
I wouldn't worry with limiting the number of items in the queue, since I doubt you'll have memory problems from just this (if you do, perhaps you do need some heavy duty stuff).
This allows the producer to simply keep working, and the "workers" will handle jobs as they appear, no need to make the Producer block waiting.
 
thats the problem - the tasks that a sub-thread needs to perform use enough of the system's resources that I cant spawn more than n threads at any given point of time
 
5:48 PM
that's why I said a pool
For instance, spawn a pool of 10 working threads
all of them connect to the queue to listen for new jobs
when there's one, one of the threads will get it
if there are more jobs than threads, the queue will do what it's made to do, hold on to the jobs until a thread is free and "pops" it.
 
there is a small problem in this case-
lets assume that there are 2 threads in the pool. first thread completes its job before the second thread starts processing the queue.
In this case, the jobs wont be running in parallel. but, it will be a sequential run because the first thread only will process the second item from the queue
or may be I'm thinking wrong
 
and?
Parallel is overrated unless you actually have enough load to force your workers to work in parallel.
 
hmmm. well said pcalcao.
 
Let's say your thread takes 2 seconds to dispatch a job. And you only have a job every 10 seconds
Your architecture shouldn't care if the same thread processes every job there is.
If more jobs come at the same time, the other threads will pick them up.
The less any thread knows about what the others are doing, the better.
 
and to support your point here. Even if the second item is being processed by the first thread, the second thread wil pick up the 3rd item and so on...
 
5:53 PM
Exactly
What I had in mind is something like this: en.wikipedia.org/wiki/Thread_pool_pattern
 
hey pcalcao, thanks a lot for clearing my doubts
There is one small thing troubling me however.
 
There's even a seemingly interesting article about threads, queues and python: ibm.com/developerworks/aix/library/au-threadingpython
 
I can sense a change in my requirement that the number of threads that require to be running can change e.g. be changed at run time.
I cant gaze at how this thread pool would scale up or scale down
 
This is what I'm imagining:
#consumer thread
subThreads = []
while True:
    #clean up threads that have finished running
    subThreads = filter(lambda t: not t.isDone(), subThreads)

    #zzz
    time.sleep(1)

    #don't do anything if there are no tasks, or if we're overloaded
    if len(subThreads) >= N or myTaskQueue.isEmpty():
        continue

    #make a thread for the next task
    task = myTaskQueue.pop()
    thread = makeThread(task)
    subThreads.append(thread)
    thread.run()
With this pattern, you can modify at run time how many threads may run simultaneously, by modifying the variable N.
 
I believe you'll always need to have a max number of threads, and that's what you want to set up in your Pool, if you do use a Pool.
 
6:01 PM
yup. but, that max can scale up or down
 
More threads will only scale so far, in any situation, depending on what bottlenecks your application has, if the tasks are IO/CPU bound, etc.
Ah, just one more thing... in case you don't know about the GIL, it may impact the performance of your solution, especially if the tasks are more on the CPU bound side of the spectrum.
 
yes. thats true about GIL. but, my tasks are more of network bound than cpu bound. so, I'm not worrying about GIL
 
I'm off for today, I hope you gained at least some food for thought :P
 
Kevin's implementation looks correct and will be of much help. but, if I'm going to use a thread pool, I'm just worried about scaling the thread pool
@pcalcao: thanks a lot for clearing my doubts.
If I come up with anything on scaling the thread pool, I'll post
 
No problem, that kind of problems are interesting, so it's a pleasure.
That might be worthy of a SO question.
See you guys tomorrow.
 
6:13 PM
see ya
 
dinner for me
you didn't even know I was here, did ya.. :-P
waves.
 
actually, this is an awesome example of a thread waiting to server the questions when 1 thread logs off :)
@pcalcao this gave me a lot of confidence on posting my question on SO without having written the code. otherwise, I was fearing of that question being closed out or downvoted
@MartijnPieters: Can you please extend the idea of a scalable thread pool that we were trying to discuss out here
 
He's indisposed with dinner at the moment. That was a goodbye wave rather than a hello wave.
 
does anyone here use sqlalchemy
 
please post your question about sqlalchemy. there are many of its users out here
 
6:28 PM
Mar 28 at 13:16, by Martijn Pieters
Just ask the question, don't ask to ask :-)
 
you actually dug it out
 
it's on the right of this window
 
6:40 PM
@Kevin: I'm partially using your solution and partially what pcalcao suggested, to create a scalable thread pool
'm off now. I'll post the code soon.
 
7:07 PM
hello, I am trying to fix this list comprehension: stackoverflow.com/a/12446208/231917 since it gives an invalid syntax error. Any suggestions?
 
Hmm, don't you need iteritems to iterate through a dict's keys and values simultaneously?
cf.insert(uuid.uuid4(), [{k: str(v) for k, v in d.iteritems()} for d in x])
But even so, I wouldn't expect a SyntaxError there. What version of Python are you using?
 
@zengr If you're using 2.6 or lower, that should be cf.insert(uuid.uuid4(), [dict((k, str(v)) for k, v in d.iteritems()) for d in x]) I believe.
 
7:51 PM
Wow - I hate to admit it, but I'm actually quite impressed by Windows 8 :(
 
Not a Microsoft fan, I take it?
 
user1125394
py.NET
 
@Kevin no - not a MS fan ;)
 
I mostly stick with MS because I like the desktop background with the rolling green hill.
It's very pleasant.
(Although it is not quite as idyllic in person, apparently.)
 
8:25 PM
Yeah - does look a bit of a let down in reality ;)
 

« first day (905 days earlier)      last day (4044 days later) »