« first day (3889 days earlier)      last day (1045 days later) » 
00:00 - 17:0017:00 - 21:00

5:26 PM
@TheShortestMustacheTheorem this is already answered here
 
If multiple threads are waiting on a threading.Lock, do they get unblocked in the order that they tried to acquire the lock, or is it unspecified which thread gets unblocked first?
Dunno how I missed it the first time reading the docs for Lock, but of course it is there.
> When more than one thread is blocked in acquire() waiting for the state to turn to unlocked, only one thread proceeds when a release() call resets the state to unlocked; which one of the waiting threads proceeds is not defined, and may vary across implementations.
 
@alkasm Tim Peters, no less, too
 
nice. Yeah this is as I expected, but I was just using a lock to serialize execution between threads and realized I didn't actually know for sure if Python did anything special here
 
I'm curious if there is a compelling use case for order-guaranteed locks. My simple threading experiments usually work whether I assume order or not.
 
@alkasm I'm curious about the use-case for this
 
5:37 PM
Then again, 95% of them use exactly two threads, so competing acquire() calls are effectively impossible
 
well, locks serialize execution. So, ordering is a natural question
I'm pretty sure asyncio.Locks preserve order, for instance.
 
But what is the context of needing to lock threads in an order?
 
I can imagine an abstract system where three threads compete over a lock, and if the user is very unlucky, then threads A and B will pass priority between themselves while thread C starves. I'm not sure how often that scenario comes up in practice.
 
My example use-case is probably not particularly compelling, but I am writing a little thing for rate limiting requests through an interceptor. On the user side this is easy, on the interceptor side its a little tricky since the framework I'm using will call the interceptor asynchronously.
Using a basic lock is an easy way to serialize the execution so that I can ship off the requests at a constant rate (leaky bucket). Queues are somewhat annoying here, because the interceptor shouldn't return until the work is done, so I would have to block inside each interceptor until the request was sent, and thus need communication from queue worker back to queue submitter which seems blah. (if this were on the user side, then a queue would be ezpz)
 
And by "I'm not sure how often X happens" I don't mean "I am skeptical that X happens often". Rather, I mean "I have no data on the frequency of X"
 
5:47 PM
@roganjosh Thank you very much for informing the link. It is very useful. Especially the np.random.rand(). :-)
 
@alkasm it just strikes me that Flask might have sorted this with thread.local()
Then the limiting could be done with nginx or something <lots of hand waving>
Or a kafka queue maybe. I've potentially tied myself in knots without knowing what the component parts do
 
Let me see if I understand the scenario. The interceptor waits for a request from the user, let's say by watching an HTTP port. When it gets a request, it checks to see if enough time has passed since the last request. If not, it sleeps for a while. Then it gives the request to a worker thread and waits until the worker is finished. Then it goes back to watching the port.
 
Roughly correct, though it's not through watching an HTTP port, the request is intercepted before going out on a port.
 
@TheShortestMustacheTheorem random.random() is faster for single values (I've not tested against the latest implementations I have), but np.random.rand()will win after some tipping point. Don't fall into the trap of thinking that numpy versions of python builtins will win-out on small data sets. If you have a list of 3 values, sum() will win over np.sum() because it has to do implicit conversion to an array first
 
I suspect I am missing something, because so far, this whole design can be implemented without using threading.
 
5:58 PM
@Kevin the thing you are missing is that the framework asynchronously calls my interceptor.
(fwiw its just grpc: grpc.github.io/grpc/python/…)
 
What is the "interceptor"?
 
In my scenario, the interceptor is the thing waiting for messages on the port. I think my scenario is backwards to alkasm's, so I guess his is... something that waits for input from the local user, and decides which port to send it out on?
 
what's good way to parallelize this?
new_dict = {k:[long_filtering_list_comp_on_v] for k,v in old_dict.items()}
len(old_dict) = 1k
sum(len(v) for v in old_dict.values()) = 3M
 
Cabbage. I'm back! :)
 
cbg
 
6:02 PM
My goodness! PM is back!
 
long time! Potato?
 
I missed you. (not sarcasm)
 
I needed a break. I didn't think it'd be quite this long...
 
Understandable
 
How the devil are you?
 
6:03 PM
pm!
 
@Kevin I make a request using the grpc framework, which does its scheduling, muxing, etc in C-land. It looks up if any interceptors are registered. If they are, it runs the request through my interceptor. I can serialize with a Lock and sleep a bit before sending outgoing requests.
 
@inspectorG4dget Are all your v values serializable?
I guess, specifically, pickleable?
 
I missed you guys too. I've been lurking for a week or two. I didn't want to come back without knowing what's been happening.
 
Gotta see how many people are saying "next time I see PM, I'm going to ask him for that ten dollars he borrowed"
 
6:06 PM
... about that...
 
oh that reminds me
 
hmm, that rings a bell...
 
Oh yeah, now I remember why we ran him out of town on a rail in the first place. Serial loan fraud!
 
Give me back my Frosted Flakes!
 
@Kevin Your nested parentheses stuff goes by the formal name of Dyck language. It's intimately connected to Catalan numbers.
 
6:08 PM
Luckily the First International Quatloo Bank of Kevin is too big to fail, and I was able to establish an aid network using my big fat bailout check
 
@inspectorG4dget you have an iterable, and an independent task on each item in said iterable. if you turned the filtering list comp into a function, voila you're done.
 
I wrote some relevant Python code a few years ago, but I haven't thought much about that stuff since then, so I'm a bit rusty. stackoverflow.com/a/41310973/4014959 There's some excellent info & links in the other answers on that page.
 
@ParitoshSingh I thought that'd work. Somehow, using multiprocessing.Pool(n).map(independent_func) took longer than doing it a single thread. I'm hoping I don't have to write my own map/reduce implementation, hoping that my usage of multiprocessing/pathos .Pool is somehow flawed
 
@PM2Ring Ooh, very nice. Your f helper function is about where I'm at in my own attempt.
 
@Kevin No worries.
 
6:14 PM
interesting, i suppose it wouldn't be easy to make a mcve of it? specifically the part where we can reproduce the speed differences
 
I'm going to try to make one. There's some stuff I need to redact, which is making things a bit challenging
 
@inspectorG4dget Maybe the processes themselves run quickly, but starting them up and passing them data is expensive.
If you're passing 3 million wossnames through the narrow pipe that connects the processes, that might be a bottleneck
 
@Kevin that's a really good idea. I should be able to test this out relatively easily. Brb
 
Why is this not something for pandas, @inspectorG4dget?
 
# single thread (0.3sec)
b = {k: [(t, c)
                 for t, c in v
                 if (c is None) or (t <= some_const)
                 ]
             for k, v in a.items()
             }

# parallel (15sec)
b = dict(pool.imap(lambda t: (t[0],
                                      [(t, c)
                                       for t, c in t[1]
                                       if (c is None) or (t <= some_const)
                                       ]),

                           a.items()
 
6:20 PM
That's really hard for me to read and I don't know what it's doing
 
@roganjosh I honestly hadn't thought to do this in pandas, because I'd need it in only the one spot. That's a valid suggestion, thanks
yeah, sorry. I'm going to try a few things before I come back with more. Thanks for the suggestions, folks
 
I wonder if multiprocessing.shared_memory is good for cutting down on the overhead of process setup. Looks like it plays nicely with np arrays, if you're into that kind of thing
 
It's still going to need a Manager obj for IPC?
 
im just surprised this gap is so stark in timings. does this mean that even though it was like 3m datapoints each...oh..i see. actually datapoints are close to 3000 items in each value roughly
 
Are you sure that the multiprocessed approach even gives the same result as a serial approach?
 
6:23 PM
@inspectorG4dget There's no point speeding up a 0.3s program via multiprocessing. Especially if it means sending 3M elements around.
 
1000 k, v pairs, roughly 3000 datapoints in each v. that's probably nothing for python in terms of individual runtimes.
 
Oh, neat, SharedMemory is accessible even from a non-child Python process that was opened from a perfectly ordinary command prompt. I can think of a couple fun and inadvisable things to do with that.
 
laurel
well im assuming this is an mcve..well..maybe me without the cv.
 
"shared memory blocks may outlive the original process that created them." I wonder if it can stay around even if all your Python processes shut down (gracefully). I wonder if the typical OS has any mechanism for noticing that nobody is using the chunk of shared memory that got requested a while back.
 
6:39 PM
@Kevin futures certainly can. I can orphan them (not sure how ctrl + c translates) but they can keep going
 
import cv2
import numpy as np


# data = np.array([[[255, 0, 0], [0, 255, 0], [0, 0, 255]], [
#     [0, 0, 0], [128, 128, 128], [255, 255, 255], ]], dtype=np.uint8)

data = cv2.imread("images/family.jpg")

fps = 30
while cv2.waitKey(int(1000 / fps)) != 27:  # press ESC key to exit

    coe = np.random.dirichlet([1, 1, 1])

    # coe = np.random.rand(3)
    # coe /= coe.sum()

    grayed = np.round(data @ coe).astype(np.uint8)

    wndGrayed = "greyed"
    cv2.namedWindow(wndGrayed, cv2.WINDOW_NORMAL)
Confused in choosing dirichlet versus rand(3).
 
There is someone here who has knowledge of java?
 
@butexa a little bit.
 
@butexa Is your question somehow connected to Python?
 
@TheShortestMustacheTheorem I am trying to convert a python script to java, I am not sure I am doing it correctly. I would appreciate any kind of help
@PM2Ring yes
 
6:43 PM
Ok
 
@butexa: Maybe you can use codegrepper.com/code-examples/python/python+to+java+converter but the generated code may have not been optimized yet.
 
@TheShortestMustacheTheorem it does support numpy functions?
 
@butexa But if someone wants to help you with this, it would probably be a good idea to do it in a different room. You have >100 rep, so you can create a new chat room.
 
Oh, sure
 
@roganjosh I'm trying to do this in pandas: for col in df.columns: df.loc[df[col]==myval, col] = new_val. But there's gotta be a way to do it without the loop. Thoughts? My brain isn't working today
 
6:51 PM
@butexa If you have a focused question about a small chunk of code, I guess it's ok to talk about it here, and you can link to the Java & Python versions on dpaste, GitHub, etc. But to discuss translation of a whole script here would probably be a bit too much.
 
@inspectorG4dget I'm not sure if I can avoid that outer loop, but I'd still need an MCVE to play with
 
@inspectorG4dget if you operated on the underlying arrays, you could essentially assign in one go. mask = arr == myval and arr[mask] = new_val
 
@ParitoshSingh wouldn't that require that I loop over the columns? If no, please show me how?
 
@PM2Ring I haven't posted it here because my question it's not yet eligible for bounty. But yeah, I don't need help with the whole script. Just a little help with some chunk of code
For instance: imageROI = inputImage[roiY:roiHeight,roiX:roiWidth]
 
@inspectorG4dget it's just those lines, if you use arrays directly, arrays can be 2d, and the mask would work. so, say, arr = df.values and then go from there.
 
6:58 PM
@roganjosh here's a better mcve (deleted the old one)
df = pd.DataFrame.from_dict({'a':[(1, 1),(2,None),(3,4)], 'b':[(4,None),(3,2)]}, orient='index').T
 
oh ew, gross.
 
I'm guessing butexa's question still pertains to stackoverflow.com/questions/67902037/…. The question is still a bit too new to meet our room rules' standard for question solicitation, but I like to bend that rule, so I allowed it / will allow it
 
I'd like to be able to delete any element that has (_, None)
 
can I get a link to a good flask tutorial
 
@piRSquared with pleasure: blog.miguelgrinberg.com/post/…
 
6:59 PM
thank you
 
that data loses any benefits you imagine vectorization would give, wouldn't it?
+1 to that flask tutorial, it made the whole world make sense.
 
@Kevin Yes, that's correct. Thank you for allowing it.
Should I go ahead and create a new room or it's fine to ask help here?
 
@butexa Have you worked with Numpy before? That statement basically extracts a rectangle from a 2D Numpy array. Sorry, I can't help, I don't know Java.
 
@butexa Hmm, I'd say create a new room. I'll join and share my thoughts, as meager as they are
hehe
 
nowai! I saw pm and wim within the last 24 hours (guesitmating) Nice to see you guys.
 
7:08 PM
Hi piR²!
 
The name-starts-with-p-and-involves-math boys are back at it again
 
I've been having fun playing with SageMath. It's built on top of Python, so you can often run plain Python in it, although you sometimes need to make minor changes because Sage like to use its own datatypes, and to treat stuff as symbolic expressions. There's a free online SageMathCell server that can run Sage (& a few other languages) in your browser. It's fun to be able to write & run Python on my phone.
SageMath can do 2D plotting (using matplotlib) and 3D plotting (using three.js). Here are a couple of examples. math.stackexchange.com/a/3991940/207316 puzzling.stackexchange.com/a/109191/36040 astronomy.stackexchange.com/a/28036/16685
 
I have to admit, I've been pretty excited about the Dune and WoT coming out at the end of this year (/fingers_crossed)
 
what's wot? i googled and got world of tanks...i have a feeling you didnt mean that
 
7:18 PM
Wheel of Time
Will be on Amazon Prime
 
it's getting a movie adoption? :o this is news to me
 
Series
 
oh, that makes more sense
(and its probably for the better)
I was fairly curious about it, but i think i've realised i wouldn't enjoy reading WoT now that it's novel techniques are not novel anymore
At least, thats the impression i got after going through reviews
 
I'm going through all of them again on audiobook. It's different than reading but I'm enjoying it.
 
hm, how's the middle part of the series according to you? manageable?
It was the part that was highlighted as a real slog, partway into the novels
 
7:22 PM
SageMathCell lets you share scripts by bzipping & encoding them into a URL. If the script is short, the URL can be small enough to fit into a chat message or even a comment.
 
After the 6th/7th book, I got frustrated. Frankly speaking, you can skim those books or read a summary and move on to book 11 or so? I'm listening to book 7 now, so I'm in the heart of where that slowness begins.
 
Oh that could be a viable workaround
 
@PM2Ring does sagemath do number theory? is that where I remember hearing about it?
 
I had heard about Wheel of time because of brandon sanderson. you've read any of his other works yet by any chance? Say, Way of kings?
 
Yes.... all of them
I am a big fan
I got into it because I was impressed how he finished up the Wheel of Time
 
7:26 PM
@piRSquared It's got lots of stuff useful for number theory. It can do Galois Field arithmetic. It can (sometimes) simplify symbolic equations. It can do lots of calculus & algebra stuff.
 
likewise! it's be an absolute blast to read his works
 
I waste a lot of time on Youtube listening to Cosmere theory. It's a lot of fun.
 
yeah, im basically approaching WoT from the other way around, curious because i really like Brandon Sanderson's works and he decided to finish WoT. But i fear that i may have spoilt myself by expecting Brandon Sanderson's novel writing
hehe, im not quite there yet, i'd like to catch up on reading a few more novels before i start listening to other people's theories on the Cosmere
I really like his "update" videos that occasionally show up on my feed though, so guilty there
 
@PM2Ring No, I never did work with numpy and I have basically zero experience with python.
 
SageMath is a free open-source mathematics software system licensed under the GPL. It builds on top of many existing open-source packages: NumPy, SciPy, matplotlib, Sympy, Maxima, GAP, FLINT, R and many more. Access their combined power through a common, Python-based language or directly via interfaces or wrappers.

Mission: Creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab.
 
7:31 PM
Until sagemath has a working model of Cosmere theory, I cannot endorse it
 
I just checked that SageMathCell has Pandas.
 
@ParitoshSingh Where are you at? I'd focus on Cosmere related content first. Stormlight Archive 1-4, Mistborn Era 1 1-3, Mistborn Era 2 1-3, Warbringer, Elantris, Arcanum Unbounded. I'm probably missing something.
 
I finished mistborn Era 1, Stormlight Archive, it's 2 novellas, and Warbringer. I did read them in sort of weird orders, but so be it. Missing the rest
I ended up re-reading mistborn Era 1 recently just so i could get into the groove of things, thinking of picking up Mistborn era 2. (or Mistborn secret history. bit torn between which one to go for first)
 
I haven't read the second novella yet. Not on audiobook yet /sheepish_grin
 
It also has requests, but they had to put restrictions on it a few months ago because some idiot tried to use the server for bitcoin mining. :( But you can still access some sites, like GitHub. And NASA, so my astronomy program that pulls data from the JPL Horizons ephemeris system still works.
 
7:34 PM
rbrb for now
 
rbrb :)
 
Here's a short Sage script that does a 3D plot of some of the branches of the complex Lambert W function.
Note that the script isn't saved on some server, it's compressed into the URL I posted in that chat message.
 
8:21 PM
                0            1
0     NCT03088111  NCT04202835
1     NCT04781400  NCT04002479
2     NCT04502966  NCT04486482
3     NCT04780074  NCT04450342
4     NCT04742842  NCT04276233
...           ...          ...
2991  NCT04640532         None
2992  NCT04644237         None
2993  NCT04646889         None
2994  NCT04639219         None
2995  NCT04633278         None

[2996 rows x 2 columns]
how to sort my dataframe based on index of row ?
something like _NCT03088111 _ where i would like to sort by [3:]
where dtypes = object.
 
assuming you want a numerical sort, just make a new temp column with the [3:] sliced, and sort by it.
 
if 100% of the values start with NCT, then slicing them off shouldn't change how they get sorted
 
Yes all starts with NCT at all.
 
@Kevin thats a good point :P
 
i would like to sort the values on numerical sort with keeping the NCT for sure. something like lambda sort by key index.
 
8:28 PM
am i correct in understanding its just a sort on the values in column 0?
 
otherwise, I need a MCVE pls.
 
@ParitoshSingh on both 0 and 1
 
depending on whether the columns are int or str. Use df.sort_values([0, 1]) or df.sort_values(['0', '1'])
 
sort_values sounds like what you want to me.
 
that's what I've used

goal.sort_values(by=list(goal.columns), axis=0)
and got wrong output
                0            1
0     NCT03088111  NCT04276558
1     NCT04781400  NCT04396717
2     NCT04502966  NCT04543188
3     NCT04780074  NCT04323670
4     NCT04742842  NCT04154488
 
8:35 PM
axis 1, no?
i think your mistake is in phrasing this as a row-sort or an index sort, if i am understanding your actual requirement correctly
 
                0            1
0     NCT03088111  NCT04511338
1     NCT04781400  NCT04202835
2     NCT04502966  NCT04360681
3     NCT04780074  NCT04549168
4     NCT04742842  NCT04469686
that's the output of axis=1
Yea, am looking for row sort, NCT1 , NCT2 and so on.
 
could you give some code that makes this dataframe? so that it's easier to play with
 
did you reassign the output back to goal because I just ran your code goal.sort_values(by=list(goal.columns), axis=0) and got sorted results
             0            1
0  NCT03088111  NCT04276558
2  NCT04502966  NCT04543188
4  NCT04742842  NCT04154488
3  NCT04780074  NCT04323670
1  NCT04781400  NCT04396717
 
oh, didnt assign. that would do it
 
8:38 PM
@piRSquared check column 0 looks good. but column 1 looks incorrect.
 
do you want each column sorted separately? That is a different question altogether
 
column 1 is tightly linked to column 0, did you want them independent of each other?
after having gone through this whole process, i think i should double down on requests for MCVE.
makes note to self
 
if you want a quick and dirty way to sort each column independently, Use: goal.apply(sorted)
 
@piRSquared oh btw re this..it's pretty good! also feels more..important and substantial.
 
             0            1
0  NCT03088111  NCT04154488
1  NCT04502966  NCT04276558
2  NCT04742842  NCT04323670
3  NCT04780074  NCT04396717
4  NCT04781400  NCT04543188
 
8:41 PM
I personally have never tried audiobooks, i tend to favour good ol paper in hand.
 
I prefer paper. But life gets in the way. I do chores around the house while listening.
 
Jab
Just noticed python is rushing to be ahead of java in questions asked. lol
Just a couple months ago it was far behind
 
Proof that Java is getting easier and python is becoming increasingly complicated
...that's why people ask questions, right? Not because they're bad at googling?
 
We need a questions per task or per person metric
 
8:56 PM
@Jab We'd probably beat Java if all Python questions had the generic tag.
Plenty of questions just have a version-specific tag. And I guess that there are Django & Pandas, etc questions that don't have any Python tag at all.
 
00:00 - 17:0017:00 - 21:00

« first day (3889 days earlier)      last day (1045 days later) »