« first day (4217 days earlier)      last day (956 days later) » 

07:05
@Wes The locking is per method call, so in the case of iteration only each item access is protected. If one process iterates while the other is writing, the iterating process may see a mixture of the old and new state. If you want to have a consistent view, do the same as for any other concurrently accessed container: create a copy (via [:] slicing) in the reading process everytime you want an update.
Alternatively, you can acquire the lock of the array while iterating, but that will needlessly increase the synchronised duration.
 
2 hours later…
09:16
New hobby: Finding/Reading brand new sentences, never before uttered since the dawn of time.
What brought this on? I discovered an anime where Zhuge Liang gets reincarnated in modern-day Japan and becomes a music producer. Yeah.
Indeed
Unexpected banger of the season
It certainly sounds like something
Wes
Wes
09:34
@MisterMiyagi i don't think i should be using that tho, mainly because seems limited in terms of the type of data i can put in it.... only primitives, right? what if i serialize an object to a string, and unserialize on the other end?
i can't even use Namespace. the main process doesn't get any change from the child process :'(
Well, you wanted to pass along an array...
10:36
Hello guys, question... what is the reason for high accuracy scores with preprocessing methods like normalization, standard scaler, minimax...is it expected that all will indeed fall in the same range?
as in what will cause a variance in these scores? I am using the iris dataset and all my scores are high, is that because its super balanced AND small? or one of the other?
also, does it make sense to apply these different preprocessors to the algorithms to see how they perform? or is it best to choose one, like normalisation and then test the algorithms or classifiers on that?
Wes
Wes
@MisterMiyagi i don't need to modify it. it's passed as a single unit and replaced with a completely different array when it's modified
Wes
Wes
11:18
damn i swear all multiprocessing guides are like how to draw an owl. they assume you already know about locks and waiting and whatever
Well, I feel that's kind of a chicken and egg problem. You need a non-trivial use-case to begin with in order for locking and such to make sense.
Hi peeps, I have a weird issue, as all issues you hear about are, namely: stackoverflow.com/questions/21163327/… And no plugging it out and back in didn't help.
@Hakaishin did you also check the global and local pager config?
in this case the code in the question seems legit too
well, not necessarily setting the global config
but querying it anyway, and trying envvars (not sure if envvars override the config or not)
And obvious question: did you do anything to your git config? Is this a new repo? Are you using a terminal?
You are a genius :) I did check it and I didn't change it, but I only checked:
core.pager=less -F -X
core.pager=/usr/bin/less
but just below that(tunnel vision) there was pager.log=false, adding pager.log=true made it work now :)
Now I am really curious how it got there, oh well another mystery to not be discovered
Wes
Wes
11:42
any idea why this hangs? pastebin.com/BzE2k4B5
11:59
Ctypes is always a prime suspect whenever the process simply freezes with no stack trace or error or anything
On my machine, it actually crashes.
No trace, though.
Clean segfault.
FWIW, if we're here because you want to extend your initial "one process writes, the other reads occasionally" from arrays to other types, it might be worth to take a step back and look at what you want to do in the big picture.
Personally, I'm willing to try literally everything else before I use ctypes
> Assigning a new value to instances of the pointer types c_char_p, c_wchar_p, and c_void_p changes the memory location they point to, not the contents of the memory block (of course not, because Python bytes objects are immutable):
This looks pertinent:
Or, hmm
FWIW, the code works when I use ctypes.c_double and just numbers.
12:13
I just tried running it. I get this output:
b Hello, World!
a 1651579938.9591005
a 1651579939.9735355
Then the cursor blinks for about five seconds, and the process ends with no apparent problems. Not sure how that's possible since every process should be inside a while True loop.
That matches how for me it crashes when the b process tries to read the changed value. Plus some Windows weirdness on top, I guess.
Wes
Wes
@MisterMiyagi the array is actually a collections.deque within the child process, because i need to trim one end while i add to the other and keep it fixed in length
@Kevin Check the return code. It's -1073741819 on my machine
Ah, the old Windows philosophy, "we must treat our customers like five year olds and hide diagnostic information from them"
a code is at least something, worst is the Error: Unknown error promt
Wes
Wes
12:16
i tried to put it in a managers.Namespace but the parent process couldn't see the changes
@Wes Ah, I was already suspecting this might be a programming language barrier. Array isn't ergonomic for that indeed.
If you need a queue-like collection that must remain a particular size, searching for "ring buffer" may give some ideas
Wes
Wes
before that i was passing each element one by one using multiprocessing.Queue but it was way too slow
I'd be interested in seeing that version
Wes
Wes
because the parent process can't consume the queue as fast as the child process fills it
12:19
Sounds troublesome, but I don't see how you could change that by using something other than a queue
Switching to a m.Value(ctypes.c_wchar_p) won't make the parent process faster or the child process slower
You want to throw away outdated values though, right? So no need for the parent process to read all values?
Queue objects have a maxsize argument btw, if you're worried about the child process stuffing zillions of elements into the queue
Wes
Wes
what i am trying to do is this, the child process is working on a collections.deque(), as fast as possible
Ok, a deque is perfectly sufficient for such a thing, please continue
Wes
Wes
occasionally (like every 500ms) the parent process needs to read that deque from the child process, for displaying purposes
12:22
Hmm, I don't like that idea. Use two deques instead.
Wes
Wes
how do i send that deque to the parent process? or how do i pull it from the child process?
The same way you sent/pulled the first one
Wes
Wes
the first one?
Yes
Wes
Wes
i only have one deque in the child process, which i need to (pass to|get from) the parent process every 500ms
12:24
I know, that's why I want you to use two
Wes
Wes
but how do i fill it? :B
You don't need to
Wes
Wes
i feel i am being trolled
My advice is sincere, and truthful to the best of my knowledge. I am admittedly being less descriptive than usual. Busy day.
Man I was going nuts, my system was rebooting randomly every 3-4min and I was searching every log to find a trace of what is causing it. Until a coworker comes to me and tells me, yo the hardware relay which cuts the power if nothing is connected to it is clicking every 3-4min. Thank god he heard that, I could have gone on a goose chase for nothing
Wes
Wes
12:34
:B ok
Did I understand this correctly, one process is working on a list of tasks, and another process periodically wants to know the contents of said list?
I'd do this by having the "printing" process signal the "collecting" process to queue an item.
That's some scaffolding but works reasonably fast and robust.
Wes
Wes
the child process is working on a single deque, which has a fixed length, it adds new entries while removing old ones with deque.popLeft()
@Wes here is a prototype of my two queue design. pastebin.com/raw/AKYfWmmg
Wes
Wes
the main process is the GUI process, it occasionally renders said deque to the user
so the main process must pull the deque from the child process, without blocking the child process, and show it to the user
12:41
Something like this: pastebin.com/gX3AUQbW
I'm half-tempted to propose a ring buffer on top of a SharedMemory block
Wes
Wes
i thought multiprocess worked without threads :'(
damn this is so confusing
Then the main process can look at the memory contents all it likes, and the child process can make changes to it without obstruction. Of course, the main process is going to see half-updated values all the time, but if it's just displaying things to the user then it's not going to cause a segfault or anything.
Wes
Wes
exactly
If we're talking numbers, then I'd say do that. If it's strings or any other non-trivial object, it's probably a lot of hurt.
Wes
Wes
12:47
do you think this is the fastest solution? the less the overhead to the child process, the better
@Wes Multiprocessing uses threads behind the scenes anyway. Every queue has a worker thread, for example. So if you need a custom queueing mechanism, you are likely to end up using threads yourself.
IIRC, shared memory is quite fast on Linux, and nothing to write home about on Windows
I have an idea for an event-based approach... I must away to my prototype cave
Wes
Wes
i like what you are doing with the double deque but it seems more complicated than it need be. said deque is just 50 elements long, and each element contains a bunch of numbers and a string that is less than 20 characters long
Hello,is anyone here fluent in the pipeline function of sklearn?
Wes
Wes
in fact i wanted to just serialize it to json, dump it to a file, and have the main process read that file
12:59
I may have misunderstood the requirements. If the worker process decides which values to add and remove from the collection, and the main process is merely watching so it can display the data on screen, then you don't need two queues. Two queues are only useful if communication is two-way.
Wes
Wes
yes the main process only reads it for displaying purposes!
it doesn't change it and it doesn't affect the child in any way
Ironically, my event based prototype has three queues
Wes
Wes
:B
still, i would love to know why i failed so hard with Namespace
Never used Namespace before, but I suspect you need a Manager too
Wes
Wes
i think Namespace is a manager tho
13:13
All I know is that the sample code at docs.python.org/3/library/… creates both a Manager and a Namespace
Wes
Wes
aaaah right
you are right ffs. i am looking at old code and there's definitely more to it
somehow i assumed that multiprocessing.managers.Namespace()
was the same as multiprocessing.Manager().Namespace()
it works now
i am such an idiot sometimes very often
Wes
Wes
13:31
i think this is good enough for me
because the value on the main process end is updated only when a property is fetched
(i suppose)
13:43
Hi. I'm planning to build a serverless application on AWS Lambda. When I was using Node there was a framework called the serverless framework. I see that it supports Python but for the Python ecosystem I'm not sure which framework to use.

There's chalice, boto3, the serverless framework and zappa.
If anyone has an experience with serverless application on python, which one would you use?
13:57
Proposal: choose whichever framework has the most questions on Stack Overflow
@Kevin I actually think this is the answer. 😄
Wes
Wes
ok so repeatedly updating the namespace from the child process is not an option
it gets transferred despite the parent process doesn't need it
that is the same problem i had with multiprocessing.Queue basically
14:17
hello
Wes
Wes
the overhead is just insane
900% of the time it takes without communication between the processes
bro damn that's a lot
try to deploy it on a service in k8s?
Wes
Wes
mmh no
14:36
I heard mongodb is webscale
Wes
Wes
14:51
so should i pursue the double queue design?
late morning cabbages, folks. Anyone here ever work with GIS? I'm trying to georeference polygons that I pulled out of a JPG of a <country> map. I was able to transform approximate coordinate values, given the coordinates of the four corners of the JPG. But this has caused some "drift" - when plotted in QGIS, the resultant polygons are southwest of where they should be and need to be georeferenced in order to "correct" or "nudge" back into position
Is there a set of terminology that anyone is aware of that I should google? I feel like I'm missing some basics here
@Wes I haven't followed the saga so far, but it sounds like you don't have an acceptable solution, so if you have a clear alternative laid out, you should go for it.
At worst you'll be where you are now
Wes
Wes
no alternative except the double queue. earlier Kevin said he had a better solution but maybe i misunderstood
15:12
@AndrasDeak--СлаваУкраїні twitter.com/soychotic/status/…
Oh the siren call of correcting someone: xkcd.com/386
Wes
Wes
i am going to do that
15:36
I'm trying to find a nice recursive function that searches the output of yaml.safe_load in a recursive way. I've used python to parse a moderately complicated YAML file that has a combination of nested objects and lists. Does such a thing exist?
The first dozen hits on SO show dump() or simple recursion; stackoverflow.com/q/59006523/4957508 gets close but I admit that's at the far edge of my understanding of python
@AndrasDeak--СлаваУкраїні cabbages!
@JeffSchaller What does "simple recursion" mean here?
You might just need simple recursion
Do you have a small example? Not sure what you're really looking for.
meaning it assumed that every level had the same data type; it doesn't look like this object is simple like that
                print "--> all"
                print yaml.dump(data)
                print "--> now just spec:selector:spec:containers"
                print yaml.dump(data['spec']['selector']['spec']['containers'])
                print "--> now just spec:selector:spec:containers:0:env"
                print yaml.dump(data['spec']['selector']['spec']['containers'][0]['env'])
ugh, sorry for all the indents
@JeffSchaller oh, yeah, sure. One could handle that inside the function. Recurse into container types etc.
15:44
you can see my caveman attempts at getting there
above, I'm hard-coding my way into the structure, but I'd like to be smarter about it
So basically arbitrarily nested lists of dicts of ...
whatever it is that yaml.safe_load gives back for arbitrary YAML files
And what would you like to find in it?
if this is new ground, I can ask a question; I was just hoping that this was plowed already
I'm looking for a particular ... "value" inside the structure and to then print all of its keys & values
you mean at one level you find a value 'potato', and you want to find the dict that contains it?
Parsing arbitrary json liad dupe, probably
15:51
I should learn more python; I don't even know what I'm dealing with :)
Call type on stuff to figure out
hmmm! sounds useful. but: AttributeError: 'list' object has no attribute 'type'
oh. other way around: type(x)
not x.type() :)
captain caaaaaveman over here
Wes
Wes
@MisterMiyagi that's useful too.
thanks!
thanks! I'll chew through those; looks like it's a mixture of dict and lists
Yeah, just handle both in the function
like falling off a log
I can take a look later, I'm on mobile for a while still
np; you've helped -- thank you!
16:18
happy dance
"> Separate source and build directories (y/n) [n]:" * me gets crushed by existential dread *
@Aran-Fey how's that sphinx killer coming along? Might need a hero over here...
I haven't made progress on that in forever, was too busy with other stuff (and got sidetracked... as usual). I intend to pick it up again soonish though, once I'm done with this new project for work
On that note, I have a problem
I'm trying to install pikepdf inside of mingw, but it fails with this error. I'm very much out of my depth here, any ideas how I could accomplish this?
There are some rather depressing hits when google'ing that error message. D:
I don't necessary have to use that particular pdf library, but it's lightyears ahead of all the others in terms of convenience :/
Wes
Wes
empty()
Return True if the queue is empty, False otherwise. Because of multithreading/multiprocessing semantics, **this is not reliable**.
why does it exist then :'(
so frustrating
16:36
Eh, those warnings are mostly fake scares.
Those methods are reliable, but in a concurrent program the state can change at any time.
It's a bit like "shoes are unreliable because you could shoot your own foot".
So if you check whether a queue is empty while the rest of your application keeps throwing new items at it – well, that might not be the cleverest thing to do.
Most of the time you don't need to use empty() since you can just call get regardless of whether anything is waiting for you
You can choose to block indefinitely until something is ready, or crash instantly if nothing is ready, or wait a bit and then crash... Throw that bad boy in a try-except and you've got designs to satisfy any need
Wes
Wes
12 seconds to count to 1 million :'(
pastebin.com/raw/ALeTLHNs this cant be right
Don't do len(deque), that's too slow. If you want to make sure the deque is never larger than ten, give it a maxsize parameter when you create it. Then old values will automatically be discarded as you add new items.
By the way, you don't need a if len(deque) > 10: deque.popleft(). deque already comes with a maxsize parameter that turns it into a ring buffer.
99.99999% of the time, you should not do if (some expression) is False:. Just do if not some expression:
Wes
Wes
16:52
that is not what causes the problem tho
if request == "update": seems redundant, since the only thing that ever gets put in that queue to begin with is "update", so you already know what it is with 100% confidence. Perhaps an Event object would make more sense...
Oops, deque's parameter is named maxlen, not maxsize
Hi everyone!
Why not something like this?
Wes
Wes
that is amazing. all this stuff is mind bending
i thought i knew what threads are but apparently i don't
:B
does push_updates run on the main or child process?
ok now it's going fast
Wes
Wes
17:06
the child then?
Wes
Wes
17:16
damn. how does one learn all this stuff
that seems to work perfectly aran tyverymuchly
18:06
Is there a library for Python for generating XSD files? I figured I'd ask before rolling my own.
Wes
Wes
18:19
libxml?
That would probably make sense. I think I was thinking of that one as being XML specific, but an XSD file, is just a specifically formatted XML file, right?
Wes
Wes
what do you mean with generating xsd files?
afaik you can validate against a schema, but you can't generate the xsd file automatically. that's just xml you should write yourself
but i might be wrong
You can convert python structures to XML: Exhibit A
18:34
@JeffSchaller does that mean you've got it?
I've never done it myself, but I imagine lxml would be able to handle XSD (I wasn't familiar with this specific format before either, so double-whammy on unknowns for me) e.g. this
18:57
@AndrasDeak--СлаваУкраїні close enough for government work, yep! Thanks again for your guidance!
19:12
{nods} I'm scraping web pages with a format specification to build an XSD file for the format.
 
1 hour later…
20:29
@inspectorG4dget yeah, that, just several years ago
@AndrasDeak--СлаваУкраїні :)
21:05
Cabbage. Is it possible to use Slurm (slurm.schedmd.com/overview.html) on Google Colab?
Uuuuh how do those two even match? I doubt it, because SLURM is a scheduler that should be running in a separate process, ideally even a separate machine. But what would you use it for that also uses google colab? These two are in very different places in my head.
Just to do job scheduling
Including to do parallel processing
Do in-python parallel processing libraries fall short of your needs?
multiprocessing, joblib, mpi4py etc.
yes
21:12
mpi4py per example
So wouldn't it be necessary to schedule jobs with SLURM in Google Colab?
I don't understand the question
then again I'm primarily an end user of scheduling systems
¿
OK, there's an English version
Oh sorry
So my (admittedly very cursory) experience with google colab is that it's a free service with fairly limited hardware resources allocated to each notebook.
21:23
yes, but there is a paid service, too
By the way, I just checked that the free google collab version only have two cpu cores
Two Intel(R) Xeon(R) CPU @ 2.30GHz
And 1 Tesla K80
Thanks anyway, @AndrasDeak--СлаваУкраїні
I conclude that due to the fairly limited cpu resources of Google Colab (at least in the free version), there would be no need to schedule jobs.
(I think)
22:41
@Marco You can't be thinking of doing anything serious on a free instance of Google Colab?

« first day (4217 days earlier)      last day (956 days later) »