« first day (4124 days earlier)      last day (1049 days later) » 

 
5 hours later…
04:48
Cabbage
05:29
@AndrasDeak Well, the philosophy that mathematics is discovered, not invented, is called mathematical realism, or mathematical idealism. en.wikipedia.org/wiki/Philosophy_of_mathematics is pretty good.
 
2 hours later…
07:53
@PM2Ring thanks
08:22
I have a bunch of data that I want to process in batches. I have a background thread that appends items to a list. Every second, I want to grab the items that are currently in the list, and clear the list. How can I safely do this? Do I need to acquire a lock every single time I append an item to the list?
Oh, I just thought of something. I can record the current length of the list every second, and then just slice out the new items. That way I don't have two threads trying to modify the same list
Yeah, and lock and clear when it gets too large
Ah, no
Then you'd still have to lock on each append...
Not if I let the appender thread clear the list. I don't think it's worth doing though, since the data will be appended to another list anyway
08:50
Depending on what else you're doing - you can use something like queuelib which is a simple disk based fifo/filo queue...
09:31
Hmm, not sure what that would gain me. I don't see a reason to take a detour through the file system
Oh, the readme says it's not thread-safe, so that's that then
@PM2Ring Nice! It's cool that it's a well-established phenomenon, especially with the well-defined frequencies for the different stages, thanks :)
09:54
@Aran-Fey maybe celery then and redis as the queue handler?
Whoa whoa whoa, I think you're very much overestimating the scale of this :D
The desire for LinkedIn to be the opposite of the mainstream social media drives people in weird directions. I've just come across a post saying, effectively, "yesterday I got sacked for performance-related issues from AWS but it's fine because I want to do other stuff". Then there's all the associated fuzzies from others on how that's ok. Well, I'm pretty sure other potential employers will see that, so that's a hefty footgun to be brandishing
It still freaks me out when I get a message from linkedin along the lines of "... People are noticing you..." :p
If it freaks you out, their marketing for premium is working :P
10:14
@Aran-Fey Maybe I don't properly understand your task, but it sounds like you can just use a standard queue.Queue docs.python.org/3/library/queue.html I thought I had an example on SO or GitHub, but I guess I never bothered to post it.
@rb3652 since bouncing from the wall (usually) results to the angle being reflected, this is about hitting a ball, you want to calculate the normal vector at the hit point but since it is a sphere... still not sure where you need the cross product
@PM2Ring Thanks, but I think that would add even more overhead than a lock? If I do the locking manually, I can empty out the list in one go. If I use a queue, I have to access every item individually, which acquires a lock every time. Unless there's a way to extract all items from a queue at once?
@Aran-Fey Ah, ok. You're supposed to access Queue items one by one.
I guess I should add some context. What I'm doing is opening a folder with 100k files inside. This obviously takes a while, and I don't want my GUI to freeze while it's doing this. So I have a background thread loop over the files, while another thread periodically updates the GUI
The background thread is doing essentially no work at all; it's just looping over a path.iterdir(). So if I have to lock every time, that adds significant overhead
Imagine that once you open a question, briefly sniff it and see that .. you left a comment under it 5 minutes ago :D That actually happened to me. Here is the question. I really for a moment questioned my mental health LMAO.
Ah, question deleted for now. The thing is that somebody created profile and used my name. It's damn confusing :D.
10:31
@OlvinRoght imitation is the sincerest form of flattery :P (or, they simply have the same name)
Doubt that it's possible. Ridiculous :D
A while back, someone "acquired" the same gravatar as me in chat. I was not flattered
The only thing I'm worried about is that this new "fan" will act rude from my name. Would be quite hard to explain what happens.
@Aran-Fey You should be ok with a simple Lock. One thread appends to the list, the other reads it & clears it. A thread only accesses the list if it's acquired the lock.
@OlvinRoght I didn't realise you were a fan of eval(). In any case, I'm not sure names mean so much on the Python tag any more. Outside of chat and a handful of names, I usually remember profile pictures/gravatars so I'd tell the difference straight away
10:41
@PM2Ring Yeah, I want to reduce the locking to a minimum though. I like my slicing idea; I'll only need a lock if I ever decide I do want to shorten the list at some point
@roganjosh, that's illegal :D
You could avoid this problem by choosing an obscure handle, like, I don't know, some kind of foodstuff or something :P
@OlvinRoght There's no rule preventing a member from having the same username as another member. But if someone does bad stuff while giving the impression that they are you, then you can let a diamond know about that, using a custom flag.
@Aran-Fey What's the benefit in slicing, rather than just cleaning out the list?
I think that would be exceptionally hard to prove. If someone creates an account in any of our names and goes on a tirade, can anyone prove that it was for defamation of character? (cc @JonClements)
There have even been cases of people duplicating mod names, although of course they can't duplicate the diamond. ;)
10:50
Wondering how he(she) came up with this idea. People are weird :D
Yeah, I have seen that on Meta in the past and I think in the past there was a way to try get a diamond into the name? I tagged Jon for the general case, though, not just imitation of mods.
@OlvinRoght again, the simplest explanation is still that they actually share the name
There's this guy, but I assume it's just a coincidence. stackoverflow.com/users/2602370/martijn-pieters
Umm.... can't divulge too much but the info for that other Olvin account doesn't quite add up but then I can't find any evidence of using it to do anything nefarious... so yeah... for now - it's a little weird... but if you notice anything just flag and a mod can take another look
@PM2Ring That I don't have to mutate it. This way only one thread modifies the list, so I don't need any locking at all
I used to study probability theory and chance that two people sharing same (pretty rare) name will be a developers and active on same website is ...
10:54
@OlvinRoght they changed their name yesterday or today, so it's probably not an accident
@JonClements, ok, thanks, I will notify if I will see anything weird (more weird that this).
The answer roganjosh posted has a "thank you" comment aimed at "JCaesar", presumably their original name.
yeah... they changed display name early yesterday morning...
@OlvinRoght Unless you are waiting for a bounty to be assigned to you this should not be cause for concern. Just ignore them and flag if you see something weird.
I always miss this stuff. I'm a terrible detective :/
10:55
I have seen people change usernames to snatch bounties, so there's that
imaginary internet points making people do weird things... who'd have thought it :p
I don't care, rly, unless he will create problems.
<notices changes in pitch for a boiling kettle, overlooks the dead body all dressed up in the corner of the room>
@OlvinRoght either they won't, or you'll notice
leaving comments like stackoverflow.com/questions/70907512/… is probably not the right approach to a potential troll
Well, let me be me :D
11:01
You're allowed to feed trolls on your own account, but then make sure not to bring them here.
@Aran-Fey Fair enough. list.append is atomic, so that should be perfectly safe.
12:03
AttributeError: module 'matplotlib.collections' has no attribute 'deque' <- Well that's the last time I let VSCode insert an import statement for me
heh
copilot would have had your back
I don't think VSCode should do that; it might be the Kite extension?
{'success': False,
 'created': False,
 'id': None,
 'errors': [{'statusCode': 'FIELD_INTEGRITY_EXCEPTION',
   'message': 'field integrity exception',
   'fields': []}]}
Although, then again, until recently I had to stop fighting the VSCode Server from correcting import pandas as pd to import pandas as pandas
Well... that's a not greatly useful error message to be repeated 136k times - thanks Salesforce... arhghghghg
12:13
@JonClements divert power to shields around []
You passed None maybe?
All fields are definitely populated... mind you - the whole product/pricebook and pricebookentry model in Salesforce is fiddly anyway - always seems to throw a few bits out but then just run it again and it works fine...
I mean, if the magnetic field of the Earth became None, I'd probably throw an integrity error for its absence :P
Fortunately, I haven't had to deal with Saleforce stuff <frantically touches wood>. I'm waiting for SAP to come onto the books
@roganjosh I don't have Kite installed, so...
12:36
How do I repro the issue? I'll be honest, I don't think I've ever used a suggested import in VSCode
I didn't even know if would do that. IntelliJ imports classes in Java, but I've not seen it in VSCode
Just type coll and press Tab. Although you may have to have matplotlib installed, not sure
it would've been even better if the attribute were valid
Well, without an import, there's no chance of tab doing that
Although, there is some "memory" in its suggestions. It still won't auto-import anything for me. That would surely be a plugin
Okay... so apparently setting UseStandardPrice when you're adding a product to the standard price book it doesn't like...? shrugs... at least stuff's going in there now...
@roganjosh Are you sure? It literally says "auto-import" on the right
12:47
:facepalm:
Totally can repro. Need to go back to bed, apparently
I can relate to that
Actually, what the hell has that done? from ipaddress import collapse_addresses is valid in a project in which that module doesn't even exist. It's not even a package I've inherited from someone else, but it's happy with the import
You've found a scary corner in VSCode with that auto-import :/
ipaddress is in the stdlib
Doubly need to go back to bed, then
That's a fun cross-over from my recent rust learning. I didn't realise python had it in stdlib but that functionality was bundled in rust too. I got very confused there
13:05
If you're learning rust it's no wonder your python is rusty!
/me groans :p
"Dad Jokes" are real, and we have them every week to open our standups. You'll build resilience, pup
That or build a device to slap people over the internet? :p
So this is it then, huh. This is how I find out I'm a dad. That's not how I expected it to go.
@Aran-Fey Congratulations?
13:11
Thanks... I guess
We should throw a room-wide party. I mean, once the jokes hit a certain bar, it can only mean one thing
:P
So a joke walks into a bar, and what then? :P
Ok, I'll stop before Jon decides to murder me
13:40
Ugh... another fantastically wonderful error message.... "Records not processed"... no hint as to why though... yamming system
 
1 hour later…
15:01
In terms of typing, does anyone actually use static analysis of code? I know a few people here go to the ends of the Earth to make sure that their type hints are correct, but how many people are running mypy over the code? Asking for a friend.
Not me, I only use annotations for dataclasses and for my docs
What do you mean "for [your] docs"?
I've published something on pypi, so that needs documentation. Sphinx extracts the annotations from the source code and puts them in the docs
You mean annotated signatures?
In that case couldn't you run mypy on your test suite?
or, uh, the library
I still don't get static type checking I guess
@AndrasDeak I was about to say "and for instance attributes", but then I realized sphinx actually doesn't do that. So yeah, the signatures I guess.
I could, but I don't. Honestly, I don't think it would be very useful, given that 90% of the code doesn't have annotations
I suppose running mypy would help me find incorrect annotations. Rather than incorrect use of annotated things, which is what it's usually used for
15:17
@AndrasDeak I think that's my stance too. I do try to make it correct, but then I never actually run anything to see whether everyone else trying to implement it was actually successful. It feels like it's just a weight on the developer that isn't even used in the real world
Here's an example. The types of the parameters and return value are all there, so you can see at a glance what the function can return. (Although sphinx renders it in a stupid way)
...and __new__ isn't really a staticmethod, as I recently discovered, but nevermind that
@roganjosh I heard opinions that it's mostly for library authors, to help downstream users' code completion.
15:42
That makes no sense if you think about it. "You should use type annotations to improve code completion" would make sense. But "You should use type annotations specifically in libraries to improve code completion for other people"? Sorry, what?
What that's really saying is "I know you don't like type annotations, but I do, so I want you to use them because they might benefit me"
I think the rationale is that library authors potentially have a lot more end users down the line
It's all fine if you don't want to use typing for your code that only you're going to use. But doing that in a library can hinder a lot of people if they depend on your code.
it's also possible that I misunderstood the argument
Sure, but that's based on the assumption that type annotations have a benefit. You can't say "Oh, the point of type annotations is that library authors can use them". No, the point of type annotations is something else. Libraries are one specific use case for type annotations.
Saying "type annotations are for library authors" is like saying "hammers are for charities, they can give them away to people in need". Why would anyone give away or need hammers if they were useless? "Charities can give them away" is not a reason to use hammers
"Library authors should use type annotations" means "I acknowledge that type annotations have benefits, and I think library authors should use them for the benefit of their users, even if they wouldn't do it for themselves"
16:05
Do you use them in other libraries?
Rarely
It's fine to suggest they help in a charity case. People donate massive tins of prunes to charity - it doesn't mean people can use it
16:37
Hello. I have a high throughput execution workflow in mind and was wondering if it is possible using the multiprocessing + standard socket library. Have a Python application listening for data. Every batch of data received goes through simulation X, which can be performed using a single core. Use up to pre-defined L cores (available on the node) for multiprocessing. Up to (L-1) cores available for simulating batches through X, 1 core available for ICP.
I'm not sure I follow, but what I'm saying is that "charities should hand out hammers" is a reasonable opinion to have, whereas "hammers are for charities to hand out" is not an argument for the usefulness of hammers
import json

def updateDF():
        url = 'http://api.coincap.io/v2/assets/fantom'
        response = requests.request('GET', url)
        dat = response.text
        json = json.loads(dat)
        price = json['data']['priceUsd']
        print(price)


if __name__ == "__main__":
        updateDF() # UnboundLocalError: local variable 'json' referenced before assignment

        url = 'http://api.coincap.io/v2/assets/fantom'
        response = requests.request('GET', url)
        dat = response.text
i think this is interesting problem
python thinks json is a local variable
is it true? but why?
Yeah. It's an optimization for functions. If you assign to x anywhere inside a function, then python considers x a local variable
Easily googled with "unboundlocalerror function" by the way
*IPC
Anonymous
Best way to shorten this code?
Anonymous
16:51
for x in name_to_variants.values():
    x.sort()
    for item in x:
        item.product_key = x[0].product_key
Anonymous
A list of product variants that needs to be sorted. Each variant should share the first product_key.
First of all, "how to shorten this" is the wrong question to ask; it should be "how to improve this". Secondly, it's fine as is
Anonymous
I know that it's good to have the code as readable as possible.
Anonymous
I was just curious! :D
seems readable to me
17:16
@Warcaith, sort seems useless here if the task is to replace all product keys with one from item with minimum value from list x.
17:27
Is anyone able to lead me in the correct direction? Essentially I want one process in my python application to listen over a socket and add incoming data to a queue, and all other processes (up to the remaining # of cores available on my node) to multiprocess tasks waiting in the queue
 
2 hours later…
19:05
I see your denvercoder and raise you a microsoft support thread solution
19:35
Was the solution to not use Windows?
Heh, probably not. There's a subsequent comment from the same user talking about a "power automate flow" which is an MS whatever.
Anonymous
20:11
Hello!
How can I convert a text like this, so every \u gets replaced with the actual unicode character?

"Anschnitt:": "Vor- ca. 5 \u2013 6, Mittel- ca. 4 \u2013 5, Nachschneider ca. 2 \u2013 3 Gang",
99% sure they already are unicode characters.
>>> "Vor- ca. 5 \u2013 6"
'Vor- ca. 5 – 6'
I'm assuming that's code (and not output) because there is no string that prints itself like that.
20:29
can anybody assist with getting a Python process listening for IPC over a port, while some number of other processes handle information added to a queue by the first process? I have some template code I've worked on, which doesn't do what I want for obvious reasons:
#!/usr/bin/env python3

import asyncio
import multiprocessing
import time

HOST = '127.0.0.1'
PORT = 9797

queue = asyncio.Queue()

async def echo(reader, writer):
    while True:
        data = await reader.read(1024)

        if not data:
            break

        print('Adding task to queue')
        queue.put_nowait(
            (data + b' :)', lambda x: writer.write(x))
        )

        await writer.drain()

    writer.close()

async def manager():
    print('Starting manager')

    while True:
I'm just browsing JustEat and there's "Pickle Rick's Pizza". I'm in awe. I wonder how many people out in the hills here will get the reference
Is there more to it than "Pickle Rick"? Because even I know what that is, and I haven't watched a single minute of the show.
@jeremy We have room rules particularly in regards to long posts of code (please post off-site and link here)
@roganjosh thanks, let me correct that. dpaste.com/AKQMLYJ3V
@AndrasDeak not that I know of, and you're missing out
20:31
yeah yeah yeah
The nihilism of the show is right up your street. I couldn't recommend the show enough!
*barfs on shirt* haha
I prefer classy trash, like Disenchantment
Can't say I've watched it. Is it something to put on now, as I'm trying to waste time?
I can help waste your time with annoying sockets/multiprocessing questions :)
async def <-- I'm probably out
20:37
@roganjosh, relatable :D
@roganjosh depends. If you iike medieval fantasy with futurama vibes then yeah.
@roganjosh Inside Job is good (imho anyway): netflix.com/watch/81050646?trackId=255824129
@jeremy I'm not sure what these "obvious reasons" are that you're referring to, but I would strongly advise against using regular sockets. Websockets are much nicer, especially if you're already writing an async program anyway. I have noticed this huge problem though:
I know who is likely to answer, so I'll leave it in their capable hands. And decide whether I wanna get pizza from a restaurant referencing one of my favourite shows or go by ratings :P
20:38
while True:
    if queue.qsize() < 1:
        continue
That's a busy loop, which is (almost) always bad, but especially in an async program
Async is about cooperation, and being stuck in a loop forever isn't very cooperative
thanks @Aran-Fey. I think that's part of it, I need to get the manager running as a separate process
I don't understand what the point of the manager is
What's supposed to happen when someone connects to your server?
the server is running (and uses the poorly named echo function) to add data points to a queue. the manager should constantly be checking the queue in parallel and perform simulations (or simply call the callback, for now, in this XY problem) on the remaining cores
Why can't the echo function do that? The manager seems like a pointless middle man to me
I will have a worker connect to the server (node) and submit N number of data points. each data point needs to go through some type of statistical simulation and get parameters back.
20:45
sounds like an hpc scheduler
I have 36 cores. one of the cores I'll devote to IO and the others I'll run simulations
@AndrasDeak precisely
So the server is IO, and the manager is... what?
I am trying to eliminate inefficient middleware by making my own?
the mananger will eventually manage simulations that need to be applied to each item in the queue separately (but in parallel)
Are these always on the same machine?
Depending on how low-level you want to go mpi4py might be applicable. Purely based on the "hpc scheduler" parallel.
@AndrasDeak i'll send data from a separate node on the same cluster
20:48
@jeremy This is too abstract for me. If you tell me "Every time someone connects to the server, I start a new process", then I have a specific problem to work towards. But "each data point needs to go through some type of statistical simulation and get parameters back" doesn't mean anything to me
@Aran-Fey sounds like one can submit "jobs" which spin off CPU-heavy calculations on processes. jeremy wants to have a supervisor process that is responsible for spawning those processes (i.e. allocating resources).
@Aran-Fey I understand. My X problem is: every time someone connects to the server (with some string of data), I want to create a new task. That task will take 1 second (manually sleep with Python), then add a smiley face to the input, then send the new string back to the client.
My Y problem is actually: a client node will submit several thousand (million) arrays of data, each of which will become a task on my server. Each task will be responsible for returning the mean and standard deviation of the array. (Or fit some other on-normal distribution)
@AndrasDeak thats correct
@AndrasDeak Hmm, alright, I can kind of understand why a manager would be needed. I'm still confused why it should run in a separate process though
@Aran-Fey since I have 36 cores available to me, I can fit 36 distributions at a time
...which I'm pretty sure I read somewhere, even though I can't find it now
20:52
@Aran-Fey separate from what?
The socket server. I think.
Ah, I see. Yeah, that's not clear. In my mental model I only need one dedicated process aside from the workers.
So you have 35 worker processes, and 1 process containing the socket server and the manager. Incoming data will be added to a queue, and whenever a worker becomes idle, it gets assigned the next item in the queue. Something like that?
yes, I am thinking that 1 core will run the socket and accept data. 35 will fit distributions
that sounds reasonable
20:55
there are packages like Parsl which are ideal for this task, but I am worried the middleware is too large of a bottleneck, since it is going much slower than when I simply use the multiprocessing API
so I'm trying to kind of do a barebones method to see if I can get suitable high throughput
The multiprocessing API is not something to be upheld as the most efficient approach
Which now makes me curious as to what your backend code is doing
faster than the multiprocessing API is even better
You're talking about mean/stdev of data? That's not a multiprocessing job.
I would assume that the bottleneck is the passing back and forth a million arrays of data. Might be easier to compare than a full proof of concept.
by backend do you mean my server? I meant mean/variance. but I'm basically fitting a distribution on the data (using statsmodels or rpy2)
20:58
Yes, I mean exactly that. And how are you going to parallelize it?
@AndrasDeak that very well could be the bottleneck. thats one reason I'm trying the sockets approach, instead of using files to pass the informatino
@jeremy most solutions are not as slow as files though...
I'm more interested in the parallelisation. I have a feeling you're just duplicating work tbh
@roganjosh I've been parallelizing using the multiprocessing API. but now my problem is, how can I do that while listening on a port?
@roganjosh I mean if a single distribution can be processed fast (compared to the overhead of passing an array back and forth) they could just offload 10k-sized batches at a time to a single process to reduce overhead
21:00
It doesn't take 1 second to work out the mean/stdev of an array of any reasonable size
I'm sure there are always large enough arrays :P
Not on the size we're talking about
I'm fitting a multiparameter distribution that requires a few other statistical tests and some MC stuff, so it takes a little bit longer, but its pretty quick
I'm struggling a bit to figure out how to make asyncio and multiprocessing together, but that's mostly because I have zero experience with multiprocessing. Plus this kind of program is a pain to test, so I think I'll throw the towel here
I need to invoke R with Rpy2 as well
21:01
Ugh
@jeremy oof
but that stuff is figured out already. but to answer your question, thats what its doing on the backend
Am I being naive for thinking that just manually sending off work to processes would be inefficient?
multiprocessing.Processes
Let's go one level higher; why sockets here?
@AndrasDeak you could be right, but thats how I've been doing it. now I'm working at orders of magnitude larger than I have in the past, and I'm running out of memory, so I can either batch my data (total pain, but I'm also doing that), or increase my ability to scale
@roganjosh sockets since i thought it might be faster than reading files
21:04
But you're blowing up your memory?
This is getting more complicated by the minute
@jeremy not sure where you're trying to scale with 36 cores, and I don't see where memory becomes an issue, but admittedly I don't really know mulitprocessing or async so we can live with that
The quick answer is that multiprocessing is just going to copy over the namespace, so if it's exploding with one process, it definitely will explode with 35. And that's before you try any IPC
@roganjosh indeed it does, I admit its a bit of an XY. I have a massive grid and a bunch of data that takes place over that grid. each point needs to go through some MC sim a bunch of times. in the past, at smaller scales, I can get away with using multiprocessing (coordinating36 proc) and keeping the entire grid in memory. Now, I cant. So I thought maybe one node can iterate the grid &submit tasks to other nodes which have Python processes prepared to accept the data and run the simulations
What are nodes here btw? You've got CPU cores; how were you divvying that work up before?
I have a bunch of broadwell nodes with 36 cores each (intel xeon)
before I had an application that iterates over the grid, then uses multiprocessing to call my simulation over each point (with 36 processes at a time)
I could do that over some number of nodes at a time
21:15
Then it sounds like a job for spark or something?
that is definitely an option, I've never used spark. I've used Parsl on this problem but faced bottlenecks. I'll see what spark may offer
I was hoping there may be a way to do this with a socket and some multiprocessing
If you go back to your original approach; have you looked at something simple like htop to confirm it even uses all the cores available?
Because mean and stdev don't. They don't use BLAS or its friends and will run single-threaded
which is my original approach? where I have the grid in memory and running multiprocessing at the same time (which I can do on a smaller scale)?
5 mins ago, by jeremy
before I had an application that iterates over the grid, then uses multiprocessing to call my simulation over each point (with 36 processes at a time)
yes. no, I didnt use htop to check this, but I seriously think they are using all available cores. I set OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS, and NUMEXPR_NUM_THREADS to 1. then I create a worker pool (using multiprocessing) with 36 (or more, for oversubscribing) processes, then I use worker_pool.apply_async to add each simulation to my pool
21:23
But mean isn't a BLAS routine
I don't actually call mean. I am invoking statistical functions in rpy2. but, I try to make sure that any computation only can take a single thread
I honestly don't even fully comprehend what BLAS is. I can check htop and run some practice run on my smaller grid to see. you suspect I'm not actually getting multiple cores running my processes?
The call into R throws me, but "I set OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS, and NUMEXPR_NUM_THREADS to 1" sounds very wrong. I'm not convinced you're even using the cores of a single machine before you scale up
@roganjosh I stopped paying attention a while back, but if they want to scale over multiple cores they should definitely set those thread limits to 1, otherwise they might overallocate
Why would you set them to run a single thread?
if you have 35 processes on 35 cores you don't want each of them to try and run on 4 cores each
21:27
@roganjosh because otherwise I will be massively oversubscribed
In which case, I clearly am not going to be much use here sorry. I back out
well, you're the only one left :D
hahaha
thanks everyone, I appreciate the effort
the problem is I see too many unknowns in your problem specification so I can't even try to reason about it
Like that "memory" issue you mentioned. Suspect can of worms stuck in a rabbit hole.
I'm happy to add more details wherever possible if you want to try to keep helping hahaha. the memory issue is because I have something like a 6000x6000 grid, each cell has 30 points (floats). thats a lot for memory already, but then I want to fit a distribution on all 30 points for each cell and then grab some probabilities from that distribution
21:32
The exact implementation of what you're doing would probably end up being important. I suspect you just had all workers have all the data and work with part of it, whereas for scaling you have to try and divide up even the inputs among nodes. If you have to do this that's a highly non-trivial constraint both on available approaches and on how you can implement it. And this is just one piece of this mess.
I don't know what your "grid" means, I don't know what a "cell" means, I don't know what each process is trying to fit on what. What the inputs are, how fixed they are, what varies across process spawning, everything is unclear and all of it could be relevant. And frankly that's just too much of a barrier to try and jump into this. Sorry.
I understand. the grid is a 2d matrix, 6000x6000, each cell (a coordinate on that grid) has 30 points, so I have an array (30, 6000, 6000)
I tried to clean out the irrelevant info in my formulation of my original question, but I understand this is a complicated problem
one reason I'm asking in chat is to see if I can get a better grasp on some direction for where to take this, and see if I have a concrete question for main, or if I can simply figure it out on my own using suggests (such as spark or mpi4py)
Well "I have an array of shape (30, 6000, 6000)" is the clearest bit of information you've given so far.
(side note, for row-major memory order if you're working with 30-point batches at a time you'd want that to be the last axis, not the first one)
Well, maybe. It's tricky because 30 is a lot smaller than 6000.
@jeremy so how does memory explode if this array has fixed size?
I see, that's good to know.
That's about 8 GB of RAM?
multiprocessing will copy it for every process
21:39
@roganjosh sure, but each process gets it exactly once, there's no blowing up to speak of
every process doesnt get the full grid, just what it needs
one second, let me do some calculations @AndrasDeak
sure thing
as long as we can stick to concrete truths like ndarrays I can try to help
(6000*6000*30*28)*9.3132257461548E-10 is 28.16 GB of RAM?
I see some surprising numbers in there
Ah, that's a really weird way to spell 1/1024**3. So why 28?
lol, sorry. I copied that from a Google search. I called getsizeof on one value and got 28 bytes
21:44
That's not what you want. A double is 8 bytes.
For a large numpy array there's negligible overhead from the array itself, and you can just work with the size of the underlying data. Which is 8 bytes per element for int64 and float64.
An empty numpy array takes up 104 bytes. Your large numpy array takes up 8 GB plus 104 bytes, give or take.
I see, thanks.
Let me see
In that case it should also be no problem if each cell has an additional 63 outputs from calculations on the distribution, since (6000*6000*(30+63)*8)*9.3132257461548E-10 is 25 GB RAM, which is still manageable for me
hmmmm who is eating my memory
OK? I'm not sure if we're going in the direction that explains your past memory issues?
And don't forget that every process might allocate this array. So you have to have 25 GB per core. Do you have that?
it was a no brainer when i thought the correct number was 28 bytes
I have 128 GB per NODE, is that what you meant
that's less than 25*36
let me do another calculation. not every core needs 25 GB, since not every core gets the full grid
21:52
Depending on how you're doing multiprocessing*, each worker might end up with a copy of the original program
I hear you, one moment
no rush, I'm just thinking out loud
so I actually have (6580*6880*((20*3)+63)*8)*9.3132257461548E-10 = 41.5 (actual grid size changed, then I actually have 20 points, not 30. but then I also have 3 versions of those values, multiply by 3)
and then the multiprocessing should take (((20*3)+63+10)*8)*9.3132257461548E-10 for each process
that float literal is killing me
lol sorry
if Python doesnt empty the trash, that second calculation times the grid size again is 1614 44 Gb
21:59
What do you mean by "empty the trash"?
after a process in multiprocessing is done, if Python doesnt remove the variables I created in the calculations
and 44+41.5 is still enough memory, so I'm lost
When a process is done and the process is killed, so does its memory.
got it, I suspected as such
I guess I need to do some benchmarking, this is a headache
It's a separate Python process. It can't help but give it all back to the OS, as I understand.
I still don't understand what all those numbers are in your figures, to be clear
((20*3)+63)? (20*3)+63+10 per process?
I know. Its 20 timesteps of the grid, and 3 different versions of those 20 timesteps. 63 is the number of outputs I have per cell. 10 is the size of an auxiliary array I pass to the computation
I need to run but I really appreciate your help. I'm going to get back to this tonight with some major benchmarking to see if I need to scale or if I'm being memory inefficient somewhere
22:07
@jeremy Please do. If you want me to keep going you again have to go back from "timesteps" and "grid" and "versions of timesteps" to something unambiguous and concrete. Arrays on nodes.
user17921218
22:28
Can anyone help me with a question? (it's quite simple)
user17921218
Who is available?
Depends on the question
Sorry, but please don't ask for help with fresh questions here. As per the rules, questions must be at least 2 days old
Here's a tip though: You should try to eliminate unnecessary details from the question. Like the database. I don't have a database with the correct schema set up, and I don't want to create one. Especially since your problem is really just getting a value out of a combobox. The database isn't exactly a critical part of the question

« first day (4124 days earlier)      last day (1049 days later) »