« first day (4729 days earlier)      last day (444 days later) » 

07:07
@DorianTurba That's also an option, yeah
07:24
talking about partial, I tried and failed to write this section of dynamic-dict-nesting with partial, just couldn't wrap my head around how to do it. I'm 90% sure it's possible though: gist.github.com/a-recknagel/…
the idea is that given a list of strings (whose length would be considered the depth of the nesting), for ["a", "b"] you'd want {"a": defaultdict(lambda: {"b": {}})}
with a partial solution probably not having to use lambdas, but ¯\_(ツ)_/¯
the solution works if the resulting construct accepts x["a"]["foo"]["b"] = "bar" and crashes for x["b"] or x["a"]["foo"]["b"]["bar"] = "baz"
i.e. no infinidicts
08:21
partial can be to lambda what for is to while.
This mean that sometimes, you can use partials or lambda instead of the other. It doesn't mean that you should.
IMO, in this case, lambda is the way to go since you don't need to "preconfigure" the call.

In this case, you may use a closure instead of a lambda, I'll show you an example in a few minutes
with partial and recursion (could achieve the same with lambda)
def f(given: list[str]) -> list[str]:
    d = {}
    d[given[0]] = {} if len(given) <= 1 else collections.defaultdict(
        functools.partial(f, given[1:]))
    return d
Hmm, I interpreted the assignment in a different way. I only made 1 level of defaultdicts:
keys = ["a", "b"]

x = {}
for key in reversed(keys[1:]):
    x = {key: x}

def identity(x): return x
x = {keys[0]: defaultdict(partial(identity, x))}
but I have a question: x["a"]["foo"]["b"]["bar"] = "baz" won't crash with {"a": defaultdict(lambda: {"b": {}})} as x["a"]["foo"]["b"] is a dict
Although the output is the same for an input of length 2 (I think)
IMO in this case: x = {keys[0]: defaultdict(partial(identity, x))}, we should use lambda: x instead, and remove identity
The partial has the advantage that the repr gives you more information though
Welp
08:31
sure, a little docstring doesn't hurt :D
I just noticed it also leads to a bug, because it always returns the same dict instance
partial(copy.deepcopy, x) should work
your code ?
keys = ["a", "b"]

x = {}
for key in reversed(keys[1:]):
    x = {key: x}

x = {keys[0]: defaultdict(partial(copy.deepcopy, x))}
"x = {}
for key in reversed(keys[1:]):
    x = {key: x}
I don't get this part, I don't understand the meaning of this
The result we want is {"a": defaultdict(lambda: {"b": {}})}, and that loop builds up the {"b": {}}
08:35
cbg
but for ["a", "b", "c"], it builds a {'a': defaultdict(..., {'foo': {'b': {'c': {}}}})}
IMO it should build something close to {'a': defaultdict({"b": defaultdict("c": {})})}
@Aran-Fey What happened to Kevin?
the "b" should be a intermediary dict with a default dict that generate the last dict, don't you think?
I thought so too at first, but {"b": {}} is just regular dicts inside of each other. The only defaultdict in the output is the one for the key 'a'
Also anybody know how to use query objects in Python? I get following error: TypeError: '<=' not supported between instances of 'F' and 'int' but according to the doc, this should work... docs.djangoproject.com/en/4.2/ref/models/expressions
08:40
@Hakaishin I don't think we know, pretty sure he just stopped showing up at some point
@Aran-Fey a pitty
cases.filter(some_field__gte=my_function(F('some_field')) and my_function does like lambda x: x+foobar_constant
@Aran-Fey that's not how I understood the statement
Here is my implementation with closures
def f(given: list[str]):
    d = {}

    def inner():
        return f(given[1:])

    d[given[0]] = {} if len(given) <= 1 else collections.defaultdict(inner)
    return d
here, because of the recursive property of f, the closure doesn't need the partial
@Hakaishin Unable to load conversation b99f9fa9-b845-447c-b84f-f36f41b7a4f9
@DorianTurba hmmm I guess they are private not sure how to make them public
08:45
@Arne interresting challenge, but because of you, I'm late on my dev schedule ^^
ah wait, the share button generates another link, can you open that?
@Hakaishin y
@DorianTurba my pleasure =]
Wow, tldr, can you explain what you expected ?
i'll have to read everything in a bit, my own dev schedule is also getting in the way right now
08:47
Like it still halucinates like crazy. It says the way I want to use the code is correct, even though it generates an error. While I feel the same way, obviously something is wrong :D Also a section later it says the below code is using F but then never uses it, although it imports it
@Hakaishin I always says that copilot and chatgpt is dumb quick assistant
We should ask them what our brain can do easily, but slower, not what require real consciousness thinking
In brain science, I would says that we have system A and B, A is quick but not trustworthy and B is slow but reliable. We can use ChatGPT to get A type results even faster, nothing more
Don't remember the author of this theory tho (about A and B, not chatgpt)
I rarely need A type, that's what computers are already for.
I really how you are using your brain A for typing a for loop :)
Ok I figured out the problem and I hate it when it happens and in Django it happens surprisingly often. They implemented multiplication/addition and other mathematical operations, but no logical once. So I can do lambda x: x+3, but not lambda x: x + 3 if x > bar. It fails on the bar comparison. Now I know why my predecessor had to use custom sql functions. Ugh
if you are familier with matplotlib, can u please explain me this line of code?
fig, ax = plt.subplots(figsize = (6,4))
08:57
@Hakaishin I will behave like an Chaotic Evil guy but... Just use fastapi and sqlalchemy :D (I want to see the world burn !!!!)
@DorianTurba lol
we use sqlalchemy but I don't think fastapi is comparable to django
@discoMonkey as a Chaotic Evil guy, I would say it create a figure with axes based on the dimensions you privided (6, 4), but I have absolutely no idea so this is ChatGPT tier answer :D
@Hakaishin No, it's so much better, it would be like comparing a bike with a rocket, there is no point ;)
Yes, I'm a sponsor of fastapi, is that so obvious ?
09:14
@discoMonkey have you read the documentation? matplotlib.org/stable/api/_as_gen/…
@discoMonkey What exactly needs explaining about it?
@DorianTurba right, I forgot about that level. x["a"]["foo"]["b"]["bar"] = "baz" is fine, x["a"]["foo"]["b"]["bar"]["baz"] = "qux" should crash
@Hakaishin I have spoken to Kevin by email several months ago. He is safe and well, he's just taking a break from programming stuff in general
12
@KarlKnechtel in i dont understand the comma before equal operator
is it a valid code in python?
09:27
@discoMonkey It's "tuple unpacking". Two objects are returned and you unpack them into two different names
ok, now I understand, thanks @roganjosh
@Aran-Fey sorry, my bad. the goal was for the innermost nesting to be a normal dict, and every one around it to be a defaultdict
(if you are not familiar with this, I would recommend that you spend more time studying Python fundamentals before trying to use big third-party libraries like Matplotlib)
@KarlKnechtel answers from the 'Gurus' like you, teaches me faster. thank you sir
also I believe in learning by doing
@roganjosh Idk why but I watched a YouTube video titled "awesome Python feature" and the author said "I don't think the operator has a name" and the code was *_, = [1, 2, 3]...
09:32
@roganjosh thanks
You watched it because YouTube is the place to get l33t skillz
@Arne then you can use my code, the one with lambda, partial and closure works the same and match your requirements
Indeed. Interestingly tuple unpacking with a * returns a list
@Peilonrayz never watch youtube videos for code. Youtube is great for a bunch of things, not for learning to code
@discoMonkey see end of the section: docs.python.org/3/tutorial/…
09:37
thanks dorian
@Peilonrayz this probably covers it
@Hakaishin I normally don't. I started watching a couple of Linux channels which can be useful. Thought I'd give Python a chance too ;)
@discoMonkey You won't have us for your whole career. I advise you to read the official tutorial. If there is a syntax you don't know, you may ask chatgpt, it may help you to find the name of the syntax structure, for you to look about it in documentation after
3
It is indeed somewhat unexpected
@Peilonrayz it's packing, packing is not tuple unpack :)
09:38
lol
@DorianTurba please can we not suggest people use ChatGPT over this chat room. As long as they are not bombarding us with questions, I'm quite happy for people to learn from us
"*_, = [1, 2, 3]" Is a swallow unpack, it take everything and throw it in the _ var
@roganjosh I'm fine with that :), won't do it again !
An awful lot of the python I know has come in part from this room. That's why I hang around as an RO and take all the crap when someone gets unhappy or spams us :P
What I mean is that it's hard to read documentation, even more to look over somehting we ignore the name. But if there is someone here to help, yep, I agree, > chatgpt :)
this room taught me more python than self research and school
09:42
"*_, = [1, 2, 3]" This is often used to get a few items at the start or end of a list like
My first programming language was C++,
a, b, *_, = [1, 2, 3, 4]  # a == 1, b == 2, _ == [3, 4]
*_, a, b = [1, 2, 3, 4]  # _ == [3, 4] a == 3, b == 4,
a, *_, b = [1, 2, 3, 4]  # a == 1_ == [2, 3] b == 4,
mistake in the second one, it's _ == [1, 2]
I think the point is that it unpacks to a list and not a tuple as in the link I showed. PEP 448 addresses that
@roganjosh I thought the syntax was called "tuple unpacking" but the PEP says "sequence unpacking" and the docs say "sequence unpacking"... Great now I have to change my terminology. I wonder how many answers I'd have to edit too
python is the best language for humans
09:47
@Peilonrayz I don't think the distinction is too great tbh. Certainly "tuple unpacking" will give you enough to work from and everyone should know what you're talking about
09:59
I think youtube is fighting back against uBlock Origin. Adverts are now starting to get through for me but the screen is blank. Unless they've decided to do audiobook-style adverts something is going wonky
Meh, for anything interesting chatgpt tells me it can't do it, so lame
 
1 hour later…
11:29
@DorianTurba thanks, I'm trying to understand it.
11:49
brief cbg folks
cbg Jon, long time no see!
indeed... how you doing?
how are things?
can't complain here thanks
We had our second son a while ago, and he finally decided to cry less and sleep more, so I'm feeling pretty great.
11:52
@Peilonrayz @roganjosh imho... "iterable unpacking" would probably be more accurate
@Arne congrats!
thanks, getting enough sleep definitely makes me appreciate the experience a lot more =)
I agree that it would be more accurate but I don't know whether there's a serious need to go back and edit lots of previous answers since it's a well-established term at this point. Also, good to see you around pup; glad you are well :)
@roganjosh I only have like 20 with the term "tuple unpacking"
@Arne ah, I don't think I knew this. Congrats!
I don't think I mentioned it here before, thanks!
12:03
@roganjosh it's certainly been a while but thought I'd pop my head in and say cbg :)
how you doing?
Always a pleasure :) I'm not too bad thanks. The stint at contracting didn't work out so well; I got 1 contract and then it dried up. But I just found out this morning that I have a new permanent role so it's worked out ok
Just annoyed that I'm now indemnified for £10m for the full year and now I don't get to go on my coding rampage :P
ultimately good news then I guess... you still in the routing stuff or new horizons ?
@roganjosh condolences/congrats!
Routing, manufacturing and supply chain is my main game but tech has taken a hit there because inflation has snuffed out a lot of the "blue sky" initiatives for old companies right now. They're seemingly after knuckling down in the short term. This one is in retail and forecasting
Thanks :)
yeah... I seem to be into retail and analytics a lot recently...
recently spent way too much time smashing my head against a brick wall getting a WiX store to work as any sane person would expect it to :)
12:11
Forecasting is such dodgy ground. We did quite a bit at my old place for a lot of different clients and it's just so fickle. Plus, forecasts are meaningless if you can't get operational change. I'll have my work cut out for me
well, I know you like to embrace a challenge - you'll have fun :p
@JonClements you could have just bought a hammer to smash the wall - you were in the right store for it :P
Have you ever seen a forecast for a supplier to Amazon?
if I had done that first and not incurred brain damage, I might have not continued doing so, and bought a hammer :p
@roganjosh kind of... but it's mixed in with multiple self-shipping warehouses, multiple fulfillment points and other shipping channels... so it's confusing
We were working on a price elasticity model for a condom manufacturer <childish giggles>. Amazon gave them an order not more than 2 weeks in advance and we reviewed their historical prices. They dropped the price of a bumper pack from £60 to £30 overnight, with no warning, and they expect the supplier to handle the new insane demand in 2 weeks, or get fined even on partial fulfilment. Nightmare fuel.
partial fulfilment is a beep in itself...
12:19
@roganjosh the kind of elasticity you can only expect from a condom, impressive
@JonClements not much you can do when Amazon will drop bombshells on you with absolutely no notice. Unless you could reverse-engineer their entire, bizarre, pricing system to predict the demand, you just have to do damage mitigation to spread your partial fulfilment across all the other retail customers.
@roganjosh I've got one client that uses one of those dropshipper/do all the seo thingies for you... a marketplace aggregation site kind of thing...
and they've got products on there half the price of their actual retail site
the rationale apparently is "because we don't want to sell them"
I've given up trying to talk any sense of reason... they pay invoices mostly on time... so it's their issue
I can't find the programme this company was on; I can find lots of Amazon schemes but this was a global manufacturer. Maybe it works for your client, for ours it was an insidious trap with fines and was a real problem with their supply to e.g. Tesco or other major retailers. You're gonna get fined by one or more of them because of this, but you can't afford not to be on Amazon for this kind of product too.
12:45
@Peilonrayz I found this in the linked questions to the post I linked you. This is a curious bug indeed (now fixed in 3.8) :)
@roganjosh Woah, pretty interesting bug
13:29
@roganjosh usually, I only says "unpacking"
makes sense
13:42
Is anyone working in kubernetes environments? Had an interesting/surprising thing happen this morning, and in retrospect, I probably should not have been so surprised.
I did... but I never directly set it up. Still we had lots of discussions about expected behaviour and benchmarking so there's an off-chance I might know
I was working with another team on parallelizing some work, and we needed to come up with how many workers to use for a ThreadPoolExecutor. I suggested calling multiprocessing.cpu_count(), and in fact, the TPE class will use the cpu count as part of coming up with a default.
So I deployed a small project, and found that it was running on a system with 64 cores and 256GB of RAM. But the pod this project was running in is much much smaller.
It looks at the underlying architecture and not the container-allocated resources. Let me see if I can remember the robust way of getting a CPU count
Yup, that is what we came up with as well.
Gah, there's a wrapper library around multiprocessing that I seem to remember has a working method but I can't think of the name
13:48
There is some interesting history on the default max_workers arg for TPE - in Py3.5, it was set to 5 x cpu_count, but in 3.8 was cut back to min(32, cpu_count+4) to address the issue of these larger systems.
I'll text one of my old colleagues and get him to ask around. We faffed with this for some time as a company. There is a python solution
Great, thanks!
someone is used to python packaging ? I try to include some files in the package in order to use it after install, but using pyproject.toml, but fastapi/starlette can't find directories I try to include
The pyproject config
[tool.hatch.build.targets.wheel.force-include]
"./templates" = "pkg/templates"
"./static" = "pkg/static"
But I never did such things before and it's very hard for me to find doc about this
14:05
@PaulMcG I haven't heard back and I can't think of the wrapper library, but I suspect it uses one of the options from the second list of this answer
14:18
joblib! That's the library I was trying to think of :P Thank god that brain worm is deaded
14:36
One of the suggestions in that SO answer was psutil.cpu_count() - I'm using that now, but getting the host's CPU count, not the container. There is another answer that suggests joblib, but a comment says that joblib looks at the multiprocessing module, which I've already determined also gives the host cpu count, not the container's. (Granted this is an answer from 2015, things might have changed since then.)
I'll give joblib a try.
Hmm, let's go into the source, then
It's certainly possible that I'm thinking about Docker on a partitioned instance and that K8s itself needs hard-coding for each instance resource in its .yaml (or whatever config file)
@PaulMcG shockingly there's an actually-informative medium article about this
14:59
I think this needs to be set in k8s itself. My reading is that if you have a 64 core machine, multiprocessing.cpu_count() is going to give you "64" all the time. If you allocate half of those cores in k8s, psutil.cpu_count() will now give you "32" but they'll be split across all containers in the node, and pods can dive in as they choose
For completeness, though, my colleague has texted back and we were using loky
I think I'll need to get in touch with our platform team that does all the deploy setups, and see where they configure this stuff. The vanilla methods so far seem to be giving me the machine total CPU count, even though I know I'm configured for just a tiny sliver of that.
I used chatgpt the first time productively to find a bash command which does a thing and then it could even make it a one-liner. I gotta say, that was really impressive
Also it tells quite decent jokes :P
I found how to do it
[tool.hatch.build.targets.wheel.force-include]
"src/poc_api/templates" = "poc_api/templates"
"src/poc_api/static" = "poc_api/static"
and a little magic with pathlib
15:18
@PaulMcG I am curious about the use of k8s here since the whole point is for it to expand into the resource based on demand. That gives you one degree of freedom in the first place (spawn more containers) and then you introduce a second degree of freedom with multiprocessing within each container?
A dumb approach would be to reduce it down to a single core per container, and drop multiprocessing. I suspect the overhead of container orchestration there would be crippling (especially their RAM requirements). But it's an interesting trade-off that I haven't considered before; that might make a fun graph when you look at optimising pod size and allocated resources
@DorianTurba I'm also unable to find much helpful documentation, but I'm guessing you're supposed to access the files via importlib.resources
...which, I might mention, is also a rather poorly documented module
15:41
The k8s world here is heavily influenced by the needs of supporting a global high-traffic website, with auto scale-up/down features, failover, etc. But we use the same utilities to deploy small internal services as used for customer-facing services. The second group is the world I spend a lot of my time in (typically configured to use 0.5 CPUs, for instance).
If that's the case, then certainly it should be possible to fix the CPU affinity for your script
Although I guess it expects every other user of the underlying instance to also do the same
Right, its not a thing I can do for just my own projects in isolation.
I might actually get a nasty letter if someone in the platform team notices.
15:57
From what you've said, I don't think this is true. I think it's the platform team that need to figure this out. My guess would be that there needs to be a proper static policy for your script and everyone else's that gets deployed
It's the latter part I'm not sure about "[...] I know I'm configured for just a tiny sliver of that.". I'm not so sure
Fair point. It does tell me that I need to raise my consciousness on the deployment configs and what impact they have on tuning apps for available resources.
Hopefully it's as simple as a missing config param :)
 
1 hour later…
17:24
@PaulMcG joblib should be doing the cpu counting right. We get hit by users doing it wrong regularly… :/
That's not on k8s though, right?
Interesting background discussion here: github.com/duckdb/duckdb/issues/6519
In this case, I think it's a subdivision of subdivisions. I could take 64 cores, split half into a k8s node and then have pods inside that that I want to only use 2 cores each. But I think it's free to choose any of the 32 cores it wants to use and it's possible (I guess?) that it could spread a job across all 32
@roganjosh No, just several container layers piled on top of each other the old fashioned way.
Ahh, a fusion of DuckDB and K8s. I forgot it was my birthday! :D
loky is mentioned in that thread too, interestingly. And it's tough as hell to search for in Google
17:44
@roganjosh Indeed I had a sudden feeling of irony when digging up that ticket. :D
Or whatever it's called... Thanks, Alanis!
The irony of the song is that most of the examples aren't ironic... or are they?
Hello. Is there anybody in there?
Yes, though please don't tell anyone!
Trying to work out why a colleague would tell you to kill em all, then ten minutes later to tell you wait a bit before doing it...
Speaking of reaping zombie children and somesuch! Don't want to give any wrong impressions...
18:18
Well this seems to be the root of the loxy "magic". I'm starting to feel that this could be the pinacle of what python can do and the rest is down to the k8s config
loky*
18:40
we had a similar issue just last week, poetry also uses some kind of cpu count to guess how much it can parallelize 3rd party package installs, and after moving our build runners to a 64 core machine all build containers suicided themselves trying to install 50 packages at once while actually only having barely one actually
fwiw, we also had to hard code the real limit, no elegant solution was found
On a related note, I learned from build-python-from-source.com about make -j "$(nproc)"

« first day (4729 days earlier)      last day (444 days later) »