« first day (4547 days earlier)      last day (384 days later) » 

 
5 hours later…
user17135505
6:28 AM
Good morning, this may not be purely python but would you have any suggestion how to store nested dictionaries in html?
 
user17135505
    import pandas as pd
    # non-nested df, matching Y axis
    d1 = {'A': {2: 'X', 3: 'Y'}, 'B': {2: 'Z', 3: ''}}
    print(pd.DataFrame(d1), '\n')

    # nested df, non-matching Y axis
    d2 = {'A': {-1: {'.': 'X'}, 2: 'Y'}, 'B': {3: 'Z', 4: {'.': ''}}}
    print(pd.DataFrame(d2).fillna(''), '\n')

    # super nested df, matching Y axis
    d3 = {'A': {-1: {'.': 'X'}, 2: 'Y'}, 'B': {-1: 'Z', 2: {'.': '', '-': 'ZZ'}}}
    print(pd.DataFrame(d3), '\n')
 
user17135505
The representation complexity arises with 3rd+ dimensions, such as in case of d3 where [B, 2] is a dictionary.

I guess it is not a good practice to have super nested data structures.
 
I don't understand the question, what does "store dictionaries in html" mean? HTML isn't (supposed to be used as) a data storage format, it's more of a GUI
 
user17135505
True, maybe differently: how to represent nested structures (3+ dimensions) in HTML? Is it bad to have tables within table?
 
user17135505
Is flat design better also as a programming practice?
 
6:37 AM
No. Use whatever is best for the task. Can be flat, can be nested. But yes, a GUI that consists of tables inside of tables sounds awful
I can't give you any advice how to make a good GUI because I don't know what your data represents
...and I'm not a frontend dev
 
user17135505
Ok, thanks. I'll research a bit more. I guess end users do not like flat but it is easier for overall manipulation.
 
user17135505
My hunch tells me every time there is more than 3 levels, one should separate L1/L2 from L3 via separate storage object / class.
 
user17135505
I also somehow feel that technical (backend) considerations are actually important for business (frontend)
 
user17135505
Is it some sort of developer's arrogance? :-(
 
user17135505
What is technically sound must be sound from non-technical perspective. I mean, technical is mathematical, concerned with efficiency and elegance
 
user17135505
6:49 AM
What is non-technical anyhow
 
7:14 AM
@learning_python_self you are possibly optimizing correctly your problem, given certain constraints, but you probably also don't see all the wider constraints that the organization has, and a lot of those are non-technical ;)
 
user17135505
7:50 AM
Hi, can 'loop until condition else exit within n seconds' be done differently / better? (Other than more explicit exception)
 
user17135505
    import time
    time_limit = time.time() + 2

    while True:
        if time.time() > time_limit:
            raise Exception('Not found')
        try:
            raise  # proxy for something failing
            print('OK')
            break
        except:
            print(time.time())
            continue
 
You should use time.monotonic, but other than that I don't see any improvements
 
user17135505
Thanks. (Interesting read about monotonic)
 
8:17 AM
Ideally, you don't spinlock but have an actual waiting strategy.
Exponential sleep often works well in practice. If the condition is based on the program state, use a synchronisation primitive like an Event.
 
user17135505
8:30 AM
Could you please expand?
 
A spinlock is code that continuously checks for a condition to become true. That's very bad for your CPU efficiency and may actually keep/delay other threads from completing the condition.
The naive approach is to have a short pausing wait (e.g. time.sleep) to not hog the CPU constantly.
 
user17135505
I see, good point
 
user17135505
What would be not naive approach?
 
How complex is the raise # proxy for something failing part of your code?
Because if it not some very trivial statement then you're probably not having spinlock anyway
 
user17135505
it reads 2 spreadsheets
 
user17135505
8:37 AM
Also good point!
 
One thing about your code is that it checks for timeout only once each iteration, so if e.g. iteration takes 10s it will need to wait this much even if timeout is 2s
 
user17135505
It is pretty trivial operation, so avoiding spinlock is a very good point. The actual timeout is 20 seconds.
 
user17135505
Would you place time.sleep just below while or inside except?
 
user17135505
I do not like to wait at the beginning of cycle but it feels more right than to do it in except
 
@learning_python_self Scaling the sleep duration exponentially works pretty well. For example, subprocess.Popen.wait on POSIX starts at a 1ms delay and then scales it by factor of 2 up to 50ms.
Such a strategy is responsive if the condition is just about to happen, and if you are waiting for a while anyway then the loss of responsiveness usually doesn't hurt.
 
user17135505
8:47 AM
If the condition is just about to happen, why would you be increasing the delta? I guess this makes sense especially when calls are somehow expensive?
 
At the start, the delta isn't increased by much since it only doubles. If you are waiting for 1ms, then 2ms, then 4ms, ... you are still very responsive in a 10ms window, for example.
 
user17135505
[i need to learn subprocess / asyncio to elevate my python knowledge]
 
user17135505
I found https://stackoverflow.com/questions/27438273/exponential-backoff-time-sleep-with-random-randint0-1000-1000 pretty interesting:

"Having a bit of randomness in situations like this is good. For example, if you have a large number of clients hitting the same server, having them use the same deterministic backoff could result in them hitting the server in perfect lockstep, which isn't desirable."
 
user17135505
@MisterMiyagi ever seen negative exponential implementation? it feels more practical - i still cannot imagine where delta shall be increasing - as long as floored
 
If you have a rough guess how soon something will happen, I can see a point in a negative exponential.
I probably wouldn't use it, since it doesn't help the relative error. But if one cares for the absolute error it may be worth it.
 
 
2 hours later…
11:16 AM
@Aran-Fey making your __post_init__s cooperative is not an option?
 
Not an option I like. Some child classes will be user-created, and I'm sure many of them will forget about the super()
 
11:50 AM
Is an asyncio.Event suitable to repeatedly inform waiters that something changed? The docs are vague on what happens if one Task does await event.wait() and another does event.set() followed immediately by event.clear(). It's unclear if the event must still be set for the wait to end.
As background, I've got two tasks where one creates files and the other processes them, but the two don't know about the respective strategy (ordering, bulk size, etc.). So the creating Task must tell the processing Task "there's new stuff" repeatedly for each created file to let it decide whether it wants to act or not.
 
Pretty sure that should work for your case, the docs say "All tasks waiting for event to be set will be immediately awakened."
Although judging by your description of the problem, I'm wondering why you're not using a queue
 
This will be a very long-running program (~months) that will get killed and restarted. That'll clear all queues but not the created files, so I cannot rely on queue content anyway.
 
user17135505
12:48 PM
One question: I want to raise an exception when my program does not manage to access some files within x seconds. Is it fine to raise TimeoutError or this error is reserved for smth specific?
 
user17135505
Simple raise Exception('blah blah') is considered poor taste by linting tool I use
 
user17135505
Per docs.python.org/3/library/exceptions.html#TimeoutError, "Raised when a system function timed out at the system level".
 
1:00 PM
@learning_python_self sounds more like an OSError. The timeout is an implementation detail.
 
TimeoutError is OSError
 
Yeah, I just saw that, thanks
yet it's a too specific subclass
(As far as semantics go)
 
From the generic exceptions I think IOError is most fitting for problem with reading from file
 
@matszwecja which is even more of OSError
 
Not according to Python
 
1:03 PM
> Changed in version 3.3: EnvironmentError, IOError, WindowsError, socket.error, select.error and mmap.error have been merged into OSError, and the constructor may return a subclass.
 
Ah, right
Confusing as they aren't listed under OS Exceptions header in the docs: docs.python.org/3/library/exceptions.html#os-exceptions
 
Because it's an alias
On my linux IOError is OSError
Which is why I said OSError originally
 
Yeah, still - confusing.
 
user17135505
    import time
    time_limit = time.monotonic() + 2

    while True:
        if time.monotonic() > time_limit:
            raise FileNotFoundError('no file')
        try:
            raise ValueError()
            break
        except FileNotFoundError:
            time.sleep(1)
            continue
        except ValueError:
            time.sleep(2)
            continue
 
user17135505
When you define multiple exception types, it seems that final raise on timeout shall refer to something more generic as it is misleading in the above case. Or would you try to split both operations (find file, valid value)?
 
1:17 PM
What is the goal of the code?
 
user17135505
Repeat execution for 2 seconds if function within try block cannot execute due to no file or no valid value.
 
If the file has incorrect value, do you expect it to change within that time?
 
user17135505
I suppose the error raised in if shall refer to overall condition: time reached, hence TimeoutError or as you say above, IOError
 
user17135505
@matszwecja No really, good point. This was more theoretical how to handle the situation where 2 errors are plausible.
 
The sketch of the code there doesn't make sense to me. If you get ValueError then you're "guaranteed" to throw FileNotFoundError('no file') on the next iteration because you've already slept for longer than your time_limit
 
user17135505
1:23 PM
Yes, this is mistake above. Let's assume time.monotonic() + 20
 
How many files are you searching for?
 
user17135505
Let's say 1 and there is a chance value error will be fixed within 20 seconds
 
Actually, what is the search? If you're going directly to the path then it'll throw basically instantly itself that the file doesn't exist. Are you trying to do a wildcard match on the file name across a lot of directories?
 
user17135505
Looking for in a location for a specific file
 
What I'm getting at more broadly is that the error handling here needs to fit the context of what you're actually doing. I could make a theoretical parser for each file name that takes 30 seconds to parse just one name, and be using it on os.listdir() with thousands of files in it... in this case your exception handler does nothing because you're still going to wait 500 minutes before you even get a chance to throw an error and break the loop
And it probably wouldn't be a theoretical parser in my case. I'm pretty sure I could work in some catastrophic backtracking with very little effort if I ever tried to write my own regex
 
user17135505
1:28 PM
the raise ValueError() in try is done only for demonstration purposes. In practice, there is a function, so the code will stop termination if other error than FileNotFoundError or ValueError is raised.
 
Exactly, but your code will be executed serially, so that function might take a long time to evaluate and your while loop is just sitting and waiting for it to finish. It's not doing anything to monitor the actual runtime at that point
 
I am undecided whether at the end you should throw Timeout or the original error, i.e. FileNotFound/Value
 
user17135505
Oh, so you are questioning the function which can take far more than 20 seconds?
 
user17135505
I guess super safe code would decorate every piece of code with timeouts
 
user17135505
Or functions in higher scope
 
1:31 PM
The super-safe code might spawn a separate thread/process to monitor the actual time elapsed so that you could throw an error regardless of whether you have a single function call that takes an unreasonable time
 
user17135505
I like that
 
I'm trying to find the canon for "timed user input" which was a question that came up all the time, as a scaffold for you to look through but I can't find the one I'm thinking of
 
user17135505
Is this done with asyncio?
 
maybe something like pypi.org/project/async-timeout
It can be done with asyncio
But that's just one of the ways
 
Folks, I don't understand why the second parameter must be exactly named `dt`. Changing to other names makes the animation no longer work.

Usually parameter is a dummy variable that be named anything.
from manim import *


class Updater(Scene):
    def construct(self):
        t = 0
        rate = 0.5

        circle = Circle(radius=3)
        dot = Dot(RIGHT*3)
        dot.move_to(circle.point_from_proportion(0))


        def callback(m, dt):
            nonlocal t
            t += rate*dt
            m.move_to(circle.point_from_proportion(t%1))


        dot.add_updater(callback)
        self.add(dot, circle)

        self.wait(10)
        dot.remove_updater(callback)
 
1:41 PM
@TheRealMasochist I have observed this behaviour as well with ROS, so if you figure out the answer, I'm curious :)
 
@learning_python_self Remember that you can define your own exceptions. There's no shame in defining a custom TimeoutError. For example, both asyncio and multiprocessing define their own separate TimeoutError.
 
@TheRealMasochist I guess that's the culprit:
def add_updater(
        self,
        update_function: Updater,
        index: int | None = None,
        call_updater: bool = True
    ) -> Self:
        if "dt" in get_parameters(update_function):
            updater_list = self.time_based_updaters
        else:
            updater_list = self.non_time_updaters
 
cabbage, world!
 
Why they wrote it like that, I cannot say
 
@matszwecja Thank you. Well done. :-)
 
1:47 PM
That's probably the most brittle API I've seen in at least a month :/
But indeed, I wasn't even 50% of the way through trying to track that one down. Nice find :)
 
@matszwecja ugh, I guess ros must be doing something similar$
 
user17135505
@MisterMiyagi Thanks
 
2:15 PM
@MisterMiyagi but the default stance is "try tp use one of the 100 built-in exceptions if you can". "Needing to catch it" being a case of "can't".
@matszwecja yikes
Someone thought this was better than introducing a new optional parameter
 
I'm not a fan of reusing the builtin exceptions until they fully match the use-case. Re-use FileNotFoundError when a file isn't in a file-system-like object store? That works. Re-use FileNotFoundError when Salvator from Archive Inc calls because they lost your paper trail? No-no.
So reusing builtin TimeoutError when it's not some OS related timeout is fishy IMO.
 
2:43 PM
@Aran-Fey Hm, you are right that a queue makes this nicer. If I change the create Task to an "create or check if exists" Task then using a queue from one to the other works nicely even on restarts.
 
3:27 PM
So I have a txt file, looking like this:
As you can see, the commas not only separate columns, but also are used in floats. The first value in the third column, for example, is 0,001 (or 0.001 if you prefer a dot). Is there a way in pd.read_csv() to let pandas know I have commas in certain column? Grateful for any help.
 
The writer for that csv should have handled that for you
 
Yeah, thought so.
 
If the field contains the delimiter, it should have been quoted, and a CSV parser would know that
I don't believe there is a sane way to differentiate delimiters and normal characters if the data is dumped as raw text. I think it's a rabbit hole. We tried. Can you fix it upstream?
 
I can't :(
 
3:48 PM
In the way you've presented it, you could skip the 1st row (based on zero indexiing) and just take the 0th row as an index. But beyond that, you're at the mercy of the values themselves
 
If the commas are in the exact same position on every line, then you could make an ad-hoc parser with fixed-width fields
 
You just have to hope there's no 1,000,000 value to throw everything out of whack. We pray that .head() is representative
 
Well, there are always three numbers after the decimal in each column, so 0,003 and 13,003.
 
If one column can contain both 0,003 and 13,003 then it's not so easy
If every line has the same number of commas, then parsing may still be possible.
 
4:04 PM
Every line has the same number of commas, but one column can contain both 0,003 and 13,003.
 
It might be easiest to have a pre-pass and replace the , in each \d+,\d{3}.
 
@schn Here's a proof of concept: pastebin.com/raw/36zrR2fE
Exercise left to the reader: take the tuple of strings returned by parse_line, and convert it into datetimes and floats, and stick em all in a DataFrame.
Or whatever it is that pd.read_csv usually gives you
 
4:19 PM
@Kevin thank you!
 
 
4 hours later…
7:57 PM
Cbg. If I have a number in float32 and I need it with 10 bits in precision (which means 4 significant decimal digits precision) do I need just round the number to 4 decimal places?
 
I can never tell if "decimal digits" means "base 10 digits" or "digits after the decimal point"
Anyway, I suspect an XY problem. Once you have your 10-bit-precision float, what are you going to do with it? Why do you need it specifically like that?
Are you going to convert it to bytes?
I guess it's not necessarily an XY problem, it's just not clear to me how you want to create your 10-bit float. Usually we don't care about how our numbers are represented on the byte level, and python doesn't exactly have many tools that help you convert numbers to bytes
 
just take the first 10 bits in each number, duh
 
Silly me!
 
too bad numpy doesn't have a 1-bit type, otherwise we could use ndarray.view() to do the conversion
 
8:16 PM
Couldn't you use True or False as psuedo 1-bit values?
 
@Aran-Fey Sorry, I meant digits after the decimal point
@Aran-Fey To lower precision
@Aran-Fey No
 
Why specifically 10 bits, then? Why not simply round() to decimal (base 10) digits?
 
@Aran-Fey because I want this precision. Well, so it's better to round() to decimal (base 10) digits?
 
It's easier, that's for sure
You said you don't want to convert the number to bytes. So then how do you intend to create a 10-bit float? How will your 10-bit float be represented in python?
 
@MattDMo you could, but the "pseudo" means you can't use ndarray.view()
 
8:26 PM
Ah. See the warning box here.
 
I don't think that's relevant
 
Maybe my mind is on a different track...
 
>>> np.arange(3, dtype=np.uint8).view(bool)
array([False,  True,  True])
>>> # cf
>>> np.arange(3, dtype=np.uint32).view(bool)
array([False, False, False, False,  True, False, False, False,  True,
       False, False, False])
 
I'm a bit lost on why this needs to be done manually. If you want to go between numpy and TF, that's surely a common path
Or, actually, between any of the formats shown in the graph. Surely they have built-in converters
 
8:37 PM
PSA: Careful about GitHub, folks. You're going to want to revoke old keys - theregister.com/2023/03/24/github_changes_its_ssh_host
3
 
@Aran-Fey Sorry, I had to leave for a few minutes. I'm just trying to understand what that need would be. First I want to understand if what I'm currently doing is wrong, just rounding to 4 decimal places. I'm using pytorch.org/docs/stable/generated/torch.round.html.
@roganjosh Good point, I will try to explain this point now better.
Well, that's because: first, I didn't know there was a ready-made function that converts FP32 to TF32, and second, the GPU I'm using doesn't support TF32, so I need to do a manual conversion.
 
Is it NVIDIA or your GPU is a different model?
 
NVIDIA, but it's a Volta one
A V100
 
I would have thought that there was some CUDA interim that could handle this for you
 
I don't know if I understood what you said, did you mean that there may be some CUDA function that can do this for me?
 
8:48 PM
It might be the case that you want cupy arrays from the start and they might intrinsically know how to handle your precision issues
(I'm not experienced with CUDA other than just sitting on the sidelines and dreaming about what I might do with it)
 
@roganjosh Thanks for this tip, I even recently tried to adapt a part of my code using cupy, but I ended up getting errors and ended up solving it in another way. But I'll check if it has a function that converts FP32 to TF32. But going back to what I was saying at the beginning, is the fact that I just round to 4 decimal places wrong? Would it be more correct for me to mess with the bits?
@roganjosh No problem, I know next to nothing about CUDA.
 
I can tell you what I think: Messing with bits isn't what I would do. The python ecosystem generally just doesn't demand this, and there are a lot of established libraries. There'll almost-certainly be functionality for this somewhere, but I don't know it off-hand to give you sorry
 
Ok, but when I said "mess with the bits" I was referring to what Aran commented here: chat.stackoverflow.com/transcript/message/56155053#56155053.
 
Sure, but it was you that originally suggested the 10 bits. He was asking about that
I think the 10 bit suggestion might just be a red herring and we should focus on something else
i.e. whether there are intrinsic conversions that you could leverage
 
Okay, maybe I'm just a little confused, sorry. But then is it something that is not so trivial to do? When I commented about rounding to 4 decimal places, I'm doing that because I read it here - https://moocaholic.medium.com/fp64-fp32-fp16-bfloat16-tf32-and-other-members-of-the-zoo-a1ca7897d407
...that TF32 has 4 decimal places after zero. So at the moment I think this would be a solution, but I wanted to confirm that this is correct.
 
9:03 PM
I really don't think this stuff comes into the domain of .round() because it's super complicated and way beyond my understanding. But, the answer is - I don't know.
 
Okay, so maybe just rounding isn't enough. I'm just rounding for now so that it's as close as possible to the decimal point precision that TF32 uses.
@Aran-Fey By now I'm doing x = torch.round(x, decimals=4)
Maybe it's exactly what you suggested.
 
Love it when the first sentence in the docs is already a lie
> Rounds elements of input to the nearest integer.
 
Hehe, yeah, there is a important note about it
 
I don't know what data type that function outputs. You might still need to convert your numbers to the correct type of float32
 
It would probably keep the same type as the input. In my case, FP32 came in and FP32 came out (but now with rounding).
A FP32 dressed as a TF32 would come out.
 
9:29 PM
Am I missing something? Why is there any conversion necessary? If you had doubles, and wanted to interface with code that uses floats under the hood, would you want to first "round your doubles to 32-bit accuracy"?
 
I didn't understand. I'm wanting to reduce the precision in the decimal places.
 
I'm just wanting to confirm if the rounding I'm doing is enough.
@AndrasDeak--СлаваУкраїні Oh ok, I think you were referring to what Aran said.
@AndrasDeak--СлаваУкраїні I don't know if the last question was for me, but my case is different.
 

« first day (4547 days earlier)      last day (384 days later) »