« first day (4381 days earlier)      last day (567 days later) » 

3:39 AM
 
 
3 hours later…
6:35 AM
cog folks, just curious, is there some argument one way or another for what's a better way to parallelize some workload? choices being parallelize on each step, or parallelize on each "document" so to speak. eg. option 1. given 100 documents, we first parallelize step 1 on each document, then run step 2 on each document in parallel. etc. vs. option 2. given 100 documents, we run one document through all 3 steps or whatnot. then treat that as a single pipeline to run in parallel
 
The second is usually much simpler since each document is processed independently. "embarrassingly parallel" is the keyword.
You've got lower initial effort for the parallelisation, and can easily scale out the parallelisation.
It's the primary paradigm for high throughput computing.
 
cool, makes sense, thanks
 
Your first approach is viable though if you don't insist on running each step for each document before proceeding with the next step.
 
I suppose in the way I'm seeing it being used here, it's doing exactly that: exclusively running one task for all documents, then doing the next for all documents, and so on
 
Good morning!
I am trying to run this python3 command: os.system("rtcwake -m mem --seconds " + str("30")), but it fails: rtcwake: /dev/rtc0: unable to find device: Permission denied.
Fron linux cmd there is no problem on execution of rtcwake.
Any thoughts around?
 
6:47 AM
os.system... eyes twitch nervously
 
I also tried: proc = subprocess.Popen(['sudo', '-S', 'rtcwake', '-m', 'mem', '-s', '30'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate(input=b'password_here\n')
 
to your point though, I have no clue what rtcwake is or so on, I won't be of much help here.
 
@ParitoshSingh rtcwake is a unix command to sleep and wake up at a certain time the pc.
 
oh. does this command need sudo to work properly?
 
It seems that python can't access this device: /dev/rtc0
There is a question here: askubuntu.com/questions/1194633/… with this error but noone has answered.
 
6:53 AM
@ParitoshSingh The main problem with such an approach is that for each step the longest document defines the performance. So if you have one document where step 1 takes the longest and one where step 3 takes the longest, processing them at once will behave as if both step 1 and step 3 are long.
 
I'm finally getting around to learn pandas, could someone confirm something real quick for me? Given this code:
>>> pd.to_datetime('2022-10-13 15:22:21.000001+00:00', format="%Y-%m-%dT%H:%M:%S.%f%z", exact=True)  #  works
>>> pd.to_datetime('2022-10-13 15:22:21.000001+00:00', format="%Y-%m-%d/%H:%M:%S.%f%z", exact=True)  #  doesn't work
I'm lead to believe that pandas has an opinion about T and whitespace being interchangeable in the format instructions, and the keyword exact serves mainly decorative purpose, right?
 
exactbool, default True
Control how format is used:
If True, require an exact format match.
If False, allow the format to match anywhere in the target string.
that's what the docs say at least... and then im not able to replicate what the docs promise me they do...
 
it would have been a good idea to look it up instead of blindly going with my assumptions
the point with T vs / in the format string remains though
 
it seems like T is literally a stand-in for whitespace
(er...or im interpreting this wrong)
en.wikipedia.org/wiki/… says "..."T" as a delimiter, and a valid time expression. For example, "2007-04-05T14:30". In ISO 8601:2004 it was permitted to omit the "T" character by mutual agreement as in "200704051430",[35] but this provision was removed in ISO 8601-1:2019. Separating date and time parts with other characters such as space is not allowed in ISO 8601, but allowed in its profile RFC 3339"
sounds like the T had a wild ride
and so did the exact keyword...github.com/pandas-dev/pandas/issues/…
 
7:13 AM
Ah, thanks. That gives at least a reference. I just checked, and python's stdlib datetime.datetime.strptime does not treat them equivalent, and then of course also doesn't mention it in the docs that pandas links: docs.python.org/3/library/…
so pandas should document it
 
8:06 AM
thanks! that solves the problem as far as I'm concerned =)
 
8:45 AM
Anyone knows - is it a bad style to modify a class attribute within instance method without explicit return that otherwise returns another value for another attribute?
Was sent to Code Review instead of StackOverflow: codereview.stackexchange.com/questions/280433/…
 
@ParitoshSingh I want to say it depend on what you're doing on the document too. See for example this SO answer you could use a range over a function that does the workload and pass that to each cores.
Also really recommend looking at other answer from the same user, since they have some really complete/good quality answers that could give you some idea (for multiprocessing):stackoverflow.com/search?q=user:9059420+[multiprocessing]
 
@aeiou Definitely bad style. Either return two values and don't modify self, or modify self and return nothing
(Sometimes it's ok to modify self and return something, but not when that something is assigned to an attribute of self)
 
9:04 AM
@Aran-Fey Thanks! I understand the mixture of styles is bad... Would you personally explicitly return or keep None return? The latter is less code but maybe not as transparent as explicit assignment
 
@aeiou The complication here is that what process returns is also assigned to an attribute but only once. So you are mixing several styles here at once.
 
If you're just going to assign the return values to attributes, then just assign them in the method
 
So avoid return altogether?
 
Yeah
 
My line of thought was - if it is explicit return, the reader knows that this instance method modifies y and z
 
9:10 AM
Depends on whether you need the return value for anything other than assigning to the attribute...
 
It is not needed anywhere, motivation by communicating the intent to reader
Which is failing anyhow since self.y is kind of implicitly modified
 
Do you only ever call this method in __init__? In that case I'd get rid of the method entirely
 
Yes. Would you also if the method was quite long and complex?
 
If it's complex enough, having such a utility method can be useful to keep __init__ easier to understand.
 
Unless it's ginormously long
 
9:12 AM
Thanks both!
One last question: would you use __ for the method?
To indicate it has no use outside instance
 
A single underscore name seems appropriate.
 
Every time I make something private, it bites me in the butt. So I only use single underscores
 
Cheers
 
 
3 hours later…
12:05 PM
How should I document a method in Python where I can't really be 100% sure about all exceptions that can be raised? For example:

def method(self, source: Path, destination: Path):
    if not source.relative_to(self.an_internal_directory):
        raise MyOwnError()

    if not destination.relative_to(self.annother_internal_directory):
        raise MyOwnError()

    destination.parent.mkdir(parents=True, exists_ok=True)
    shutil.copy2(source, destination)
Would you even document all exceptions that can be thrown by pathlib and shutil?
 
New high score - made it into the top 3 for posts on our company slack out of 478 people. Does anyone have a fire extinguisher handy? Send help
@Warcaith Personally, no. You've covered your expected cases so other failures are not based on your own API. What could you even really say about them that would be useful?
 
Soon: Josh gets pulled into a performance review for chatting on slack too much :P
 
To be fair, it would give me an excuse for not having to somehow be involved in everything!
 
@Warcaith That's not how relative_to works; it returns a Path, not a boolean
And given that the inputs for the method are two paths, I don't see any need to mention file-related exceptions
 
@Aran-Fey Sorry, was going to use this method: PurePath.is_relative_to(*other)
@Aran-Fey Oh okay, so you assume that the developers should know that exceptions like FileNotFoundError can be raised?
I don't know to what degree I should document, as some libraries are extremely bad at documenting their own exceptions.
We had a small chat about this a week ago I think, but went back on this task today
:P
 
12:13 PM
I assume that the developers won't know what exceptions can be raised, but will be able to discover them during routine testing, and revise their code accordingly
I assume this because I am that developer every day
 
@Warcaith Yeah. You call a method with two paths and it doesn't return a new Path? That method is doing file operations. Kind of a no-brainer.
 
@Kevin That seems like the way to go, mainly because Python doesn't require anyone to explicit say which exception can be thrown (unlike Java for example).
 
Although, if this is function is part of your public API, then maybe it is worth documenting it more extensively
 
I don't begrudge Java for being very rigorous about declaring/documenting possible exceptions, but it's not my style
 
Yeah, sure, but I still have the issue here that some libraries only describes their own exceptions, so I don't know what I can except coming from it.

Example:
https://docs.python.org/3/library/pathlib.html#pathlib.Path.mkdir
I mean, wouldn't mkdir raise some OSErrors also?
 
12:17 PM
If all else fails, OSError ¯\_(ツ)_/¯
 
Haha, yeah, but would be awful to document OSError everywhere
:')
Is there any error suitable for "This path is not relative to your other path", or should I make my own?
 
ValueError would work, but I like the idea of a custom exception
 
Yeah, ValueError is kind of suitable. NotARelativePathError does look ugly though
NoTaReLaTiVePaThErRoR
 
Half-serious devil's advocate: OSErrors are not caused by Python, but by the OS that Python is running on. Documenting every OSError in the Python documentation would be like a car manual documenting every possible way you can get in a wreck.
 
@Kevin Haha laughing out loud IRL atm
 
12:22 PM
"AI dun it"
 
@Warcaith Erm, it seems you missed the introduction of the module
> Many of these methods can raise an OSError if a system call fails (for example because the path doesn’t exist).
 
@MisterMiyagi Oops, missed that part. What about this one though? docs.python.org/3/library/shutil.html#shutil.copy2
I mean, what is an "Raises an auditing event"?
 
There's a link right there on the "auditing event" text...
 
I know
"Raise an auditing event and trigger any active auditing hooks. event is a string identifying the event, and args may contain optional arguments with more information about the event. The number and types of arguments for a given event are considered a public and stable API and should not be modified between releases."
 
here is the related PEP:peps.python.org/pep-0578
 
12:26 PM
Does the exception come out as non-Exception object?
 
Probably they shouldn't have used the term "raises" there, because it's not related to exceptions at all
 
What exception? It says event
 
@Kevin That's what got me confused. But yeah, as you can see, shutil.copy2 doesn't document a lot about the actual exception that it can raise.
 
@Warcaith sadly, there is a lot of "doesn't document a lot" in python docs, but I probably said this too much already
anyway, the pep I linked give some more explanation about how it work and what it does (and possible application):peps.python.org/pep-0578/#suggested-audit-hook-locations
 
@NordineLotfi Yeah, I saw that. I'll try to look into that a bit more. :)
 
12:35 PM
@Kevin probably they thought "raise" as in, "heyyy, raise hand right here, yep, frantically shake hand I'm here for whoever want to see this!"
 
I'd be happy to have truly totally complete documentation. But perhaps this is an impossible goal. Let us try to make peace with our imperfect world.
 
12:51 PM
I wouldn't bet that anyone really knows all the OSErrors that may be raised and their conditions.
At the very least, it would be a long list akin to "under Linux kernel … you get this"
 
Yeah I imagine the truly totally complete documentation would have dozens and dozens of lines describing the possible OSErrors for every OS
Including KevinOS, whose possible set of OSErrors is uncountably infinite. The documentation will have to represent it graphically as a fractal.
 
KevinOSError: 🤷‍♂️
 
1:08 PM
KevinOSError("FileNotFound'); DROP TABLE Documentation;--")
 
 
9 hours later…
10:21 PM
stackoverflow.com/questions/13340856 What's going on in this question?
 
 
2 hours later…
11:52 PM
stackoverflow.com/questions/74075490 Surely there's an appropriate duplicate for this? (perhaps it shows that I'm very tired of dealing with this user, and at risk of snapping)
 

« first day (4381 days earlier)      last day (567 days later) »