« first day (3917 days earlier)      last day (1017 days later) » 

2:39 AM
cbg
 
 
3 hours later…
6:03 AM
cbg
 
Hey people!
Can you please let me know how to install pandas module such that I can use it in IDLE?
 
6:51 AM
you just install it like normal, a sure way is to use python3.9 -m pip install pandas and open IDLE for python 3.9, usually avoids problems that will occur if you have more than one python version and pip will belong to something that is not what you want
 
people actually use IDLE? :D
 
7:16 AM
'python3.9' is not recognized as an internal or external command,
operable program or batch file.
@python_user
@ThiefMaster yup..
 
I mean use whatever python version you have, if it is windows I guess python or py should work
 
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
 
Why can't PyCharm autocomplete an Abstract Generic class?
It always resolves to 'T' instead of the type I've set the subclass to
 
py worked :)
Collecting pandas
  Downloading pandas-1.3.0-cp39-cp39-win_amd64.whl (10.2 MB)
     |████████████████████████████████| 10.2 MB 1.7 MB/s
Collecting pytz>=2017.3
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 3.3 MB/s
Collecting numpy>=1.17.3
  Downloading numpy-1.21.0-cp39-cp39-win_amd64.whl (14.0 MB)
     |████████████████████████████████| 14.0 MB 3.3 MB/s
Collecting python-dateutil>=2.7.3
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
 
class ProductBuilder(base.ProductBuilder[pd.Series]):
    def _build_key(self, data: T) -> str:
        pass
Here's my problem: PyCharm autocompletes to type 'T' instead of 'pd.Series'
 
7:24 AM
@RandomPerson Can somebody please explain the warnings?
 
@RandomPerson that most likely wont be there if you install from an admin cmd, if all went well you can use f2py you could use that without having to change env / path variables in your system
 
@python_user OK..
 
you might want to refer onine ;) I have not used windows in a while
 
Maybe the issue is because of this:
I didn't select "Add Python 3.9 to PATH" while Installing python in my PC..
 
8:25 AM
Guys when wanting to automate the running of many separate scripts would it be a good idea to use os.system to call the scripts individually from a single script or what would be the best method?
 
are they python scripts? if so, just import them
 
yeah they are and tbh that could be better option... just been asked to find a way to automate the entire process so that it could automatically happen weekly without us touching the files so thinking how to do it
 
also, just as a general fyi, use subprocess* over os.system whenever possible, subprocess is recommended over the latter
 
did not know that thanks paritosh
 
in this case, if you're just coordinating python scripts, an import will just run the code. brownie points if you re-write the scripts as functions (if they arent already) and invoke them using function calls
 
8:30 AM
@Kwsswart always prefer subprocess over os.system. Otherwise, it's a good idea.
 
curious why is it prefered?
 
from docs: The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
basically it's a more powerful interface, with more control.
 
Excellent will read through it thanks
 
iirc, though i dont seem to find mention of it here, there was also some benefit in the context of shell injections
either it was the fact that subprocess doesnt invoke the shell by default, whereas os.system does, or something similar.
 
8:45 AM
Yes, os.system invokes a shell. That also includes all the ugly of escaping and such.
The usability of subprocess is a lot better.
 
Seems interesting I am playing with it at the moment
actually quite like this
 
9:01 AM
also, invoking the utilities without it happening through a shell is, in theory, faster/cleaner. I've never benchmarked it though.
 
9:47 AM
Essentially everything is better
 
 
1 hour later…
10:53 AM
is it considered bad practice to have a default value in an argument in argparser? I have a repository with multiple modules -- and sometimes instead of passing all values in bash for the python script. I like to keep default relative paths (paths within the repository -- infact just a couple of well defined folders). A co-worker has taken that and completely uprooted it. Now, in my pipeline, there are 140 lines of creating paths -- which are all subsequently passed to various tasks.
 
Default values for command line arguments are perfectly fine. But how on earth do you end up needing 140 lines of code just to define some paths?
 
I made around 20 packages. I made around 10-12 tasks of those packages in airflow. I went to another project. Couple of guys worked on the pipeline I had made. They made 40 tasks out of the 10-12 tasks, then decided on the path thing. Every little thing was passed as a path. to 40 different tasks. Then one of them left the company
Good days
 
I don't know anything about airflow, but generally speaking... passing around arguments isn't a bad thing
 
neither is default values right? I can always log all default value args to the console with a warning message.
 
11:09 AM
Ok, I'm starting to be really lost here. Why would you log default values as warnings?
 
I don't want to, but two guys in my team consider it bad practice to have default values, they consider it as 'hidden' values. So I'm considering a logging message:

running "make_data" with args:
{dictionary of args}
 
It is not bad practice to have default values. There are tons of programs out there which keep the modern world running and have default values.
 
Oh, I see. It's probably a good idea to log that anyway
I can think of a few reasons to avoid default arguments, but they're very niche
 
Reifying default values when calling a program can be harmful if you don't actually care for their meaning, since it prevents the program from changing recommended settings.
 
@Aran-Fey can you share? may be i have that case too.
 
11:18 AM
devil's advocate. if these are paths that should be visible/configurable, then keeping them as defaults can cause problems, and there's some validity in saying they seem "hidden"
so i believe, purely speculating, the optimal point lies somewhere in the middle between your stance and your coworker stances.
 
It essentially boils down to this:
Advantage of defaults: Convenience. You don't need to pass as many arguments.
Disadvantage of defaults: Makes it harder to debug if you accidentally forget to pass some arguments.
This disadvantage only really matters if you suck at debugging, so...
 
@ParitoshSingh that is understood, nothing that is meant to be configurable is hidden from the dev. It is well described in readme. It is displayed before the package runs. it is tested for existence. These are intermediate files and folders that won't be an output of the pipeline.
 
The most realistic scenario where the disadvantages outweigh the advantage is if the defaults are pointless - i.e. if your function is always called with all the arguments, why do you even have defaults?
 
I think the real question is whether they should be your defaults or the program defaults. Just because you always need them doesn't mean they should be defaults.
It's perfectly fine to have a default of /dev/stdout, but /home/user27813124/work/projectXYZ/tmp/foologs is stretching it a tad.
 
That's not fine at all; it's platform-specific. Do you want your code to be portable, or do you want it to break when you try to run it somewhere else?
 
11:30 AM
Currently, there is 140 lines of real-estate in initializing paths, based on 1-2 lines of config. It doesn't make sense.

The way I was doing it is similar to r`/results` . It was checking for existence, creating if didn't exist etc.
there was no local, or OS specific path
 
Most platforms provide standard directories for different types of data to be stored in. I haven't been following this discussion, so I don't know what kind of data you're storing.
 
@CodyGray All platform specific tools disagree. :P
 
Wrong things are wrong.
 
they might have vested interests :P
 
Even platform-specific tools can't make assumptions. What if I renamed stuff on my system or moved it around?
 
11:34 AM
I'm just going to remove myself from this discussion now.
 
I accept your concession. :-)
 
I hope it is clear that this wasn't meant as such. Rather that I am feeling extremely uncomfortable to continue.
 
Oh... I didn't mean to make you uncomfortable.
 
12:33 PM
Not causing trouble I hope @Cody :p
 
Apparently I am!
 
Ban him!
 
Ban him? Ban him! <insert two spider men pointing at each other>
 
12:53 PM
spider men, thats fancy :P
 
1:29 PM
i started trying to figure out logging today, but could use some help. my "info.log" file does not seem to get info and debug logs. code
i guess i must have messed up with the levels somewhere, but i can't figure out where. any idea how do i get this to log correctly?
 
1:43 PM
As far as I can tell you must lower the root logger's loglevel, but I have no idea why
logging.getLogger().setLevel(logging.NOTSET)
 
oh ok, ty. that's...uh disappointing i suppose
okay, worked like a charm, thanks Aran!
 
2:36 PM
@Kwsswart Thank you..
 
3:32 PM
re logging, found this in docs
> When a logger is created, the level is set to NOTSET (which causes all messages to be processed when the logger is the root logger, or delegation to the parent when the logger is a non-root logger). Note that the root logger is created with level WARNING.

The term ‘delegation to the parent’ means that if a logger has a level of NOTSET, its chain of ancestor loggers is traversed until either an ancestor with a level other than NOTSET is found, or the root is reached.
so apparently NOTSET invokes delegation behaviour.
 
 
1 hour later…
4:47 PM
@roganjosh ummm: They have been notified but have not yet accepted or declined the gift.
 
Sorry @JonClements the yam has hit the fan at work! I wasn't ignoring you! I'll get back to you shortly pupster and I will claim the gift :)
Many thanks for sending the gift too!
 
no worries mate... thought you were just cowarding out and admitting defeat for your inevitable
annihilation upfront :P */me coughs coughs coughs*
 
Hahaha, me?
I think our DB cluster has hit some kind of concurrent writer limit, so it's now firefighting all of the projects until it can be investigated properly and fixed
 
@roganjosh oh come on - DB stuff is a piece of cake... it's not like stuff happens and you wake up screaming in the middle of the night about it... sheesh... :p
 
Hello all. I'm pretty sure the answer is "there's no problem" but just to confirm it since I'm debugging. Say I have a binary file, and I'm using ProcessPoolExecutor to run parallel tasks. Each task when launched will read the content of this file. Can there be a problem when reading (and only reading) a file with multiple parallel calls?
 
4:55 PM
@ihavenoidea as long as no process tries to get an exclusive read lock on the file - no, I can't see why.
 
didn't you mean to say, i have no idea why it would be a problem...
 
Seems a bit of a waste though if you've got multiple process that all read from the same file - seems a bit odd?
 
You're right! The read from file code is in some class constructor which I instantiate in each parallel task. It definitely can be refactored, it's just not the priority atm. Thanks for the answer!
 
cbg
Anyone know about Django window functions? docs.djangoproject.com/en/3.2/ref/models/expressions/…
 
@ihavenoidea since each task will start reading the file from its beginning anyway... just seems odd you'd want that in parallel...
(best guess is you'd want the master process to read and put stuff into a shared queue etc for a task to pick up)
@Code-Apprentice do you have experience with window functions in pure sql?
 
5:11 PM
I didn't say before because it was irrelevant to my question but each parallel call can read different files, it just that sometimes it will happen that some calls will read the same file
 
@JonClements nope...I just learned that these are a thing a week ago
and they solve part of my problem...but now I have two problems
 
@Code-Apprentice my advice would be - understand how it works in sql by itself before trying to ORM it
 
@JonClements thanks...I'll have to read more on the underlying SQL stuff
For what I need, I have the ORM version figured out for a calculation. The next step I need is a way to filter records based on the result of the calculation from Window(Lag()).
 
@ihavenoidea no worries - I took a binary file as meaning a single file...
 
5:57 PM
Encountered a wacky numpy internals issue: stackoverflow.com/q/68282430/314290
basically some narrowing conversions raise errors and some don't
wonder if I should file a bug
 
6:41 PM
I passed 100k a few days ago. Just wrote up some stats and thoughts, I could write a whole blog about it probably: twitter.com/davidism/status/1412843907952435200
8
 
Congo on that
 
bread and peanut butter
 
 
2 hours later…
8:55 PM
Please give hint what `or [1]` means. I don't know what to search in for in Google / Stack Overflow.

This is legacy code, Python 2.

`years` is list of strings, most are like '1998', '2004', ....
```
max_year = max(map(int, filter(lambda x: x.isdigit(), years)) or [1])
```
In other words, does or means something special in max() function ?
 
x or y is a poor man's x if x else y
 
oh, and I was wondering why it isn't returning bool :P
@Aran-Fey thank you, seems in Python or doesn't return bool
 
nope, it returns one of the operands
 

« first day (3917 days earlier)      last day (1017 days later) »