« first day (4240 days earlier)      last day (700 days later) » 

7:55 AM
@KarlKnechtel There is jinjasql but I'm not a fan of it at all. You can use {{ table_name | sqlsafe }} which isn't what it appears to be. The sqlsafe is a declaration by the programmer (as in, "I know this is safe"), not a command to escape the input. IMO it just gives people the impression that they can parameterize things that otherwise couldn't be parameterized
 
 
3 hours later…
11:23 AM
cbg folks. stumbled upon polars today (looks fun, it's a dataframe library, so a rust based pandas alternative perhaps, but it supports python!) and while tinkering around with it i notice they depreciate column access after groupby df.groupby("a")['b'].sum(). A dive through git blame led me here, but i see no explanation why. Anyone have any clue why this choice was made, or theories as to why this style of accessing is not desired?
fun fact...i stumbled upon polars starting from a youtube video of a top 10 list...of iconic piano riffs...i dont even know how i managed to go off tangent this much, but i did.
 
The deprecation print(???) mentions that it's deprecated in favour of the .agg method, so perhaps there are hints there. Edge cases that are properly handled there, important parameters that can be passed etc.
 
that could be a lead, let me take a look
no comments stand out, but they seem to do a lazy call before the agg and collect right after. I can't think why that would make a difference but this is the one part that does stand out.
 
12:15 PM
Consider asking the maintainers. The issues page is fairly active so I think your odds are good
 
12:44 PM
gulp
 
I considered asking myself, but my message would be, in its entirety, "why is column access depreciated after groupby df.groupby("a")['b'].sum()?" I think that level of terseness would lower my odds.
 
1:01 PM
Thanks Kevin, I ended up posting, this would be my first github issue ever i think
on an unrelated note, i noticed a typo. in terms of github etiquette, do repo owners appreciate typo fixes in docs or so on? And if yes, is it fork + pr, or just raise an issue and let them deal with it
(if it adds context, the typo is inconsequential and very minor. )
 
1:19 PM
I don't remember the specifics, but I have been asked to submit a PR for a minor issue once. As if I'd clone the repo and figure out how to make a PR for something they could fix in 1 minute. But anyway, it means a PR is probably appreciated.
 
1:35 PM
raising the question as an issue ended up being pleasantly painless and nice experience, got a quick response. looks like the primary idea was to keep api bloat low.
 
Even with a p.communicate(), my wrapper around Popen returns just a millisecond before the command finishes. This leads to problems where I'm stopping a service, but the command returns before it actually stopped, so the next check for the service's status still reads Started.
 
Well, how are you stopping your service, and are you sure that whatever command you're executing waits until the service has stopped?
 
If I do service stop $SERVICE; service status $SERVICE in bash, it rightly shows me that the service is stopped. However, doing Wrapper("service stop $SERVICE"); Wrapper("service start $SERVICE"); in Python, returns that the service is still up.
I check whether the service is up by checking a certain file exists, so maybe os.remove is the one that returns early?
 
The man page isn't very helpful, but as far as I can tell, service $SERVICE stop just sends a "please shut down" message to the service and returns
 
Oh no, service is my own script.
 
1:48 PM
does the service report that its stopped* if you check Wrapper("service stop $SERVICE"); after waiting a bit more?
 
Yes, if I add a time.sleep(2) it works fine.
If I add a os.listdir("/tmp") after the stop, the file still exists.
 
then Aran is pretty much spot on, no? ie. the stop command just "initiated" but no claims are made about whether it's completed or not, so you'll need to yourself correct this or account for this
a naive approach could be to just check if the service is still up once the stop command is initiated, and keep sleeping and checking.
 
But by the time stop exits, everything should be shutdown. This only happens in my Python script, and not in a shell.
 
since this is a "timing" thing, just because it doesn't happen in the shell unfortunately doesn't guarantee anything. it's entirely possible initiating the command via shell is just "fast enough" that the 2nd command for checking the status only sees the service closed. perhaps if you tried this many times, maybe even the shell might show it up once. i am not sure.
 
Well, I mean, if you wrote the service program yourself, then this is easy. Does service stop wait for the service to stop or not?
 
1:57 PM
@Aran-Fey Me too :-) I once found a small problem in the Python docs. Basically a one-character change. Took me like two hours to submit a PR.
 
Quite a bit of that was one-time authorization stuff that I won't have to do if I notice a second problem. I guess it keeps the most deranged maniacs from submitting pointless PRs
 
So essentially, subprocess.run('service stop') executes a Path('/tmp/whatever').unlink(), and if you do subprocess.run('service stop'); Path('/tmp/whatever').exists() it still returns True?
 
Yes. A little more digging shows that this goes a bit deeper: stop also runs some umount commands. However, if I add a listdir right after status, it still shows that the folders still are mounted.
 
Hi guys, if Im using pdffilemerger, and trying to loop for each excel column and only have it working for the last row, do I have my loop in the wrong spot in my function?
 
2:06 PM
@DrownedSuccess Hmm. I'm afraid I have no clue what could cause this
@Chaotic.Python Minimal, Reproducible Example please
 
for header in models_needed:
    for index, row in df.iterrows():
        merger = PdfFileMerger()
        if type(row[header]) == str:
            merger.append(row[header])
            merger.write(.test.pdf")
        merger.close()
 
Are you overwriting the same file in every iteration?
 
I'd like to append to the file for the length of the column, then reset it for each header
Excuse my noobism, I'm very new
 
Then yes, you're creating and closing the merger in the wrong loop. Not sure about the merger.write though. That should probably be called right before the close()?
 
@ParitoshSingh I was gonna suggest the same. I've been wanting to use polars for a while because the codebase is night and day with the pandas codebase, mainly because there's 50 different ways to do everything :/
I read a blog a bit ago that mentioned a huge number of cache performance changes the maintainer made, so the lazy evaluator might do some chunking for large frames?
 
2:25 PM
@Aran-Fey I moved the creation of the merger to below the for header line, and moved the merger.write in the for index loop and it works now, but only for the first header in my list =/
 
The first one?
 
My list has 3 elements which correlate to the columns or headers in my excel
It only loops the first element in that list
 
Yeah, no, that makes no sense
I don't think I can help there without an actual MRE
 
I think I figured out why: does os.kill(pid,SIGTERM) wait for pid to end?
I think what happens is that my stopping script sends a TERM signal to the running service. However, the kill may not wait for the process to actually end before continuing.
I can't use os.waitpid because the pid is not a child of the stopping script.
 
@DrownedSuccess Nope
 
2:39 PM
That's probably why then. I'll just use the wait command to wait before continuing.
 
Ok, I did some research to be safe and I hereby reduce the confidence in my "nope" to 90%
 
@DrownedSuccess Nope
 
Hmm, running wait doesn't seem to work either.
 
ask yourself what did it actually wait on
 
It waited on the service PID (the main one, that coordinates everything else)
Oh, it seems that wait only works for direct children.
I guess I'll just have to loop and check every 0.25 seconds or so.
 
3:14 PM
Yes, got it to work.
 
@roganjosh Yeah, I'm getting tempted to give this a go in our projects too. Sadly i dont think our current one will accept such a major change with our existing code base
 
Thanks to everyone that helped.
 
nicely done.
 
3:35 PM
@ParitoshSingh My only real concern is API coverage, although they are developing it pretty quickly. If we hit something where it's "well, this is super-simple to do in pandas" and it can't (yet) be done in polars, I think that could cause issues
 
aye, it's a valid concern
im currently playing with some of the codes* we have from memory to see if i snag onto anything.
 
Ah cool. If you do happen to stumble over any, I'd be really interested to hear about it
 
Well, for starters it's behaviours around Nulls are different. One issue with new libraries is that googling isnt as effective, so my hunts are not very fruitful so far.
import polars as pl
import numpy as np

data = {"a": [1, 2, 1, 2, 1, 2,], "b": [3, 4, np.nan, 5, 6, np.nan]}
df = pl.DataFrame(data)
this is what im going to try resolving, essentially how to initialize a df with nans in a way that polars doesn't hate me.
 
4:00 PM
Ok... that's weird
 
it's also got two kinds of nulls, something else that may bite you if you're not aware (but may be very handy once you get used to it, or so i hope.) Currently in the being bit by it side of things (ive managed to get a nan and a null. im not sure which is which.).
 
Are you sure np.nan is valid in there at all?
docs should say something about nullable things
 
It looks as though you can pass numpy arrays to the df constructor but it's weird that it tries to interpret it as int type
 
cbg
yeah, i'd have no qualms if it coerced for me, or gave me a flag to coerce to float to make the null work. still going through the guide though, haven't found anything concrete yet. (well, other than the fact that the data is by default backed by Arrow. for those that know arrow, this probably clears a lot of things up. not so much for me yet.)
i should also mention that the other big change is no indexes.
so while your pandas experience will help initially, it may also hinder you. the iloc loc stuff is all gone, and the expression api is favoured. (initial impressions, this is a good thing, not a bad one)
 
4:17 PM
oo... solars looks interesting
will stick with dask... but that's going on the "look at it later" list
 
My initial impression is quite positive so far. I just need to find more concrete info on the Null stuff.
A blog post did compare this to dask, and the key difference is that this is designed "ground up" for utilizing all cores on your machine, as opposed to dask's "take the single threaded libraries and parallelize that work", so i imagine it will feel great to use if you're coming over from dask.
 
 
3 hours later…
7:11 PM
cbg,
I was looking at this question: https://stackoverflow.com/questions/72396419/python-recursively-adding-to-a-class-variable-gives-different-results ... and I just don't get it. In short, using `self.count += self.perimeter_calculator(x + 1, y)` instead of ` a = self.perimeter_calculator(x + 1, y) ; self.count += a` (`self.count` is just an int) gives different results. There must be something evident I'm missing, but I just can't see what. Any idea?
 
I don't think I can spot a difference based on that message
Can you reproduce the issue in the question?
Oh, and everything is supposed to just be an int?
 
If self.perimeter_calculator can also modify self.count, perhaps that does something
 
good point it does
I wouldn't be surprised if the individual prints were different but the final value of count were the same
the recursive calls also all mutate the same self.seen set that is the basis of a branch in the recursive function (and the increment only happens on one branch)
 
class Fred:
    def __init__(self):
        self.value = 0
    def f(self):
        self.value += self.h()
        print(self.value)
    def g(self):
        x = self.h()
        self.value += x
        print(self.value)
    def h(self):
        self.value += 1
        return 2

Fred().f() #2
Fred().g() #3
A slightly simpler example
For the record, f still shows 2 if you change it to self.value = self.value + self.h()
 
Ah, += combined with recursion, I get it
 
7:25 PM
OK, so with f, we add to the value self.value had before calling h, and in g, we use the value it had after calling h. Got it, thanks!
 
Yep
 
the mistake was recursion
 
Ideally one does not encounter problems like this very often. You can usually design the algorithm so that it keeps track of state using an attribute of self, and doesn't bother returning anything; or vice versa.
 
I noticed that messy design, but it never would've occurred to me that it's the source of the bug
 
An illustrative example. pastebin.com/raw/kJsV3uq1
 
7:43 PM
How do you folks test code that checks length of its input against the allowed maximum? Like if you have a function validate_input_string_length(string) and you need to test its return value for the two input strings of lengths MAX_STRING_LENGTH and MAX_STRING_LENGTH + 1, it makes sense to just generate test strings of the required lengths if MAX_STRING_LENGTH is small, but what if MAX_STRING_LENGTH is 10000 or longer?
 
Can you just throw hypothesis at it?
 
Nope, we don't do that here
 
it was worth a shot
What's wrong with 10k-long strings?
 
It just feels wasteful, since we don't care about their contents (only their lengths), and they're just going to be discarded after N milliseconds
 
>>> sys.getsizeof(' ' * 10_000)
10049
10 kbytes? Is that correct?
 
7:49 PM
Yeah, but still :(
 
I would assume that correctness and simplicity are more important than performance in CI.
the trickier your test is, the more likely it is that you have a bug in the test itself, or miss some edge case
 
Alright, you convinced me :D I was mocking the len function for the module containing validate_input_string_length, when I stopped to think "what the hell am I doing"
Yeah
 
You could use a fixture with sys.intern(' ' * 10_000) inside :P
 
If your function is literally just def validate_input_string_length(s): return len(s) <= MAX_STRING_LENGTH, maybe you don't need to test it.
 
@Kevin unless there's a hard constraint on code coverage
 
7:54 PM
# pragma: no cover
 
I've written at least two bugs inside no cover pragmas...
 
oof
 
I don't believe in hard constraints, it's incompatible with my belief of a just world
 
usually "forgot the f in the f-string of this obscure error message"
 
Good evenings everyone!
 
7:56 PM
@PéterSzilvási hey :)
 
Yo
 
8:49 PM
https://stackoverflow.com/questions/72397699

Is there seriously not a good duplicate for "don't repeatedly open the file"? I tried to find one but all I got was stuff about `'a'` mode, which is not the right way to solve the problem.
it seems like kind of the counterpart to sopython.com/canon/100/…, but none of those is a proper duplicate closure either.
 
well, because that's a different problem
on a scale of -10, how bad is it to close as a dupe of stackoverflow.com/questions/15369884/…? :P
 
@AndrasDeak--СлаваУкраїні About -20. There should be a language-agnostic canonical for this, like there is for "is floating-point math broken?".
I'm sure I answered a question that repeatedly opens the same file in a loop. But it was years ago, and it might be hard to find. And it's probably Python 2.
I agree that that "Python but meh" example is too meh. And probably a bit confusing for the average newbie.
 
 
1 hour later…
10:21 PM
+1 for an agnostic canonical. Is anyone making a list of missing canonicals?
>https://stackoverflow.com/questions/25115140/python-only-last-line-is-saved-to-file is Python but meh.

This one has the same issue as the ones I was finding - it doesn't address the loop issue.
I feel like it would be within scope for such a canonical to give a basic 'anatomy of a file object' - explain the concept of a stream, talk about how the file doesn't have an "insert mode" but only "overwrite mode", that sort of thing.
but it's hard to say where to cut it off, then.
 
@KarlKnechtel Sounds good to me. It may be worthwhile to create such a thing. But ask on SO meta if it already exists, and get feedback from users of various languages.
On a related note, we had a discussion in this room, not long ago, about how many younger computer users have a very hazy understanding about file systems. That's actually not a new problem, but it's a bit alarming when even kids who are starting to learn how to code are clueless about the file system(s) & directory structure.
 
Indeed. - Isn't there a SE site for programming educators? I think I recall posting on one once but I'm not sure
 
I remember (years ago) asking a guy "where did you save that picture?" He replied, "I saved it in MS Paint, of course". And I couldn't get him to understand how or why that that was an inadequate answer to my question...
 
10:39 PM
They probably have some great & relevant info. But we want a canonical Q&A on SO, so it can be used as a dupe target. Of course, it can be handy to link to other sites on the network too. I often link between Astronomy, Physics, and Space Exploration.
 
@PM2Ring I guess if I realized I was dealing with that level of... well, that level, the next attempt is "if you wanted to open the picture again, what steps would you take?"
but that probably gets an answer based on some kind of recently-opened file cache. :/
 
11:10 PM
Hot off the press. Cara Vel Wicked Game by Chris Isaacs.
 
11:22 PM
stackoverflow.com/questions/55900256/… this old question needs some cleanup of some kind and I'm not really sure what to do with it.
 

« first day (4240 days earlier)      last day (700 days later) »