« first day (4525 days earlier)      last day (438 days later) » 

1:05 AM
Once upon a time I wrote a markov chain text generator where the inputs were fed from all the messages that came into the room, and the output is produced via the 2nd order markov chain over all the inputs, with the probability of previous/next words is based on the occurrence across the full set of messages. The whole process taught me that any attribution of intelligence produced by those bots are just purely coincidental, because sometimes the output is uncanny intelligent and in context.
 
1:42 AM
stackoverflow.com/questions/75657243 I think the overall intent of the project can be understood here, but it needs major editing if there isn't already a canonical...
 
 
1 hour later…
3:04 AM
Here's an alex turtle question I answered in 2014: stackoverflow.com/q/26228483/4014959
@NordineLotfi Yes, but it's not on my phone, and it's a bit messy. IIRC, it uses Tkinter for the output, but it doesn't have a GUI, or even a proper command line interface, you need to edit the code itself to choose a hash function & set various parameters. I'll make a simplified version sometime in the next few weeks & let you know.
 
 
3 hours later…
5:51 AM
@PM2Ring :D Thank you
 
 
5 hours later…
10:52 AM
Noob pandas question: Is there a way to filter rows and columns in a single operation?
>>> df
   foo bar  qux
0    0   a  NaN
1    1   b  NaN
2    2   c  NaN
3    3   d  NaN
4    4   e  NaN
>>> result_i_want  # df[df['foo']<3][['foo', 'bar']]
   foo bar
0    0   a
1    1   b
2    2   c
 
Not that I'm aware of; that seems to me to be the most idiomatic approach
Unless the predicate for the column filtering is "All rows are NaN"?
 
Sadly no
I was worried about performance, but I guess I can just compute the boolean mask first, then select the columns I need, and then apply the mask
Since I have a df with ~150 columns and I only need 2
 
Your reasoning there being that df[df['foo']<3] will have to prune data out of the other 148 columns? I guess that makes sense if you consider a df being a dictionary of Series objects - it's not something I've thought about before. I might have a go at timing that out later
 
11:08 AM
I admittedly do have a track record of prematurely optimizing unnecessary things
 
In this case I think it's pretty valid since you could trivially swap the order of those two operations if there was a noticeable difference. I'm curious myself though I have a feeling that pandas can treat row filters more intelligently than my previous analogy of a dict of Series objects
Nice. It's smarter than my analogy but there is still overhead
my_dict = {x: np.random.randint(0, 500, 10000) for x in range(200)}
df = pd.DataFrame(my_dict)


def row_first(df):
    df = df[df[1] < 250][[1, 2]]
    return df


def col_first(df):
    df = df[[1, 2]]
    df = df[df[1] < 250]
    return df
%timeit row_first(df)
6.22 ms ± 1.29 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit col_first(df)
2.36 ms ± 392 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
 
11:29 AM
Interesting. Wonder how that works under the hood
 
11:41 AM
I'm struggling to track that bit down. The Frame object is absolutely massive and at some point I'm guessing it goes into Cython but I haven't yet found the actual link to what it calls
 
Is there anything in the standard library that allows me to iterate destructively (as in while items: item = items.pop()) without turning the thing into a list?
 
What is it if not a list?
 
Hm, the answer is "a generator", for example, but that's actually a good point.
Having a list is the special case, not all the other things it could be.
 
 
3 hours later…
3:02 PM
@KarlKnechtel "the question is about whether transsubstantiation is legal". That's given me a proper laugh :D
I'm impressed that the Bible apparently wrote in units of litres. Way ahead of its time.
 
 
2 hours later…
4:37 PM
the NIV translation does some unit conversions, apparently. (I wonder if/how that impacts on the verses that imply pi = 3)
@MisterMiyagi how many non-list things will be mutable in the first place?
if you only had an iterator in the first place, then you'd need to access the underlying container somehow in order to do anything "destructively", and the iterator protocol doesn't offer that access.
and of course if it's a tuple then you certainly won't be able to iterate destructively. I suppose the common case is a set, really
 
@KarlKnechtel I admit to drastically overthinking this until Aran made the right comment. Basically I have *args or an iterable. *args I can deal with manually via list and the rest I have no chance of improving anything as you say.
 
*quacks happily*
 
Socratic triumph
I wonder about your use case that is such that just iterating over an iterator is insufficient
 
I could imagine a use case where deleting lines/chunks of a file as you go through it is necessary, but I don't think this is what Miyagi is doing
 
It's for some library code for which I'm decently sure that the contents of the *args/iterable usually hold state that should be gotten rid of ASAP. So holding on to the items I already processed is a waste if I can help it.
 
4:52 PM
it sounds like a weird place to be responsible for cleanup
So is the solution that when you have a list then you just pop, otherwise just iterate (or just convert to a list and pop)?
 
If I get *args then convert them to list and pop-iterate. If I get an iterable then for-iterate and hope the user did it right.
 
But then you're not getting rid of any state, right?
unless the user has a lazy generator, which is probably what "user did it right" means
@AndrasDeak--СлаваУкраїні I mean you'd still have the reference in args for any unnamed expression passed to the function
unless you meant args = list(args); that would probably work
 
Yeah, that's what I do immediately after getting the *args.
 
5:07 PM
Popping from the left side of a list is inefficient in its own way, but I guess you're dealing with a small number of very large objects
 
Well, technically I do args = list(args)[::-1]. :D
 
Tsk-tsk.
 
puts on the hat of shame
 
Are you sure that doing the above will let go of any references?
or did I fudge something up here?
>>> from weakref import ref
...
... class Foo:
...     pass
...
... def foo(*args):
...     r = ref(args[-1])
...     args = list(args)
...     print(r())
...     args.pop()
...     assert not args
...     print(r())
...
... foo(Foo())
<__main__.Foo object at 0x7ff3d566ca60>
<__main__.Foo object at 0x7ff3d566ca60>
unless it's the args tuple that stuck around without being destroyed, but then that might apply to your real use case as well
 
5:13 PM
Interesting. I don't think you did anything wrong there
 
let me try outside ipython
nope, same
 
maybe GC comes in only when there are too many references or on other context (eg: instead of systematically)
 
Interesting indeed.
 
@NordineLotfi this is not GC, this should be reference counting which is usually super prompt
 
ah, got you
 
5:15 PM
FWIW, I do throw away the initial function frame (it's __init__).
 
I don't understand that sentence
 
<__main__.Foo object at 0x0000022EA7690A90>
None
^ my output
 
Huh. Python version and type?
I'm on 3.9 CPython on linux
 
@AndrasDeak--СлаваУкраїні Basically I do this:
class Bar:
    def __init__(self, *args):
        self.r = ref(args[-1])
        args = list(args)
        print(self.r())
        args.pop()
        print(self.r())

    def bar(self):
        print(self.r())

b = Bar(Foo())
b.bar()
# <__main__.Foo object at 0x10fde3e80>
# <__main__.Foo object at 0x10fde3e80>
# None
 
Python 3.11.1 on Win10
 
5:18 PM
Python 3.10.8 on Intel MacOS
 
@MisterMiyagi but if you wait for the function to end, do you win anything compared to just keeping *args?
 
Can repro Andras' output with 3.9 and 3.10
 
nice, royal flush
or whatever the other one is :D
 
@AndrasDeak--СлаваУкраїні Well, I do use the "list args" in some methods.
 
SET? Something like that.
 
5:19 PM
Can reproduce too on 3.8. I guess something is done differently in 3.10.8 and 3.11
 
But whatever causes Python to keep onto the original *args should be gone.
 
@NordineLotfi we can't rule that out
 
I'm guessing CPython might hold onto the tuple it constructed for *args until the function is done.
 
 
3 hours later…
8:31 PM
@KarlKnechtel Hey hey, you can totally write a list comprehension [... while condition] as [... for _ in iter(lambda: bool(condition), False)]...
 
8:50 PM
Huh, that's a pretty neat trick.
 
9:04 PM
I thought about doing something similar once, but I don't think I made decent progress and went to do something else. The idea was to make a for loop with a range that could be modified (without using a while loop). By making the range for the end grow, and shrink at the start, you could make a "while loop" using only a for loop
it's definitely less elegant than the lambda trick though...
 
9:29 PM
The best way for a while loop is a while loop
 
9:40 PM
duplicate stackoverflow.com/questions/49136597 ; the canonical is stackoverflow.com/questions/8347048/… (since OP already knew how to handle the nested list structure)
@KellyBundy True. If you want to maintain state like in the collatz example in the duplicate, you'll also need the walrus operator I guess, but I suppose it could theoretically be written. If you want to make the appropriate edits for this information, be my guest. It already seems too long, though.
Proposals for other refactorings of the canonical answer are also welcome, of course.
 
@AndrasDeak--СлаваУкраїні I agree. Just wanted to see if there was something useful you could do with it aside from using it in a list comprehension.
 
9:56 PM
I'd also be interested in having, e.g., Pandas-specific approaches added as separate answers
 
@KarlKnechtel done
 

« first day (4525 days earlier)      last day (438 days later) »