« first day (3312 days earlier)      last day (1639 days later) » 
01:00 - 21:0021:00 - 00:00

1:11 AM
Does plt.imshow require plt.clear if being used repeatedly?
Or can does it just overwrite the previous image without causing any sort of memory leak?
 
 
4 hours later…
5:38 AM
cabbage
I know this isn't necessarily python related but I'm running a linux vm via vagrant from windows
Getting a few errors with pip and whatnot and I think it might be an error with file endings
I have mixed endings some are CRLF and others LF and I'm just wondering how to enforce windows endings locally and somehow making them work in the synced vagrant folder (and also enforce them with git ... the project has already been pushed btw)
 
 
2 hours later…
7:54 AM
hmm
I wonder if there is ready code for "extracting json objects from html"
I guess it would be sufficient if there was a json parser that wouldn't check contents after the last character of the first object
ah raw decode
 
8:50 AM
cbg guys o/
Since Pythion doesn't have "switch" like Java. What is alternative?
 
if
 
That's recommended way?
Thought there would be better way.
 
@TheLittleNaruto Pretty much that's the only simplest way.
And, oh yeah! Cabbage!
Woah. I haven't been here for like a year.
 
@SeanFrancisN.Ballais Simplest != Recommended/Correct :)
@SeanFrancisN.Ballais cbg
 
put it this way, we consider switch statements to be silly :P You do have the option of using a dictionary as a switch statement too if you really want to.
 
9:01 AM
like that dispatch thingy
 
@TheLittleNaruto Guido didn't give Python switch because it can often reduce readability.
 
@TheLittleNaruto exactly!
 
@PM2Ring uhm? How can switch reduce readability? On the contrary I think it'll make code look more readable.
 
Yes, you can do dispatch using a dict. But it will often be less readable than equivalent if..elif..else code. So please don't just use a dispatch dict because it looks cool.
 
Yeah right! I have if for now
 
9:06 AM
@TheLittleNaruto Ever seen a switch in C code that spans hundreds of lines? And then there's the issue of drop-through cases (cases that don't end in a break), although switch in some languages doesn't permit that.
 
The local scoping behaviour of inner functions you explained also occurs when an outer variable is used in a conditional statement in an inner function, not just during variable assignments. — Sean Francis N. Ballais 11 mins ago
@SeanFrancisN.Ballais Not sure what you mean there. Could you give me an example?
 
```
@Aran-Fey

Say I have:

`def outer(some_var):
` def inner():
` if some_var is None:
` ...
 
Speaking of switch, in her recent Stackoverflow blog post, Ellen Spertus describes her mistakes in optimising the code generation for switch in the Microsoft C compiler.
 
In that case some_var will be a local variable of outer, but not of inner
 
Running that code would raise an UnboundLocalError.
 
9:18 AM
no
 
Well, I did something like that and it did raise an UnboundLocalError.
 
def outer(some_var):
    def inner():
        if some_var is None:
            pass
        print('success')

    inner()

outer(3)  # success
 
@PM2Ring Thanks
 
There's good info about Python's scoping rules here: stackoverflow.com/q/291978/4014959
 
@Aran-Fey, I stand corrected. The UnboundLocalError must have occurred since I call the outer function in a lambda. Said lambda then becomes a slot function for a Qt widget signal.
 
9:26 AM
where you call outer really shouldn't matter. That has no effect on the scope of the variables in outer and inner
there must be something that makes the variable local to inner. Could be any kind of assignment (x = ..., for x in ..., with ... as x:) or a type annotation (x: ...) or a del x
 
@SeanFrancisN.Ballais Ok. I think we'd need to see a MCVE. My guess is that your problem was due to late binding, which is kind of related to scoping.
 
Ops. That was not intended.
 
@PM2Ring, here you go.

@classmethod
def _create_dialog(cls, model, model_instance, attrs=[], title=None):
...
if model_instance is not None:
# We're an "Edit Record" dialog.
model = type(model_instance)

for attr in attrs:
...

save_btn = QPushButton('Save')
@pyqtSlot()
def _save_action():
if model_instance is None: # UnboundLocalError raises here.
...
save_btn.clicked.connect(_save_action) # Okay, so the outer function was not actually called in a lambda.
I managed to fix the problem by using nonlocal model_instance inside _save_action().
 
9:44 AM
@SeanFrancisN.Ballais Ok. BTW, that code us hard to read without proper indentation. And it's not a MCVE because it's not runnable.
 
yeah, that's not a MCVE. Like I said, there has to be something inside _save_action that makes model_instance local. Posting only 1 line of that function won't help anyone figure out the problem
 
I think I see the issue. The complete version of the conditional in `_save_action()` is:

if model_instance is None:
model_instance = model()
I think model_instance = model() is what causes model_instance to be local.
 
yep
 
I didn't expect that really.
Thanks, @Aran-Fey
 
Ok, but the important thing there is that an assignment is being made to model_instance, the conditional is irrelevant.
 
9:49 AM
So the scoping rules in my case is pretty much, "Hey! If you have an assignment, even though you might not run that statement, it's local."?
 
yes, it doesn't matter if the assignment is executed or not
 
An inner scope can easily look up outer names, no problem. But if it's doing an assignment (or the other stuff Aran-Fey mentioned in chat.stackoverflow.com/transcript/message/47802945#47802945) then it's working on a name that's bound in the local scope (unless the name is explicitly marked as global or nonlocal).
 
Woah. Quite an interesting facet of Python. Thanks, guys.
 
@SeanFrancisN.Ballais Correct. Even if the assignment is in unreachable code, that name gets marked as local. So if False: x = 42 tells the interpreter that x in that scope is local. You could even do return 0; x = 42 and x will still be local.
 
I am guessing this is because Python code gets "compiled"/evaluated (not sure with the correct term) first before being executed.
 
10:12 AM
@AndrasDeak Ah, thanks for sharing that. Your ping came through as I happened to be watching the storage container section of Silence of the Lambs, so it added an interesting jump to the scene (since they're always disproportionately loud to whatever I'm watching/listening to) :P
 
10:30 AM
@roganjosh Hello, Clarice
 
fly, fly, fly. It happened to be on the front page of Netflix and it's been ages since I originally watched it. I tend not to like re-watching films so much but glad I did in this case
 
@user76284 by default it creates a new figure, does it not? If you pass it an Axes I'd expect replotting to be safe.
@SeanFrancisN.Ballais for some reason nobody told you but chat.stackoverflow.com/transcript/6?m=47683672#47683672
 
11:00 AM
@AndrasDeak Thanks, Andras! I should have used the sandbox before I sent formatted messages. Either I didn't know there was a sandbox, or I forgot about it already. I haven't been here for a looong time.
 
It's not your fault, chat formatting is a mess
 
11:34 AM
Andras :)
 
cbg
 
I saw "Enter The Dragon" popup the other day... haven't seen that one in ages...
 
Andraaaaaaaaaaaaaaaaaaaaaaaaaaaas @AndrasDeak
 
An interesting numpy question that my brain keeps telling me to use groupby on, but I equally feel it's possibly not great form to start out with a pandas approach. Hmm
 
11:38 AM
@Saha please calm down
 
Hi ^_^
 
hello
 
every time I am doing a project I remember you :D :)) because of your gande damaghness
 
@roganjosh Yeah... pd.Series(data).groupby(index).transform(lambda v: v - v.median()) is remarkably tempting though :p
 
@JonClements sure is! But even just looking it at, I'm not convinced it beats the vectorize approach of the existing solution when doing transform. I feel I have some Divakar magic along these lines baked into some old code but I don't have a copy at home
 
11:49 AM
@roganjosh well, the building of media_dict seems overly expensive...
 
Perhaps we could ditch the transform, initialise a median column to 0, then populate that column with the result of a groupby... then divide the two columns through in a vectorized way, thus dodging lambda. I might start playing with that
data =  [1.00, 1.05, 1.30, 1.20, 1.06, 1.54, 1.33, 1.87, 1.67]
index = [0,    0,    1,    1,    1,    1,    2,    3,    3]

df = pd.DataFrame({'data': data, 'label': index})
df['median'] = df.groupby('label')['data'].transform('median')
df['result'] = df['data'] - df['median']
subtract* through, not divide sorry. Time to put some timings together
 
@AjayMishra Kevin gave you some good hints. But also see the permutation algorithm of Narayana Pandita from the 14th century, which can be found in Wikipedia.
 
@roganjosh umm... isn't that doing the same work as what I posted above but with the overhead of a dataframe on top?
 
quite possibly. I'm pinning my hope transform doing better than lambda when given "median". I'm testing them now with bigger inputs
Hmm, it looks like there's an error in the existing answer
import numpy as np
import pandas as pd

data =  [1.00, 1.05, 1.30, 1.20, 1.06, 1.54, 1.33, 1.87, 1.67]
index = [0,    0,    1,    1,    1,    1,    2,    3,    3]

data = data * 500
index = np.sort(np.random.randint(0, 30, 4500))

def df_approach(data, index):
    df = pd.DataFrame({'data': data, 'label': index})
    df['median'] = df.groupby('label')['data'].transform('median')
    df['result'] = df['data'] - df['median']

def series_approach(data, index):
    pd.Series(data).groupby(index).transform(lambda v: v - v.median())
Surprising:
 %timeit df_approach(data, index)
5.38 ms ± 50.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit series_approach(data, index)
26.3 ms ± 1.41 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
 
12:12 PM
Umm... maybe not... I'd imagine doing the calculation by group means aligning the values at the end and will be the overhead... (although admittedly, slightly surprised by quite how big that overhead is but...)
5.45 ms ± 25.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
31.6 ms ± 138 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Seems similar...
 
I don't know Pandas, but lambda v: v - v.median() looks expensive to me. I assume it has to recalculate the median on every lambda call.
@Dair stdlib random randint / randrange is uniform. It achieves that using a potentially infinite while loop, which @Kevin wants to avoid.
 
> 4.27 ms ± 109 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
65.2 µs ± 1.01 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
:P
how does Divakar's time?
 
12:29 PM
sec, let me scale back an order of magnitude
 
I don't think it's complete, I can't subtract the result from the original. I think it only computes one median per group, rather than a whole array?
I think you were timing apples and oranges there, @roganjosh
 
In which bit?
 
Hi. I have a list of dicts.. I was wondering how can I eliminate the dict, if the 'to' value appears in 'from' like in this example pastebin.com/L4U5R8En
 
Divakar's version. Ours computes data - median of shape 4500, his returns a median of shape 30,
 
Just seen your comment
 
12:33 PM
E.g. this value 0x2b0c8440c53edf28b4da14815b3ddcef85a6a67f appears in the 'to' column, so that dict sould be deleted
 
@MikaelKen that's half an MCVE; what exactly do you mean by "eliminate the dict"? Remove the dicts where the to and from are the same values?
 
Remove the dict if value from left column appears in the right column
 
So from a collection of dicts you want to remove any dict for which the "from" value appears in any of the dicts as a "to" value
 
yes
 
that's not a pandas series/dataframe of dicts by any chance, is it?
 
12:35 PM
no
 
okay :)
I'd first loop over the dicts to collect every "to" value that's forbidden in a set, then loop over again and check if "from" is in the set
 
But maybe a pandas kind of dict would be better any way? I have like hundred of thousands of dicts
 
@MikaelKen if you have thousands of 2-item dicts with the same keys then yes, it might make sense to have a dataframe with two columns instead. Not sure how you'd do this reduction in pandas, though, because I don't use pandas. But "thousands of dicts" usually sounds like there's a better tool for the job :)
even thousands of tuples could be better, if you always have (from, to)
 
@AndrasDeak It looks like it's a simple .isin() for a df
 
could be
 
12:38 PM
df = df[~df['from'].isin(df['to'])]
 
hmm, I'm getting an error with your df version, @roganjosh
oh, it's because I'm running np.allclose on them
 
That is for a pandas dict? cuz for a normal one it doeasnt work
 
lol, because your function doesn't return df :D
@MikaelKen a pandas dataframe, yes
 
@AndrasDeak oops. I timed all of them without a return though (which I'm not sure would make much difference, but they're all equal in that regard anyway)
 
@roganjosh now that I made yours return df['result'] and with Divakar's updated version, I see the same result for OP's example but Divakar's gives something else for your large input
(before you time anything you should first check np.allclose(this_one(), that_one()), otherwise timing is meaningless)
 
12:43 PM
Divakar's approach has gone beyond my point of following the logic properly
 
>>> np.allclose(diffmedian_jit(data, index), df_approach(data, index))
True

>>> np.allclose(diffmedian_jit(data, index), bin_median_subtract(data, index))
False
@roganjosh same, we can still check the result
 
In the meantime, I've removed the bit at the bottom of my answer, and can remove the upvote since it was edited... until he gets it back on track (but it becomes a bit meaningless if it becomes completely incomprehensible)
 
Well it's not meaningless. A working convoluted piece of code can still be superior to slow readable code. It all depends on how desperate the asker is. If they spare 30 minutes with their real dataset they can spend it figuring out what Divakar's code does.
 
hey, can i get some advice on making this recursive (lines 12-17 i think): gist.github.com/tjt263/6772d315793c0c3ddb158db121d373c7 i started rewriting with os.walk() but i'd like to finish this one
 
@roganjosh why'd I unupvote?
oh, you mean your upvote on his answer
 
12:47 PM
Yep
 
@voices ooh, generating TikZ with python, bold goal ;)
@voices just to be clear, is the requirement for it to be recursive, or for it to work?
 
@AndrasDeak i figured it's just text manipulation. so it should be pretty easy. and well, it's not too bad, but it needs to descend into subdirectories. which isn't a tikz problem. .... well, it works. it's just.. incomplete
 
@MikaelKen not a pandas "dict" but a DataFrame. A list of dictionaries can be converted directly: df = pd.DataFrame(the_list_of_dicts)
 
@voices so are you trying to draw a directory tree with tikz by using the actual directory tree as a template?
@roganjosh he fixed it :)
 
@AndrasDeak yeah, sort of. not a template so much, but the actual input. but i think that's what you mean, right?
@AndrasDeak you can see here overleaf.com/read/mbbrgyhdbpww
 
12:53 PM
@voices yup
 
but the tikz stuff is mostly superficial really, the problem is a bit lower down i think
i don't suck at python, but i'm not great at it either. i still get lost sometimes
 
@AndrasDeak It tramples the pandas approach, coming in at 573 µs ± 7.48 µs per loop for me.
 
@roganjosh neat
I'm wondering how foolproof I should make mine. I assumed the labels are contiguous and sorted, which I'm pretty sure they are.
 
I don't really "get" numba with the timings. Is yours a combination of the two timings; the first for jitting, and the second for then running?
 
No, the first timing is the native python function, the second timing is the jitted function. If you pass a signature to numba.njit it compiles ahead of time.
 
12:57 PM
@AndrasDeak think you can help me out getting it to descend into the subdirs?
 
@voices Not sure, sorry, and I don't have much time right now. I just wanted to clarify the problem for the benefit of potential helpers.
OK, OP says labels are sorted and contiguous
 
Cor blimey... who fancies explaining to the OP what's going on here :)
 
For completeness I'll try run your timeits on my system since it looks like jit is the winner
 
sure thing, thanks
@JonClements I just see an indentation error...
 
I'm sure there's a dupe for that, I just can't find the motivation to search for it
 
1:06 PM
Which one? The median one?
 
Jon's one, sorry for being unclear
 
@voices Well, your first step to making this recursive is to put it into a function or two. ;) Practice on a simpler problem first. Make a recursive function using listdir that prints the filenames in some directory tree, with indentation. Once you have that working, it should be easy to adapt it to your actual task.
 
@Aran-Fey ah, right, makes sense (forgot about that one)
 
@AndrasDeak I get TypeError: No matching definition for argument type(s) array(float64, 1d, C), array(int32, 1d, C) trying your jitted code, I'll have to do some research before I can give a timing on my system sorry (both inputs are already numpy arrays)
 
@roganjosh oh yeah, because on windows you get int32 by default, not int64
meh
 
1:09 PM
I guessed as much, I'm just trying to find the fix :)
The OP has also asked me to rerun my timings all over again with an edited input :/
 
wait until they comply to have sorted, contiguous indices :P
 
@voices I did notice a couple of things about your code. Eg, for directory in directories: directory = 'child{node{'+directory+'}}' probably doesn't do what you think it does.
 
@roganjosh the fix is to have proper JIT, but then you have to execute the function once so that it can be compiled...
that's what I tried to avoid with the signature
 
Write a custom function with Cython and compile it? :p
 
Jon's back in the race for speed :)
 
1:12 PM
@roganjosh OP's new data is 2d so I'm not even sure if the methods work...
mine will certainly break :D
 
That's nuts. Surely a rollback case?
 
@voices Also, the sorted function copies its list arg then calls the sort method to sort the new list in-place. So it wastes time & RAM to do seq = sorted(seq). Instead, just do seq.sort().
 
@roganjosh dunno, OP's not very responsive to comments
 
@PM2Ring i'll look into it, but i'm trying to focus on the problem at hand
 
okay, they've sorted and flattened their data I think?
 
1:16 PM
i'm running out of ram myself on this one
 
Seems so, but I've lost interest when someone just throws that in at the end and then asks me to do it all again.
 
so it should be consistent with the original version
 
It's just as arbitrary as my initial, inflated, dataset. I'm not sure what it's going to reveal, the fun has already been made by having some pretty good answers using a total variety of approaches. The OP can try adapt any one they wish
nm, they've since backtracked :) <jimmies unrustle>
 
well, with the larger data I get 3.5x speedup rather than 65x
 
@voices Sure, that's why I started that post with "Also". As for the recursive listdir stuff, you might find this old answer of mine helpful: stackoverflow.com/a/33914896/4014959
 
1:25 PM
Oh wow. Correct me if I'm wrong, but the new data blows Divakar's solution out. I'm getting 3.02 s ± 31.9 ms per loop vs the pandas 495 ms ± 16.9 ms per loop
 
@PM2Ring what do you think it does? or what do you think that i think...
 
Ahh... looks like @Michael's posted a Q (thought it sounded familiar...)
 
Divakar has since edited again
 
oh boy, I was looking at an earlier version of the code
let me start over :D
 
The Q that keeps giving :p
 
1:39 PM
@roganjosh his new version gives different results...
@roganjosh why can't you test mine locally?
 
Because it throws the int32 error
 
@voices I think it creates a bunch of strings, and throws them away.
 
if it's about int64, say groups = groups.astype(int64) to use int64, or remove the signature from the jit decorator and ignore the warning from timeit
depends on whether you want to time int64 or int32
you can also edit the decorator to have i4 instead of i8
that would be fairest with your timings, actually
 
Ah hold on, I need to read the text you added sorry
 
use @numba.njit('f8[:](f8[:], i4[:])')
that will be fair with the other timings because you're using int32 everywhere
(and check that the results agree :D)
 
1:45 PM
@PM2Ring why does it throw them away
you might be right
 
@AndrasDeak done
 
>>> deak = diffmedian_jit(data, groups)
... div = bin_median_subtract(data, groups)
... rog = df_approach(data, groups)
...
... print(np.allclose(deak, div))
... print(np.allclose(deak, rog))
False
True
@roganjosh you can even omit the .py_func timing, that's just to show how much the jit does. It's not for prod.
 
Starting to feel a little secretarial to this question now :P I've made that edit, if Divakar's answer doesn't work, it's already blown out by the bad timings in the benchmarks and I don't wanna have to keep on top of edits coming in while I'm mid benchmarking (which they did)
 
you have only yourself to blame :D
 
tis... true :'(
 
1:55 PM
here are my int64 timings:
>>> %timeit diffmedian_jit(data, groups)
... %timeit bin_median_subtract(data, groups)
... %timeit df_approach(data, groups)
93.7 ms ± 1.17 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
3.3 s ± 67.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
266 ms ± 2.83 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
 
Well, now you're just showing off the crappiness of my laptop :P
 
maybe :P
for what it's worth the data type has to matter
I guess I can check that nah, too lazy
 
<wipes brow>
 
Wow, ndimage.median is very slow
 
@voices In each iteration, for directory in directories: binds the current string in directories to the name directory. Then directory = 'child{node{'+directory+'}}' builds a new string, which it binds to the name directory, replacing the previous binding, but it does not change the string in the directories list.
 
2:05 PM
>>> %timeit diffmedian_jit(data, groups)
... %timeit diffmedian_ndimage(data, groups)
93.5 ms ± 398 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
2.82 s ± 70.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
heh, it seems slightly slower than my original python loop
 
@voices If you're not sure what I mean by name binding, please see the links below:
Oct 17 at 8:44, by PM 2Ring
@djsmiley2k Have you seen and Facts and myths about Python names and values by Ned Batchelder? Also see Other languages have "variables", Python has "names" for a briefer version of the same stuff, with cute diagrams.
 
@PM2Ring i thought i was adding it to a dictionary
 
2:30 PM
@voices Oops. directories is a dict. Sorry about that. But my previous description is still basically correct, the strings in directories don't get changed.
 
2:45 PM
@PM2Ring No wonder people criticize python for being so slow :P
 
@PM2Ring i'll pay ya to help me if you're interesrted
 
I'm fairly sure PM isn't cheap :p
 
Where is that 1000$ thinktank quote?
 
This could be interesting. Where do we start the bidding war? :P
 
Aug 20 at 9:35, by PM 2Ring
Imagine that you won a competition, and the prize is a free session with a Python think-tank who normally charge $1000 per hour. Don't waste that prize!
 
2:54 PM
@AndrasDeak Nice, thanks. Also appropriate that it's by PM2Ring.
 
ya'll are welcome to help me for free
 
Oh, well then PM can have the contract :P
 
that's what i thought
 
But on a serious note, free help is what you'll get here as long as you're receptive to feedback and don't push things too far
 
You should hire me to be your consultant. I have advance knowledge of for loops, while loops, if statements and other control flow.
 
2:58 PM
@Dair you're going for free?! That's the offer on the table right now :P
 
@roganjosh push what things too far
 
@roganjosh (Shh, that's what voices thinks I'm doing but then I can bring out the good ol' labor laws)
Minimum wage here I come.
 
@voices some people come and ask, say, 10 things a day and don't contribute anything back. I don't suppose there is a firm definition of "too far", but as long as you're mindful that we're not paid to hang out here, it's unlikely you'll get to that point
@Dair Still happy for you to take that contract :P
@voices anyway, I'm not a room moderator, so it's only opinion from me, I can't speak for them
 
(the term is still "room owner")
 
I kinda guess that that term isn't really intuitive?
 
3:03 PM
whether it's intuitive or not - that's the actual term :)
 
It's just that you keep saying "room moderator" and then later people ask me if I'm "a mod" and I don't know what to say. Similarly to how "multithreading" is correct in general, but in terms of python it makes a huge difference if someone's really talking about multiprocessing, so we might as well stick to technically correct terms.
 
"Anyway, I'm not a Room Owner, so I don't moderate here" possibly is better
 
besides, we occasionally talk about ROs
 
@AndrasDeak people ask me if I'm "a mod" and I don't know what to say - how about: "No... how crazy do you think I am!!?" :p
 
@AndrasDeak fair point. I'll change my wording sorry
 
3:04 PM
@JonClements well I usually say "no" and then say :P to my laptop
 
@voices Thanks for the offer, but this chat room isn't the place for job offers. I'm confident that you'll be able to write the code you want. But recursive code can be confusing, which is why I suggested doing it in stages. Once you have a simple recursive directory lister working, it'll be easy to modify it to print in the exact format you want.
Hopefully, the answer I posted earlier gives you some useful ideas. And Ned's article about names should clear up that misconception you had about the directories for loop.
 
@roganjosh thanks
 
@AndrasDeak Imagines someone typing "No" then physically sticking their tongue out at the computer.
 
i'm inclined to help people for free whether it's with programming or anything else. but i'm trying to solve a problem. i'm just trying to provide an incentive
 
You can't really provide an incentive. You can just ask nicely and wait.
people here default to helping when they can, so the only trick is not being annoying enough to stop them from doing so
 
3:10 PM
I'm annoying you am i
 
I didn't say that
 
That certainly wasn't the intended reading of my comments
 
and I was only reflecting on the "how can I provide an incentinve to get help" aspect
 
that's okay, it happens
thanks for saying so
 
@Dair :) Well, in the worst case the expected number of loops is less than 2 (1 + .5 + .25 + .125 + ...), so it shouldn't be too slow, on average. It uses getrandbits to generate a uniform random number, and re-rolls while the result is too big. Eg, randrange(10) keeps generating 4 bit numbers until it gets one in range(10).
 
3:14 PM
heh... image a weird glitch where the 3rd and 4th bit always ended up set :)
 
@PM2Ring is the worst case not an infinite loop? :P
 
@AndrasDeak The worst average case. :P
 
Average Monday?
 
the lim sup complexity (?? need to review analysis lol)
 
3:18 PM
@roganjosh actually I think it's clear enough...against all odds
 
@PM2Ring I'm thinking of one those brain memes where the brain gets more expansive starting with 4 then 2 * 2 then 2 (1 + .5 + .25 + .125 ...)
 
(I think 2 was just the sum and the parenthesized expression the explanation)
 
@AndrasDeak we're going with Jupyter? It'd be a lot of faffing for me to test since I don't use Jupyter. Spyder also has a graph panel
 
@roganjosh OP says Jupyter, so yeah
 
Oops! I missed that edit after my comment
 
3:22 PM
@roganjosh I've never found Jupyter awkward :)
 
@JonClements I used it a few years back maybe twice. When you throw IPython into the mix then I'm less certain about my assumptions, I have enough of those over Spyder
 
the main complaint I've heard against jupyter is how cell data can become stale, but this only happens if your rewrite history, in which case you can only blame yourself
 
One thing I use to (stupidly - because I didn't know better) do was copy/paste bits out of it to create a plain .py file... then I realised one can use nbconvert :)
 
@AndrasDeak Yes, but the probability of the loop going forever approaches zero. :) But yes, if your randrange is the worst possible size, of the form 2**n + 1, almost half the n+1 bit numbers produced using getrandbits are too high. But you should only expect to get 20 "too high" results in a row once per million calls.
 
@roganjosh found an exact dupe \o/
 
3:27 PM
@AndrasDeak many thanks :) And did I miss your gold badging?
 
maybe
 
You literally just hit it? Congrats!
 
Few days ago, but yeah. Thanks :)
(today's upvotes from the numpy thing haven't shown up in the tag score yet due to caching)
 
@JonClements That would be... unfortunate. But if Mersenne Twister did stuff like that, it would've been noticed by now. Sure, it could produce huge runs of "undesirable" bit patterns, but it can't do it forever, since it's got a finite period. OTOH, that period is rather large: 2**19937 - 1...
 
@AndrasDeak Aha, that explains it. I was looking through chat for "gold" too and didn't catch it in chat. Apologies for only just spotting it.
 
3:35 PM
I didn't mention it because it doesn't really matter
 
Congrats from me too.
 
thanks :)
 
@AndrasDeak tell me, have the lambs stopped screaming? :P
<Silence of the Lambs reference from the start of today falls flat>
 
Yeah, I got that. Just watching that scene :D youtube.com/watch?v=fd7e1fXYIuM
 
Oh good, I wasn't completely off-the-mark, then :)
 
3:44 PM
was trying to figure out if there was an appropriate response, but there isn't one, alas
 
baaaa ?
 
oh no, call an ambulance, the puppy is confused :P
 
nnnnnnaaaaaay!
 
... scrapie?
 
now he's a horse, it's worse than I thought
 
3:48 PM
snarfff snarff...
oh no wait... that's a cartoon... errr...
right... couple of bits to do... rbrb for a little
 
rbrb
 
Wowsers... wonder how the OP ended up with this structure
 
4:04 PM
probably a lack of familiarity with dicts and other structures
 
4:20 PM
Quick question for those familiar with RNNs: RNNs for translation (english -> french for ex), the input sequence and target sequence don't have to be the same length right? Like if my input sequence is of length 100 but my target is length 500, that's possible? I'm not doing natural language translation but my problem can be expressed as such so I'm trying to explore it
 
yep, it's possible. (and needed for any good translation really)
 
Whew thats great
 
4:47 PM
@JonClements Oh dear. All that gratuitous use of reduce & map, with lambda args. And there's now an accepted answer by Ajax, written in classic Ajax style...
 
sounds interesting - will have a look in a bit :)
@PM2Ring cor blimey guvnor!
Although overkill, pandas makes it fairly convenient... pd.DataFrame(map(dict, in_data)).T.reset_index().fillna('-').values.tolist()
Still... using native Python it's still possible to write it less obfuscated :)
Umm... might need something to ensure name comes to the top but given the example data- it's not really needed
 
5:20 PM
@roganjosh Divakar's fixed his answer
>>> %timeit diffmedian_jit(data, groups)
... %timeit bin_median_subtract(data, groups)
... %timeit df_approach(data, groups)
95.5 ms ± 914 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
3.28 s ± 92.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
274 ms ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
 
So it's even worse
This has all been quite shocking to me, tbh. They have produced some amazing code that I cling to, but this case seems to not be working for them
 
there's no guarantee that convoluted numpy black magic will be faster
 
This is the perfect example of that. For all the flak that pandas gets, it isn't that unclear, and incurs no startup time
 
what do you mean by "incurs no startup time"?
 
@numba.njit('f8[:](f8[:], i4[:]') stalls the interpreter, no?
 
5:26 PM
@JonClements RomanP posted much more readable code, but it has a couple of minor issues.
 
@roganjosh ah, yeah
660 ms wall-clock time
if you are calling that exactly once on OP's example it indeed isn't worth it (unless we include pandas' import time which is also significant :P)
but when you're running that exactly once then performance shouldn't be your main concern
 
Sure, there's lot of ways to look at the issue, but the failure of digging bigger and bigger numpy holes here has been interesting to me
Just use the 3 lines of pandas, damn it :P (to Divakar) Stop adding more and more that we can't even understand
 
5:45 PM
Anyway, heading out for some food. Divakar is still the god of numpy for me, it's just interesting how this backfired. rbrb for a bit
 
@roganjosh well, it has 7 calls to numpy functions, two of which are sorts
 
@PM2Ring @roganjosh very similar to Roman's:
dcts = [dict(entry) for entry in in_data]
header = ['name', *set().union(*dcts).difference(['name'])]
res = list(zip(*[header] + [[dct.get(k, '-') for k in header] for dct in dcts]))
 
@roganjosh I just plugged his function through a profiler, and lexsort takes up 70.6% of runtime (in a single call), argsort 17.1% (again a single call)
 
Although... perhaps ['name', *{'name'}.symmetric_difference(set().union(*dcts))] might be better
 
@JonClements Maybe, although it's drifting towards Ajax-like unreadability. ;)
 
6:02 PM
Cabbage
Any Keras user here?
 
cbg
 
@PM2Ring a little perhaps... but certainly not quite the same league of it :p
 
 
2 hours later…
vp7
7:36 PM
I have a question, I was trying to generate distribution of a value based on the count. I am trying to use this :

def f(x):
total = x.sum()
minimum = 10
dividers = sorted(random.sample(range(minimum, total-minimum, minimum), len(x) - 1))
return [a - b for a, b in zip(dividers + [total], [0] + dividers)]

this is based on

https://stackoverflow.com/questions/3589214/generate-multiple-random-numbers-to-equal-a-value-in-python/3590105#3590105

and am using it as :

joined_df2['value'] = joined_df2.groupby(['Date_1','Date_2'])['c_b_i'].transform(f)
 
whoever can help will probably also need an MCVE
the superficial problem is that you're using a float type as an array index but this might just be a symptom of a higher-level issue
 
vp7
@AndrasDeak Thank you for the same, however, can we put MCVE in the chat itself?
 
hey guys, just wanted to make sure, is pandas reset index guaranteed to preserve the order of the previous index
 
@Skyler preserve it after what? I don't think that's guaranteed at all
 
@Skyler well, if you reset it, it's just going to generate a new RangeIndex while moving the previous index to a new column... so, yeah.
arhghghghghgghghghghghghghgh
who has £2.99 they don't need sitting in paypal (for litterally 12 hours) or something
 
7:51 PM
Oh, Jon, has it come to this...? :P
I don't have PayPal though, sorry
 
@roganjosh lol
 
@JonClements can send you sausage rolls in the post, though... I'm curious about what's happened?
 
you know that time time of the month after all bills are paid, but invoices aren't paid, and there's a tiny bit just to stop your bank charging you overdraft fees, but they will and then... meh
 
... There's a time without overdraft fees? :)
 
and then you regret helping a friend of the family out with their rent and taking 'em shopping when you know they should manage money better, but at the same time you don't want them or their children to go without food because "Universal Credit" is beep
 
8:04 PM
@JonClements good, just wanted to make sure i havent been making a mistake after sorting my dataframe
 
@vp7 sorry, got distracted. If the MCVE is short enough (and it should be), it's fine here. If the code is more than ~ ten lines you should post it in a code paste service such as pastebin or dpaste or github gist
 
i figured thats the most likely use case for reset index so it was unlikely to do anything wacky
 
@JonClements A slippery slope. I have fallen into that trap before and it's cost me £1000s to keep lights on for others. I don't think this is the best place to continue this; message me if you're stuck
 
@roganjosh on a side note (just going through my inbox): I'm not "secretly American" - I just happen to love American Country music :)
 
whats really messed up is when a parent is doing that while you are in college
(not like living under their roof, away for college)
 
8:15 PM
@JonClements enough to be a big Tom Petty fan, though?
 
wim
8:26 PM
cool trick that is not well known, you can mount a .whl file
>>> from distlib.wheel import Wheel
>>> whl = Wheel("pytoml-0.1.21-py2.py3-none-any.whl")
>>> whl.mount()
>>> import pytoml
>>> pytoml.__file__
'/Users/wim/pytoml-0.1.21-py2.py3-none-any.whl/pytoml/__init__.py'
 
can you mount it in a linux shell?
 
wim
hmm, what would that mean?
you could do PYTHONPATH=/Users/wim/pytoml-0.1.21-py2.py3-none-any.whl to the same effect (which will make it available using zip importer)
 
$ mkdir np_wheel
$ sudo mount numpy-1.17.3-cp37-cp37m-manylinux1_x86_64.whl np_wheel
mount: /home/user/tmpdir/np_wheel: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
:(
 
wim
oh mount as a filesystem? nah
but you don't really need that (a .whl is just a zip file + metadata)
 
that's just what came to mind when you said "mount"
 
wim
8:31 PM
I guess the cool thing about distlib mount is it can allow you to temporarily use a different version than is installed in that python env
kind of a glorified sys.path prepend
 
stackoverflow.com/questions/58791047/… unclear. The indentation is a mess
 
@wim that's pretty neat if you put it that way
 
01:00 - 21:0021:00 - 00:00

« first day (3312 days earlier)      last day (1639 days later) »