« first day (3139 days earlier)      last day (1806 days later) » 

6:00 PM
not that many posts
 
btw cs95 refers to a course number in your uni ?
 
no, just me :P
 
I know, I was trying to get at where the name came from, or is it counter strike?
 
I'm gonna guess that they were born in 1995
 
6:09 PM
there was a hint in his phrase there :P
dang it rogan, lol
 
cs is shorthand for an older username I used to use here and 95 is just there to get over the 3 character limit
 
CS is half-computer (as evident from the avatar) and 95 is their family name. Distant cousins with Windows 95.
6
 
makes windows logging off sound
 
That is my new favourite explanation of what "cs95" means
 
so you are half-trapped in the half-computer cs95?
towards the lower left of the lower half?
 
6:13 PM
I may use lambda, splats, comprehensions, and other functional thingamajigs... but yammit, its all PEP8.
 
CS95 is trapped in a computer in the same sense that all humans are trapped in their meat suits
 
@Kevin One of Bruce Springsteen's more obscure ballads.
 
@piRSquared are you having an existential crisis?
 
I was referring to his location from his SO profile Trapped inside the Macintosh on the left
 
@roganjosh wait, I have another joke before I think of a response. @PM2Ring I hear that is Trumps next campaign song.
 
6:16 PM
well we have our general election results in two days time :)
 
Profile pages have long been the ideal incubator for angsty existential quips
 
Hahaha; "hold that thought, guys". Your stand-up needs a little refinement :P
 
I can only hope that the server holding my teenage years AIM profile is at the bottom of the ocean right now
 
Kevin I feel the same way
I was such an edgy kid:my WoW name was Octobersorrow and my AIM was lifeonstandby10 >.<
 
I think I saw AOL once or twice, it was Yahoo Messenger and Orkut when I was a kid
@roganjosh how's your flask thingy coming up
 
6:21 PM
Today I had to build a new feature in my other app so I can't report back sorry :/
 
@roganjosh no worries :)
 
Tomorrow I'm in the office and have to deploy the changes (And actually test them first). If you're interested in the outcome of the scanner, I will message you an update
That's if IT doesn't shut me down. Time will tell :)
 
@roganjosh yessir, thanks, I am interested to see how something which looks so straightforward pans out :)
 
You've probably missed my previous rants but, suffice to say, this project undermines the existing IT dept and I'm a contractor. They will shut the project down unless I can advance it past the point of no return so... it might not make it
 
haha, even I am a contractor, I know how much trust is put on us by our client company :D
 
6:29 PM
Ok Pandaneers: Z-Score time. Using groupby.apply is slow. What is the fastest PANDAS ONLY solution you can come up with to z-score grouped by the first level of the index.
np.random.seed([3, 1415])
s = pd.Series({(d, i): np.random.rand()
               for d in range(100)
               for i in range(3000)}).rename_axis(['Date', 'Id'])
This is your bogey s.groupby(level=0).apply(lambda x: (x - x.mean()) / x.std())
 
grabs popcorn and waits for cs
 
No using of Numpy allowed.
 
Pandas are cute...I know nothing about Python Pandas tho.
 
I've got an entry but withholding for a bit.
 
I assume it will be cracked before I get home in 20 or so mins, but "No numpy" is a tough restriction
 
6:38 PM
spoiler Appears to be 10 times faster on my machine
 
no spoilertext today?
 
Ah, I thought we were just throwing out ideas
 
I don't even know pandas, but pretty sure someone will come out and whine about it
 
@DeveshKumarSingh whine about what?
 
spoilers
24 hours ago, by cs95
at least not as painful to read as endgame spoilers
 
6:41 PM
that the answer is revealed before people were done solving it, or they wanted to not spoil the answer for otehrs
 
Meh, spoilers are for riddles, but this one is more like a race
 
race implies more than 1 person has a chance though. :P
 
yesterday, by wim
yeah I was just looking at this and it was already spoiled. have some common sense man.
 
"Find exactly the solution I'm thinking of" vs "invent the fastest possible solution, which is probably faster than the one I'm thinking of"
The fact that pi gave a working implementation immediately after posing the challenge is telling of its nature
 
@ParitoshSingh reminded me of youtu.be/Gb6XdhEqPVc
 
6:45 PM
hah, that's pretty funny
 
I'm surprised
g = s.groupby(level=0)
(s - g.transform('mean')) / g.transform('std')
I thought that'd be a tad quicker
sparing the time to join on unequal indices. But no.
 
Broadcasting also handles index alignment. This broadcasts first, and then the arithmetic has to check the indexes again. Maybe that's why.
I hope I explained that well :P
 
My untested suspicion is that converting the lambda to a real function, and passing x.mean() etc. As arguments, pre-calculated once, would be a cheap gain
 
Also, don't forget about groupby.pipe
s.groupby(level=0).pipe(lambda g: (s - g.mean()) / g.std())
 
pipe is faster with lambda? o.0
TIL
 
6:50 PM
what will be a good starting point to learn about pandas/numpy
 
Roughly 2000 questions on stack overflow ought to do it :P
 
question:
is it possible to use multiprocessing, inside a methode of a class?
(which calls a methode of that class)
 
If you just want to gain a working knowledge, the 15 minutes to pandas tutorial is a good start
 
yes, in theory. but in practice, it depends on what you're trying to do.
 
@cs95 or your canonicals? and it's now 10 minutes :)
 
6:52 PM
the main thing is whether the problem itself is "trivial" to parallelize or not.
 
i've a random forrest,
and for each tree, i want to fetch the contributions of the variables
 
While I am proud of my canonicals, they are mostly deep dives into very specific topics. You will likely never need to know everything there at once. What you're looking for is breath coverage, not depth. Although my canon list has grown large enough to cover that too.
 
this is a non-winner as well
s.groupby(level=0).agg(['mean', 'std']).join(s.rename('s')).eval('(s - mean) / std')
 
this instead of for tree in self.estimators_: i want to use some parallelisme
because i can have 500 trees
 
"parallel-is-me"
 
6:55 PM
@cs95 great! And how much numpy background is assumed for going through 10 minutes?
 
you don't really need to know numpy, just know it exists and some basic topics i.e. vectorisation and broadcasting.
 
'p.map( self._predict_tree, self.estimators_ )' this does not werk ...
work*
predict_tree is a method, self?estimators_ is a list of trees
 
@DeveshKumarSingh they say 10 minutes but in reality it takes much longer for all of it to sink in :P
 
@DeveshKumarSingh here's a tip: the inadequate use of "whining" is not among the things you could learn from coldspeed
you're on the "trying our patience" side of things these days, so try not to use demeaning language when referring to people other than yourself
 
@Dieter are you able to do the same task using a loop at the moment?
 
6:58 PM
yes
 
@Dieter Realize that each multiprocessing worker is its own separate Python process, so each Tree instance must somehow get instantiated
 
wim
I have a hunch O(n * log n) should be possible here
but the dupe I hammered to is also O(n ** 2)
 
With some tricky datatype?
 
"whine" is your favourite word, not mine @Andras :P
 
im guessing you're asking for best solution time complexity? because yes, as-is, its O(n**2)
 
7:02 PM
@PaulMcG - it works
 
@Dieter Hooray! What works?
 
the mutliprocessing
 
@AndrasDeak and @cs95 apologies! Will keep this in mind for future
 
@Dieter - you are still leaving out some key details, but I'm happy for you
 
actually, come to think of it, its probably worse than O(n**2). shouldn't the string length itself play a factor?
 
wim
7:04 PM
@Kevin That is exactly why I didn't write it like that. Don't want to catch TypeError during iteration and possibly mask bugs.
@ParitoshSingh yes but then you have to add another axis
 
Oh good, I came to the right conclusion... Eventually :>
 
please trash those messages from mine @AndrasDeak
 
@PaulMcG, writing a typo (and looking over it, for more then 20 minutes is not something to be proud of ;-) )
 
@piRSquared is that french for "I don't understand"?
 
@Dieter Only 20 minutes? Wait until you write an if-test inside an except clause, and use the wrong indentation level for the else.
 
7:13 PM
@wim i see. so barring string lengths or anything in the "lower level", i see this problem as essentially making all possible 2 length pairs from a list. (and then checking membership). i Dont think that can be made faster than O(n**2). Sure, you can reduce the number of checks you have to make, but not at the level of changing orders i wouldn't think.
 
@wim I have a feeling Aho-Corasick might speed up the string comparisons themselves.
 
wim
meh, Barmar reopened it and posted his answer
which is wrong of course because Barmar has about 90% chance of being wrong in anything he posts
 
cbg! Been awhile since I hung out in this room. Anything crazy happen lately in the land of Python?
 
But his rep? /ducks
 
I'm fixing
I can successfully encode and send windows commands and get answers.
 
wim
7:23 PM
@piRSquared 20,376 answers
 
whoa my yam!
 
wim
it's actually about pruning completely overlapping regex match ranges. pretty interesting question actually, there is probably an elegant solution involving sorting the ranges by start and/or endpoints somehow
I think mathematicians call it a "covering set" or something
 
yes, numbers make this a much better problem to tackle smartly
 
I've been mulling it over. I'm no algorithms expert but yeah. Minimal covering
 
wim
hmm, shouldn't have searched "minimal covering" at work
 
7:27 PM
lol
 
laurel
 
wim
too many brazillian bikinis
 
hahaha
 
Girl_with_certain_hair_color #1: Last night I slept with 2 Brazilian guys. Girl_with_certain_hair_color #2: Wow, I don't even know that many guys!
 
wim
groan
so what happens if you hit the rep cap 200 and then offer a bounty for 100
 
7:37 PM
I remember that joke from the Bush Jr administration
 
wim
I think you just stay rep-capped at 100 right?
 
yes
 
wim
seems that way
that hash of infinity thing went HNQ so I bountied my most recent unloved Q
 
Which seems totally nonsensical, considering I didn't get any rep and it's a meta post (I assume)
 
woah, what is this pipe magic. I have basically avoided it entirely in my exploration of pandas so far, but I'm surprised to see it be so much faster than the apply
 
7:42 PM
Hello
Can any body tell me please why when I try to write some floating numbers in form of scientific number into a file there are some unintented \n in my file?
 
wim
it's weird that you're allowed to recoup more rep from downvotes but not from bounties
 
@roganjosh badge on meta
There's secret meta rep
 
Hmm, interesting
 
@wim no it's not. rep cap is strictly from up/down votes.
you can accumulate above 200 if you have answers accepted
accept answers to questions you ask
 
meta.stackoverflow.com/users uses fake rep I think
 
7:45 PM
receive a bounty
I forget how they handle the 1 point penalty you incur from down voting an answer yourself.
 
@ali73 Not without a Minimal, Reproducible Example
And that is now the first hit for "mcve" on Google, so the change has come
 
not for me.
having said that, i am genuinely surprised they'd just choose to "leave it like that" after the kind of response it generated.
 
@ParitoshSingh click it
Ostensibly it's the old mcve link, but it takes you to stackoverflow.com/help/reprex
 
@roganjosh sorry, i meant my google search for mcve still uses the mcve url. the redirct was already implemented before the announcement (apparently, cause logic...)
So, it's all backwards. it was already implemented before being announced. I just, expected it to be reverted.
 
7:51 PM
Yep, and now it looks wonky in search results too. We're all winning here
 
you know what, im going to dig that meta post up. perhaps its time for comment 2 on meta.
 
def write_into_file(self, name):
        file = open(name, 'w')
        for key in self.bi_grams.keys():
            key = key.replace('\n','')
            file.write('{0}|{1}|{2}\n'.format(key.split(' ')[0],
                                              key.split(' ')[1],
                                              str(float(self.probability(key.split(' ')[0],
                                                                  key.split(' ')[1],
                                          str(self.probability(key.split(' ')[0],
one of the outputs is now like this:
and|\n
 
Ok. Is that not what you want?
 
it should be something like
and|blah|1.203e-10
 
The fact that you're repeatedly calling split on the whitespace is an issue
 
7:58 PM
Is this exactly the code that you're running? Because it has 18 left parentheses and 15 right parentheses and this usually means that the code cannot run
 
It would perhaps be easier to do it once before write
 
def write_into_file(self, name):
        file = open(name, 'w')
        out = []
        for key in self.triGrams.keys():
            words = key.split(' ')
            key.replace('\n', '')
            out.append('{0}|{1}|{2}|{3}\n'.format(words[0], words[1], words[2],
                                                  str(float(self.probability(
                                                      key.split(' ')[0],
                                                      key.split(' ')[1],
                                                      key.split(' ')[2])))))
 
Then, if it doesn't split the way you expected (a value is missing), you can handle that it in one place
 
one of the real outputs are like this:
and|profan|\n
i|4.1787054964947076e-21\n
 
If your assumption that each key splits nicely into 3 parts is not valid, then either scrub your data or handle the exceptional case.
 
8:02 PM
@roganjosh
I've done it in last piece of code.
If I really got what you mean
 
And you still have 3 extra calls to keys.split() that should reference words instead
 
@ali73 No, because you continuously call .split(' ') and use different indices
 
And even if the multiple key.split calls aren't the cause of your problem, they're still a bad idea. String splitting isn't particularly fast, so you shouldn't do the exact same split on the same string multiple times. And especially not in a loop, where you're doing that on every key string.
 
@roganjosh Not the likely cause of the bug, but certainly some code ugliness
 
@PaulMcG for sure, I was just taking a second look :)
 
8:07 PM
I know that it is a bad cod style and to be honest it's horrible.
At first I'm trying to have a working code to not to miss the homework deadline.
:D
 
I don't know whether we can answer this without a more complete example. It feels like guesswork
 
May the file size effect it?
It's about 12 MBs and even one of them is about 40 MBs
I mean output file
 
No, the file size won't play a part in a logic issue, other than making it more likely to hit a record that doesn't fit a standard format
 
If self.probability expects to be called with 3 strings, and you're certain that key.split(' ') will create a list of 3 strings, you can do self.probability(*key.split(' ')) However, I'm curious as to why that call is wrapped in str(float()). What type does self.probability return?
 
Are you reading these keys from a file? and are you opening that file in binary form using "rb"? If so, then that might be why you are getting \n's embedded in your keys.
 
8:12 PM
It will return a floating number.
And a friend told me str(...) may fix the problem
 
Have you read a good python tutorial yet?
You've received several focussed and helpful suggestions and you don't seem to be able to make use of them. This is usually a good sign that you need to get to be more familiar with the basics of the language first.
 
@PaulMcG
the keys are stored in a dictionary name tri_grams or bi_grams based on the class.
 
@ali73 If it returns a float, converting that to a float does nothing, apart from wasting a little bit of time.
 
Hey guys, I have a directory with many subdirectories inside of it. Each subdirectory has a serial number and a date as its name and it has four .dat files whose names are the same in all subdirectories. I want to make a data frame containing the information inside the .dat files plus the information in the subdirectory names. Should I post it as a question or some of you can help me?
 
@PM2Ring
Again, that friend of mine told me to do so. It looked illogical to me. But since his experience is more than me I trusted him.
 
8:21 PM
@MarlonHenriqueTeixeira What would be your question title on the main site?
@ali73 I think it's worth reading the comments and thinking about this independently (not disrespecting your friend)
 
when it comes to programming questions, Stack Overflow is your greatest frenemy
 
@ali73 I think it would be: how to me a DF including the directory name inside the DF? How is it?
 
Horrible
 
:(
@AndrasDeak what do you think would be a better one?
 
I'd have no idea what you're asking. Plus you replied to the wrong person.
 
8:25 PM
@roganjosh
I've been reading comments and reediting my code since I got here and most of the comments were completely true about the code style and ...
I'll try more on this.
Hope figure this out.
Thank you all for your comments.
 
@ali73 good luck
 
@AndrasDeak Sorry
 
@ali73 best of luck mate. All of the comments were constructive in some way, I think, and hopefully you can piece them together to be on to some winning code :)
@MarlonHenriqueTeixeira I asked because I was curious about what part of your process was causing you difficulty
 
@roganjosh it is the part of getting the information on the subdirectory name. :/
 
If the choice is between asking in chat or asking on the main site, the proposed title should give us some idea of the specific issue
 
8:31 PM
*from
 
heck, as an exercise, you can come up with a proposed title, and then see if just searching for it online gets you any help or not as a first step
 
Alright! I'm gonna do it! :D
 
learning to break your main task into small problems, and then searching for those small problems online is a very useful technique to pick up while programming
 
Good advice! :)
Cool! I've found something promising!

https://stackoverflow.com/questions/44112286/python-store-directories-and-filenames-as-dataframe-columns
 
nice. :) Make sure you try to take the code apart line by line until you're comfortable with what the code is actually doing before using it. Sometimes you do not need every step that is written in the answer. But you will be able to piece together your entire solution step by step if you just repeat this process.
 
8:36 PM
rbrb all
 
rbrb
 
Nice! Thanks guys! By the way, what does it mean "rbrb"? :)
 
@MarlonHenriqueTeixeira It's short for rhubard here
 
8:39 PM
part of our optional but fun little code language here
 
@cs95 that's a neat trick to use broadcasting to create the dummies. Seems to be a lot faster than using get_dummies
 
8:59 PM
In the last few days I wasted about 10 hours trying to figure out why MyApp.delete_config_file() wasn't deleting the config file... only to realize that MyApp automatically re-saved the config file on exit :/
 
^ that's called outsmarting yourself.
 
@Aran-Fey only because you can't bask in the glory of "Why don't even the debug prints work??!....oh, I've been using an executable compiled earlier by mistake"
 
I went down a rabbit hole because I actually recently started using pyfakefs to prevent my tests from creating actual on-disk files... I couldn't find a bug, so I thought it'd have to be that library's fault... I was ready to submit a bug report and everything. That would've been embarrassing
 
@ALollz Not sure at what point I came across using that broadcasting trick with mini lists, but it works! Also, completely missed the use for pd.get_dummies there :D
guess I'm somewhat rusty
get_dummies didn't hit me because of the added requirement that they wanted columns for certain values only (not all)
 
wim
9:17 PM
don't you have a hard disk? LOL
 
I like to think that I'm doing people with SSDs a favor :D
 
*looks at funding* no, no, use HDDs, HDDs are bestest
 
worse hardware makes for better software
 
wim
9:41 PM
import numpy as np

def setter(x):
    np.array([float(x)])

def getter():
    return np.empty((), dtype=np.float).tolist()
>>> setter(1.23456)
>>> getter()
1.23456
>>> getter()
1.23456
>>> setter(4321.)
>>> getter()
4321.0
free storage yo
 
Uhhh... why exactly does converting an empty numpy array to a list return a float?
Actually, do I really want to know?
 
>>> np.empty((), dtype=np.float).tolist()
1.0

>>> np.empty(0, dtype=np.float).tolist()
[]
Hmm, because the first one is a scalar, and the second one is a 1d array with length 0.
numpy scalars are always weird
 
10:05 PM
suggestion to educate the masses by force github.com/numpy/numpy/issues/13525 :D
if sys.version_info[:2] >= (3, 7):
    __all__ += ['__don_t_do_this']

    def __getattr__(attr):
        if attr == '__don_t_do_this':
            warning.warn("Don't do this')
            return None
        raise AttributeError
meant to prevent star imports of numpy
 
shot down, as many good ideas sadly are
 
Well Ralf is right, it's not the responsibility of library code to enforce best practices.
 
11:06 PM
cbg
 

« first day (3139 days earlier)      last day (1806 days later) »