« first day (3094 days earlier)      last day (1855 days later) » 

2:15 AM
Another case where the repr or str representation of a value is mistaken for the actual value.
 
 
3 hours later…
5:23 AM
In Pandas data frame, I have two columns to store timestamps say timelab1, and, timelab2. I want to add timestamp '19:00' to either timelab1 or timelab2. If timelab1 is empty, update the 19:00 to timelab1 otherwise update it into timelab2.
I tried the following using np.where,
`updated_old_price_df['timelab1'] = np.where(updated_old_price_df['timelab1'].values == '', update_time, '')`
Here I am trying to update 'timelab1' column with timestamp if `timelab1` is not exist. But I don't know how to update `timelab2`, if value of `timelab1` is exist.
 
 
4 hours later…
9:14 AM
@SmithDwayne Please see An Illustrated Guide To Formatting Code In Chat. It's not quite clear to me what you want that code to do
updated_old_price_df.loc[updated_old_price_df['timelab1'] == '', 'timelab1'] = updated_old_price_df['timelab2'] maybe
 
 
1 hour later…
10:20 AM
What are "dummies" in pd.get_dummies? It's a brutally hard term to Google but it keeps confusing me. The docs use "dummies/indicator variables" but I don't understand why the name "dummy" was chosen and it keep screwing with my understanding
In other words, is there a way I can contextualise this where "dummy" actually makes sense? To me, there's no dummy when you're expanding out actual values
 
> Abstract: Dummy variables are variables that take the values of only 0 or 1.
doesn't sound like a very good name
perhaps we're just dummies
 
Ok, that makes a bit more sense, but the name is indeed confusing to me since I think of dummies as like stand-in values
Ha, the paper even uses pretty much that exact phrase and yet we're talking about real data
 
it's probably most often "a variable that represents one-hot encoding"
there's no "variable" term that can be derived from "one-hot encoding"
I guess it could be "one-hot code" and nobody would understand it :P
 
I think the disconnect here is that it's being used in Pandas in a bit of a hacky way
Not the implementation, but how it becomes useful in entirely different problems with strings
Or actually, I've just completely missed the context of what people were eventually trying to solve all this time. It neatly expands out a delimited string into columns and it looked a bit like a magic bullet for certain problems
 
11:30 AM
Is this actually how to use Django? Accessing __class__?
 
11:41 AM
Weird. I haven't heard of dummy variables for decades, back when I learned Fortran, when they were what are called parameters or formal arguments these days, as mentioned here. The Pandas usage seems to be what this Wikipedia article is on about.
 
My issue was not seeing it in terms of even variables, just something to do with strings since that's how I see it literally applied.
 
12:30 PM
@PM2Ring yup
 
12:54 PM
Hi. Q about the hare and tortoise algo. They say the time complexity of the algorithm is O(\mu + \lambda). Where mu is the initial tail length and lambda is the length of the cycle. But isn't that just O(N+1)? Where N is the length of the linked list
So the tortoise will loop \ptopto \mu times and hare will loop \propto \lambda if I understand correctlty
 
@isquared-KeepitReal I'm unfamiliar with the algo but if mu <= N and lambda <= N then O(mu +lambda) = O(N) = O(N+1), yes
Note that "initial tail" is a bit of an oxymoron
 
Why is it an oxymoron?
thanks Andras
 
@AndrasDeak spoiler; the tortoise wins
 
That O(\mu + \lambda) is for the general case, where you don't have a finite list to inspect, so there isn't a sensible N.
@isquared-KeepitReal A tail should be at the end, not at the beginning.
 
1:13 PM
Say you want to calculate the decimal digits of 1/98. There'll be a leading part that doesn't repeat, followed by a repeating cycle of digits. Now it has to repeat in <98 steps because there are only 98 different possible remainders when dividing by 98. But the cycle length might be shorter than 98 (and in fact it is).
 
this chat is legendary
THIS. IS. ROOM. 6!
 
So you could say the time complexity is O(N), with N=98 here, but the algorithm may do better than that.
 
OTOH, if you do 1/97, the repeating cycle is maximal: 96 digits.
 
How does that denominator determine the cycle length, though. All rational numbers have a reapeating set of numbers after decimal point?
 
1:19 PM
And for all 0<a<97, the digits of a/97 all share the same cycle. So 97 is called a cyclic prime. Some other cyclic primes are 7, 17, 19, 43. This property depends on the base, so in hexadecimal there are different ones.
 
@isquared-KeepitReal yup
With a possible prefix
 
In base 10, a fraction converts to a terminating decimal if and only if the denominator only has prime factors of 2 and 5.
 
Your number theory is strong
thanks for all of this
does it have to be 2 and 5. Wouldn't only 2 work as well? 7/2, 6/2, etc.
 
Yeah, any powers of 2 &/or 5
If you know what the powers of 2 and 5 are, it's pretty easy to calculate how long the decimal will be. And if there are other primes as well in the denominator, the 2s & 5s tell you how long the non-repeating prefix part is.
Investigating the cycles of various primes was part of the foundation of modern number theory, and guys like Fermat, Euler, and Gauss figured out the major results.
 
2:12 PM
Can you link me up with something on this please, this is fascinating
@PM2Ring
 
2:38 PM
@isquared-KeepitReal There's some info at en.wikipedia.org/wiki/Cyclic_number But you should check out en.wikipedia.org/wiki/Fermat%27s_little_theorem which is a special case of en.wikipedia.org/wiki/Euler%27s_theorem and leads onto en.wikipedia.org/wiki/Primitive_root_modulo_n There are probably better places to learn that stuff than Wikipedia, but those articles are ok starting points.
 
Wis
3:07 PM
@PaulMcG @PaulMcG can your parsing python library pyparsing be used to parse the python commandline, cpython implementation: github.com/python/cpython/blob/…
 
Wis
or is it easier to isolate this algorithm from cpython and use CFFI to make a python commandline parsing library?
 
There's also stdlib argparse
also third-party things like click
Make sure not to reinvent any wheels
 
@Wis Why do you want to parse the python interpreter's commandline args?
@roganjosh I don't know pyspark, but that looks like a XYWTF problem.
 
3:23 PM
The OP has tied themselves up
For a moment I thought I was hasty with the close tag but it's going downhill
 
3:49 PM
@roganjosh I haven't voted on it yet, but I don't think there's an urgent need to close it, unless people start posting stupid answers. I guess the OP's posted enough info that a pyspark expert would be able to steer them in the right direction.
 
@PM2Ring the reason I flagged here is that it makes no sense and I'd be pretty confident it will drop off the radar
 
Wis
4:01 PM
@PM2Ring to handle edge cases in my answer stackoverflow.com/a/55413882/4178053
 
@roganjosh FWIW I voted unclear
 
But by the rules and practices of this chat room, I was simply invoking the need for a reason. The ayes to the left 1, the no's to the right, 1. So nobody has it, nobody has it. Unlock!
 
@Wis Ah, ok. But you could use argparse for that, or even plain old getopt
@roganjosh Now on 2 of each. :)
 
I have failed in my duties as Speaker :/
 
4:29 PM
I've gotten a few more of those Announcer badges in recent weeks. My guess is they're mostly due to those dodgy sites that republish Stack Exchange answers. I can't figure out why they do that, or why people visit them.
 
Probably a dupe too
There's a target in one of the answers stackoverflow.com/questions/1837874/…
 
4:46 PM
@AndrasDeak Probably a dupe, but that target's tbe wrong solution. roganjosh is right, those student numbers need to be strings.
 
They are against strings but they don't know why, that's all
 
I'm not having any luck finding a good dupe target, though. There's plenty about zero padding numbers on output, or differences between Py 2 & 3 re handling octal, or Py 2 input() issues.
 
Guys do you have a block of code in python that has in it a lot of stuff and it would be good to do a practice in a lot of things inside? Thanks in advance
Pun intended: I am using the PYTHONPATH and i became a Pythonopath
 
I have tonnes of functions with a lot inside, none of which will run stand-alone
But there wasn't a pun :'(
 
it was not a pun against somebody of course just a play of words
 
5:01 PM
Ugh
 
PYTHONPATH(Environmental Variables) // Pythonopath (having a hard time learning python) hahaha
 
5:25 PM
@PM2Ring We collaborate quite a bit with Australians at work and it was funny when I learned that “dodgy” in Australia is a term for sketchy or unreliable. Considering that Dodge is my last name I guess I’ll always be a dodgy programmer. LOL
 
Wait, "dodgy" is well known and means exactly the same in the UK
 
Hi guys, I have a problem
Input
The first line contains integer n (1 ≤ n ≤ 105) — the number of groups of schoolchildren. The second line contains a sequence of integers s1, s2, ..., sn (1 ≤ si ≤ 4). The integers are separated by a space, si is the number of children in the i-th group.

Output
Print the single number — the minimum number of taxis necessary to drive all children to Polycarpus.
 
Like a dodgy dealer (someone who rips you off) or a dodgy insert_anything_here. Where are you based?
 
import math
n = int(input())
s = list(map(int, input().split()))
print(math.ceil(sum(s)/4))
 
@roganjosh In the US you’ll never hear that. It was news to me a couple of years ago when I learned the expression
 
5:28 PM
I wrote the above code but its failing for a test case
Input
3
3 3 2
Participant's output
2
Jury's answer
3
Checker comment
wrong answer expected 3, found 2
Can anyone explain why so? My question why it should be 3 instead of 2?
 
@Dodge how strange. It doesn't strike me as a colloquialism but quite embedded in the language
 
ok nvm I got it , I missed a condition in the question.
 
It’s not strange. Slang is slang. Far from universal
 
I'm saying I didn't think it even qualified as slang. I thought it was universal
 
Wait. Do you guys call slang “slag” or is that a typo?
 
5:34 PM
Hahaha, sorry
I had a really bad auto-correct yesterday on my phone, but that was a genuine typo
"Unless everything is a consistent 12 hour shift, 7 days a week and you don't cross daylight saving time, the results cannot possibly match the length of the existing datetime values. " But my phone apparently didn't think "shift" was the right word so booted a letter out
 
Oh geez... that would be a long one no doubt. Ok. I’ll stop now.
 
:)
 
6:02 PM
@Wis It probably could, but as others have already posted, there are probably available libs specifically targeted for this task.
 
@Dodge FWIW, "slag" is an insult in British & Aussie slang. It's like the American "bimbo", but probably worse.
 
I was tempted to ask whether that word existed in the US but skipped it. My typo made me chuckle :)
 
"Slag" was the racial slur referring to the alien refugees in Alien Nation.
 
Good to know that unless one is in a foundry it’s probably a bad idea to use the word slag, Earth or elsewhere.
 
Always brightened up my chemical engineering lectures
I think the answer is "no", you never really get tired of the word
 
6:15 PM
Mind you "Bimbo" was an innocent term until the late 20th century. See en.wikipedia.org/wiki/Bimbo_(song)
As an insult, "slag" doesn't just imply a woman of loose morals. It also implies ugliness.
 
Trollop is a much better, though less used, word. It happens to be my favourite word. Something about how it sounds.
 
Trollop sounds like a tasty treat or a small innocuous creature of some sort
 
@roganjosh It's a good word, and probably a more sophisticated insult than slag. But I think slag is probably a worse insult.
 
Far from a tasty treat. Always read the label.
@PM2Ring so it exists over there, too?
 
Trollop has the implication of prostitute, slag doesn't. But this conversation is getting difficult to continue without breaking Stack Exchange rules...
@roganjosh Sure, but I don't know if the teens & 20 year olds use it. Older people certainly know it.
 
6:25 PM
It comes with all sorts of implications, but your distinction isn't one of them. If anything, the opposite would be true in how I understand the word
 
@roganjosh Ah, ok. Over here, a slag is too unattractive to be a prostitute.
 
@Dodge The US stole one of our shows an made it unbearable (at least for me). Shameless is set in the city I live in and the type of place I grew up. It's an absolutely fantastic series but it tried to run for too long. Not ashamed to say I cried along with one of the main chars at the end of the 1st series
 
I wonder if slag as an insult is more popular in steel towns? Sydney never had steel mills, but Newcastle to the north and Wollongong to the south did.
The steel mills have been closed for more than a generation, and they're now both thriving university towns.
 
I would think it's pretty universal. It's used a lot here
I would expect it's used no more in Manchester than in London
 
@roganjosh "The US stole one of our shows an made it unbearable" They do that a bit, eg The Office. But they'd never steal (or comprehend) Coronation Street :)
 
6:33 PM
Interested thought PM about the use of that word. I’d say yes without knowing anything. @roganjosh Shameless is the title you say? I’ll check it out sometime
 
Oh God, please do not think that represents my city!
Coronation Street needs to go out to pasture. Please.
@Dodge yes, Shameless. But not the American version. I had to stop watching that.
 
OTOH, they turned Steptoe and Son into Sanford and Son, which kind of worked.
 
There is a mega divide between US and UK humour
@PM2Ring not saying it can't work, but I was really upset with the Shameless translation
 
@roganjosh I hated it since I was a kid. It's pretty amazing that it's managed to last so long.
 
But then again, maybe I just lost the connection of relatability
@PM2Ring We can't even address Coronation Street before tackling Eastenders
 
6:39 PM
@roganjosh Aussie humour has been influenced by both.
 
Because I know which I'd rather be tortured with
 
Eastenders is for people who think Coronation St is too sophisticated. ;)
 
Lol, my sampling method does not like that input at all
 
I am using pool.map() to multithread some python code. The function called returns an list of list. I'd like the list returned by each thread to be combined, instead of being appended. Eg code here. Any suggestions?
 
dataset.extend([10,20])? Does that work?
Also, you're not multithreading. This is multiprocessing and there's quite a lot of overhead in maintaining a pool of workers. Unless the data is gigantic, there's a good chance you'll get slower results from this
 
6:49 PM
that would make the list returned by function into a single list. To be specific, what I am doing is I need to process 100s of Ks of files. each file gives one sample for my dataset. The function I have written handles files related to one class, hence returning a list of list. Maybe I could use a matrix of some kind? But even then, wouldn't the matrices append to each other in a wierd way, instead of combining
 
I don't follow at all. I think you need to describe the problem more clearly before we try work on that code.
"wouldn't the matrices append to each other in a wierd way, instead of combining" what is the issue you envisage?
 
I guess I will get a list of matrices, instead of a big matrix?
I'll try to explain in detail
 
Are you sure you need multiprocessing? Have you tried this wirh threading? How many cores does your CPU have?
 
I think we need to take a step further back still.
 
I have 100K's of files, belonging to some 50-100 classes. I am supposed to read each file, process the data, and get one row of output, something like [0, 1, 2, 1, cat]. Now I wrote a function which processes all files related to one class at a time. Then I create pool and assign one class to each process
 
6:56 PM
If you do need multiprocessing, I can't offer much help, but I have done a bit of threading.
 
(Also question for later) Why is it called threadpool if they are processes?
So one function call will return all rows of data for cats, as list of lists
Then from each function call, I have list of lists for cats, for dogs, for unicorns, and so on.
 
Do you have something working without multiprocessing?
 
@roganjosh Yes
 
Can you share it?
Off the top of my head; get a 2D lists of files the fall into each category (assuming they can be filtered by name) and distribute those lists out to different Processes
 
I can't. It's open sourced, but it's meant not to be shared yet. It's a network security project.
 
7:03 PM
That's fine. But I'm wondering if there can be a pre-processing step that can segment the work and you can pass it out to Processes rather than using a Pool
 
@roganjosh As of now, function takes a single string input, and finds all relevant files based on that input.
 
But, as you said, you have different categories. So is there nothing that distinguishes all the files other than parsing the lot?
 
There is some misunderstanding
 
Or you have to read the files and they contain mixed categories
 
files are distinguished based on names. But their content needs to be processed to get the output, which is a row vector per file.
one file == one row in dataset
 
7:10 PM
Ok, and I'm asking whether there is any categorical data contained in the file names that would allow you to batch similar files together
 
Yes, the file names are class name + random string
 
And really, if each file contains 1 row and we're talking about a few thousand files, is this even worth it?
 
It's taking hours to process the whole stuff :( . And more data is incoming
 
You could probably read several thousand files in the time it takes to launch your processes
Are you appending to something like a dataframe or numpy array?
No. It's a list. Something feels wonky about this
 
finally I convert it into pandas dataframe and then to csv
 
7:15 PM
@RegisteredUser So you really don't need to have coordinated multiprocessing. You can run multiple independent invocations of the script. One invocation processes all the cat files, another all the unicorn files, etc. And then have another script to combine the processed data.
 
You can launch multiple processes and feed them their own batch from within the script
There's something else going on here I think
 
@roganjosh True. I'll just shut up & watch. :)
 
If you have a loop appending to a df, for example, you'll get catastrophic runtimes
 
So you say I should write the output of each class to a file, and then combine them?
 
No, absolutely not
What I was saying is that you might batch the files, based on their names, and hand each batch to a different process
but I think your runtime problem happens elsewhere and this is an XY problem
 
7:22 PM
@roganjosh But wouldn't this be similar to what I already have? only difference being each process is finding it's batch of files on it's own. The finding files + reading them is not time consuming. Processing each file is very time consuming. some files take few seconds to get processed.
 
A single line in a file cannot take seconds to process
unless the file was totally ridiculous
 
It's not a single line
 
You said earlier "one file == one row in dataset"
 
Some files are few MBs in size. Most are around 100KB
Now I get where the confusion came from :)
@roganjosh Sorry. I meant one file converts to one row in output dataset.
 
Ok, so it's a parsing problem
And we've established that you might be able to group files into categories based on their name
How long does that take? take all the file names, sort (hopefully) by the category, then use itertools.groupby
os.listdir will give you the list of file names
 
7:31 PM
Should be pretty quick
 
What is the shape of the chunks? How many sublists do you get?
 
each sublist(each row) is of same length. Number of sublists per class would vary. So basically a matrix of variable rows by constant columns.
 
It wouldn't be after groupby
I'm interested in the size of the groups because we might divvy them out between processes
 
Some files are corrupted, or have other issues detected only at runtime. But take around ~4000 per class and around 60 classes.
 
Ugh, you haven't made this easy :P
I'll have a go at putting untested code together
Actually, I think it's too much of a stretch to even be useful
This is what I might do (no idea if it will work)
1. batch the 60 classes into lists of cpu_count -1 so at least we have target file names
2. Pass each list to a Process
 
7:43 PM
@RegisteredUser Let's say you are processing the 'cat' class, which contains multiple files. To what degree could the files be processed in parallel?
 
3. Set up a list-type Manager to collect the results
 
And can each class be processed independently?
 
@QuestionC curious where this is going. How could they know that?
 
@QuestionC You mean single file being processed in parallel? The parser doesn't support reading nth line in constant time, so that wouldn't be possible
 
We're trying to avoid a file being parsed by different processes
That's the point of batching the file names
 
7:48 PM
I mean if you have cat1.txt and cat2.txt, they can be processed independently, yes?
 
Yes
I think that was a derailing
@RegisteredUser did you follow up to (3)?
 
@roganjosh This should work. But as you said, it's lot of effort :P
 
Oh man it's a lot of effort
But you will control what process does what, rather than throwing to a pool and having all the overhead of managing that and somehow working out the categories
I assume I know who starred my messages, bu they are different to upvotes :) I try harder to make witty quips to be starred, honestly.
Though it does mean I dislodged on Aran on 3 messages on my starboard :P
I also know the party-pooper that is getting rid of them :P
 
I'm trying to catch up on the problem but the standard "I gotta parallelize this" approach is to just have an input queue, an output queue, and however many threads you want.
Instead of each file individually you could do it by class. Makes the code simpler since you presumably have a step of putting the results for each file into a vector or something.
 
For I/O, sure
 
7:59 PM
Not everyday I get to witness 3 supernovas within 5 seconds. RIP roganjosh's stars.
 
Burned too brightly I guess
It's a common problem for megastars. I've come to live with it. Now I'll just suck everyone else into the crushing defeatism.
 
What's wrong with just using a thread queue to hold all the class names and letting your threads consume it?
It shouldn't matter if I/O or processing is the problem. It sounds like processing is the heavy lifting so for the most part you're going to have all cores crunching.
 
Uh oh, seems like those stars turned into a black hole...
 
Tis the joke. You're already lost
 
8:03 PM
Oh. Guess I'm a bit slow :D
 
:P
 
The thing is if processing's the bottleneck, python threading only helps so much. Slow / 4 is still slow.
 
threading slows python down
 
The problem is I/O based
 
> The finding files + reading them is not time consuming. Processing each file is very time consuming. some files take few seconds to get processed.
 
8:06 PM
guess I'll not stick my nose into a topic I haven't been paying attention to. Back to youtube I go
 
I thought the python slow threads thing was a thing of the past
 
The GIL hasn't gone anywhere
I explained how they might batch the files and use separate Processes to parse the files
 
Using a manager seems to be a good idea. That way, I could use multiple computers.
 
ooo, I'm not so sure about that
 
But then I could use Hadoop
 
8:10 PM
This is not trivial
 
But I won't. Because I don't want to get entangled deeper into this!
 
Batch the files and send them to different processes
if that doesn't work and you want to look to Hadoop (which really is overkill for reading files) then you're probably in for a shock
 
Yes. That seems to be the reasonable solution. For now, I got to complete my TA work.
 
8:59 PM
new wired stuff static and dynamic name resolution sometimes the documentation is writen in a hard way for learners
 
yo
any selenium users here?
have a lil issue
my browsers keep me logged in on certain websites, but when accessing those pages through webdriver.get(url), I'm logged out in this particular browser window. when I use the manually opened page I'm normally logged in
trouble is it requires a mobile number and then a code for this particular page I'm trying to automate
 
9:15 PM
@PawelFlajszer I don't know selenium, but that sounds like a cookie issue. And I guess the phone number thing adds an extra layer of complication. Does the site let you visit that page on multiple browsers / devices simultaneously?
 
hi PM 2Ring. No, it doesn't. 1 active tab at a time
it's weird because if I manually open a new window and visit the site I'm already logged in, but when the automation is on it acts almost like it's "incognito mode"
 
Hey guys
Why does my FOR loop start at 0, even though I specified x to be 1?
/**x = 1

for x in range(10**2+1):

BLAH BLAH**/
 
9:31 PM
cause it's not the same x
if you write
 
oh what
 
for item in range(10**2+1)
it'll work exactluy the same, but you then have to use "item" to refer to each number in that range
 
@BigCoder101 You need for x in range(1, 100):
 
x=1 before the for loop doesn't make any difference here
 
ohhh, foolish me
thanks guys
 
9:33 PM
Also, /* comment */ doesn't work in Python.
 
yeah. either # or """
PM 2Ring you think clearing cookies would help? it seems to me like it's a safety feature of browsers that when they're in the automation mode they automatically log you off all the sessions
 
@BigCoder101 And if you try to change x inside the loop, it'll only work temporarily, until the bottom of the loop, and then x will get set to whatever the next x is supposed to be at the top of the loop.
@PawelFlajszer Not sure. The phone number authorization thing is probably the killer. I assume selenium is smart enough to handle cookies automatically under simpler circumstances. But it's probably a good idea to read the selenium docs relating to cookies, since that authorization stuff is related.
 
well it's the same on other websites too. always logged off
but that could be sorted somehow, phone number thing is really crap
like... my app runs on pyautogui now, but it's much better to search for html tags, classes etc, then to look for pixels on the screen... not really responsive, nor reliable
 
10:13 PM
"As far as I am aware, each instance of loading a FF with Selenium opens a stripped browser with no history/cookies/extensions etc. It's as if you have just download FF for the first time and its a fresh copy. I am not sure how to solve this issue. Maybe if you keep the session open by not closing the browser down but log out manually and not closing down the browser. Or open a new tab and opening the same address."
 
 
1 hour later…
user11093202
11:34 PM
@PM2Ring It only works in javascript and I think Java
 
user11093202
Also, @PawelFlajszer dont cuss
 
I think saying 'crap' is ok
 

« first day (3094 days earlier)      last day (1855 days later) »