« first day (2826 days earlier)      last day (2121 days later) » 

12:02 AM
cbg
there is an answer here, that i can't get to work: stackoverflow.com/questions/44439586/…
it's the only answer to the question, it's a python script for accessing google cloud print ; but the main method refers to a refresh key and i can't see how to get that. There is a brief mention of it in the answer but it makes no sense to me
 
Rhubarb all.
 
 
4 hours later…
4:07 AM
Hey i'm working with some code that makes use of 'GetUrl' ; but I'm working in python 3 not 2.7 ; is there some way I can still access GetUrl in python3?
 
 
1 hour later…
5:27 AM
cbg
@wim what has been going on with Chicago weather lately, I visited a month ago and it was 20 degrees hotter than Texas and then it was perfect the last few weeks
 
5:42 AM
cbg
@Simon nice. Also, someone's inching closer to that 5k mark!
 
 
1 hour later…
7:09 AM
cbg
 
7:26 AM
Hello All,
Needed a quick help regarding a situation.
I am trying to check a condition using if statement but the tricky part is is that if the condition is satisfied I don't want to do the action for that line but the lines that immediately follow.
Any help/ideas?
 
if not condition:
    that_line
else:
    the_lines_that_immediately_follow
^ like that?
 
@shad0w_wa1k3r Yeah it's slow going though.
 
7:41 AM
@Aran-Fey Not exactly getting what you meant
 
can you explain it in example?
 
Ya sure.
I have a text file with a number of lines. I want to an append some number to a list if a line starts with 'open' but not to the line that starts with 'open' but to the next line that follows. The line that starts with 'open' will have a NULL appended to it.
But now the working solution that I have is appending numbers to the line that starts with 'open' as well as I am not being able to achieve the neeeded solution
 
I still don't get it. Could you show us an example? 3-4 lines of input, and the desired output?
 
like:
line
another line
open line
hey please append to me
line

this?
 
7:54 AM
you split the file by "\n" symbol, then you go through the loop
lemme try writing example for you
>>> multiline = '''line
another line
open line
hey please append to me
line
open another line
and me too
line'''
>>> splitted = multiline.split('\n')
>>> for x in range(len(splitted)):
	if splitted[x].startswith('open'):
		try:
			splitted[x+1] = splitted[x+1] + '*appended data*'
		except IndexError:
			splitted.append('*appended data*')


>>> processed = '\n'.join(splitted)
>>> print(processed)
line
another line
open line
hey please append to me *appended data*
line
open another line
and me too*appended data*
line
>>>
and thats the solution
then it is up to you - write that back to file, or continue doing something else
 
8:25 AM
Anyone available and knowledgeable in Pandas?
 
cbg
I thought the World Cup was tense last night but the new battle seems to be the upvotes/downvotes on this question
@Stacksofoverflow What is your question?
 
@roganjosh I have a dataframe with 3 cols: datetime, user_id, minutes_consumed. Based on 3 predefined time windows (ex: 4am - noon) I want to find which window a user consumed the most content based on minutes_consumed.
I tried splitting the dataframe into 3 based on the windows but I don't know where to go from here
 
I have a feeling Jezrael gave an answer very similar to this not long ago, let me see if I can find it
 
@roganjosh thanks; I've been researching a couple hours and can't seem to find how to do it
I know there is a rolling_window functionality but not sure how to apply (no pun intended) it.
 
You don't want a rolling window because you have pre-defined limits
 
8:33 AM
ok glad I didn't waste my time chasing that dragon then
I'm also trying to avoid cumbersome for loops if possible
 
If you've already split the DF into 3, maybe you can use something like:
df = pd.DataFrame({'user_id': [1, 2, 3], 'mins_consumed': [4, 5, 4]})
user_id = df.loc[df['mins_consumed'].idxmax()]['user_id']
 
I want to find the window for each user
So user 123 watched most content in window 1
125 watched most content in window 3
etc
 
oops, I read it back to front
 
No problem
appretiate the time
 
I have some ideas but I have quite a bit on at the moment. I will ping you if I get chance to test them and if you haven't got an answer by then
 
8:45 AM
no problem, any general direction you can point me in?
thanks again
 
I would probably start by adding a new column to give a category number for each datetime
i.e. what window it falls in
you can then groupby that value and user id and sum up the time spent in each window
 
ok thanks @roganjosh will keep trying
one issue I come across is not all the users ids consume content in all windows
so joining/merging options don't seem feasible
 
Sorry, I don't know what you're joining or merging
You said that you had a single df with 3 columns?
 
cbg
 
8:59 AM
Cabbage Robert, roganjosh
 
cbg Simon
 
if I could merge the 3 dataframes after grouping I could find which column has the max value
 
My approach would be not to split the df in the first place
 
whats cbg btw? embarassed
 
You're just adding a new column for a category
@Stacksofoverflow sopython.com/salad
 
9:02 AM
@Stacksofoverflow More percisely sopython.com/salad/?highlight=Cabbage+%2F+cbg
 
We have our own mini language here :)
 
With my soln you split the dfs, groupby + sum, then you can merge/join them to find which column has the max minutes watched value?
 
I'm pretty sure that makes your life harder
 
9:05 AM
My IDE is pretty much locked up with testing my simulations so I can't get a chance to put an example together :/
 
ok, want to PM? can guide me through your idea
 
Now hating: Google Chrome
anyone can tell how I can view "Last Modified" on Google Frikkin Chrome?! The "For dummies edition" please!
 
Whatever criteria you're using to split the df, just use the same thing to add a number to a new column to start rather than splitting it into a new df
 
@heather depends on what you're going to do with it. But anyway you'll probably want a tuple or similar; either to do first+second, first-second or first_sol, second_sol later.
 
@roganjosh ok
@roganjosh I'm guessing there's someway to group now based on user ID and max minutes?
then just get the window column?
 
9:25 AM
The first thing you need to do is groupby timeslot and user_id and sum up the time spent
 
yeah I have the window groupby done
I did it before assigning the window
 
Cool, so then use a similar approach to the one I started with originally
 
I just need to find the max for each user
+ the associated window
df_sliced[['UID', 'mou', 'Window']].groupby(['UID']).agg({'mou': max}).reset_index()
so I think that's the call
but it's not returning the window columns
column*
 
You can edit/delete messages for 2 minutes after posting
 
derp nevermind
UID Window mou
0 34 1 34424
1 34 2 35885
2 34 3 10417
still returns all the rows
df_sliced.groupby(['UID']).agg({'mou': 'max'}).reset_index()
this works, but if I add window to the groupby it doesn't
 
9:46 AM
Because then it searches for each user and window
Don't reset the index and use the index to get the window?
 
df = pd.DataFrame({'user_id': [1, 1, 1, 2, 2, 2],
                   'mins_consumed': [1, 2, 3, 6, 5, 4],
                   'time_slot': [2, 1, 3, 1, 2, 3]})
df = df.sort_values(by='mins_consumed')
df = df.groupby('user_id')['time_slot'].last()
There's probably a cleaner way but I'm rushing sorry
 
10:07 AM
@Stacksofoverflow did it work?
 
@roganjosh inputting now
sorry got some water xD
 
Unacceptable :P
 
@roganjosh it does!
do you know how to boolean index with timestamps?
currently I'm still splitting the dataframe into 3 to add the windows
 
"With timestamps"? Boolean indices should be...boolean
but timestamps support relational operators if that's what you're asking
 
What is a boolean index..?
 
10:09 AM
(time1 <= timeseries) & (timeseries < time2)
@poke a numpy array with bools inside, works as a mask along a dimension
 
df.loc[(timestamp < some_value) & (timestamp > some_other_value), time_slot] = 1?
 
ah, filters
 
yup
 
ok thanks @AndrasDeak
 
@poke (sorry, not python) how harmful do you think it could be if the last invasive line is kept in this git log answer?
it's only a copypasta of aliases, but still I'm a bit concerned
 
10:20 AM
Seems like there’s a matching gwip alias missing
but yeah, seems like a bad idea to include that as part of a “here are a bunch of log output aliases”
 
thanks, I might remove that then
 
10:50 AM
@roganjosh since you seem knowledgable I'll drop another question here
I have a groupby with the following format
UID Genre
34 Drama 29524
Comedy 10275
Movies 8914
74 News 31917
Romance 27703
Drama 27259
Is it possible to get this in the format
UID Genre1 Genre2 Genre3 etc.
where each row is a UID
 
@Stacksofoverflow I think you probably want to read this answer
 
watermelon
 
11:08 AM
@roganjosh is there any method to get a mapping from UID to the string values from 'Genre' above for each user in the groupby?
This is somethign else I've been researching that I cannot find
Or the solution is extremely slow
 
11:20 AM
i have a simple python function that sends email if log file isnot updating
I read a configfile and if my section is logging_active then i run this function in a thread, else i run some other function in a thread. However i think i am doing something wrong in the threading part
my try is https://pastebin.com/89PbcqpP
 
jpp
11:32 AM
 
@jpp why don't you hammer yourself? Looks good enough to me
 
jpp
@AndrasDeak, I made a mistake of attaching a wrong duplicate and reopening (I should have changed the target). Now it's locked.
 
Ah, yes.
 
11:48 AM
@Stacksofoverflow as in, as a dictionary?
 
yeah that would be awesome
I need to filter another dataframe based on these groups
 
I'm not really sure what you're looking for, does to_dict() not do it?
 
'UID': 'Genre'
just calling to_dict() on the groups will do this?!
(12724882, 'News'): 227,
 
So I think you probably just need to play around with them for a bit
 
cool thanks again man
 
11:54 AM
You're welcome. If you have future general questions, it's better not to direct them specifically to me btw, since there are other people in this room far more knowledgeable so it's better just to open it to the room. Also, I was on lunch :)
 
Ok understood.
df_group = data[['UID', 'mou', 'Genre']].groupby(['UID', 'Genre']).agg({'mou':sum})
g = df_group['mou'].groupby(level=0, group_keys =False)
g_final = g.apply(lambda x: x.sort_values(ascending=False).head(3))
currently applying those operations
mou
UID Genre
34 Drama 29524
Comedy 10275
Movies 8914
74 News 31917
Romance 27703
however I cannot access the UID and Genre columns...
It outputs that
 
IIUC you now have a multi-index
 
g_final.index.names
outputs UID and Genre
but when I index those names I get key error
 
jpp
@Stacksofoverflow, Just a hint, you're more likely to get faster and more helpful responses if you post a question on SO rather than here. For one, code formatting is poor here.
 
12:10 PM
yeah I understand @jpp
but I have cascading questions
don't want to make a spammy post
 
yeah, those are not a good fit for SO
 
once complete I'll probably summarize and self solve on SO
 
and you'd need to work on the questions a lot before posting on main
 
yeah once my issues are solved I'm going to make a clean post for other people looking for similar answers
 
Have you done any pandas tutorials?
 
12:11 PM
A couple
 
jpp
Yep, SO as a concept is Q&A rather than a help-desk. Not a criticism of your problem or how you're laying it out. Just don't think it's the right place.
 
I agree! Just don't know where else to turn haha
 
Depending on how localized your problem is (and it sounds pretty localized) I'm not sure others would benefit a lot once your problem is solved. There are already hundreds of specific problems with solutions on SO :P
 
Well for example the groupby stuff could help people for sure
 
Are you sure there aren't already posts that demonstrate the groupby stuff?
 
jpp
12:12 PM
There are thousands of questions on pandas groupby
 
3 days of searching...
 
I agree with Andras. I think if you try formulating this as an MCVE you'll either find the solution yourself in the process or you'll have a question that you could post
 
jpp
@Stacksofoverflow. You can play it this way. If you think you have a Minimal Complete & Verifiable Example of your problem that isn't answered elsewhere, post it on SO. I think it's highly likely we'll find a duplicate, but we may be wrong :).
 
or else jpp can FGITW answer it :P
 
jpp
Sometimes it's a matter of knowing what to search for.
@AndrasDeak, Haha, that's also an option.
 
12:14 PM
Ok well you can scroll up and see what was solved. Kudos if you can find it. I looked and also @roganjosh looked
 
Getting your post dupehammered should be a happy occasion, because someone found the solution to your problem
 
exactly haha
 
yup
 
jpp
Make sure you choose a good title and tag correctly. We want duplicates to be good sign posts :)
 
@Stacksofoverflow I did start losing track of what you were trying to do in the end though. Just the process of formalising the problem into an MCVE solves a lot of problems I have
 
12:16 PM
Sounds good guys :D I'm on a tight deadline which is why I went this route initially
 
"Here is the exact code required to get my dataframe. When I do df.genre, it gives an AttributeError. How do I get the genre column?" Seems like something that would get answered on the main site in like a minute
 
plot twist: df.genre is already the genre column if it exists
cue "column doesn't exist" canonical
 
Scrolling up I see you're calling it "a groupby" so maybe it's not the same kind of object as a dataframe. idk, I am a simple llama farmer that wandered into this public library whose computer already had this page open
 
jpp
In my opinion, be explicit: use df['genre'] for a column, df.loc['genre'] for an index, df.genre for an attribute. These are all different things. Pandas (IMO) confuses the issue by allowing the period for multiple purposes.
 
I don't know why they chose to support that syntax
 
12:21 PM
convenience, which is a huge driving force for pandas
 
jpp
To me, df.genre is not more convenient than df['genre'].
 
also explains a lot of the "multiple ways to do things" in my opinion
@jpp to me it is :P
 
it's a groupby object
so when I call those columns it says key error
 
jpp
@AndrasDeak, Fair enough :). At least on SO answers, I always advocate the explicit way. The last thing you want is having to explain an error when a space is added to a column name.
 
KeyError, AttributeError, whichever. The important part is "here is the exact code required to get my [object]"
 
12:23 PM
I'm not sure I agree with the existence of the feature, but df.a < df.b is much much simpler to type than df['a'] < df['b']
 
Which is a step above "here is the output when I do print(my_object), I assume all you readers already know how to turn this back into something useful using the special magic incantations handed to all numpy users during their induction ceremony"
 
the problem is this
df['life lover'] works
df.life lover
does not
 
yes, hence jpp's remark about a space
 
oh sorry
 
same thing as creating a dict using keyword arguments, really
dict(a=2, b=3) # OK
dict(2='a',3='b'): # not OK
{2:'a', 3:'b'} # OK
interesting, in def foo(**kwargs) the keys must be strings but not necessarily identifiers
>>> def foo(**kwargs): print(kwargs)
...
>>> foo(**{2:'2'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: foo() keywords must be strings
>>> foo(**{'a b':2})
{'a b': 2}
 
12:33 PM
Visually though, using df['key'] syntax does help with one-liners which can get out of hand and with approximately a billion methods and attributes in the library, I find it better to have that clarity
 
AttributeError: 'SeriesGroupBy' object has no attribute 'columns'
im losing my mind
has no columns or levels attribute
so where the fk is the info stored lmao
all this should be done using SQL not fkn pandas
 
jpp
why would a series have columns
dataframes have columns, series have an index and values.
compare a series to a dictionary [assuming index is unique]
a dataframe is a combination of series to make a table.
 
UID Genre
34 Drama 29524
Comedy 10275
Movies 8914
74 News 31917
so why is it giving me column header
 
jpp
Because print doesn't determine data structures.
I strongly advise you read a good book / tutorial on Pandas. Unfortunately, SO (main or chat) isn't the best tool for this. Not because we don't have the knowledge, but because there's simply too much information to impart and we don't know exactly where to begin.
 
Yup I'll stop
any good texts you recommend?
 
jpp
12:41 PM
I'm a fan of the official docs, it's actually well indexed so you can pinpoint the bit you want fairly easily.
 
Ok; thanks.
 
jpp
It includes a page on Tutorials, if you like the tutorial-based approach.
 
I guess while everyone is annoyed at me I'll ask one last question
 
I couldn't really recommend a book, I'll leave that to others, but another approach you can try is going through the pandas tag. You'll quickly realise who the most prolific answerers are. It's worth going through their approach to solving problems because the docs won't necessarily give you a sense of the clever ways methods can be combined
 
Yeah I'm going to go with a mix of tuts and reading
the problem is I'm neck deep in projects that use this software atm
UID Genre
0 34 Drama
1 34 Comedy
2 34 Movies
3 74 News
4 74 Romance
5 74 Drama
Is there anyway to map the unique UIDs to their associated row values?
Last question I promise...
{34: [drama, comedy, movies], ... etc}
 
jpp
12:45 PM
86
Q: grouping rows in list in pandas groupby

Abhishek ThakurI have a pandas data frame like: A 1 A 2 B 5 B 5 B 4 C 6 I want to group by the first column and get second column as lists in rows: A [1,2] B [5,5,4] C [6] Is it possible to do something like this using pandas groupby?

 
that doesn't look like what I want...
ok maybe if I call dict it'll give me what I'm after
sorry everyone
thanks for not flaming me
yeah im running into speed issues again aarrgg
is the pandas code parallelized well? I notice whenever I monitor cpu pandas uses 1 core
maybe it was just the use cases I monitored it for
 
Huh... apparently a module I installed placed its tests folder directly into my site-packages directory, and now my pytest imports those tests instead of mine... that's really not what I expected the problem to be :D
 
@Stacksofoverflow native python code is subject to the global interpreter lock (GIL). Lower-level stuff can go around that but that happens under special circumstances (such as numpy calling multithreaded blas)
 
Right... so Pandas doesn't implement any multithreading natively?
I'd have to use the python module to optimise where it's possible?
 
I'm not clear what you mean
 
12:57 PM
^
 
Or try and turn what I'm doing into numpy operations?
Some python packages such as numpy
use blas for multithreading
 
I merely meant that most library code runs on 1 CPU
 
Well, lambda will run in python time. So if there's some pandas/numpy method to get the same result, that will probably be faster
 
If you need pandas use pandas. Numpy won't help you with db manipulations
@roganjosh that's another issue about stuff implemented in C, don't confuse them :P
 
that's what I mean some packages compile into C/Cython or whatever
and are optimized
it seems like everything pandas does is not
 
1:00 PM
There's a lot of stuff that's vectorized but single-core, and yes, apply+function will be even slower
 
Yes, don't assume that all pandas stuff will run faster simply because you're using pandas
 
I just don't understand why pandas is all single core
while something like numpy isn't
is it a development decision or a restriction based on what's being done?
 
That's not the case. Pandas will utilise numpy methods when possible
 
oohh ok
@roganjosh that makes sense then
 
1:02 PM
@Stacksofoverflow numpy is also single-core
 
ok then I clearly have no idea wtf i'm talking about
 
But does that count the Cython code, Andras?
 
I get the GIL, how does using multithreading python module go over this?
 
except when it isn't. Emphasis on except.
 
1:03 PM
multithreading doesn't
 
There's no multithreading module
 
so this works for multiple CPUs
 
@roganjosh probably does
 
but not cores?
 
Threads in python are still bound by the GIL, they only give the impression of concurrency
 
1:04 PM
this is actually something I've been trying to understand for a while
because when monitoring resources some python will use all my cores
and most won't
 
@AndrasDeak which threads are we talking about here?
 
@roganjosh I insist on "threading or multiprocessing; there is no multithreading"
Users misspeak way too often
 
Fair enough, then we each had a 50/50 chance of being right in our earlier exchange since multithreading could go either way :P
 
in the face of ambiguity resist the temptation to guess :P
 
1:09 PM
multiprocessing copies the name space and passes it to multiple processes, each of which is bound by its own GIL
 
when you say passes it to multiple processes
what does that refer to
 
I will very quickly get out of my depth down this route, I'm trying to find a decent resource for you
 
what I can't understand is why some python uses all the cores of my machine and some use only one, is it just the case that most python code can't be compiled "far down" enough to be passed to multiple cores?
idk if that makes sense
but I can see python using all cores in some cases and only 1 in others
 
When you say "some python uses all the cores of my machine and some use only one", how can you tell?
 
1:14 PM
I open up my terminal
type
"top"
it shows % cores being used and what is using it
IE 400% Python3
or 100% python3
 
Most likely in the 400% case, the Python code is calling C code. The C code isn't bound by the GIL.
 
right so my question is
why doesn't more python code do that haha
for example pandas which has so many operations which would benefit from being parallel
 
when you say "more python code" I worry that you assume that some Python code isn't bound by the GIL, when in fact 100% of it is
 
right
so more python libraries sorry
 
(ignoring multiprocessing since it hardly counts if they're in different processes)
 
1:17 PM
is that a better way of wording it?
I understand the GIL lock and don't understand how it relates to packages that compile down to C and why more don't
how's that?
is this case pandas haha
 
Pandas used a lot of Cython to compile the code
 
I see because when monitoring a lot of pandas code that seems like it would be easy to make parallel it isn't...
I'm ignorant about it's probably way harder than it looks of course
 
\o cbg
 
cbg
 
I find it kinda hard to answer "why doesn't mode packages compile down to C to get around the GIL" (whatever compile down to C means). It's more or less up to those who maintain the Library/created it. Seems like an opinionated question which has no short sweet answer...
 
1:24 PM
Just picking up a conversation that went on too long in comments here (stackoverflow.com/a/17811862/575530) with @MartijnPieters. Does anyone have first-hand experience using screen-readers with Python and how they voice indentation? Martijn recommended looking at NVDA, EdSharp, and Orca.
 
at least to me that is ...
 
@Stacksofoverflow the thing with python is that you use it for convenience. For critical parts library maintainers might consider implementing parts in C (or fortran), but the default is, you know, python
much harder to both write and maintain C code
 
pandas is a Python package providing fast, flexible, and expressive data structures
pandas is fast.
I guess they mean relative to other python packages?
 
and compared to rolling your own loops to do all those groupbies
 
All measurements of speed are relative if you think about it :-)
 
1:31 PM
as I said, vectorization is a thing
 
but shouldn't a lot of these operations be handled in something like an SQL?
Which are actually fast?
 
Nothing stops you from doing that. Python and speed-critical don't mix well together.
 
Right but I think that misleads people haha
to think that pandas is a replacement for sql
 
Where do they market themselves as a replacement for SQL?
 
I'm not saying they explicitly do
 
1:35 PM
I bet pandas can do many things that sql can't do. I'm just a llama farmer, so I don't have any specific examples, but I bet there are some.
 
yeah I'm playing devils advocate for sure
these are just things I think about
and also deal with issues because organisations I work with hold those philosophies
I don't know where they get them
but a lot of people seem to treat pandas as an sql replacement
and don't recognise how slow it is for large data in a lot of cases
this is more personal venting and strife than anything...
 
MAX
python bag
 
TIL that regex match objects can be indexed
>>> re.search("foo", "foobar")[0]
'foo'
I've been using .group() up till now
Still can't iterate over them, though, which is weird since I thought we determined earlier this week that __getitem__ by itself is sufficient to support iteration
>>> class Foo:
...     def __getitem__(self, idx):
...         if idx > 5:
...             raise IndexError
...         return idx
...
>>> [x for x in Foo()]
[0, 1, 2, 3, 4, 5]
>>> import re
>>> [x for x in re.match(".", "coconuts")]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '_sre.SRE_Match' object is not iterable
 
1:57 PM
Interesting. I suspect C-level hacks.
 
Maybe SRE_Match explicitly implements an __iter__ method that unconditionally raises a TypeError
 
>>> re.match('','').__iter__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: '_sre.SRE_Match' object has no attribute '__iter__'
 
Ok, changing my vote to "C-level hacks"
Really, I shouldn't have expected it to work in the first place, since the documentation does say you need __len__ as well
Undocumented behavior in Foo should not lead me to expect it anywhere else
 

« first day (2826 days earlier)      last day (2121 days later) »