« first day (3120 days earlier)      last day (1820 days later) » 

1:14 AM
>>> l='c**b**e*'
>>> s='abags'
>>> s2 = iter(s)
>>> ''.join([next(s2) if i=='*' else i for i in l])
'cabbages'
>>>
 
2:07 AM
Hey whats up everyone
Does anyone have any idea on how to binarize certain words. I have a column in a pandas dataframe that has certified and denied as object values. When I print to excel I want those values to print 1 as certified and 0 as denied.
 
 
1 hour later…
3:13 AM
Flask-Mail on pypi is no longer being maintained. What alternatives exist?
 
 
1 hour later…
4:19 AM
@PM2Ring Appreciate it! It kinda hard but I'll be fine.
And that's it! All I need to do now is collect my diploma in a couple weeks.
 
need help getting 404 when downloading pdf from django
 
@Martlark github.com/mattupstate/flask-mail/issues/169 asks the same question and has a comment which suggests to copy the Django one ... from a quick look Flask-Mail doesn't seem to offer very much by way of Flask integration so any module to wrap email and smtplib would get you 90% there
@LukeLuvevou 404 means wrong URL basically
there's an email package author who often posts a link to his offering on questions which seems pretty slick for general-purpose email, the name of the package escapes me at the moment, something like dsmtp or demail?
 
5:02 AM
hi guys... how do I parse package name versions compatibly from the filenames, i.e. spam-0.0.1.tar.gz or similar I want to get spam and 0.0.1, was there anything in setuptools/pkg_resources/whatever...
and I am on python 2
or will I throw in a regex?
 
@AnttiHaapala in the completely general case extracting the setup.py from each is probably the most robust
 
@tripleee can't
I have the name only
the name conforms to what the setup.py would build however.
 
if you don't have weird cases like package-2.1-name-continues-3.14159pi2-version-continues-random-stuff-123 then I guess a regex could handle it
 
the complications are: there are package names with dashes, there are package names with dots, there are package names...
 
exactly -- the completely general case is probably not easily solved with a regex
but if you have a constrained set of packages then a regex which handles just those cases can probably be reasonably simple
 
5:08 AM
hmpf
 
6:01 AM
@cs95 Good Luck! Hope it went well!
Thursday cbg :)
 
6:25 AM
@AnttiHaapala Are you trying to do this programmatically, from outside on a (Linux) command line? or from inside Python, after doing import? Sounds like the former.
 
no and no.
def name_and_version(package_file_name: str) -> Tuple[str, str]
 
Name regex: https://www.python.org/dev/peps/pep-0508/#names
Version regex: https://www.python.org/dev/peps/pep-0440/#public-version-identifiers
I spent some time trying to make sure that I'm not mixing up definitions of "package" in this context, but I'm only like 70% confident that that's the right one I linked.
Or were you already aware of those @AnttiHaapala ?
 
6:43 AM
@Arne ahha so a version cannot contain a -
 
huh, true. I had the impression that postfixes like alpha/beta/dev should have a leading dash, but apparently not in python
 
7:06 AM
@AnttiHaapala Then where are you getting 'package_file_name' from? Inside Python (without import) or outside it?
 
cbg
 
@smci what does it matter, I've got a package file name like 'zope.component-4.5.tar.gz' or 'mysql-connector-python-8.0.16.tar.gz'
 
7:25 AM
hi, i am trying to drop all rows in my pandas table where a certain column has a 0 value. I am trying to use this syntax:
df.loc[df['column_name'] == some_value]

but it returns empty. When I use this:
df.loc[df['column_name'].isin(some_values)]

it does have values. What am I doing wrong?
 
You just want to drop columns containing 0 values?
What are the datatypes of the columns? And what is some_values?
 
some values is meant to be 0 but i dont seem to get any results with that at all
but yes, that is what i want to do
i cant use df.loc[df['column_name'].isin('0')] since many entries contain 0 as apart of the entry
 
I'm confused. What I understood from your question is that == 0 doesn't work, but isin(some_values) does work.
 
for my testing - i am using the value "65900000018530" since "0" doesnt work at all
 
.isin('0') is not numerical zero, that's a string. That's why I asked what the datatypes were
 
7:32 AM
oh i see. its int64
ah ok! i removed the quote sign and it worked. thanks and sorry for the silly question
 
sneaky datatypes!
(thank god for them. please dont ever silently convert things, python!)
 
although im not sure why 65900000018530 worked with the big_data.loc[big_data['column_name'].isin(['65900000018530'])]
 
lol, that may be exactly what Paritosh just wished didn't happen
 
oh..god. groan
whelp, you say jinxes are not real? heres proof
so, uh, can i blame this on pandas? :D (sorry roganjosh)
 
Yeah, for sure
I'm looking into it. I think it might overflow and fall back to strings
df = pd.DataFrame({'a': [65900000018530]})
df['a'] = df['a'].astype(int)
df
Out[78]:
            a
0 -1978171294
Ahh, Windows :)
 
7:39 AM
nope, not overflow.
oh? :o
import pandas as pd
df = pd.DataFrame({'a': [1, 2, 3, 4]})
df['a'].isin(['3'])
0    False
1    False
2     True
3    False
Name: a, dtype: bool
apparently, its not the only this instance.
try this on for size.
df = pd.DataFrame([['3', '11'], ['0', '2']], columns=list('AB'))
df.sum()
A     30.0
B    112.0
dtype: float64
 
Wut
 
heres the explanation i was trying to search for isin behaviour, and ended up there.
 
I still don't understand the first example, though. The docs don't make any mention of type but they do say "match exactly"... which they don't
df = pd.DataFrame({'a': ['1', '2', '3', '4']})
df['a'].isin([1, 2, 3])
Out[85]:
0    False
1    False
2    False
3    False
Name: a, dtype: bool
Does not compute :(
 
7:58 AM
oof
 
It's good one way, but not another.
Ok, well it's explained by the source code
If the column is integer type, it will also try to cast the other values to int
But it also catches OverflowError which I suspect was the case with 65900000018530 and it falls back to object dtype
Actually, no, it can't be, because it converts to int64, not the native integer type of the OS
 
8:16 AM
cbg
 
Huh, scheduled emails in gmail
Living in 3019
At least they'll never implement silly things such as PGP
 
wow, you're way ahead of us man
 
@AnttiHaapala Because find $PYTHONPATH/site-packages -maxdepth 1 -type d | egrep '\-[0-9]' ... is a one-liner, and probably way simpler than a native Python approach.
 
So you don't even have to type to send out the daily "any movement on this?" on some stupidly complicated task?
Managers across the world rejoice.
 
it can send emails by reading your mind...
 
8:27 AM
@AnttiHaapala find $PYTHONPATH/site-packages -maxdepth 1 -type d | egrep --color '\-([0-9\.]+)' should get you started
 
8:38 AM
@roganjosh if you could do it by the afternoon meeting it would be grrreat
 
Afternoon meeting??
 
@AndrasDeak Synergising backwards overflow and squaring the circle, so we're all aligned, is pretty important work. I'll do my best.
Or is it circling the square? I forget.
 
9:18 AM
@alkasm you changed your name from Alexander Reynolds to alkasm?
 
@U9-Forward shhh...
 
9:34 AM
@U9-Forward I did!
 
@alkasm Happy that you agreed, unlike cs95 :-)
 
I have real trouble remembering cs95. Mentally I convert it back to the (second edition and ping-able) original name
 
Sam
Is it possible to break out of all tests if a single assertion fails? I'm trying to test functionality which requires the user to be signed in for any of these tests to pass, so in my first test I want to see if they are signed in, if not return an error message with something along the lines of please sign into your account before running tests
 
Running tests how? If the user needs to sign in to run them then it doesn't sound like a standard test suite (at least that I know of)
 
@roganjosh I remember it by the Macintosh :-)
 
9:45 AM
@Sam if you use pytest you can run that check in the conftest.py
 
Sam
@roganjosh My application is purely Selenium based, so I'm trying to test a lot of functionality which manipulates the DOM
 
Ah, ok, that makes sense. I'm no help at all here sorry
 
Sam
@Arne I shall check this out
 
Sam
9:56 AM
@Arne I've just seen capsys
 
the pytest fixture?
 
Sam
Yeah, looks like what I want. Although I don't know how to use it
 
you can use it to capture everything that the program sends to sys.stdout
which does not include loggers, as I learned over way too much debugging
 
Sam
so can I use it to get an email from a user for instance?
 
uhm
 
10:02 AM
you want to take user input during your tests?
 
I don't think so
 
Sam
@Aran-Fey Yup
 
are you sure? You want to enter your email every time you run your test suite? Why not use an environment variable, or a config file, or something similar?
 
by 'email' do you mean a string that is an email address, or an email that is actually sent to some account?
 
Sam
Because signing in will be unique to whoever runs the tests and the page uses MFA so a unique code is required each time
@Arne A string
 
10:08 AM
That is quite cumbersome for a test suite, but I'll assume that it's reasonable in your use case. Anyway, you don't need capsys, you can just execute input() in the conftest.py and ask for email / pw / mfa code.
 
@U9-Forward i dont get it
 
When coldspeed changed his name to cs95, he was keep acting like another user :-)
 
he changed his name?
 
Yeah, you didn't realize?
 
i've been AFK for a little while, just moved
or at least AFSO
 
10:10 AM
Wow, to irvine?
 
ya
 
Sam
@Arne You able to show an example? Struggling to understand. Fairly new to pytest
 
Wow
 
do you know what a conftest.py is and does?
 
Sam
I do not
 
10:12 AM
The summoning is complete
 
heyyyy i made it into the top 20 on
wonder how long that's gonna last...ive been lazy!
 
soon you can single-handedly all the dupes
 
lol, it's gonna take a long while to get to gold badge...
but im more than halfway there! just a few more...years....
but nice pun anyways
 
Not so sure about that. The field is only gonna grow and you're in at the start
 
openCV isn't exactly an emerging technology
 
10:17 AM
Yeah. But there's only 6 gold badges in OpenCV
 
@alkasm 43 minute drive to LA, to meet cs95
 
@U9-Forward lol, that's currently as it is at 3:15 AM. during the day he;s probably more like 1:30 or 1:45 from me.
 
No, not emerging, but uptake will grow. There's countless examples of established technology that goes along with its daily life and then booms
 
oh wait is he at USC or UCLA? dont remember...
ah USC. okay not as far as UCLA..
 
@alkasm Oh yeah, that's true
 
10:19 AM
at least in openCV even RTFM answers deserve rep, because the docs are a horrible invisible labyrinth
 
they are bad
but the questions are even worse
(partly because of poor api choices, to be fair)
 
I see the ones that overlap with MATLAB (which has a toolbox for openCV). "Here's a selfie I took with my cabbage patch in the background, how can I use image segmentation to count each cultivar of cabbage? Please also include the ones behind me in the picture.
 
i have literally helped someone segment a cabbage before.
computer vision will change the world they said
 
Sounds like I need to spend more time on that tag
 
> why my cabbage not fully detected
I mean it's where I got almost all my rep
 
10:24 AM
Cabbages?
 
obv
cbg
 
You have a claim to ownership of this room (maybe I've watched too much GoT)
 
I didn't know much about OpenCV when I started on SO, I just knew the very basics. But early on I saw a few questions that I knew where to link the answer to because I had just looked them up recently, and so on.
 
most of my very little openCV score comes from a weird "feature" with interpolation
 
via remap or something else?
 
10:28 AM
@alkasm separately, I did actually check out your website when you spoke of updating your SO profile. It still shows you working at your old company
 
Yeah I'm going to update it soon, but previously it was just hardcoded HTML, I wanna do a static site gen with pelican or something and learn something new.
 
@alkasm yup, former
 
@AndrasDeak ya i just checked it out. interesting, nice answer!
 
Ha, thanks.
most of the recent openCV questions I came across had to do with perspective transforms one way or another, and I was never interested enough to read into that
 
rbrb just as everyone shows up
 
10:32 AM
rbrb
 
@AndrasDeak ah you should, that is the best part!
okay, was trying to get this build system working, but i am unsuccessful
and its 3:30. im going to bed.
rbrb yall
 
rbrb
 
good night, maybe tomorrow with a fresh mind
 
nah, tomorrow with coworkers that i can ask for help, lol
more accurate :P
 
10:35 AM
hehe
 
A luxury!
 
night
 
I am back
When everyone is leaving, eek
 
10:51 AM
One person left (or at least said bye). The amount of conversation going on in the room is no indication of how many people are watching. There just has to be something to talk about.
 
2 days ago, by Andras Deak
@U9-Forward you love being redundant, don't you?
 
@roganjosh That's true
2 days ago, by U9-Forward
@AndrasDeak :(
I don't...
Damn, i thought, my code is working, it's not, i totally took that back
 
Seems I either really do have a troubled mind or YouTube's recommendation system needs work ibb.co/dLtbS9y
 
I know my big problem now...
 
@roganjosh hahahaha. I don't know if it's Youtube or your troubled mind, but I don't think you have to worry too much unless you decide to watch both videos... at the same time
 
11:07 AM
@Aran-Fey That would be crazy having those two at the same time, one half of your mind would be exploding, other half relaxed...
 
Don't tempt me!
 
@roganjosh Haha, i was ready to ask you to do that
 
@Aran-Fey I mean, the "&..." links them perfectly :)
 
10 kilotonnes of TNT in D minor
 
Oh, I love that song!
 
11:20 AM
rbrb
 
11:38 AM
hi, im struggling with some date formatting stuff
def get_date_patterns(series, patterns=[ '%d-%b-%y']):
    datetimes = []
    for a_date in series:
        try:
            datetimes.append(pd.to_datetime(a_date, format='%d-%b-%y', errors='raise'))
        except:
            datetimes.append(pd.to_datetime(a_date, format='%b-%d-%Y', errors='ignore'))
    return pd.concat(datetimes, axis=1).ffill(axis=1).iloc[:, -1]
hmm sorry i was trying to fix the formatting
 
Before you post any more, please see sopython.com/wiki/…
 
thank you
 
Where does the data come from?
 
im getting the error TypeError: cannot concatenate object of type "<class 'pandas._libs.tslibs.timestamps.Timestamp'>"; only pd.Series, pd.DataFrame, and pd.Panel (deprecated) objs are valid
it comes from multiple spreadsheets (monthly) which i have had to combine and put into the same formats
 
What do you want to end up with when converting a list of timestamps?
 
11:42 AM
Ok, and do those spreadsheets use a consistent date format?
 
A series? A dataframe? If the latter, what should the column name be?
 
well i have narrowed it down to those two patterns as well as just empty columns
im passing the function a column from the dataframe
 
So you do definitely have a mix of date formats across the various spreadsheets?
 
also don't use a bare except:, just catch what you expect
 
hmm. i had this previously without the except and it worked but ive tried to add in an additional pattern
 
11:45 AM
Why?
 
Don't use a bare except.
 
But, IIUC, they're saying that except doesn't actually do anything here, so I'm not sure why they need to add another date format
 
i have been slowly trying to get to a working function that can cater for multiple patterns, otherwise could run it with different formats i guess and adjust the pattern each time manually
 
def get_date_patterns(series, patterns=[ '%d-%b-%y']): is a mutable default argument, so you probably want to avoid that
 
nevermind, it doesn't help trying to have two parallel conversations
 
11:46 AM
the except here is meant to be a different pattern
%d-%b-%y vs %b-%d-%Y
 
pandas already does that for you
 
i had quite some trouble with pandas guessing
 
In this case, it's easy enough to check for the format between the two because you can either convert the 0'th index of split() to an int, or you can't
 
I am hoping to accomplish a more general solution though because there are more spreadsheets with more date formats im going to have to deal with at some point.
 
you should first figure out the error you're asking about, then generalize the parsing step
whether or not you can parse other formats you'll end up with a list of timestamps
 
11:58 AM
ye, i dont really understand the issue - datetimes should be appended with timestamps in any case, so when I have the function without the try except it works
 
Jun 7 '18 at 13:13, by PM 2Ring
Bare except is bad enough, but except: pass is the Python equivalent of sticking your fingers in your ears and shouting "La la la, I can't hear you".
 
can confirm, works 10/10 times when you need to ignore siblings. debugging program errors though, not so much.
 
And Vigil has the equivalent of "why are you hitting yourself?"
 
"Its primary distinguishing feature is its supreme moral vigilance. Using swear and implore, functions may specify what they accept and return."
Me: Yes, yes, yes please!
"Non-compliant functions are automatically deleted from the source file."
Me: NOOOOOOOOOOOOOOOOOOOO
 
Well, that's my internet gone :( I tried turning the router off and on again twice and it didn't work. There's nothing more I can do
 
12:04 PM
@roganjosh :( Pressing F to pay respects
 
"Vigil is an esoteric programming language similar to Python" Depending on how you read it, that sentence can basically be the equivalent of shots fired for python
 
@AndrasDeak it is much appreciated in these hard times. I never did hear the end of the EXPLOSION Symphony and it's too much for my data plan on my phone
Oh, it's back! The prayers worked!
 
so i have a solution (yay) its not very quick or elegant but it seems to work:
def fix_date_MMM_D_YYYY(mydate):
    return pd.to_datetime(mydate, format='%b-%d-%Y', errors='ignore')

def fix_date_D_MMM_YYYY(mydate):
    return pd.to_datetime(mydate, format='%d-%b-%y', errors='ignore')

big_data['BIRTH-DATE(MAIN)'] = big_data['BIRTH-DATE(MAIN)'].apply(fix_date_MMM_D_YYYY)
big_data['BIRTH-DATE(MAIN)'] = big_data['BIRTH-DATE(MAIN)'].apply(fix_date_D_MMM_YYYY)
 
I'm pretty sure the .apply trashes any speed from to_datetime
 
ye.. its a bit slow but the result is correct
 
12:18 PM
wat
 
thank you :)
 
Did you just return a statement? :P
 
mmm, yes I did :/
 
The code I posted is nonsense
 
12:19 PM
you need return df['column']:=..., silly!
 
Oh man, I'm an unwitting supporter :/
 
sorry do you mind elaborating?
 
Can anyone please explain what is the use of * in python in terms of below code:
str = ["flower","flow","flight"]
for x in zip(*strs):
print ({*x})
 
Andras is joking about assignment expressions, and my code was absurd so it's not even worth saying what was wrong there
 
12:24 PM
@AmanJaiswal That's argument unpacking.
 
it might help you to just print x, and observe 1 asterisk at work, rather than having two mixed in at once.
 
yes I saw that loop one is zipping index wise and print one used to remove the dups, I waht to know it's internal working
 
for x in zip(*strs): is equivalent* to for x in zip("flower","flow","flight"). {*x} is equivalent to set(x)
(*assuming that strs is equal to ["flower","flow","flight"]. The code snippet doesn't actually define a strs variable, but I assume that str = ... is a typo and you meant to do strs = ...)
 
@ThelurkerLurker This is what I meant, I just cocked it up
def get_time(df, column, time_format):
    df[column] = pd.to_datetime(df[column], format=time_format)
    return df


df = pd.DataFrame({'a': ['2019-01-05', '2019-02-03']})
df = get_time(df, 'a', '%Y-%m-%d')

df2 = pd.DataFrame({'a': ['05-01-2019', '03-02-2019']})
df2 = get_time(df, 'a', '%d-%m-%Y')
 
12:30 PM
but this will both mutate and return the dataframe...
not what we usually like
 
> An asterisk * [when it appears in the context of a starred_list] denotes iterable unpacking. Its operand must be an iterable. The iterable is expanded into a sequence of items, which are included in the new tuple, list, or set, at the site of the unpacking.
 
@AndrasDeak worse than apply?
 
more than one thing can be bad at the same time
apply is bad from a pandas point of view, mutating-returning functions are usually frowned upon from the native python side
get_time that mutates its input is especially bad
 
So is there a good way to make a general function?
 
I'd either rename it to something obviously mutate-y and return None, or bite the bullet and copy inside
 
12:32 PM
> If the syntax *expression appears in the function call, expression must evaluate to an iterable. Elements from these iterables are treated as if they were additional positional arguments. For the call f(x1, x2, *y, x3, x4), if y evaluates to a sequence y1, …, yM, this is equivalent to a call with M+4 positional arguments x1, x2, y1, …, yM, x3, x4.
 
note that you could just return pd.to_datetime(df[column], ...) and assign back on the caller side
 
Yes, that would be better. I think I recoiled too much from my initial post and went too far the other way :P
 
ok thanks ill try that
 
But why "copy inside"? I intend the function to mutate the df
 
then don't call it get_time
 
12:36 PM
Yeah, the choice of function name was bad
 
call it convert_times
 
cabbage
 
cbg
 
def fix_date_MMM_D_YYYY(big_data):
    return pd.to_datetime(big_data['BIRTH-DATE(MAIN)'], format='%b-%d-%Y', errors='ignore')
 
Despite my choice of function name (which Andras is right about), I did show how you can pass the format string to the function
 
12:41 PM
ah yes, i will add that
can i also pass the column name in?
 
Um, did you try my example?
 
there's literally nothing stopping you from passing the format string
 
def fix_date(big_data,column_name,pattern):
    return pd.to_datetime(big_data[column_name], format=pattern, errors='ignore')
 
Looks better, does it work?
 
no im getting KeyError: 'BIRTH-DATE(MAIN)'
 
12:44 PM
Actually, I'm now thinking that function is pointless
Basically everything you specify can just be done directly to whatever DF you're working with.
The function only makes sense if you want to .apply it and guess at the format
 
hmm. well it initially it was a function because i wanted to try different patterns
but that didnt work
 
But we've gone full-circle
 
sigh im afraid so
 
never go full circle
 
Disregard what I've said. You want a function to guess the format, so use dateutil or something in the function. It will be slow, but that's the only way to guess
Apologies, I think I derailed that one from the original question
 
12:48 PM
actually, the original question was about getting back a dataframe so this is still on topic
 
@Kevin oh sorry, Thats a typo,
@Kevin so what will be the time complexity of this in terms of Big O notation ?
 
@AmanJaiswal it's syntactical sugar
 
Of the entire function? About O(N*M), where N is the length of strs, and M is the length of the shortest element of strs.
 
@AndrasDeak means ?
 
meaning that using unpacking or imagining that you write the same thing by hand should have the exact same complexity
it shortens development time (both reading and writing the code), not runtime
 
12:56 PM
Agreed. {*x} has the same complexity as set(x) and zip(*strs) has the same complexity as zip("flower","flow","flight").
When I describe two approaches as "equivalent" I typically mean that the outcome is the same and the time complexity is the same.
 
okay
 

« first day (3120 days earlier)      last day (1820 days later) »