« first day (3629 days earlier)      last day (1312 days later) » 

12:37 AM
@smci *hangout :p
Ooh, looks like I got my python bronze badge over the weekend, nice.
 
12:58 AM
100% conda, miniconda
 
1:36 AM
Hello Folks, Any Pandas professional here? Need a little help with finding trend from a csv data.
 
@SatishJonnala hello. There are few people here at this time of the day, and in any case you should just ask your question and if someone can help, they'll get back to you
 
Thanks @AndrasDeak. Let me post my question.
 
if there's a lot of code consider posting it to a code paste site and just linking the code here
for formatting smaller blocks of code in chat, please see sopython.com/wiki/…
 
I have a dataframe with these columns: activity_date, facility_city, score
I need to find which city's score is improving the most and which city's core is deteriorating the most. Basically a trend of the scores sorted by dates. Kindly can someone help with this?
 
1:53 AM
@SatishJonnala Sounds like you need a data scientist rather than a pandas expert
 
2:33 AM
@SatishJonnala Please read the quickstart 10 minuites to pandas, then start writing code, and post a specific question here when you get stuck. But you can't just post a spec and ask other people to write your code.
 
 
4 hours later…
7:00 AM
@Code-Apprentice 90% venv, 10% bare. 3rd party libs don't provide enough extra for me to justify the hassle of managing them as well.
 
@Code-Apprentice 50% manually createt venvs, 30% venv handled by pycharm, 20% venv handled by poetry
 
 
2 hours later…
9:16 AM
Does someone know how the standard CPython REPL works? I've found the code that invokes sys. __interactivehook__ , but not any explicit REP loop.
 
Or are you looking for the actual while loop?
 
I did, but needed a second nudging to not skip over the PyRun of stdin line
TY
 
saucy stuff seems to be at github.com/python/cpython/blob/…
 
that's some helpful code right there
sadly, it thwarts my plan to easily help some poor SO fella to shoot off their own foot with a nuke. :/
 
9:33 AM
seems like this is the actual REPL
 
Yep. I was hoping it would call into some Python hooks, similar to how import allows hooks.
Apparently, it does not.
 
Well you just have to edit the source a bit and recompile python...
 
Since the motivating SO question has changed from "how do I get the REPL to re-interpret "foo" as "bar"?" to "how do I break the fundamental consistency of object identities?", the effort-to-bonkers ratio is no longer attractive enough for me.
 
Clearly if one runs into problems with if val is "potayto potahtoe" the obvious solution is to hack identity.
 
@AndrasDeak That is a step I have so far avoided. Madnessssss lies that way...
 
9:42 AM
on a debian-like you control it shouldn't even be that hard; with build-dep you can get everything you need to compile
I tried buildng python on an hpc cluster once (as a pleb user), it was not fun
 
brew regularly builds new Python versions for me. It's the ultimate powaaah that I'm afraid of.
there are some knobs you should not let a person like me play with.
 
> You want MiyagiScript? Because that's how you get MiyagiScript.
 
 
2 hours later…
12:03 PM
Poll: path / 'foo' / 'bar' or path / 'foo/bar'?
 
won't the right thing break on windows? either way, left looks a lot better to me
 
Nah, they both result in the same path
My problem with the left one is that it gets increasingly annoying and harder to read the more directories you have
 
12:20 PM
f'{os.path.join(path, *("foo", "bar"))}'
 
Im trying to add two columns together except if the value in one of them is 0. Im only finding suggestions as to how to do it with nans.

Basically have df['c'] = df['a'] + df['b']
 
@SB18 and what should happen when one of them is 0?
 
@AndrasDeak Your vote will be considered invalid (:
 
Oh right, then take df['a']
 
@SB18 since I don't like numpy.where I'd do df['c'] = df['a'] + df['b']; inds = (df['a'] == 0) | (df['b'] == 0); df['c'][inds] = df['a'][inds]
but I'm not a pandas user so take this with a grain of salt :P
the where equivalent would be np.where(inds, df['a'], df['a'] + df['b'])
 
12:31 PM
Cool, ill give it a shot
 
I bet there are about 20 jezrael answers on SO main that show how to choose values conditionally from columns
 
I had a bit of difficulty, but what i did now is just do the addition but then conditionally replace it with 0 again afterwards
 
1:32 PM
@SB18 You never said anything about "replace it with 0"
 
Aaah, I had in mind that when not doing the addition youd take df['a'] anyhow, which would be 0
 
late morning cabbages, folks
 
Something like bool(df['a']) * (df['a'] + df['b']) then?
Warning: not really a regular pandas user, so that might not be legit
 
hmm, this website I'm on fails to load if I navigate to www.example.com, but it loads if I go to example.com. I didn't know that was possible.
 
@PaulMcG yes, you can't call bool on a Series
@Kevin oh yeah
Sep 5 at 16:04, by roganjosh
Thanks. It looks like the issue might be by dropping www but I've been told about if while I can only test on a phone
 
1:41 PM
I figured that in terms of internet architecture etc, the www isn't mandatory and doesn't have any special powers. But I sort of assumed my web browser would try to add it to the url on my behalf after it got a 404 the first time
Also surprising that the semi-reputable site would elect not to have a redirect there
 
we also have pages at the university that only work with www
 
Websites developed by overworked TAs may be missing certain niceties :-)
 
Then looking at your previous comment as guidance: df['bool_a'] = (df['a'] != 0); df['c'] = df['bool_a'] * (df['a'] + df['b'])
 
@Kevin I have no idea what you mean ;)
@PaulMcG yup, assuming they want zeros. Although there's no reason to assign that temporary series as a column, probably
 
Oh, sure - (df['a'] != 0) could just replace my erroneous bool(df['a']) from previous.
 
2:04 PM
@roganjosh here is the event that fires when you click on the "x questions with new activity" button. It's basically making one async request to the server saying "I've got X question ids here, please give me the html for them", and it inserts them into the body when it gets a reply.
(function extracted from cdn.static.net/Js/full.en.js?v=<lots of hex goes here>)
 
2:24 PM
Up until thirty seconds ago I was inclined to agree with shadow_walker's theory that gravatar loading is the bottleneck. But now that I'm watching the network activity panel, I see that gravatar images are loaded as soon as the "x questions with new activity" button appears/increments
Hmm, now I'm confused never mind, the inspector was not displaying the full response content
 
Guys I was trying to answer an interesting python question on site, and ran into a really subtle bug. Despite debugging for quite some time I cannot find it.
Can anyone take a look please? https://stackoverflow.com/questions/64005887/need-a-algorithm-to-make-slots-which-contain-unique-pairs-and-those-pairs-are-u/64005950#64005950
 
@solid.py what's the bug?
 
The expected output is a dict of tuples with non-repeating elements and non-repeating tuples. I managed to write the first part correctly but not the second.
 
@solid.py shouldn't you be looking at non-empty intersections rather than subsets?
if there's any overlap, an element was reused
 
Do we know what the exact correct output is for the sample input? If not, what about for a smaller input? N=4 maybe?
 
2:35 PM
@AndrasDeak Hey that's actually pretty good, I'll implement that later and tell you how it went.
 
@solid.py the way you'd normally debug this is look at the first iteration that breaks, and print out (possibly in a debugger) what the objects are in your expression and why the result isn't what you expect
 
It's unclear to me how you can have six slots of three pairs each, for a total of eighteen unique pairs, but you're only selecting from a list of 15 pairs. Isn't this just impossible?
 
@AndrasDeak I did exactly that but the any() statement yielded false, and I assumed its correctness, while debugging I mean.
 
@solid.py yes, but a bug by definition means that one or more assumptions of yours have failed. The any() can't be right because if it were your code would work.
 
Or am I misreading the OP's expected output? There are six rows, but the last row is labeled "row 5"...
 
2:38 PM
@AndrasDeak I see, I'll try your suggestion with intersection.
@Kevin Must be a typo, didn't notice it before. I thought he meant 5 slots. Where he said S slots, where S is: S = N-1
 
He does mean five slots, the first "slot 2" should be ignored
This also answers my question of what the expected output is: everything in that block except the first "slot 2" line
I was throwing the baby out with the bathwater earlier by assuming that the presence of a "wrong" line implied that every line after that was also wrong
 
Sam
Hey all, I'm working with Selenium and was wondering if anybody knows if its possible to run a Selenium instance in a Docker container which isn't headless? I've only seen images which run headless
 
How does docker change the picture? Is there non-headless selenium normally?
 
Sam
Yeah. By default it has a head. I'm not sure if, with Docker, trying to run a "visual"/headed instance is possible. It is probably my lack of understanding with docker
 
I wonder if it's useful to imagine the pairs problem as a graph problem... Given a fully connected graph with N nodes, the challenge is to color every edge so that, for each node, the edges touching that node have no colors in common.
And you only have N-1 colors.
Here's the graphical solution. I wonder if it's noteworthy that it's symmetrical over the y axis.
I also wonder if this is the only solution for N=6. What if you started with a slot 1 of (1,2), (3,6), (4,5)? Would you always arrive at a dead end?
 
2:55 PM
@Kevin not entirely, is it?
the vertical lines are purple vs blue, for instance
 
True. Perhaps it's better to say "every color set is either symmetrical, or it's asymmetrical and is the mirror image of another color set"
 
I'll allow it
 
red, green, and yellow are symmetrical. Blue and purple are asymmetrical, but mirror each other
 
@AndrasDeak The intersection part solved the bug. It generated the expected output.
Thanks a lot!
 
no problem
 
3:00 PM
I suspect there is at least one other solution for N=6, because hexagons are more than y-symmetrical
 
there are also 5! trivial variations
I guess that just applies to the coloured representation, not the underlying problem
 
Here's another solution. It's distinct from the first one even if you switch colors around or rotate the image.
 
you can actually permute the vertices with a given colouring
 
And it's symmetrical-or-mirroed over the x axis
@AndrasDeak I was just about to say, if I could completely rearrange the vertices, would my solutions be equivalent? I'm inclined to say yes but I don't have a firm proof
It reminds me of how a Rubik's cube has a zillion permutations, but not all permutations are reachable from a given starting state. There are 12 independent "orbits" of permutations that can't cross over into one another.
So the question is whether this graph coloring problem has one orbit, or more than one orbit, under the operation of arbitrary vertex-relabeling
For extra credit, prove it for Ns larger than 6
I bet there's a smart proof involving lots of factorials that I can't be bothered to work out
Honestly I'm just filibustering until someone comes in and says, "you fools, this was solved in 1358 by Steven Graph, inventor of graphs"
He knocked it out over dinner, and during dessert he created the pie chart
In combinatorial mathematics, Baranyai's theorem (proved by and named after Zsolt Baranyai) deals with the decompositions of complete hypergraphs. == Statement of the theorem == The statement of the result is that if 2 ≤ r < k {\displaystyle 2\leq r<k} are natural numbers and r divides k, then the complete hypergraph K r k {\displaystyle K_{r}^{k}} decomposes into 1-factors....
"Already known in the 19th century" for r=2 (i.e. the case we care about)
And this links to the article on Round Robin tournaments, which was already referenced in trincot's answer on the original post. Full circle!
Interestingly this is a problem I've encountered in real life. Magic: The Gathering tournaments are frequently round-robin, and tournament organizers that choose pairings at random for the early rounds often find themselves having to hold more than N-1 rounds. Imagine N-2 frustrated nerds waiting around for the last two players to duke it out
There's a plethora of software that will give you correct round robin pairings, but I think most people never even consider that winging it could result in a dead end
 
3:34 PM
Are universities using discord as an assignment? Two questions on the topic in rapid succession.
 
@Kevin ah neat, thanks for doing some digging there :)
 
@inspectorG4dget Doubt it. Looks way too messy for assignments. I would not want to review that as a tutor.
 
lol! Aren't those usually graded by unit tests?
 
@roganjosh No problem, I was curious about the inner workings myself. Still unclear from whence the slowness comes, but it was a fun dive.
 
wim
@Kevin all-escapes codec on pypi, essentially copied from your paste but added some errors/ignore/replace handling and py2 support. This question seemed to (legitimately) need it.
 
3:44 PM
Started getting used to a 13" screen. Jumping back to my own 17" laptop has the same disorientating sensation as walking off a particularly fast travelator at the airport. It's not a screen, it's a wall
 
@wim Cool 👍
I've always wanted my dark influence to reach Pypi
I don't actually remember writing an all-escaper in the last couple of moons, but that's neither here nor there
 
wim
Jun 3 '19 at 17:17, by Kevin
@MisterMiyagi tadaa
 
Ah ha, it was approximately 15 moons ago, which is beyond my short term cache. That explains it.
 
wim
codecs are tricky to package so I didn't bother to do it until a real use case came along (most people that think they want this feature are actually just confused)
 
Yeah the XY problems probably outnumber the legitimate use cases by a significant factor
"Because I think this way looks nicer" is a legitimate use case for the record
 
3:50 PM
@wim Can't remember any of that, but :thumbsup: Genuinely missed such an encoding several times.
@roganjosh Finally got your Mac?
 
Yeah, it arrived yesterday. I wasn't sure if they come in different colours so I held off buying new clothes until I knew what would match
 
"Black turtleneck" is the only piece of clothing you need from now on.
I would have added "and pants", but so is 2020.
 
Next year's hottest fashion item is the iTracker, a fashionable ear accessory that will let Apple customize your experience by gathering even more detailed biometric information from you.
Don't worry about installation, Apple's dedicated iMercenaries will tranquilize you when you least expect it and take care of everything at the nearest iBlackSite
 
It's a reverse Big Brother. You first get shot in the back of the head and then you start loving Apple.
 
All for the modest price of $999.99. Don't bother trying to opt out, you already agreed to it in the last ten iTunes EULAs.
 
4:02 PM
I have indeed had to sign several agreements without reading just for the sake of needing to move fast. There's probably a number of demons fighting over who devours my soul
 
aha! that's when you give your soul to your partner so these guys can't lay any claim to it
 
That's... so cheesy
<-- also, single. That might factor into my view of the comment :P
 
s/partner/computer
 
Don't ROs pledge their soul to room/6? All those crazy RO powers must be fuelled by something.
 
That's the last line of The Oath. You repeat it verbatim and only then you go "wait, what?" but... it's too late
 
4:36 PM
I have to lol at "please solve my prolem"
 
Sam
@AndrasDeak I'm wrong, it's completely reasonable to have a visual chrome instance running in Docker. I just wasn't using the right search terms when I tried to research it :D
 
cbg patch
 
@Skyler cbg
 
@MisterMiyagi I don't suppose you had any chance to play with this yet? I'm interested myself so please do ping if it ends up working/not working
 
does np.testing asserts have any good way of just testing for equivalent values (out of order) or should i just sort my arrays
 
4:51 PM
@roganjosh Fighting with the underlying crypto library right now. The M2Crypto we used before accepts our pkcs12 key, but the oscrypto underneath smail does not. :/
 
Maybe try the predecessor smime?
 
what python crypto libraries are mature and safe to use in projects btw
 
@roganjosh they only offer to encrypt, not sign.
 
Although, it looks like python-smail bundled the same functionality as smime and just added to it, so that might die fast
Ah, poop. Deaded even sooner than I thought as an avenue :/
 
@Arne Is there a difference between venv and virtualenv?
 
4:55 PM
I thought one was shorthand for the other that they were therefore synonymous. Now I'm doubting myself
 
@roganjosh It's a bit frustrating, actually. We have the time/skill to roll out our own mail layer on top of M2Crypto, but not to fix oscrypto to support PKCS12. :/
@Code-Apprentice venv is the builtin module, virtualenv is a pypi module
 
@MisterMiyagi does this mean that I can do venv myVirtualEnv with a vanilla python install?
 
I got a multimeter :) Also I was in the worst restaurant today, there where no joke around 15-20 bugs in all our salads. That stuff was grose
 
@inspectorG4dget in principle yes, but for instance on debian you might need the python3-venv package
 
sry
 
4:58 PM
understood. Thanks. That was actually brand new information for me :)
 
Is the sensitivity to profanity also this big in the UK or mostly in the US? I wonder if it's a language thing or a country thing
 
I have zero sensitivity to it. That's not the point; we aim to keep the language clean just in general because this room is not just UK/US and under-18s can use SO
 
you didnt know, they were experimenting with their brand new bug salad Hakaishin
 
@MisterMiyagi and they have similar functionality, correct?
 
@Code-Apprentice some differences are listed in the virtualenv docs
 
5:02 PM
I've only used virtualenv. For some reason I use "venv" and "virtualenv" pretty much interchangeably.
@MisterMiyagi thanks for the info
 
@Hakaishin FWIW, in answer to your question, I know more people than I would like who cannot make a single sentence without swearing twice. I'm not exaggerating. So, I think the answer is "no"
 
yammin' right, ya li'l yammer!
 
 
1 hour later…
6:15 PM
@MisterMiyagi hahaha. My 2 locals are shut, and I've found myself in the most hipster bar I've been in for years listening to The Joker - Steve Miller Band. But I brought my breezeblock out with me, not the Mac. I've found my new haunt, though!
 
6:36 PM
While point 1 in this answer is probably quite true for the flask server, I can't help but feel that it's security through obscurity and it'd better if it wasn't said. Am I just being a grump or is it actually a legitimate concern? I stumbled over it looking for a dupe for an SO question
 
7:01 PM
If he's saying "the dev server is secureish but not stable so it's probably OK for your little demo", I don't consider it security through obscurity. It's not like he's saying "as long as they don't discover you're using the test server, you're safe"
FWIW davidism says "The development server is not intended for use in production. It is not designed to be particularly efficient, stable, or secure" which somewhat contradicts any narrative along the lines of "stability is more a problem than security". It sounds more like they're equally issues
 
I have my own opinions on why that message is parroted but it's sufficient for me to join the choir. It'd defeat the purpose to elaborate on that. I posted an answer instead to the question because I can't find a decent dupe
 
I'd love for the devs to go into grisly detail about the weaknesses of the server, but maybe keeping quiet about it is a kind of security by obscurity that we should actually uphold :>
 
<implicit agreement by silence>
 
"Don't bring our footgun to the firing range" is sufficient warning for 99% of cases
 
7:26 PM
@Kevin I keep toying with the idea of making a "canon" for that by just battering the server with requests. I've crashed a Java server before (both running on localhost). I can't know all the ways to break the dev server but I could probably have a go breaking it one way. A community wiki, maybe?
 
Alternate theory: perhaps the development server has actually been fairly well tested and is free of obvious security issues. But the team is only 95% confident of that, and for production-quality software you need something more like 99.99% confidence
 
I don't know if there's use in demo'ing 1 way to crash the server and have that as a "here's the reason" or whether people will just work around it by being more inventive
 
question for bash/zsh users: I'm using zsh on osx. I have a tail -f running, to monitor a log file. Is there a way to restrict the output of tail -f to use only 5 (or any other specified number) of stdout lines? (i.e. overwrite the output with the next line)
tail -f path/to/file.csv | stdbuf -oL cut -d ',' -f2 | tr -d $'\n' | echo -n && echo -n " " && echo -n $'\r' does not work
 
I wonder if you could do something with ANSI escape sequences to put the cursor back at (0,0)
@roganjosh Might be tricky to make a post on the main site and keep it up to date. Anything you discover could get patched out in the next release.
 
I don't think it's any of pallet's concern. They add the disclaimer, so I think that their dev server would always be crashable from my idea
 
7:39 PM
I suppose there are unpatchable issues like "the server goes down if I DDOS it" but that's not as fun as "enter hack_the_planet() into the url bar for admin access"
Certainly pallets doesn't have to patch the server every time you post a vulnerability, and in fact they might pointedly refuse to do so on the principle of the thing.
 
The problem is that it seems like the arguments I have in SQL. People contort themselves in, frankly, disgusting, ways just to keep to their string formatting approach rather than parameterization. I reckon they'd do the same to get around my demo so they can keep using the dev server (I really wish gunicorn worked on Windows so I could bury their argument)
 
Hmm, it's a real possibility that you'll get a perpetual stream of commenters saying "but what if I half-assedly sanitize user data like this in addition to all the other half-assery proposed so far"
 
@Kevin my foo is nowhere near that strong :(
 
"What if I use nginx to distribute requests across a range of googlesheets with cells containing JSON and then have another script using gspread to read those sheets at a set rate, and making API requests to get to the dev server and then...." just to get around a single-line of launching an actual server
 
ANSI escape codes are easy, if the console you're using supports them. Admittedly, I have no idea if osx/zsh supports them.
stackoverflow.com/questions/30568258/… suggests yes, if you use dollar signs
Here's how the cursor-positioning escape sequence looks in my Python prompt. (I have to import colorama to trick Windows into supporting it)
C:\Users\Kevin\Desktop>python
>>> on 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import colorama
>>> colorama.init()
>>> print("\x1b[2;2H")
This puts the cursor at (2,2) and then Python puts the next ">>>" on line 3, partially overwriting the Python version name
 
7:56 PM
hey guys, I was wondering, is there an easy way to access 1 element of a pivot table without unstacking and doing all these extra operations, something like pivot['a']['1'] or such
 
If you're thinking "I know the syntax for the escape sequence, I just don't know where to stick it in my existing command", me neither.
 
@Skyler MCVE please. You're asking about a multiindex?
 
@Kevin hmm... thanks. That might in fact be the fix. I'll check it out. Thanks again ;)
 
👍
 
8:14 PM
@inspectorG4dget there are things like unix.stackexchange.com/questions/108500/…. Using watch tail -f inp might actually be simpler, albeit different
 
ooh! thanks @AndrasDeak. It does look a lot more complex, so I'll take a look at it when I get a minute
 
Sorry, I meant watch tail -5 inp
 
8:39 PM
I'm having an issue with pandas.read_csv (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html):

ParserWarning: Both a converter and dtype were specified for column label - only the converter will be used

Is there any way to apply a converter before a dtype constructor (CategoricalDtype)? I need to strip the last column of periods (before each newline).
 
Probably incorporate the type constructor into the converter. This way pandas doesn't have to guess whether you are converting and then constructing, or constructing and then converting.
Warning: I am not a pandas expert
 
@roganjosh just a few, was getting on flight
 
The converter seems to work at the level of individual items, though, while dtype takes a dictionary whose keys are column names.
I'm not sure how to specify that I want a CategoricalDtype for the whole column in converters.
 
You don't. Since you are running the converter per value anyway, construct the desired type in the converter also.
Something like last_column_converter = lambda x: float(x.rstrip('.'))
 
I'm pretty sure you can only create categories after the fact
You have to see all items to define an enum
 
8:53 PM
dtype={
    col_name: CategoricalDtype(categories=col_category_list, ordered=True)
    for col_name, col_category_list in zip(col_names, col_category_lists)
    if col_category_list is not None
},
This is what I was using, for instance.
I'll also try modifying filepath_or_buffer directly.
 
But this is just a naive numpy picture. We need an actual pandas expert.
 
/me going back under the dining room table
 
@PaulMcG myself excluded :)
 
filepath_or_buffer=StringIO(''.join(
    line[:-2] + line[-1]
    for line in open('filename')
)),
Well this works. Bit ugly though.
Maybe there's a more elegant way to do it.
 
9:15 PM
I need more context on the problem you're trying to solve.
 
I’m reading a CSV file whose last column unfortunately ends with extraneous periods.
 
I don't think that's cause for StringIO
If you get rid of your converter, what do you get?
 
I know relatively convoluted ways to get around it, but was wondering if there’s a simpler way.
The values of the last column will have extraneous periods at the end.
And of course don’t fit in with the categorical values, so the whole column just becomes NaNs.
 
I don't see how your converter actually accounts for that. What if it ends with `"................."?
 
My converter for the last column strips the item of its last character.
Multiple ending periods aren’t present in this file. Only 1 (per line).
 
9:21 PM
Whatever that character might be
 
Yes. It’s always a period because every row in this file ends with a period (excluding the newline that’s already accounted for by pandas’ row parser).
But I could also do e.g. item.rstrip(".").
 
It's another case of needing an MCVE. I'm happy to test if I'm around but it'll need regex and I'm not the person to figure that out
 
I think I’ll just apply the “categoritization” for that particular column after the fact.
After calling rstrip or something on the whole column.
Would be nice if pandas added some kind of row preprocessing option though.
 
rstrip what? There's plenty of times I end with ... for emphasis
 
This is a CSV data file. I mean rstrip the values in the last column.
 
9:30 PM
I'm bowing out of this sorry. I can only make your life hard by giving corner cases, but regex or text processing is not my domain and I can't give a solution. That sucks, sorry
 
It's hard to know when to take a person's stated problem and try to anticipate potential unexpected cases, vs. many of these data scrubbing tasks which are often one-off reformats of a mysterious source issue. Like getting the last column of floats in a csv that all have (for some inexplicable and unavoidable reason) a trailing '.' after the decimal part.
I run into this a lot in parsing questions too.
OP: I'm trying to parse "X".
me: Here is a parser for "X".
OP: But sometimes it is spelled "Ex".
me: ...
It looks like @user76284 just want to take a column of values something like "Alice.", "Bob." and "Fred." in the last column, and just make them "Alice", "Bob", and "Fred". But use .rstrip('.'), not .replace('.', ''), in case one of the possible values is "Mr. Snuggles". (See, there I go again trying to explain corner cases for something pretty simple.)
 
@PaulMcG Yes, exactly.
 
replace with \.$ regex
 
But if one of your values is a big blue company named "I.B.M.", then rstripping('.') from "I.B.M.." could be trouble. Hence your solution of just clipping the right most char, no matter what it is (and we presume your presumption that every row has a trailing ".")
 
Yep. In this case, the last column belongs to a specific set of values (these are the labels).
And yes, every line always ends with a period for this file.
 
9:41 PM
df=pd.DataFrame({"Name":["Jet","Jay","Jet","Jay"],"Hobby":["games","swim","bbal","games"]})
tab=pd.crosstab(index=df["Hobby"],columns=df["Name"])
sorry about the delay for mvce, had to pull out 2nd laptop to be able to make mvce since the dev environment is stuck behind phone tfa and im 30000 ft in the air without cell reception
 
^ .... dedication :)
 
is there a simple way I could just access the crosstab score for "Jet" and "games" for example
i could do stuff by unstacking the table but it'd be convenient to be able to directly read from the crosstab
 
10:10 PM
Maybe you want a pivot table instead of a crosstab. The docs for pivot_table say that you get something that is MultiIndex'ed
 
It doesn't give the expected outcome, so it's impossible to know. That's why I asked for an MCVE. I do admire the the dedication but we don't know what we're shooting for
 
10:32 PM
hmm, when i ran those two lines in the interpretter it looked like they were working
 
So put some corner cases in your example to differentiate
 
df=pd.DataFrame({"Name":["Jet","Jay","Jet","Jay"],"Hobby":["games","swim","games","games"]})
tab=pd.crosstab(index=df["Hobby"],columns=df["Name"])
if we do that, then both values for Jay are 1 while Jet has 0 and 2 (corresponding to swim and gamerespectively)
so if we tried to get Jay,games we should see a return value of 2, if we tried for Jay,swim we'd get zero
all other cases lead to 1, so how would we extract the value 2 from the crosstab directly
sorry, Jet,game gives 2
# pseudocode
# tab[Jet,game]
2
# tab[Jet,swim]
0
# tab[Jay,game]
1
# tab[Jet,swim]
1
is that clear for expected outcomes
 
10:47 PM
But you're not getting that?
 
im asking if there's a syntax to directly read those values from the crosstab
 
The pandas docs are okay on creating and displaying these computed tables, but not so great on accessing the contents. The pivot_table method looked promising, since it used the word "MultiIndex" in its description of what you get back, and it sounded like that was a meaningful term to you earlier.
 
When I tried to see if I could access a series or do loc with a command like tab.loc[tab["Hobby"]=="Jet"] (just to see if I could get the whole series before I then slice that, I'd get a KeyError
oh ok, so like you'd create a pivot table and aggregate on count
 
So when you write that, it's not clear what key is the error. Try just tab["Hobby"] first
If that doesn't work, then the larger expression doesn't have a chance.
If that does work, then try tab["Hobby"] == "Jet".
 
yea, tab["Hobby"] is an invalid input that gives keyerrors
 
10:56 PM
If you do a pivot table, is that indexable by Hobby?
 
ill try that, might appear offline for a few, you wouldnt happen to already have the command for taking this and making the equivalent pivot table, been a bit since i worked with em
 
From the docs, here is an example:
table = pd.pivot_table(df, values='D', index=['A', 'B'],
                    columns=['C'], aggfunc=np.sum)
Ugh.
Out of curiosity, how big is this CSV that you are reading?
Not counting the 4 line Jay-Jet hobbies example
 
in production it'll get to millions of "edges" (since you can think of each of these as edges in a bipartite network), and the module I write may get use later by other collaborators in in other projects so that can get a lot larger
 
11:58 PM
Anyone know how to get the categorical codes of all categorical columns simultaneously?
I know for an individual column it's df[column].cat.codes.values.
 

« first day (3629 days earlier)      last day (1312 days later) »