« first day (2117 days earlier)      last day (2814 days later) » 

10:01 PM
I have two csv files, each row corresponding to an event (and all events are present in both files). I'm reading the rows into a database. One of these files has an indicator field for row i indicating that row i should be excluded from A.csv and from B.csv, it should not be read into the database. I'm struggling to come up with an elegant design for this situation.
 
@Hatshepsut cabbage
 
@AndrasDeak forgive me, i don't know what that means
 
@Hatshepsut I'd probably do something like a blacklist...
If the rows are in order it makes things nice, but it's not that bad even if not...
 
@WayneWerner they are in order
I have a function that reads each file. How would you want to format or store the blacklist?
 
blacklist = set()

for row_a, row_b in zip(reader_a, reader_b):
    if row_a['do not write']:
        blacklist.add(row_a['event id'])
        continue
    elif row_a['event id'] not in blacklist:
        write_to_db(row_a, row_b)
something like that
 
10:10 PM
why not open both files simultaneously in a with, then loop together? something like that ^
 
if each row is a unique event then you can just skip the blacklist and just do if row_a['do not write']: continue
and with that... rbrb - lawn mowing time!
 
So I work in a lab at a university, and I'm apparently the only person in all of the labs who knows Python. I don't know how that is, but it is, and now I'm being asked to teach the entire lab "how to structure big data using Python."
I haven't been an analyst for very long so I have no idea what that means.
Can anyone help me out and tell me what's being expected of me?
Specifically, what is "big data" as it relates to Python and programming at large?
 
lol
they're screwing you real good
 
@WayneWerner lemon... ?!?
 
Someone who didn't didn't read the question, or know how taxes work: stackoverflow.com/a/38731320/344286
 
10:19 PM
@Cosmo I don't think python = big data, and even if you knew data-y python, that would still not necessarily mean that you know "how to structure big data using python". And if "what is big data?" is a question in this context, they're probably asking the wrong guy, or the wrong questions:/ Don't take this against you: I'm surprised (angry? outraged?) at your colleagues. But I'm not one of the data guys, so maybe my issues are unfounded:)
 
You can do big data with python... but I suspect they're talking about medium sized data, as big data involves terabytes of data
not simply jiggabytes
 
Oh, they're definitely asking the wrong guys, haha.
The problem is that the "right guy" doesn't want to do it (he left the lab), and now I have to do it.
 
What size of data are they talking about?
 
so as this is a lab at a university....can't you just tell them "up yours"? Or maybe "I'm sorry, but I don't think I'm qualified for this task" to soften the message?
 
You could also just turn it all into a pandas dataframe I guess ;)
okay, rbrb for reals
 
10:21 PM
Because what you described reminds me of one of those nightmares where you show up to your finals without studying, and realize that you're also naked.
 
So far, I've only been in email correspondence with the member who wants me to teach everyone.
And she refuses to give me any details until we meet in person.
And at the point, I'll be giving my "lesson."
"Were all new to python over here and are generally wondering about the best and most efficient way to package our data in python and then the best ways to do matlab type matrix operations in python."
That's the most I've got, lol.
I've already recommended that they check out Anaconda.
 
@AndrasDeak no you're correct big data != python and even data analyst != works with big data
so the question is just silly
 
Particularly numpy, scipy, matplotlib, etc.
 
thanks
@Cosmo oh dear
 
because it's like calorie counts vs. average person - if you don't know what the average is
 
10:24 PM
"We heard python is great, you know python, WOOOO TEACH US HOW TO DO SCIENCE"
 
Exactly. :(
 
So is "screw them" not an option?:D
you can give them a numpy intro
MATLAB <-> numpy goes very easily
with all the caveats of course
 
Haha, "screw them" isn't an option because the person who is basically my superior recommended they talk to me.
 
hmm...the first part of this question (what should I do about expectations) could be asked on academia.SE with likely better results (trust me you'll need that site soon if your in this position already)
 
I did tell them that I'm probably not qualified for whatever they want, though.
 
10:27 PM
actually it probably has been asked there in some form (this is not uncommon)
as to data - what kind of lab?
 
@JGreenwell Thanks for pointing me toward this SE site!
And it's a neuroscience lab.
 
wet lab or just bio-analytics? either way the standards that you already meantioned are a good start (add use pandas notebooks)
and there are a number of really good tutorials on YouTube from PyCons that I would start with
 
@AndrasDeak I'm definitely going to take that suggestion, thanks!
 
...I'll link in a sec
 
Both wet and analytics.
 
10:30 PM
oy vey
 
Yeah...
 
let me dig up a reference for that
 
and the rest of the group; would definitely recommend starting with those as they are interactive demos that can be immediately shared
However, your first problem is "how much programming do these people know?" (Python or otherwise) cause you may have a deeper problem if no dev experience (or analyst) at all
 
I have a guess.......
if Python is the bringer of light, there must be horrid things underneath
maybe it's localized to the boss...let's hope for that
 
Again, thanks so much for all the help, guys!
 
10:34 PM
yeah, definitely start by finding out (test them if possible) knowledge of python and programming concepts - have to know your starting point
 
@Cosmo May the odds be ever in your favour.
 
Actually, I have no idea why the entire lab is suddenly deciding to move from MATLAB to Python (they've been ignoring my evangelism since I got here). I desperately hope that you aren't correct.
Is there any way to reward you guys with rep or something for your help?
 
Andras is never wrong. Ever.
He's programmed that way
 
We have deliberately caught all false statements to never be posted here
 
10:36 PM
MATLAB -> python is great because MATLAB costs an arm and a leg
MATLAB can probably outperform python in many ways, but then again there are many python modules which are not available in matlab
and python is a proper programming language
 
yeah, I went R -> Python and couldn't be happier
 
well, but R sucks, right?:P
 
Yeah, my typical schtick involves shitting all over MATLAB's totally unintuitive syntax and elevating Python as the free, open-source savior of humankind that it is. Of course, no one cares.
"Python's hard and stupid because I don't know it and I've been using MATLAB for SO LONG and EVERYONE uses MATLAB!!!"
 
well, MATLAB has pretty simple syntax, simpler than python's at times:P
 
depends on what your doing :P
 
10:39 PM
So I honestly have no idea where this sudden mass switch is coming from.
 
but it really does
 
At least for the things I do, Python trumps MATLAB functionally and morally.
 
it is happening a lot of places I notice - but then I am probably bias
 
I'd also be worried that you'll have a bunch of people who write godawful MATLAB code, who'll start writing the same godawful code in python. But at least they would be able to program somehow
@JGreenwell it's funny because MATLAB's new execution engine makes it considerably faster, and the new graphics engine makes it considerably prettier (figure output-wise) by default
@Cosmo if they are advanced enough to know and love bsxfun, you should get familiar with numpy.einsum and sweep them off their feet
but those are advanced MATLAB and numpy, respectively
 
@AndrasDeak policy changes at universities have how much to do with capabilities?
 
10:42 PM
yeah, I know:P
 
compared to say..... fill in the blank
:) :P ;)
 
they probably started thinking about switching in '95
 
hey, give them some credit: '96
 
Considering I didn't know what bsxfun was until just now, I doubt they do.
 
OK, you'll be fine;)
 
10:43 PM
Haha, thanks.
 
(Lying is a good thing if I do it for someone's feelings, right?)
 
So wait, how do I give you guys rep through chat?
Y'all missed that message before.
Or is there no rep system here?
I've honestly never used SO's chat system before.
 
oh, you can't:)
chat is just what it is -- chat
rep comes into play in that you need 20 or so rep to talk
don't worry about it, we're here to... chat.
 
Ahh, well, I guess all I can give you is my gratitude, then.
 
good enough for me
 
10:46 PM
yup:)
you seem to have enough on your hands anyway, you need all the help you can get:D
 
@AndrasDeak I have a lecturer friend whose university just changed his lab to Python because they decided to change the main CS program to Python - even though he knows R and Matlab pretty much exclusively - guess what he has a semester to learn (well enough to teach ;)
 
@Cosmo while you're here... I'd consider uploading the gif in your matplotlib question to Stack Overflow's imgur account (just use the "insert image" functionality of the editor)
I think you can upload animgifs, at least I remember doing that recently
your gfycat link seems like something that could rot at any minute (or year)
and if you're not bound to python 3, mayavi is really good for kick-ass 3d plots (I mean replacing matplotlib)
 
Oh man, I've moved pretty well beyond that project by now.
 
that is a horrible reason to move off of 3
 
@Cosmo you can also just delete it:) That's always fine with quesitons that have no answers.
@JGreenwell they might be using 2 anyway; I'd never suggest going back to 2 just for this
also, my own scientific data processing applications are very narrow in scope, and I can play around with versions as I please
my own applications happen to be pretty version agnostic:P
with key differences being print() and list(zip())
 
10:54 PM
tongue in cheek and all that
 
you didn't
 
Oh, I did ;)
supper time rbrb all
 
rhubarb
 
11:12 PM
So what's up with Python chat on SO anyway? Are you guys a pretty tight-knit community?
 
Unable to parse; specify tightness of knit in inch units.
 
This all now makes a lot more sense.
Three-and-a-quarter inches sound a bout right?
 
that sounds real loose
Anyway, the real people are away:P I'm fairly new here.
 
Lol, "the real people."
I'm totally gonna hang around here while I work.
 
welcome:)
 
11:16 PM
Is there a chat room for fortran?
 
Melons.
What's a "help vampire"?
 
@chanzerre not that I know of
awareness is crucial, only you can stop the disease
 
So most of my coworkers, then?
 
that might be the case, yes
depends on their reaction if you make them face an actual problem they should solve
if they hiss and draw back to the shadows....ready the garlic
 
11:30 PM
SO even has it's own answer to help vampirism
 
@JGreenwell nice, thanks, haven't seen that yet
favourited it for future reference:)
I'll probably be using it...
heh
We have been doing all these for several years now. They don't seem to be working. — Raedwald Mar 10 at 18:17
 
yeah, why do you think I have more moderatation badges then actual question and answer ones ;P
 
I figured, pure masochism?
 
I like to fix things....so in a way, yes
 
hey everyone, I was hoping to get a little direction... been looking at documentation all day and have just turned into a swirl of thread, processings and async coroutines.
 
11:37 PM
just to be clear... SO documentation, or real one?
 
so your Tron?
 
I'm trying to create a server that will consistently listen to a socket, once data is received, I need to determine what type of "packet" it is and then pass that packet onto another event based action that will run in the background
@AndrasDeak both actually, but the real one is the one that I've been mostly looking at
 
good, keep doing that:P
the real one, I mean
 
what I'm struggling to do is figure out when I should use threads, processes or the async co-routines
 
Oy, I've nearly all the SOD badges?! I don't want 'um
 
11:40 PM
@AndrasDeak I figured that's what you meant
 
SO docs is written by rabid monkeys (generally speaking)
 
hey! I wrote one of them and I agree with that statement
 
It's OK, the rabid monkeyness is an emergent property of the system;)
 
consistently? or constantly? (meaning it checks at regular intervals or it just waits and watches)
well, listens
 
Not as cool as Kelvin--Helmholtz instabilities, but it's something
I'll let you do the actual programming thing now:P
 
11:46 PM
I swear I heard someone call that a "dragon's tail" once
 
Hi all! I have a question regarding the "in" operator in python. Does it always return True or False? And how does it work in the for loop?
 
I'm not sure that's the "in operator" in the for loop
for <var> in <iterable>: is the syntax of the for loop
while <var> in <var> is an expression returning a boolean, yes
this is the point where usually someone comes along and demonstrates that I'm stupid, so don't take my word for it
 
I'm a rabbit! Honest!
 
Okay. Thanks. It's just the syntax. Very confusing :P
 
docs.python.org/3/reference/… <-- membership test operations, <var> in <var>
The docs don't seem to anticipate any confusion
if you realize that the for syntax is pretty strict, it's not confusing;)
for_stmt ::=  "for" target_list "in" expression_list ":" suite
              ["else" ":" suite]
and the same for an "in" inside the loopy part of a list/dict/set comp or genexpr
not to be confused with membership tests inside those comprehensions:D
>>> lst1=[1,2,3,4]
>>> lst2=[3,4,5,6]
>>> [k for k in lst1 if k in lst2]
[3, 4]
 
11:54 PM
Got it, thank you very much.
 
don't mention it
 
How do you have all that just ready to go?
Damn. :P
 
Andras is a bot
 
:D
 
:D
The reason I hang out so much here is that chi is just dripping from the walls. It oozes out of Martijn and I soak it all up.
yeah, sounded much less disgusting in my head
 
11:58 PM
Lmao.
LAURELS.
 
true though
very true
 

« first day (2117 days earlier)      last day (2814 days later) »