last day (3176 days later) » 

2:52 PM
cat("Hello, world!\n")
 
3:05 PM
Wow. This is dangerous.
Interesting how this doesn't integrate with the rest of the SE chat...
 
Yes, this has the potential for being really useful and/or for being a tremendous time sink. Shall we put a post in the [r] stream to make it more visible?
 
Sure. You mean on twitter?
 
What's twitter?
Recall I said [r] not (cough) #rstats.
So I meant a SO question where we'd abuse the question notation for an 'announcement'.
 
How can you announce this on SO?
Oh...
Sure...go for it.
I'll downvote it.
:)
We should also add it into the [r] wiki.
I added it to the wiki.
We should probably try to clean that up a little; some of the other tags have put a lot of effort into it. (e.g. stackoverflow.com/tags/java/info)
Whoa! I didn't downvote your post.
 
I was wondering... Feel free to upvote it to set it back to even.
 
3:18 PM
I did.
Someone voted to close too.
Some people take themselves way too seriously...
 
Yup. And not unlike the people constantly call for community-wiki.
BTW I didn't see the button for CW anymore once it was posted. What am I missing?
 
I think that I may have seen something about a bug related to that...
 
Four close votes, and -2 score. Whoa Nelly.
 
Ha! I'm not overly surprised. Meant to suggest that you use CW...
Ironically, you're still ahead in the rep. on this question, right? Although I think that it might get downvoted into oblivion before we're done...
 
I up-voted. Dirk, I've got your back. ;-)
 
3:33 PM
How many of you are using R 2.12 now?
Haven't tested it yet, but looking forward to it (with all the Rcpp discussions...).
 
@Shane I'm using R-2.12.0 at home, but it's been a pain to get packages updated at work (Windows, bleh).
 
@Joshua What OS do you use at home?
 
@Shane Ubuntu
 
@JoshuaUlrich That's a relief. Was worried you might be a Mac guy.
 
Josh: Thanks :)
Re 2.12.0 -- I just updated the (Ubuntu) dev boxesn at work this morning with the CRAN binaries. Have been running 2.12 pre-releases at home for once at Rcpp needed it.
I hadn't realized the Windows pain with the switch to gcc 4.5.0. What a mess.
 
3:39 PM
Definitely makes it a non-trivial switch.
 
Question now closed. Wow. What a bunch of nitwits.
I could use my precious SO capital to re-open ;-)
Oh well better move on though.
 
Where did the people who voted to close the announcement question come from? Perhaps they're just looking for chat questions to close?
 
They would have been monitoring the front page.
 
I think none of them were R users. Shane is right -- there are a lot of self-appointed SO police men around.
 
@Shane Ah, chat police... I see.
Okay, OT question: how do I change my picture from this pretty pink pinwheel? Don't get me wrong, I like pink, I just don't want to be known by it. ;-)
 
3:45 PM
@JoshuaUlrich Just click edit on your profile.
And then click "change picture".
 
So I have to sign up for this gravatar thing?
 
@Marek Why don't you change your picture too?
Yes, I think so.
@JoshuaUlrich Are we going to be graced with a hockey player?
 
Another OT: maybe someone vote up stackoverflow.com/tags/data.frame/synonyms
 
@Marek Done.
 
@Shane You already are; I'm just disguised in pink at the moment.
 
3:47 PM
@Shane I got it on my to-do list ;)
 
4:02 PM
Dirk, your mistake was you didn't ask "what is the best way to have a real time chat about R without being on IRC"
see then you could have even answered it. It's very Jeopardy-esque
 
Ah, you get the honorary "Trebeck" for that :)
 
thanks... add it to my bling
 
As we're all empiricists, why don't you put that to the test. If we all vote it up quickly ....
 
@Shane My mirror does not yet have windows R 2.12 so I went to the main CRAN site and my download manager showed me the number of bytes downloaded with just 1 and then 2 digits. I will try later when my mirror has it. (CMU in Pittsburgh)
 
The SNL Trebeck series is among my most cherished Will Ferrell memories.
 
4:05 PM
@DirkEddelbuettel With Sean Connery? Fantastic stuff...
 
"Take that, Trebeck" ;-)
 
I'll take swords, for $500 Alex
 
@JDLong Dirk, you're up...
 
Already replied and upvoted.
 
Wow. Something feels very wrong about this.
 
4:08 PM
Ok now what?
 
@Farrel That's up to your imagination.
 
Can one be looking at a question and answer thread, see that some other poster is online at the same time and move to a chat? It would be quite cool to have chats linked to specific questions
 
That's a great idea. Maybe post it on meta?
 
OK so one row per observation
what if I have 10 observations and 25000 features?
 
Then you have 10 rows?
 
4:15 PM
wait a minute. I am reading the FAQ. There is something similar but not what I was thinking about. You know how a|b != b|a. Well here comes something interesting but not what I wanted. Look at my next post
1
Q: How can I get the output I want from a lubridate subtraction

FarrelI started using lubridate today. I have one variable called Started which is the date on which human sujbects enrolled in a study and another variable is called dos1 which is the date upon which the subject last had surgery. I want to work out how many months since their last surgery to the day o...

 
@MikeDewar That's a lot of features...would be curious about what you're doing.
 
mike, genetics?
 
genetics. yes
10 rows is OK
but not very R happy
or maybe that should be "is very R happy"
 
hey guys
 
I don't know beans about bioconductor, but it seems like they would have a standard way of handling this, Mike
hey Marek, greetings.
 
4:19 PM
yep - bioconductor gives rows as features
hence I often end up writing x = t(exprs(exprset))
 
Sounds like you should follow that convention, then.
 
exprset is a bioc object
exprs returns a matrxi
I have to interact with conventions
 
do you use other bioconductor functions? if so there's a strong arguement to do as shane said :)
 
which is why I started thinking about this in the first place
 
Or just go with whatever is easiest and works, and ignore convention.
 
4:20 PM
so I use a lot of bioc, but also a lot of regular modelling tools in R
 
@shane, that's what I do. Then paint myself into a corner
 
so I end up transposing things a lot
 
this really sounds like the sort of thing that would benefit being run by bioconductor folks and see how they do it
 
@JDLong Somehow, I'm not surprised that you paint yourself.
 
but I'd like to automate some of this stuff, as it's getting boring, so I wanted to settle on some convention. But maybe I should just think that bioc <- t(rest_of_R)
that's not hard
 
4:22 PM
you might make a few wrapper functions for handling bits that get tricky
 
Oh, here we go. Getting closer. On chat.stackoverflow.com/faq there is a question How can I discuss a particular question? which starts off If you want to discuss a certain question that has been posted on Stack Overflow, you can create a room for it.
 
yep. Nice to know row-centric observations dominate though. Thanks for the help guys! Cheerio!
 
4:36 PM
@JDLong Seems to have lasted longer than Dirk's question, but eventually met the same fate.
 
fascists.
 
Sorry JD
 
Ah. Goswin's law. We are coming full circle.
@JorisMeys FWIW I disagree with you here.
 
this IS still the intertubes @dirk
 
I know Dirk
Jon Skeet doesn't though
his site, his laws
And I like your initiative, I also had a good laugh with the question of JD
mind the difference between a closing vote and a downvote...
 
4:39 PM
@JorisMeys Do I care? I don't intersect with his .Net world. And he works for Google, not for the SO overlords, doesn't he?
 
cult of personality wins again. But it's not the cult where I like to kill my chickens. But still has authority over me. Alas.
 
Sorry, I meant Splosky
Spolsky
no Skeet
 
But great that we're wasting time on meta questions rather than talking R. Sigh.
I also do not care one bit for Spolsky, at the end of the day this all builds his equity and not mine. Hence my reservation about getting too involved with all the spinoff sites.
"Do not one care bit" is too strong. Make it "am at best lukewarm". I am appreciative for this site.
 
I don't know the guy either. My mom just told me to be a bit polite when playing on other people's servers
 
Nothing wrong your mommy. But how is chasing other people because of somebody else's rules "polite"? ;-)
 
4:47 PM
you got me there :-)
I'll make it up by marking my next question wiki, what do you think?
 
You'd be ahead of me there :)
 
5:03 PM
sup, ppl
 
greetings Roman
 
JD, thanks for bringing up the chat topic
didn't realize this was an option on SO
 
Wow, look at this nerd convention...
 
now i can run a script that will talk to Dirk, so i can have some alone time with the next question
hey Drew
all the familiar faces
too bad i missed the conference at the NIST
 
welcome to the nerd sausage fest @drewconway
 
5:10 PM
not a lot of women into R, init?
 
hey drew, I got nerd points last week for knowing you. Seems our devs in Dublin use NetworkX
 
Yes, the R community could use some Title IX excemptions
@JDLong really? That's great! Aric and I just confirmed we will be doing the NetworkX workshop again at this year's Sunbelt
 
I get title 9 and prop 13 confused sometimes. leads to some awkward dicsussions
 
i'm organizing a workshop on R next week and maybe i could lure some women into the love den of R
 
There is a slow, but sure, creeping increase in the number of women coming out to the NYC meetup. It is truly a welcome balance
 
5:13 PM
@RomanLuštrik Where is your workshop?
 
Ljubljana, Slovenia
at the department of biology
but we'll be doing very basic intro stuff
 
Is R popular in your department?
 
a few of us are striving to build a base community that will use R for research
a few of us are using R, but the rest is on SPSS. we're hoping to change that gradually
 
Isn't your univ. also the home of the (Python-based) Orange data mining thingie? That's the comp sci folks though, right?
 
dunno. our uni is quite big and heterogenous
and we've got all sorts of people
Gregor Gorjanc works at the same faculty as I do, fwiw
he's the LyX/Sweave guy
 
5:19 PM
www.ailab.si, and yes, eventually they reveal the Uni Ljub url too.
Say Hi to Gregor. If it is two of you, you should get in line for hosting useR and then we all come visiting :)
 
that's not a bad idea :)
 
Count me in
 
in any case, you're all welcome to visit. if you fancy natural heritage, slovenia is the place to be :)
we've got sea, mountains, karst... you name it
 
karst?
 
i have to be going now. i'm visiting a friend before she ships out for the Island
 
5:23 PM
is there a rule about on or off topics here?
 
the word comes from the region that is located on this type of geography. it's called "Kras"
another term in geography that comes from here is "dolina"
i'll shut up now :)
talk to you guys in a little while. peep peep cheerio
 
@wahalulu I don't think so...
 
We could declare question about what is off-topic to be off-topic.
As per the old saw of "to know recursion you have to know recursion" ;)
Time for more coffee...
 
As I have noted to @JDLong in the past, every time I use recursion I make a comment next tot he snippet to the effect "#Recusion FTW!" In fact, just saw a bit of my code this morning with that exact comment.
 
5:43 PM
so, if we post a new R question to SO, like this one (stackoverflow.com/questions/3961705/…), should we poke our heads in here too? Or is R chat meant to be more meta?
 
@dnagirl There are no rules here. It's the wild west. So, you can poke your head in and see if people can help.
Probably more useful for meta discussion, though.
 
@dnagirl There is no "meant to" although I predict that chat will be a useless place to get a thoughtful answer. I think chat would be a great place to discuss a question and answer.
 
@Farrel I'm thinking you're probably right. Maybe chat can stand in for the commenting back and forth that some questions need.
 
6:34 PM
I need to document some of my models like this. Not R related but a good example of explanatory documentation. pauladamsmith.com/articles/redis_under_the_hood.html
 
@JDLong Looks like a lot of work...
 
twice when I left previous employers I've had to do something of that sort to explain my models. lots of work, but I wish I had done the docs as I wrote the code not at the end of my tenure. Would have been easier
 
I've gone through the same thing.
Definitely painful.
If possible, I prefer to either (a) use Sweave in a very clear manner or (b) comment and make the code itself very self-explanatory.
That only gets you so far, though...especially on a big project...
 
the problem I run into with most inline comments is that they don't really put the whole thing in context. Seems I always end up writing a document that basically says "here's why the hell we're doing a model to begin with and here's the general logic"
 
Right.
 
6:47 PM
then I end up creating diagrams that look like FAS 133 process documents
and as I tell my kid, "that's why daddy drinks"
 
It's also important to have standards. In my last firm, I was really diligent about using documentation for everything, but no one else did. That really discouraged me from doing it too...
The other reason that I hate standard documentation is that code is often an ever moving target.
That's why I try to put all my models in Sweave, so the code and analysis are side-by-side.
 
 
1 hour later…
7:52 PM
hey Hadley
and Neil, and Greg. hola
 
Challah!
 
no thanks, I already ate
Neil, how goes the Hadoop work?
 
lol. is this what google wave could have been?
actually, the hadoop-ing is good, i just converted my cluster to use CDH
 
oh Cloudera's dist. Is it much different?
 
truthfully, i wish the documentation was better for us tarball-installers. the documentation pretty much assumes you're on linux. I'm running OSX and don't have yum/apt-get/deb
the dist is allegedly faster but i haven't executed any large jobs, so I don't have my own benchmarks
 
7:57 PM
ok, makes sense. Clearly the solution is for you to get some cheap hardware and switch to Linux
 
the output looks more readable. there are are a few gotchas with respect to security but no show-stoppers. i went with cloudera dist #1 because they ensure there are no compatibility issues within the ecosystem.
they suggest a version of pig, hive, etc. from there, i dont have to worry if pig version x.y.z will work with hadoop .20.1.etc
 
compatibility is good
have you seen a descent short guide on when to use each tool in the stack? ie when should I use Pig, what does hive offer, etc
I'm not up to snuff on the tools. I've used Hadoop streaming and that's about it
 
I dont know if that exists. A hadoop-tools guide so-to-speak. The cloudera site lists some of the tools but there are many more.
are you at least up to speed with pig and hive and what they do?
 
there seems overlap between Pig and Hive and that's what confuses me
 
yes.
as far as i understand, they both do a similar job, each having their own dialect. hive uses a variant of sql. pig uses its own data-flow language
 
8:04 PM
they are both abstractions on top of Map/Reduce... both allow "querying" of sorts
 
i picked up pig a while back and love it. i feel that learning hive will be a no-brainer because of my SQL experience so I stuck with pig. pig represents a much different way of thinking compared to SQL. i feel its both good and bad.
i highly encourage you to try your hand at either. pig seems (IMO) to be more well-supported.
i believe EMR will let you create pig jobs.
 
yeah, I use an "interactive pig session" to test on EMR
 
additionally, you can play with pig on a local machine without requiring hadoop to be present. it's how i develop most of my programs.
 
I don't actually do Pig, I use that menu choice so the instances don't shut down
 
and its a snap to install. work with a small enough input data set(for dev/testing) and your local-mode pig jobs complete nearly instantly.
 
8:06 PM
oh really? that blows my mind a little. what does it mean to use pig without Hadoop?
oh... so it's like having a one instance hadoop cluster
without hadoop
 
yes and no. it doesn't HAVE to run against hadoop.
 
ohhhhhh
I was missing that abstraction
 
obviously i wouldn't send a billion records through standalone mode
 
natch
 
but when testing things, i'll create a file with a few thousand rows and run it through pig local mode.
that way you don't have the 'overhead' of iterative development using hadoop
 
8:07 PM
ok, so Pig code is plain text?
 
yes
 
you motivated me to read up
 
i think i have some examples on gist.github.con and on neilkodner.com
i wrote a pig script to show top-n hashtags
i then modified it to show by the day and then by the hour
 
can pig run R code against "groups"
 
oh i dont think pig can run R code, it uses pig latin
what do you mean by groups?
 
8:09 PM
so use Pig to parse data into a subset, like a group by
then use an R function as the aggregate function
 
oh
streaming R? i believe that's possible.
 
assuming that the subset is small enough to fit into memory, of course
 
i haven't tried. i know that pig can call out to .py, i'm not sure about R. when i get this cluster finished i'll give it a shot
that's a good question.
so pig, you can group by nearly anything..it's nice
i prefer sql(and need to play with hive) but i can see why people like the way that pig defines data flows.
 
I need to grok these tools better so I know which ones to lean on
as you know I use EMR mainly as a poor man's grid
my most common reducer is cat
 
you're running linux aren't you? Installing pig is just a matter of tar -xvzf
then pig -x local and you're in
you'll find no shortage of people to lean on for pig help.
 
8:13 PM
no debian package? oh well ;)
 
you know SQL don't you? Look at neilkodner.com for an article about hacking seinfeld tweets using pig
and i show a few differences between pig and sql
 
yeah, I SQL
 
Wow...you step away for a minute and the conversation degrades into Hadoop?
 
lowest common denominator shane
 
I was going to start in on some of my fav fart jokes next
 
8:16 PM
Neil: While we're on the subject, have you tried Cascalog? github.com/nathanmarz/cascalog
 
no, i sorely need to give it a try.
 
question: I have a list of Raters by ID number, a list of subjects by ID number, and a score. what is the best way to count the number of actual (not possible) rater-subject combinations
 
so many amazing tools now. i spent a few minutes chatting with @LuciousPear and he told me i need to jump on it.
 
I was doing unique and then table but I wanted to know if there was something faster/more R-like
 
@stedy that sounds like a unique() question to me
 
8:19 PM
yeah thats what I was thinking as well,
 
Yep; unique() would have been my first impulse.
 
@stedy, include only the columns you care about: unique(myDf[c("colA", "colB")])
 
ok maybe I just need to tool with it a bit to make it work
 
@NeilKodner I don't use Hadoop, but Cascalog definitely sounded more up my alley than pig...
 
@nei
 
8:20 PM
Thanks! @JDLong
 
@neilkodner Brad told you to jump on Cascalog?
 
yes. but ive never really done anything with clojure, so i'm staring at a decent learning curve
i have so much on my reading list it's ridiculous. going to hadoop world only complicated things.
 
yeah I looked at it, read about the pain of installing clojure and had flashes of Java and then I took a nap
I had more patience when I was younger
 
For what it's worth, Clojure is much more enjoyable to use than Java...
 
i need to focus on hbase, all the while, work on my R and statistics as well
 
8:23 PM
Although installing it can be a little more painful (this is true).
 
yeah it's the installing it that scared me
 
Just stick with R. :)
 
in the meantime, i haven't run into too many limitations of pig and/or python streaming w/hadoop.
i dont use R enough :/
 
a lot of "big data" world is sort of the BI slice and dice on big data. query, group, aggregate
@neilkodner, have you used Kettle with any of the Hadoop suite?
or any of Pentaho tools with Hadoop, for that matter
 
i haven't used kettle with pentaho, but they had a strong showing at HW
speaking of which, you still liking the kettle book? I've done 99% of my etl by hand but am impressed with kettle as a product
i might give pentaho a shot on this cluster.
 
8:33 PM
I'm maybe 15% through. I'm having some conceptual issues though. Mind if I bounce them off you?
 
i'm excited for this cluster because i'll be able to leave it running full-time. Before, my main laptop was my namenode/tasktracker so the cluster went down when the laptop was off.
sure, but i can't promise that i'll be able to answer :D
 
no worries... but you might stumble on the answer later and email me ;) I do ETL from my corp admin system. I want to query only current year policies (i.e. 2010) and copy those to my local risk data mart. I want to use the same ETL over and over, so the year needs to change. So do I set up a variable named policyYear and set it to 2010? Is that a parameter? I'm just curious how I should do this in a very Kettle friendly way
 
btw, i like Roland and I think he's done some brilliant things pre-kettle so I'm probably going to support him and pick up the book.
 
Yeah I've almost emailed him this question but feel like it's low level and should be in the book
but I can't quite find it
 
yeah, the year gets defined as a parameter. then reference the parameter in the jobs.
IIRC defining parameters wasn't exactly intuitive in kettle; hopefully things have gotten better.
 
8:42 PM
yeah the terminology was eluding me
ok... gotta run.. talk to you later
 
i'm doing a pig install on a different machine, i'll let you know what i think of the current docs and process, if you're interested
might screw around with hive also for good measure
 

  last day (3176 days later) »