« first day (83 days earlier)      last day (3218 days later) » 

8:36 AM
morning everyone
@DavidArenburg I am using a big function that does date manipulation but since my data is of around 3m rows the following is inefficient: minute(df$Timestamp) <- ifelse(minute(df$Timestamp) < mins, 0, mins)
and hence my question about replacement functions and data.table
The funny thing is that I have used your POSIXct + seconds idea many times but I have been relying on lubridate lately and I forgot all about it :). Thanks for reminding me that there is an easy, fast base-R way
I guess it looks easier to use second(5) or minute(10) than 5L or 600L respectively..
 
8:51 AM
good morning
 
9:15 AM
@zx8754 Morning
 
good morning @David
 
@AvinashRaj Hi m8
 
m8?
 
mate :)
 
:-)
 
9:16 AM
@LyzandeR Yes, I'm old school and when it comes to date/minutes manipulations that's good usually
data.table also have some time manipulation functions, see ?as.IDate
@LyzandeR You shouldn't use ifelse neither. It is very inefficient and completely unnesacery
You can do something like this instead dt[minute(Timestamp) < mins, Timestamp := Timestamp + mins*60L]
Or something similar (as I don't have your real data) but you get the idea
Instead of evaluating the whole vector (what ifelse does), just modify the subset
ok, gtg
nice to see you guys
 
9:51 AM
good morning (for a few minutes ;-) )
 
10:10 AM
@Jaap morning
 
@PauloCardoso hi! good to see you here
 
10:51 AM
Would this be faster if it is in data.table style? BIM <- BIM %>% group_by(V2) %>% mutate(dupeRank=row_number(V1)) , BIM is class of data.table.
@DavidArenburg Good morning David
 
11:02 AM
gtg, see you later
 
@zx8754 Are you using row_number as ranking function or just a sequence of 1:n()?
So @PauloCardoso what do you do in life?
 
@DavidArenburg 1:n
 
@DavidArenburg David. I'm biologist working as consultant in the last 15y
@DavidArenburg what about you David?
lunch time...
i ll be back soon
 
@zx8754 You could do BIM[, dupeRank := seq_len(.N), by = V2] but I don't think you'll see a big efficiency gain as dplyr uses := undrneath mutate when the object is a data.table. My guess that just BIM %>% group_by(V2) %>% mutate(dupeRank=row_number(V1)) will update by reference without the <- assigment operator
@PauloCardoso interesting
I'm a statistician that don't like stats much rather programming
 
@DavidArenburg Thank you David
 
11:16 AM
@zx8754 NP
 
"will update by reference without the <- assigment operator" you mean there is no need to assign it back to BIM?
 
@zx8754 I think so. Can you test?
 
hmm, BIM is 3Mln rows... it took awhile
 
As I said previously, dplyr uses the := operator when you run mutate on a data.table object
@zx8754 so try it on a subset. Test <- BIM[1:20] ; Test %>% group_by(V2) %>% mutate(dupeRank=row_number(V1))
 
@DavidArenburg testing... I will post the results
 
11:21 AM
@DavidArenburg You are absolutely right Dave. Thanks so much. I am still a beginner in data.table terms. I had never seen its real capabilities (up until now that I construct a large scale product for my company). Still learning. I was struggling to find a way to replace ifelse as I know that it is quite slow.

To be fair even with ifelse it did run quite fast but it will be even faster by using data.table 's i index. Thanks Dave. You helped me a lot in these last two days. I really appreciate it.
 
@LyzandeR NP, glad you like data.table :)
 
@DavidArenburg nope didn't update
 
@DavidArenburg I didnt want to use data.table at the beginning as I am afraid it might clash with specific functions that internally use sth like this: dt[, 'Timestamp']. This will output 'Timestamp' on data.table but will work fine on data.frame. Don't know if this is an actual problem though. Has anyone seen it?
 
@zx8754 Ok, good to know
 
@DavidArenburg It makes sense, why should it?
 
11:26 AM
@LyzandeR Yes you need to add with = FALSE whenever using SE or selecting columns by their location, e.g., dt[, 'Timestamp', with = FALSE]. But you could just do ` dt[, Timestamp]`.
@zx8754 I explained twice already :)
 
@DavidArenburg oops, sorry. I should really learn data.tables.
 
@DavidArenburg Yeah I know about with = FALSE but what happens if you use an external package that has a function that uses this: dt[, 'Timestamp'] in a function. You cannot change the function (or you can if you actually rewrite it) and it will create an error if you feed a data.table to that function instead of a data.frame..
 
@LyzandeR yes that's a problem. There is a discussion regarding this on GH
 
@DavidArenburg I see... Thanks Dave. This is the only thing that kept me from using data.table so far.
but I do see its pros and they are a lot
 
@DavidArenburg I thought that data.table checked to see if packages were data.table aware and tried to behave accordingly?
 
11:34 AM
@BrodieG Hi, never heard of that, link?
@Peque hi
 
@DavidArenburg: Hi there :-)
 
@Peque What's up? Welcome
 
I was asking myself why would Vim adds a new line at the end of a file by default, and found this question in SO (stackoverflow.com/questions/14171254/…). It has been marked as duplicate, but as I have commented there (first comment on the question), I beleive that is not correct. The user is asking why, not how to avoid it. Answers are all about hot to avoid it too... (to be continued)
 
@LyzandeR See here
 
In Meta I've seen I could edit the question and then it could be unmarked as duplicate, but I don't think it needs to be edited, so, how could I point that out? ask for a moderator to review if it is really a duplicate
 
11:38 AM
@DavidArenburg github
 
@Peque You flag it to moderator and ask to reopen
but from experience, SO is treated as a democracy
and mods dont mess up with community decisions
 
@DavidArenburg: yeah, didn't do that because I read this: meta.stackoverflow.com/questions/252844/…
 
other than that, we can help you reopening it
 
I'll flag it then, for me that is a clear error of the community
 
@Peque we can help you instead
 
11:40 AM
@DavidArenburg: what do you think about this then? meta.stackoverflow.com/questions/252844/…
I would appreciate your help, thanks :-)
 
@Peque I voted to reopen
this will put the question in the review queue
Hopefully some other people will vote to reopen
 
Thanks for your help :-)
/me crosses fingers
 
@Peque The problem is that I usually don't vote on questions out of my area of expertise, but you seem right here
 
@DavidArenburg: yeah, I guess in this particular case it is easily understandable for anybody (why is not the same as how to avoid it)
 
@BrodieG interesting. How did you find it
btw, I'm listed in the contributes there (proud of my self :))
 
11:44 AM
Tashkent, Uzbekistan, my home town on fire
 
Hope that device is broken xD
 
@zx8754 Is that Celsius ???
 
Yep
in the Sun it is 68, in the shade 40
 
@zx8754: if it is not broken, you have just won a new world record :-P
 
@zx8754 hmm, never measured in the sun
but we also have around 35 in the shade here (I think)
 
11:50 AM
@Peque it is probably aimed at the wall or something, still it is pretty hot
you don't want to lean on that wall :P
 
xD
 
@DavidArenburg A bug came up related to it; can't quite recall the context.
 
@BrodieG btw, did you see my comment yesterday?
 
@DavidArenburg Congrats! That's a goal I have for myself someday. So far I've just gotten to caught in my own to contribute elseplace.
 
it seems sapply beats colMeans on a big data set with not so many columns
 
11:52 AM
@DavidArenburg Yes, I was thinking about it last night.
 
@DavidArenburg: last question: is there a way (a link perhaps) to track the reopen voting? or did it already failed? :-P
 
@Peque I don't think it can fail
It will either get enough votes or not
regarding tracking, I don't think you have enough rep
@akrun just deleted a perfect answer, strange...
 
@DavidArenburg: ok, thanks for your help :-)
 
@BrodieG got my epic badge yesterday :)
 
@DavidArenburg You save the matrix copy, and additionally I think there is new function in 3.1.3 (I don't have it) that does something like any.na so you could use that to make it even faster.
@DavidArenburg Awesome. Every time I get to 200 I think I'm on track to get the Epic badge and then I realize how many more of those days I have left to go...
2
 
11:55 AM
@Peque no problem, you can ping me or anyone here with more than 3K rep in order to check progress in the future :)
 
Feels like you should get a gold badge for epic. And platinum for the next one up or something.
 
@BrodieG I had these moments :), but decided to give it a try and finnaly made it
@BrodieG Yes, I was also thinking about platinum badges for long time now
Like 3K votes on a certain tag
I could have a platinum r badge now
 
@user227710 it does :-). Try openInEditor(DT). I've a snippet that calls this function, and have bound it to the shortcut Cmd + Alt + o.
 
@LyzandeR It seems like Matt added it as a mile stone for v 1.9.8, I'm guessing it will be present in the 1.9.7 devel version already
 
@user227710 a little more on in what way you think so would be nice (for improving). Architect builds on top of Eclipse and StatET, and therefore comes with all the advantages (and disadvantages) of Eclipse... Every plugin available from Eclipse community can be used here. I find that quite powerful.
 
12:01 PM
@Arun So your company is basically a Rstudio competitor?
 
Marketing makes a huge difference :-).
@DavidArenburg it's not the only thing our company does. But in this regard, they'd be, yes.
 
@Arun interesting. I never heard of this IDE
Why, aren't you do marketing ? :)
 
@DavidArenburg that's what we'd like to change :-)
@DavidArenburg not as much as I'd like to (and I am not a marketing expert). Not certainly as Rstudio does.
I find Architect very similar to data.table, in the sense that it has a lot to offer, but doesn't have a really-easy-to-learn interface, and therefore people can give up before understanding what it's capable of.. Perhaps we can improve that in the coming versions..
 
Good morning folks :)
 
@Arun Interesting. I'm certainly should try it
Morning @StevenBeaupré
 
12:07 PM
Congrats on that Epic Badge :)
 
@StevenBeaupré thanks :)
 
@DavidArenburg cool. Thanks.
 
@Arun btw, did you see this
This kind of blows my mind
unless it's just a silly printing method error
 
@DavidArenburg It's not a perfect answer. I check the efficient part after I posted the answer
as I don't want to get downvotes :-)
 
@akrun oh
I dont think you will get downvotes
there are plenty "base only" lovers on SO
 
12:11 PM
@DavidArenburg If I can get downvotes for some duplicate questions, why not here
It's okay. Can you add that base only solution to your post
 
@DavidArenburg yes, I think I know the reason.. don't have time to fix it yet.
 
@akrun People will think I've stole it from you
You can add it if you want
 
@akrun, do you mind sending me an email (for me to have your id)? You can find mine here...
 
Ok I will add it.
@DavidArenburg Do you think stri_split_fixed/as.data.frame wins here in benchmarks as there is an open challenge?
 
@akrun I think its bollocks and tstrsplit will "eat" it without even trying
@akrun From my experience, the simplify = TRUE part makes the code very inefficient
But I can't say for sure
 
12:23 PM
@DavidArenburg I also thought so. It would have been better if he added some benchmarks to backup his words
 
@akrun I would have, but I'm supposed to be working instead of answering questions on SO :\
 
Thanks @DavidArenburg . Really interesting. Thanks for sharing.
 
@LyzandeR NP, you can use the fact that @Arun is here in order to ask him some questions directly :)
 
@akrun which Q are you talking about?
@LyzandeR, what Q?
 
@Arun It is this
 
12:29 PM
nice: "I challenge anyone to find a faster solution." :-)
Challenge accepted ;-)
2
JK
base::strsplit is quite fast for fixed = TRUE, and I've not seen stringi beat it. Which means, tstrsplit might just beat it here..
 
@Arun Yes, I need to add fixed = TRUE but was too lazy
 
Hi @Arun. I asked a question yesterday about data.table and replacement functions if you want to have a look. Apart from that we had a general discussion with @DavidArenburg about the package and potential issues with data.table and I totally support this idea which as far as I understand will be implemented in the next release.
Overall awesome package :)
 
I'm not sure if we can get rid of with=FALSE.. :-( at least not cleanly. I'm skeptical against it just creating more trouble. We'll see. It'd be nice if you add your thoughts directly in that post though.
 
@Arun I saw Matt added it to 1.9.8 milestones
 
@LyzandeR thanks!
@DavidArenburg I know :-). Doesn't mean it's going to be implemented. It'll be resolved by then..
 
12:35 PM
@Arun Ok, good to see @MattDowle back in action though :)
 
Thank you and the rest of the developers and contributors @Arun for this package! I will add my thoughts later on in the day there.
2
 
@DavidArenburg lol. I agree.
Thanks again, LyzandeR. Sure.
 
1:13 PM
@PierreLafortune hi, welcome
Hi @Jaap @user227710
 
@DavidArenburg: hi
 
Hi all
Didn't know about this room. Thanks for the invite
 
@PierreLafortune yes, its kind of new
im gathering here some r tag superstars in order to build some type of a community
2
 
great idea. I appreciate the endorsement
 
1:31 PM
@Arun : Thank you @Arun. I have tried it and it looks great. I do also like the layout when compared to RStudio, and there is also built in ggplot and ftable packages which is nice. It took me an hour to figure out how to use it, but that is not certainly time intensive. I do not expect it to be RStudio, but it nice to have good competitor so that users can benefit from good competition. Code completion (like in RStudio) is something, that I would expect in Architect.
 
@DavidArenburg back again :-)
@PierreLafortune nice to see someone else who's also involved in education :-)
 
@DavidArenburg: can you provide more details on what kind of community?
 
@Arun some better screenshots would be my advice: it would give me (and others) a better idea of what to expect
 
@user227710 code completion also exists.. Eclipse has had it for ages :-)... What clearly lacks is documentation.
We're working on it, albeit slow at the moment.
 
@Jaap I taught college Econ and high-school History (yikes!)
 
1:41 PM
@Jaap Thanks. Definitely. I think an easy to get started tutorial, followed by advanced usage would be awesome to have.
 
@user227710 hmmm, just some professional users gathering up, discussing stuff on/off topic, developing ideas, contributinng to open source, solving issues, whatever
3
Havnt though it thru TBH
 
I prefer analysis much more
 
@Arun: That is good news.
 
I saw this question on twitter and SO stackoverflow.com/questions/31385964/…
 
@DavidArenburg: Thanks for the info.
 
1:44 PM
I'm thinking the loop can be quicker without that growing vector
 
@user227710 who are you, and what do you do :-)?
 
@Arun: I am still a student (economics).
@PierreLafortune: restarting R is the only option, I guess. I do that often.
 
@DavidArenburg: may I ask how's the voting going?
 
2:08 PM
@Peque 2 votes in favor
 
how many are needed? :-)
 
@Peque hmm, dont remember
either 3 or 5
 
@DavidArenburg: ok, thanks, I'll check it again tomorrow then :-)
 
@Peque it seesm to be 5
 
@DavidArenburg: hehehe. Thanks again! Wont bother you again with this.
 
2:14 PM
@Peque its ok, np
 
@Arun looking forward to it!
 
@PierreLafortune that's an interesting question. I wonder if it will run better within the usual R IDE (not RStudio)
@Peque 3 in favor
 
way to go
 
2:54 PM
@Arun in addition to that: write a nice introducing blogpost about it, contribute that to R-bloggers, post it on twitter under #rstats, post it on reddit & DataTau (and let us know so that we can give it upvotes ;-) )
 
@Jaap very nice idea. Thanks much! Would be great to get this all done asap :-).
 
 
2 hours later…
4:31 PM
@Arun I don't see a difference between the questions or answers...?
the other question also seems concerned with labeling according to runs, and rank & runs just happen to coincide... hard to parse what they were really after, though
anyway, un-duped
for anyone else's reference, talking about this: stackoverflow.com/posts/31411973/revisions
 
@Frank I think it's a dupe, why did you reopen?
 
Arun left a comment suggesting my first dupe target was not a good match because it was concerned with "dense rank" (even though the OP there posted a bad example for which rleid worked fine), i think the "another candidate" i mentioned above is a better fit
could probably also include this one in the duping: stackoverflow.com/questions/29376178/…
just figured i'd ask here/Arun, in case i was missing something
 
@Frank Anyway, I think each one of these are good enough so killed it
Was too lazy to look for dupes, wonder if I should delete the answer
 
okay
 
4:53 PM
I can delete also if need be
 
@PierreLafortune oh, no need to delete. nothing wrong with answering a dupe as far as i'm concerned
 
@Frank, the first one has dense rank as accepted answer, which IMO is incorrect.
The second one is more like it, but doesn't involve grouping.. so not exactly the same.. Again, imo.
But, I'm pretty sure that this Q has a dupe.
 
5:12 PM
@Arun oh, fair point
 
 
1 hour later…
6:15 PM
It seems there is a bounty question right up your alley @BrodieG
Actually two of them
Some S4 and extension of generic methods stuff
 
 
2 hours later…
8:22 PM
@akrun no need to delete answers to dupes, i think... stackoverflow.com/q/31414785/1191259 besides, as far as rep goes, you were going to come out ahead there
 
8:42 PM
@Frank Undeleted, I thought the OP is not interested in that answer after the duplicate tag
@Frank Also please consider to post the modified answer to the data.table question
It was your idea that I modified it a little bit
 
8:59 PM
@akrun not sure which question you mean. you haven't answered a question tagged data.table in a few days stackoverflow.com/…
 
9:17 PM
@Frank I meant this
 
@akrun oh, i'll leave that to you. i fear answering and having the asker make a followup or modification to the question. also, i don't know wpct enough to know that its results would align with those of unique
 
@Frank I also think that OP is a goof that has no idea what he wants.
 
@Frank I also didn't know wpct enough to answer it. But I checked the results in a lapply loop and it seems that it sorts by the unique values
 
@akrun ok cool, you can go ahead and post. you've done the research; i just corrected some bad/ugly syntax
 
It's okay. I will leave it that way as I think he is also not very sure what he wanted
 
9:27 PM
@DavidArenburg yeah, well, i'm often in that position myself :)
@akrun ok
 
@DavidArenburg I had seen that guy posting answers which are not the expected ones.
 
@DavidArenburg lol
 
@akrun Yeah, but library(dplyr) never fails it seems...
 
@akrun, mind coming online for a minute?
 
@Arun sure, I will be there now
 
9:51 PM
@akrun upvoting despite the mildly awful df[-1,c(1,7,2:6)] :)
 
@Frank You can post a data.table option, thanks
 
that's okay. i think the base r solution is what i would use in practice (and not bother rearranging columns, of course)
 
@Frank It was just to make all.equal
See u all later.
 
10:13 PM
@akrun bye
 
yes, i thought so. cya
 
11:15 PM
@DavidArenburg thanks for the heads up. Been too busy to answer stuff recently, and might stay that way for a bit.
 

« first day (83 days earlier)      last day (3218 days later) »