« first day (421 days earlier)      last day (3390 days later) » 

9:21 AM
@tonytonov Do you mean flags for questions or for answers? There is also "not an answer" and there are several more for questions, but I believe those result in a close vote. But I might see different things since I'm above 10k rep and I don't remember how it was before that.
 
10:19 AM
@Roland Ah, I see, flagging answers is still the same, thanks.
 
11:11 AM
Hello
 
Hello @ilir.
I think tagging this question with data.table would be frowned upon, as the original question is not directly on data.table.
 
Hi @Arun ... I saw you are one of the authors of data.table?
Oh, OK... I was meaning to ask about that... I tend to add it to the tags if the accepted answer uses it
 
@ilir Yes, I started contributing sometime in June or July 2013 I think.. And in September, was promoted to author.
 
but I don't know the etiquette
Thanks for that, excellent work :-)
 
@ilir Yes, I can understand that. I used to do the same before, until someone corrected me. I think the reason was "tags are meant for questions, not for answers".
 
11:14 AM
@Arun that's what I meant to ask about... no way to tag answers that I can see
 
@ilir Yeah, right. I am pretty sure this has been discussed though. I don't follow 'meta' posts that closely.
 
OK, I will stop doing that. Thanks for the heads up.
 
It also becomes a bit difficult to find all questions that have "data.table" answers (for some nice stats when you're bored :D) a bit difficult - as you've to use the search feature of SO a bit differently.
@ilir sure. I was under the impression that your intentions are right, but you might not be aware of it.
While we're on it, one question: how familiar are you with Python/Pandas?
 
@Arun I know what you mean. I'm new here and have been using SO mostly as a source of bite-size challenges to see if I understand the syntax (mostly of data.table) well enough to have a solution
 
@ilir more or less what I did as well :). Nice way to learn things.
 
11:19 AM
@Arun somewhat familiar with python in general, working of understanding the syntax/structure of pandas better
 
@Arun @ilir thank for the help
 
@Arun would you like to send that to a private chat instead?
done
 
@ilir, great!
@MOMO sure.
 
@MOMO you're welcome... look into it and you wont look back
 
 
7 hours later…
6:38 PM
@Thell Hello
 
6:50 PM
Hiya @ilir !
 
@Thell I am sorry if my reply seemed snarky. I did some basic fact-checking before I made the comment. And the offer to suggest an edit was genuine
I just think it would be overkill to plaster every answer for a quick speedup with benchmarks
 
No problem, didn't take it as snarky. Just that the benchmarking doesn't seem to support the statement.
 
I don't know how to track R's operations in a more detailed way... It does seem to rewrite the data frame to a new address every time I add a column
 
IMHO any question labelled "Performance", "Optimization", etc... should have some data to support the answer (and the question too, but I don't expect that to ever happen) :)
 
this is 3.0.3... I head (again hearsay) 3.1 is much better in this
 
6:54 PM
@ilir, it doesn't matter if a copy is being made if the copied operation is faster than the non-copied... that why I posted the benchmarks and suggested the edit.
 
OK, perhaps I conflated it with memory use as well
my own code at work did speed up significantly when I switched to data.table... i am talking about the new column addition here, not the grouping/summarizing
@Thell I have made changes to the answer. Again, feel free to add to it if you are confident in your own results.
 
No big deal, really. The questioner obviously found the performance gain to be enough to not even want to test the alternative...
 
@Thell and to think you wanted him to already come with baseline benchmarks :-)
 
@ilir, thanks. I removed the comments, but left the request for arun's insights. It is a little odd to see such a difference.
I know... I expect to much from askers.
 
@Thell very odd. I may give the old method another try when I get around reviewing the model at work. But I am also memory-constrained there.
 
7:08 PM
The memory constraint could be it... Do me a favor and let me know the results if you do get around to testing.
 
Will do. Cheers
 
 
3 hours later…
10:33 PM
@thell, @ilir, here's the timing on my system. Not sure what's happening with yours...
1000:
Unit: microseconds
          expr      min        lq    median        uq       max neval
   dt.addC(dt)   20.586   24.0700   29.5970   32.5840    63.822   100
    df.add(df)   57.479   71.8890   76.6815   81.2150   414.324   100
   dt.addB(dt)  460.727  515.2825  563.0275  618.4745  1511.532   100
   dt.addA(dt) 1258.341 1580.2455 1628.5075 1778.5280  4709.371   100
 dfadd2dtB(dt) 1315.149 1571.8510 1632.3610 1753.1560  5604.546   100
 dfadd2dtA(dt) 2499.322 2574.1585 2657.9075 2853.6085 84829.764   100
Note that the size of this large data.table (or data.frame) is 115MB - which isn't that huge. Try creating a data.table with 2 character columns (with many unique values) and 3 numeric values and a total of say 1e8 or more rows.. and copying that (if you've enough free memory still left). You'll see the amount of time spent on copying.
 
@Arun, very interesting. I'm re-running now and will put sys and session info in the post.
I was definitely shocked with the results; so shocked I almost made it my first SO question. :P
 
11:22 PM
@thell, I think I understand what's happening here.. Give me a minute.
 
@Arun, kk, thanks.
 
Let's create a data.frame df and a vector y as follows:
df = data.frame(x=1:5)
y = 6:10
And do a tracemem(df):
> tracemem(df)
[1] "<0x7fa39b44a148>"
Now, when you do:
df$y = y
tracemem[0x7fa39b44a148 -> 0x7fa39b440b28]:
tracemem[0x7fa39b440b28 -> 0x7fa39b440c18]: $<-.data.frame $<-
tracemem[0x7fa39b440c18 -> 0x7fa39b440d38]: $<-.data.frame $<-
This is what I get. However, this is not a deep-copy. This seems to be just a shallow copy. Why? 1) The time to do this is very small (from your benchmarks) and 2) because the address of y and df$y are the same.
> require(data.table)
> address(y)
[1] "0x7fa39ad99e00"
> address(df$y)
[1] "0x7fa39ad99e00"
Now, what happens when you touch df$y?
> tracemem(df)
[1] "<0x7fa39b168b50>"
> df$y[2] = 10L
tracemem[0x7fa39b168b50 -> 0x7fa39b168df0]:
tracemem[0x7fa39b168df0 -> 0x7fa39b168e60]:
tracemem[0x7fa39b168e60 -> 0x7fa39b17ea40]: $<-.data.frame $<-
tracemem[0x7fa39b17ea40 -> 0x7fa39b17ef10]: $<-.data.frame $<-
Here's some copying taking place. But it's not clear if there's a deep copy of entire data.frame or deep copy of just column y.
> y
[1]  6  7  8  9 10
> df$y
[1]  6 10  8  9 10
> address(y)
[1] "0x7fa39ad99e00"
> address(df$y)
[1] "0x7fa39ad9a160"
Seems like column y has been copied in df now.
 
That makes sense, and there wouldn't be a copy from "usage" of df$y, just reassignment, right?
 
(just checked. column x is not copied)
So R 3.1.0 is clever. It shallow copies on assignment and deep copies only columns which are sub-assigned.
Now, to the difference.
When we do: DT[, y := y] (where y = 6:10, from before)
> dt <- data.table(x=1:5)
> y = 6:10
> address(y)
[1] "0x7fa39ab8ca98"
> address(dt)
[1] "0x7fa399099690"
> dt[, y := y]
> address(dt$y)
[1] "0x7fa39b2c0208"
> address(y)
[1] "0x7fa39ab8ca98"
As you can see, address of y and dt$y aren't the same.
This is because we don't point to vectors that's already been assigned. This is because data.table modifies by reference. If we just pointed to that vector, then a sub-assignment would also modify y.
That is, R does a copy-on-modify, which data.table doesn't do. As it does things by reference, it has to make sure, y=6:10 isn't affected when we do: dt[2, y := 20] for example. So, it allocates column y in dt and puts the values there.
Now, this is a special case in R3.1.0. I'm curious to see, if you run a function and assign it..
for example:
foo <- function(n) runif(n)
df$y = foo(5)
 
Very interesting. I wonder if 3.1 uses a ref count on y's addr ptr so the if rm(y) is done df$y still points to the same addr, would make sense...
 
11:37 PM
Here foo(5) isn't materialised (I mean it's temporarily created)
So, df$y has to be created..
Let me run the benchmarks again by creating a function instead of a constant pre-allocated vector.
 
lol, those kinds of tricks are gonna skew quite a few first-glance benchmarks.
 
@Thell unfortunately yes! :)
Also note that tracemem(.) does not differentiate between a shallow copy and a deep copy... which is also annoying!
library(data.table)
library(microbenchmark)

n <- 1e7
foo <- function(n) {
    start = sample(10,1)
    seq(start, start+n-1L, by=1L)
}

df <- data.frame(x=1:1e7)
dt <- data.table(x=1:1e7)

add_dt <- function(x) {
    set(x, j="b", value=foo(n))
    set(x, j="c", value=foo(n))
    invisible(x)
}

add_df <- function(x) {
    x$b = foo(n)
    x$c = foo(n)
    invisible(x)
}

microbenchmark(a1=add_df(df), a2=add_dt(dt), times=25)
Unit: seconds
       expr      min       lq   median       uq      max neval
(the sample(.) is not necessary. Just realised.. can't check now for equality)
But you see that the timings now are the similar...
 
Indeed. I'll need to update the post again... sighs. Unless you feel like making an 'educational' answer on the topic ;)
 
I think it maybe better to update your post with R 3.0.3 (or if you wish, both 3.1.0 and 3.0.3) and then create a new question and explain these tricks in R3.1.0
Or maybe better to "answer your own question". I'll think of an appropriate title and answer it. Perhaps you can then just link to it?
It's easier for you as well I'd suppose..
 
I'll definitely update the answer to include the 3.0.3 info; I have a win laptop with it installed. It'll be late tonight or tomorrow though; getting ready to do some grilling.
 
11:51 PM
@Thell Great.. Maybe I can self-answer and link it to your question, with a link to this chat discussion.
 
Before I go though, I remembered I have a question for you. We have .I, .N, et al; and to 'use' .SD we can 'copy' within lapply... but is there a way to have .SDCols as a char vector available in j?
s/.SDCols/.SDcols
 
Not quite sure I understand the question. Could you provide an example of how you intend to use it in j or how you think it'd look like?
 
I was just playing around with some alternatives to building eval(parse(text="...")) expressions and tried a solution with DT[,eval(parse(text=paste( .SDcols, collapse='+'))), .SDcols=grep(names(DT)[names(DT) %like% pat] # where pat is passed into the func containing the DT...
 
Ah I see.. maybe as names(.SD)?
 

« first day (421 days earlier)      last day (3390 days later) »