« first day (1500 days earlier)      last day (2311 days later) » 

3:10 AM
library(microbenchmark)
library(data.table)

ng = 1e5
n = ng*10

sample_dt2 <- data.table(
    FW = 1L,
    CP = sample(ng, n, TRUE),
    AE = sample(2000, n, TRUE),
    SD = sample(c(-1,1), n, TRUE)
)

microbenchmark(times = 10,
    GForce = sample_dt2[, .(sum.AE = sum(AE), mean.SD = mean(SD)), by=.(FW, CP)],
    namedList = sample_dt2[, f(AE, SD), by=.(FW, CP)]
)

# Unit: milliseconds
#       expr       min        lq      mean    median        uq       max neval
#     GForce  46.77016  47.36718  49.03106  48.73407  50.56145  51.83276    10
 
@Frank, so it means that the advantage of GForce will be obvious when there are many unique values for grouping variables compared to named list?
 
4:14 AM
@mt1022 yeah, the advantage of GForce is most noticeable when there are many groups. actually, this is true of many of data.table's efficiency features (they are most obvious with many groups). the "named list" thing is kind of a side-note, even though the message for it is long. the main difference here is coming from the GForce, which is only triggered when DT[...] can actually see sum and mean (which are hidden by your function)
microbenchmark(times = 10,
    GF = sample_dt2[, .(sum.AE = sum(AE), mean.SD = mean(SD)), by=.(FW, CP)],
    namedList = sample_dt2[, f(AE, SD), by=.(FW, CP)],
    noGF = sample_dt2[, .(sum.AE = sum(AE)+0, mean.SD = mean(SD)+0), by=.(FW, CP)]
)
    Unit: milliseconds
          expr       min        lq     mean   median        uq       max neval
            GF  45.70551  46.96255  49.2317  47.5768  48.32168  66.40802    10
     namedList 468.25457 474.67816 483.8825 484.5419 493.27464 498.17650    10
so by adding +0 on the end, i prevent DT[...] from recognizing the functions, and it's much slower
 
 
3 hours later…

« first day (1500 days earlier)      last day (2311 days later) »