last day (14 days later) » 

6:47 AM
6
A: Reduce computing time for reshape

Ricardo SaportaBenchmarks Summary: Using Stack (as suggested by @AnandaMahto) is definitely the way to go for smaller data sets (N < 3,000). As the data sets gets larger, data.table begins to outperform stack Here is an option using data.table dtt <- data.table(data) # non value columns, ie, the...

 
You get a +1 here for sharing an answer and a benchmark ;) I don't know much about benchmarking, though, but I think your previous benchmark, where you tried to replicate a larger dataset before benchmarking would result in different relative times here. If you have the time, it would be great to add in my suggestions too, and test on a larger dataset.
 
@AnandaMahto, please see updated answer
 
Thanks for the work you've done here. Again, I don't know much about benchmarking, but I'm still hesitant to fully accept these results. Try clearing your workspace and running from scratch, but only doing one replication. I consistently get the second alternative I mentioned as fastest, stack as second, and data.table as third. I am guessing this is because of your use of := with data.table, which makes it so that particular step doesn't need to be repeated on further replications.
 
@AnandaMahto, you are 100% correct. Something in my workspace was throwing off the results.. editing the answer now. Thank you for pointing that out!
 
Good luck. I was trying to figure out what's going on too. (I'm proud of my answer, even if data.table comes out on top in the end.) Looking forward to your edits, because I'm learning something about benchmarking in the process. (But as it's past midnight where I am, I might call it a night.) :)
 
6:47 AM
@AnandaMahto, Your method is slick! And looks like for N<3K, it is the way to go. For N > 3K, data.table outperforms
 
@RicardoSaporta, great work on the benchmarks.
I've played a little bit further, but I'm on a slow system and I'm impatient, but a few other notes:
The OP's data are 3000 rows x 8000+ columns.
When I set the columns larger than a 1000, data.table gives a strange warning, but as it is a warning, and not an error, I'm not sure what to make of it. I get the following:
Warning message:
In `[.data.table`(dtt, , `:=`(Date, d)) :
  tl (16006) is greater than 1000 items over-allocated (ncol = 8003). If you didn't set the datatable.alloccol option very large, please report this to datatable-help including the result of sessionInfo().
Also, FYI, add the following to your benchmarks for a surprise:
Manual <- quote(data.frame(data[1:3],
                           vals = as.vector(as.matrix(data[-c(1, 2, 3)])),
                           date = rep(names(data)[-c(1, 2, 3)], each = nrow(data))))
Anyway, I'm sure there's also a threshold on how wide the data are before data.table takes over, but overall, I must say that this is a good testimonial for some of the base R functions, because I find the base R options quite readable, and in many cases, shaving off a few minutes of processing but reducing general readability is not a trade I like to make :)
(But I do love data.table and use it when I can, and fread is f-ing awesome.)
 
 
2 hours later…
9:00 AM
@RicardoSaporta, in case you're interested, this Gist reproduces the warning for me. I've shared it with Matthew too, so hopefully there's an answer soon!
 
 
1 hour later…
10:23 AM
I've also posted this RPubs document that you might be interested in: rpubs.com/mrdwab/reduce-computing-time
Any idea what's so magical about 3000 rows that makes base R choke so badly?
 

last day (14 days later) »