« first day (2137 days earlier)      last day (1156 days later) » 

7:42 AM
Happy friday all :-)
 
 
6 hours later…
1:12 PM
@Cath TGIF :P
 
1:33 PM
@Jaap ... and, I'm on vacations tonight ^^
 
nice! I had a short vacation last week :-)
 
1:50 PM
Hello
yay, kids on holidays next week... meaning all the same as usual for past 10 months :/
 
2:05 PM
@Jaap @zx8754 Based on some preliminary benchmarks on the files of the sizes I expect, data.table::fread seems to be the faster solution out there.
 
@RomanLuštrik Oh did you mean to use sqldf to subset data before importing?
 
Basically I have many files and I need just one line from each of them.
One option would be to create an index and use that to skip certain amount of rows before reading one line. Turns out if I read the entire file using fread and subset one row, the extra cost of is 0.1s. And this solution is, as you can imagine, far faster than anything read.table can spit out.
 
I prefer to use d <- fread(cmd = "grep something myFile.txt")
 
Ha, I'll add that to the benchmark, I totally forgot about it.
 
If you know which row, I think you can tell fread to skip n, then read n rows?
 
2:10 PM
That's what I tried, but there's really no speed gain for some reason.
Or very little I should say.
> system.time({
+   i2 <- fread(lof.i, skip = 280000, nrow = 1)
+ })
   user  system elapsed
  0.702   0.029   0.735
> system.time({
+   f2 <- fread(lof.i)
+   f2 <- f2[f2$gene_id == "ENSG00000185787.11", ]
+ })
   user  system elapsed
  1.419   0.000   0.847
 
I see, then try cmd something like: fread(cmd = "sed -n '2 p' myFile.txt") # extract 2nd row using sed
or yes, use to grep the gene name
 
> system.time({
+   z1 <- fread(cmd = sprintf("zgrep ^ENSG00000185787.11 %s", lof.i))
+ })
   user  system elapsed
  0.878   0.093   0.791
 
not sure comparison of such small time is reliable in a "one-shot" manner. Time can depend on what your system is also up to at the specific moment
nice cmd parameter of fread :-)
 
 
1 hour later…
3:33 PM
@RomanLuštrik it's better to use f2 <- f2[gene_id == "ENSG00000185787.11", ] when using data.table (so ditch the f2$ part)
 
I don't know...
> system.time({
+   f2 <- fread(lof.i)
+   f2 <- f2[gene_id == "ENSG00000185787.11", ]
+ })
   user  system elapsed
  2.129   0.000   1.144
> system.time({
+   f2 <- fread(lof.i)
+   f2 <- f2[f2$gene_id == "ENSG00000185787.11", ]
+ })
   user  system elapsed
  1.716   0.000   0.958
 
@RomanLuštrik hmm, didn't expect that; might that be due to what @Cath said as well?
 
I tried a few runs and it's pretty consistent. I'll do a more comprehensive testing using proper benchmark tools over the weekend.
It's not exactly my forte to debug a test in python and do R at the same time. :)
 

« first day (2137 days earlier)      last day (1156 days later) »