« first day (2582 days earlier)      last day (702 days later) » 

5:43 AM
Goodmorning all
 
 
1 hour later…
6:48 AM
Hi @Sotos and @all :)
 
Hi @Cath
Question: Is foverlaps() the most efficient way to find a date between two dates in data.table?
 
I'd say so
 
7:53 AM
hello
@Sotos depends where you get the 2 dates from; the between function could be an option too
 
8:09 AM
@Sotos I don't think so. Try non-equi join, between or inrange
foverlaps is pretty heavy, don't think anyone using it anymore
 
This is what I had to do to work with `foverlaps`
transactions_df <- transactions_df[, c('start_date_promo', 'end_date_promo') :=  salesdate]
setkeyv(transactions_df, c('intcode', 'start_date_promo', 'end_date_promo'))
setkeyv(Leaflet_df, c('intcode', 'start_date_promo', 'end_date_promo'))
indx <- foverlaps(transactions_df, Leaflet_df, type='within', which=TRUE, mult='first')
 
Try something like transactions_df[Leaflet_df, on = .(intcode, start_date_promo >= start_date_promo, end_date_promo <= end_date_promo), mult = 'first'] or such
 
I was trying @Jaap solution here
 
It looks the same idea to what I wrote, no?
 
@DavidArenburg Yes yes exactly. That's why I mentioned it :)
 
8:20 AM
@DavidArenburg oops...
 
@DavidArenburg things I did with foverlapspreviously, I do now with non-equi joins
 
I don't think I ever compared between them, so don't take my word for it
 
131 messages moved to Trash can
 
 
1 hour later…
9:49 AM
It seems that sqldf package might be more efficient here, i.e.
sqldf('SELECT *
       FROM Leaflet_df v
       JOIN transactions_df p
       ON v.intcode = p.intcode AND
       salesdate BETWEEN v.start_date_promo AND v.end_date_promo'
          )
 
@Sotos it runs faster?
 
@DavidArenburg As a first glimpse yes but I ll do some benchmarking after the break
 
that would be strange. What data size?
 
Just a subset for now
 
10:07 AM
so what are time differences? milliseconds? I would guess it makes sense for huge datasets, for small ones, you can use whatever I guess
 
11:07 AM
@DavidArenburg Can I keep start and end dates here? It drops them after the join
 
@Sotos If you''ll do assignment or such I guess
 
11:43 AM
@DavidArenburg yeah, DT way faster
 
Surprised? :)
 
@RomanLuštrik More amazed :P
 
12:15 PM
awesome
 

« first day (2582 days earlier)      last day (702 days later) »