« first day (945 days earlier)      last day (2349 days later) » 

12:13 AM
It's nice to run in parallel on 8 core machine.
Instant runtime speed-up in 8 times. :D
 
 
2 hours later…
 
1 hour later…
 
2 hours later…
6:38 AM
@Michael you now have write access.
 
Thanks! :) I have a small question about duration arithmetic in ISO 8601. Not sure if it's a good fit for SO, and I do not know any reference implementation of the standard I could use to test the case.

The question is, what is the last day of the (closed) interval starting 2017-03-20 with a duration of P1M12D (1 month and 12 days in plain English)? If we shift by month first then by day, we come to 2017-05-02 because April (04) has only 30 days, while if we shift by day then by month, we come to 2017-05-01 because March (03) has 31 days.
 
7:13 AM
Hello
 
Morning all
 
7:40 AM
good morning
 
Hello :)
 
Uwe
@MichaelLeBarbierGrünewald The lubridate package distinguishes between periods and duration (see the vignette). Adding a period of 1 month to a given date may add 28, 29, 30, or 31 days depending on the specific date.
 
Hello hello :-)
 
Uwe
7:56 AM
@m0nhawk If the dates are in ISO8601 week format, none of the "%Y %W %w" or "%Y %U %u" formats will give the correct results.
 
Uwe
@m0nhawk (Shameless self-advertising on) Try ISOweek::ISOweek2date(c("2015-W01-6", "2015-W01-7")) which gives "2015-01-03" "2015-01-04". See also longer explanation here. (Shameless self-advertising off)
 
@Uwe In this case it is pretty warranted I'd say
 
Uwe
8:24 AM
The ISO8601 week definition has the benefit that all days are contiguously assigned to a period of 7 days without gaps or overlaps (or "crippled weeks" with just 1 or days as with the US and UK conventions). But there is a penalty that the year in the ISO week date may differ from the calendar date, e.g., ISOweek::date2ISOweek("2016-01-01") as well as format(as.Date("2016-01-01"), "%G-W%V-%w") return "2015-W53-5". So, Jan 1st, 2016 belongs to the last week of 2015.
Note that "%G" has been used instead of "%Y" to get the correct year. Unfortunately, "%G" doesn't work on input.
 
8:43 AM
how can I call base intersect when I have dplyr loaded? By default when I write intersect it makes a call to dplyr intersect.
 
@RonakShah base::intersect()
 
@Sotos cool..thanks! I didn't knew there is something as base:: as well. I thought its only for additional packages.
 
@RonakShah YW. yup...that's a thing :)
 
@Uwe Thank you! Adding only months or only days to a date is pretty well defined, as I illustrated it in the example things needs to be clarified when using durations mixing months and days.

And yes the name “duration” used by ISO 8601 does not describe accurately the objects it names, “period” could be a more sensible name – and the leading P in the notation P1M11D hints at “period” as well.
I'll take a loot at lubridate to see how the perform the compuation.
 
9:10 AM
@RonakShah base is also a package. Just the base one. :)
@Uwe Thanks. But I found out that this is neither of UK, EU, US or ISO.
It's some cripple combination.
 
Uwe
@m0nhawk Yep, people like to re-invent the wheel instead of looking for exisiting standards.
 
Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.
5
 
9:31 AM
 
So I tried R/lubridate and OCaml/calendar which both maps 2017-03-20 + P1M12D to 2017-05-02
 
Hello @MichaelLeBarbierGrünewald
 
Sounds the perfect case to use regexes ;)
 
9:53 AM
@RonakShah
a <- c(123, 234, 432, 223)
b <- c(234, 238, 342, 325, 326)
regex <- paste0(substr(a[1],2,3),'(\\d)')
sapply(a,function(x) { z <- sub(regex,paste0(x,"\\1"),b); z[!b %in% z] } )
       [,1]   [,2]   [,3]   [,4]
[1,] "1234" "2344" "4324" "2234"
[2,] "1238" "2348" "4328" "2238"
Would it fit your needs ?
oops
Missed a point
unlist(sapply(a,function(x) {regex <- paste0(substr(x,2,3),'(\\d)'); z <- sub(regex,paste0(x,"\\1"),b); z[!b %in% z] } ))
[1] "1234" "1238" "2342" "4325" "4326" "2234" "2238"
 
@RonakShah This is terrible but maybe a base to improve : unlist(sapply(common, function(sub) apply(expand.grid(a[sub_a %in% sub], b[sub_b %in% sub]), 1, function(x) paste0(x[1], substr(x[2], 3, 3)))))
 
Checking @Tensibai and @Cath .
 
I think, probably some grep with value=TRUE and sub(...) with capture to replace. Maybe there is a way with strsplit after 2 first digits for a, last for b.
 
@Jaap version with setnames is great
The clever one :)
 
thx :-)
 
10:01 AM
yeah this is the one
 
Time to benchmark :p
> microbenchmark(Jaap(a,b),Tensi(a,b))
Unit: microseconds
expr min lq mean median uq max neval
Jaap(a, b) 894.032 936.795 1264.1629 979.986 1099.0090 15363.32 100
Tensi(a, b) 129.429 134.561 204.1791 148.815 158.0805 4703.93 100
 
Eh, Matt stole my answer :D
 
that's a clear win for you ;-)
maybe include in your answer (with possibly also a the other answers included)
 
On such a small vector yes but I'm trying on larger ones
that's the plan, adding a benchmarck of all answers
 
@Tensibai does Matt have any chance?
 
10:11 AM
Unit: milliseconds
        expr      min       lq     mean   median       uq      max neval
  Jaap(a, b) 348.4489 350.0728 355.3469 354.1167 357.7211 366.8373    10
 Tensi(a, b) 402.7534 409.4800 413.4044 411.4019 420.0698 422.5065    10
With vector a of 400 obs and b of 500 obs
@zx8754 not included now
a 400 b 5000:
> microbenchmark(Jaap(a,b),Tensi(a,b),times=10)
Unit: seconds
        expr      min       lq     mean   median       uq      max neval
  Jaap(a, b) 3.865072 3.866360 3.930771 3.904093 3.960393 4.126447    10
 Tensi(a, b) 3.978255 4.000754 4.039312 4.020096 4.047571 4.176594    10
 
@RonakShah in real case what is the size ratios of a and b? I am guessing a is a lot smaller than b?
 
@Tensibai looks like the difference stays approximately the same after a certain minimum size of the vectors
 
@Tensibai Hi :)
 
@Jaap I assume mine would be longer with larger A
(I loop over A after all)
 
Hmm Matt's solution almost same as Jaaps, merging dataframes, right?
 
10:16 AM
> microbenchmark(Jaap(a,b),Tensi(a,b),Heikki(a,b),times=3)
Unit: milliseconds
         expr      min       lq     mean   median       uq      max neval
   Jaap(a, b) 341.9176 341.9987 346.7905 342.0798 349.2269 356.3740     3
  Tensi(a, b) 407.4237 409.8235 411.7927 412.2234 413.9772 415.7311     3
 Heikki(a, b) 127.8528 135.3049 143.0249 142.7569 150.6109 158.4649     3
 
This would be extremely slow I guess...
 
withh 400/500
 
sub_a <- substr(a, 2, 3)
sub_b <- substr(b, 1, 2)
d1 <- expand.grid(a, b)
d2 <- expand.grid(sub_a, sub_b)
i1 <- d2$Var1 == d2$Var2
d1 <- d1[i1,]
d1$Var1 <- substr(d1$Var1, 1, 1)
do.call(paste0, d1)
 
Matt's solution won't be benchmarked, I don't with to add dplyr :)
(I mean to install it)
 
could be converted to base merge
I will update, wait
 
10:17 AM
@Sotos post as an answer?
 
@Jaap Ok. Will do :)
 
In real case, size of a and b almost remain the same however, there is a loop on top of this which would keep on increasing the number of digits for a from 3 upto 9 keeping the logic same. (last 2 , first 2)
 
could you post a message here with the complete benchmark code?
(also including how you created the larger vectors)
I will then include a complete benchmark
 
a <- rep(c(123, 234, 432, 223),1000)
b <- rep(c(234, 238, 342, 325, 326),1000)

Jaap <- function(a,b) {
  a <- setNames(a, substr(a,2,3))
  b <- setNames(b, substr(b,1,2))

  df <- merge(stack(a), stack(b), by = 'ind')
  paste0(substr(df$values.x,1,1), df$values.y)
}

Tensi <- function(a,b) {
  unlist(sapply(a,function(x) {regex <- paste0(substr(x,2,3),'(\\d)'); z <- sub(regex,paste0(x,"\\1"),b); z[!b %in% z] } ))
}

Heikki <- function(a,b) {
  sub_a <- substr(a, 2, 3)
  sub_b <- substr(b, 1, 2)
unfinished
but 4000 and 5000 are a pain for my machine
 
BTW @Cath , your solution works too. :)
 
10:22 AM
Unit: milliseconds
         expr      min       lq     mean   median       uq      max neval
   Jaap(a, b) 333.3214 335.1851 337.0343 337.0489 338.8908 340.7328     3
  Tensi(a, b) 405.1740 406.4612 415.3688 407.7484 420.4662 433.1841     3
 Heikki(a, b) 126.6629 138.7892 145.0983 150.9155 154.3160 157.7165     3
  Sotos(a, b) 147.8146 148.5218 149.8334 149.2289 150.8428 152.4567     3
 
@Tensibai Matt's now has base solution.
 
(added sub_a and sub_b definition in Sotos part as others don't have this need=
 
Somehow benchmarks do not correlate with upvotes :/
 
Yes, I like to pick up where others left off...Follow their train of thought :)
 
Unit: milliseconds
            expr      min       lq     mean   median       uq      max neval
      Jaap(a, b) 341.0224 342.6853 345.2182 344.3482 347.3161 350.2840     3
     Tensi(a, b) 415.9175 416.2672 421.9148 416.6168 424.9134 433.2100     3
    Heikki(a, b) 126.9859 139.6727 149.3252 152.3594 160.4948 168.6302     3
     Sotos(a, b) 151.1264 164.9869 172.0310 178.8474 182.4833 186.1191     3
 MattWBase(a, b) 286.9651 290.8923 293.3795 294.8195 296.5867 298.3538     3
@Sotos I did the same for Heikki answer
 
10:25 AM
Yup
I thought It would ve been way slower than that :p
 
surprised with Sotos's, thought would be dead slow...
 
So I guess expand.grid is not as expensive as I thought
 
@RonakShah yes but it's (very very) ugly !! ;-D
 
I didn't really compared the answers not added unique or other things to get identical results
 
Thank you guys for all the answers. I was not able to find a single solution an hour back and now I have 5. This is overwhelming. I am going to let the solutions sink in, apply it to my real case and then accept the best answer.
2
@Cath you can maybe beautify it a bit and post a solution, we are having some surprising results till now. Who knows ;-)
 
10:36 AM
@RonakShah ah yes maybe I'll try that but unfortunately I don't have the time right now :-(
 
60 points for the worse solution but... with a benchmark :p
 
that answer skyrocketed me past the 40k milestone :-)
3
 
@Jaap With a nice bronze badge at the same time
 
that benchmark is taking ages on my system to complete :-\
 
And to think I was too embarrassed to add mine :)
 
Uwe
10:56 AM
@Jaap congrats
 
Uwe
11:39 AM
@Tensibai That's pretty good, a total of 44 upvotes cast on 1 Q and 5 A (now 6 A). My experience is quite different.
 
12:25 PM
@Tensibai new benchmark:
Unit: milliseconds
             expr      min       lq     mean   median       uq      max neval cld
      Jaap1(a, b) 629.8433 642.3807 750.2660 784.9615 846.7310 847.4137     5   c
      Jaap2(a, b) 219.9508 226.1149 242.8806 239.2496 255.0196 274.0683     5 ab
      Tensi(a, b) 677.4269 683.9475 701.2738 689.6502 724.7414 730.6030     5   c
     Heikki(a, b) 256.1418 261.9554 329.6054 295.2794 303.7601 530.8905     5  b
      Sotos(a, b) 268.9945 280.3224 332.7791 295.4542 307.0244 512.1002     5  b
(also added to my answer now)
 
12:39 PM
@Jaap @Tensibai maybe post benchmark as separate answer as wiki?
 
will do
 
12:55 PM
Can this be duped with this?
 
@Jaap do you mind adding mine to your benchmark?
 
@zx8754 will do in a sec
 
@zx8754 Nice. Math oriented answers always put a smile to my face :)
 
After all those answer, had to think outside the box :D I am sure it has some logic holes...
 
@zx8754 very nice! I suspect nobody can beat that
 
1:25 PM
i'd do a join... looking at those benchmarks i'm not clear what larger vectors y'all are using to scale it up
library(data.table)
aDT = as.data.table(tstrsplit(a, ""))
bDT = setnames(as.data.table(tstrsplit(b, "")), c("V2", "V3", "V4"))
merge(aDT, bDT)[, .(V1, V2, V3, V4)]
so many answers, i'm not even sure if that's there already, i see "merge" already
 
1:42 PM
@Frank it is not there, it is either base merge or dplyr::join
 
@Frank a small adaptation:
aDT <- as.data.table(tstrsplit(a, ""))
bDT <- setnames(as.data.table(tstrsplit(b, "")), c("V2", "V3", "V4"))
merge(aDT, bDT, allow.cartesian = TRUE)[, paste0(V1, V2, V3, V4)]
 
@Jaap replacing sappy to forloop for my solution, seems like makes it 3x faster
 
2:07 PM
@zx8754 nice improvement! will try to run a new benchmark on solutions
the adapted solution of @Frank seems to be the fastest
 
2:26 PM
@zx8754 suprisingly when i run a benchmark like Doscendo did, the for-loop is 3x slower ??
anyway, updated the cw-answer with the benchmarks; including @Frank's solution
 
3:24 PM
Lovely to see this room returned to one of its key functions: getting insane performance out of data.table ;-)
 
3:36 PM
@DirkEddelbuettel It'd be nice to also see some Rcpp performance :)
 
4:32 PM
@Jaap Thanks, forloop didn't scale well :)
 
5:22 PM
@SymbolixAU, have you ever figured this out? Perhaps you can close that otherwise.
 
arg, 4 kids born practically at the same month- I'm starting to hate birthday parties
 
 
1 hour later…
6:28 PM
@Jaap ah yup, need allow.cartesian if there are multiple matches. thanks for the fix and adding to the benchmark
@Jaap also, i wonder if there's a more efficient way to convert to data.table... oh right, setDT actually works: bDT = setnames(setDT(tstrsplit(b, "")), c("V2", "V3", "V4"))
i expect that'd be faster, though maybe as.data.table.list does similar magic already
 
7:09 PM
added a new benchmark (with more runs); your solution is on par with Docendo's
maybe post as an answer?
 
7:33 PM
@Jaap will do, ah, i'd never use outer but that's a good solution for this special case, since the result won't grow combinatorially, just to 10x10. actually, a solution should probably start with a <- unique(a); b <- unique(b) .. or maybe the sample data in your benchmark shouldn't have duplicates to begin with (not sure case for @Ronak 's real data). the space of possible values is quite small, 900^2
chaning as.data.table to setDT seems to make little diff:
Unit: relative
             expr        min         lq      mean     median         uq       max neval   cld
      Jaap1(a, b)   8.451413   7.977024  7.148966   7.844091  6.8270031  4.602984    50   c
      Jaap2(a, b)   2.192496   2.075547  1.912818   2.184319  1.4527040  1.718718    50 a
      Tensi(a, b) 116.574869 118.628236 91.841228 119.072724 79.3152157 44.046794    50     e
     Heikki(a, b)   8.158465   8.005155  6.987243   7.833101  6.4463358  4.717050    50   c
      Sotos(a, b)   9.937472  16.821282 14.621707  18.068715 13.7778652  9.000426    50    d
(testing with your a and b with dupes dropped)
if we were working with chars instead of digits, i guess the outer way would break for a sufficiently large char set ... not really sure what that would be (as an monolingual american)
@Jaap added Matt's, translated to data.table and it's comparable too:
> microbenchmark(Jaap1(a,b), Jaap2(a,b), Heikki(a,b),
+ Matt_base(a,b), Matt_dplyr(a,b), MattDT(a,b), Docendo(a,b),
+ zx8754(a,b), zx8754for(a,b), Frank(a,b),
+ times = 15, unit = 'relative')
Unit: relative
             expr      min       lq     mean   median       uq       max neval cld
      Jaap1(a, b) 8.108384 7.730540 7.619380 7.625632 7.702138  7.308563    15   c
      Jaap2(a, b) 2.067804 2.010881 1.986351 1.966517 1.921280  2.129412    15 ab
     Heikki(a, b) 7.619632 7.501333 7.345213 7.405260 7.199062  7.031387    15   c
 
7:51 PM
@Frank thx :-)
 
now my only remaining thought is.. did Ronak actually care about performance? :)
4
 
8:31 PM
@Axeman - no I didn't, and I can no longer reproduce it so I've closed it
 
@Frank we will see when he reports back ;-)
 
9:04 PM
@SymbolixAU alright, cheers
 
 
1 hour later…
10:24 PM
@DavidArenburg So lucky, make one party for all and be done with it.
 

« first day (945 days earlier)      last day (2349 days later) »