« first day (940 days earlier)      last day (2354 days later) » 

lmo
12:13 AM
Maybe delete answers for text book problems.
close: no effort
 
del stackoverflow.com/a/47341325 (not sure if it's among lmo's)
 
lmo
@Frank It's not. I've been looking at questions. Just voted on that one.
 
ok, thanks
 
lmo
It's gone
 
yup, rare opportunity for me to use a flag
 
@lmo any reason not to let roomba delete it instead?
 
lmo
Not sure. You build up "delete points" (yet another invisible internet point system) by successfully deleting bad posts that then add to your ability to delete more. This is probably a perverse motivation.
 
@lmo ha ok, no problem in doing it, i think. i find 40 delete votes plenty. is there somewhere you can see how many votes you can cast?
one minor benefit of leaving briefly undeleted is that commenters can reply if @-ed
 
lmo
I suspect it maxes out at 100 like the other votes, though am not sure. In terms of contemporaneous "how many votes," I see that with the "are you sure?" pop up.
Yeah. I agree about the time to respond. My current rule is to give at least 3 hours before casting that vote (and about 1 hour for a close vote) unless the question is like (or worse than) the most recent link I posted.
 
ok. ah i see 29 remaining, guess i have fewer than 40/day
 
lmo
12:40 AM
Yeah. I think it increments every (5?) thousand imaginary internet points and every 10 = (successful - failed) deletes
Also, I seem to recall that some posts put on hold are not roomba'd if the user is of sufficient reputation. Not going to double check this at the moment though.
probably delete
 
1:20 AM
@lmo sure, occasionally I do visit there..I used to be very regular there at one point of time..
going through some previous discussion of making this room as public. I would vote against it. The primary purpose of this room is of moderation and I would like to stick with it. Having said that I am ready to accept whatever decision the majority of room members decide. :)
 
lmo
1:44 AM
@RonakShah Regarding the LQP and other review sites. It is unpleasant to get a review audit on something that you disagree with and then get either a warning or ban. I've more or less given up visiting there as well.
 
 
1 hour later…
 
1 hour later…
 
2 hours later…
6:43 AM
aloha!
 
6:59 AM
@lmo I doubt whether the balance of succesful delete votes adds to your max number of delete votes per day; it hasn't resulted in an increase of my limit
 
7:16 AM
(as far as I can see, I haven't kept an accurate track record)
 
Morning ppl
 
Morning!
 
7:32 AM
Hello all :)
 
7:46 AM
Hello hello :-)
 
7:58 AM
@Queen k
:D
like the car? :)
or hammered (drunk) :)
 
8:13 AM
@Sotos no, @PetterFriberg just needs to add you to Queen's 'master' list
 
@que add user 5635580 sotos 2
 
User: (5635580) sotos as Hammer updated
 
@Sotos done, sorry but I need to add manually hammers (no info from api), thanks @Jaap for notification.
 
Great. Cheers @PetterFriberg :)
@Queen k
ok got it :)
 
hahaha.. :D
 
8:21 AM
1 message moved to Trash can
 
9:19 AM
 
9:30 AM
@Queen k
6 messages moved to Trash can
 
 
2 hours later…
 
2 hours later…
12:56 PM
does    Error in left_join_impl(x, y, by$x, by$y, suffix$x, suffix$y, check_na_matches(na_matches)) :
  std::bad_alloc  mean I m out of RAM?
 
it seems like it
 
I m working on a server with 160GB RAM...
 
maybe it's allocating in an infloop? if you think that's a bug can you file an issue please?
 
1:12 PM
I ll work on it a bit more and see how it goes. I don't think It's a bug. One of my columns is a string with about 120 words each entry
 
1:24 PM
@Sotos Are you using 32- or 64-bit R version?
 
64
 
Is this dplyr code?
Maybe some underline library are 32-bit.
Then it will only allow to allocate 3 GB.
 
It's dplyr yes...not sure about the 3GB though. Task Manager shows that all 160GB are being utilized whilst trying to do that operation
 
Then it's just an Out Of Memory issue.
All joins can eat a lot of memory.
 
1:43 PM
Yup... match worked btw
 
Would be curious about the data format and the join done there
 
lemme share some strs
 
A left join should not exhaust ram unless each Y has a match in x OR x and y are already using 70% of the ram
 
1:59 PM
'data.frame':	1000000 obs. of  6 variables:
 $ id         : chr  "53200537909" "53200537911" "53200537913" "53200537915" ...
 $ cdatetime  : chr  "2017-09-11 23:55:20" "2017-09-11 23:55:20" "2017-09-11 23:55:20" "2017-09-11 23:55:20" ...
 $ eventid    : chr  "4624" "4624" "4624" "4624" ...
 $ username   : chr  "4454" "4454" "4454" "4454" ...
 $ machinename: chr  "ROOTDC04.boc.local" "ROOTDC04.boc.local" "ROOTDC04.boc.local" "ROOTDC04.boc.local" ...
 $ par20      : chr  "%%1833 An account was successfully logged on. >> Subject: >>  Security ID:  S-1-0-0 >>  Account Name:  - >>  Ac"| __trunca
'data.frame':	110950 obs. of  8 variables:
 $ id         : chr  "53546372602" "53546372605" "53546372608" "53546372611" ...
 $ cdatetime  : chr  "2017-09-15 00:09:35" "2017-09-15 00:09:36" "2017-09-15 00:09:36" "2017-09-15 00:09:36" ...
 $ eventid    : chr  "4624" "4624" "4624" "4624" ...
 $ username   : chr  "dbkofaxsvc" "dbkofaxsvc" "dbkofaxsvc" "dbkofaxsvc" ...
 $ machinename: chr  "SSV-DB2KOF-UT1.resources.boc.local" "SSV-DB2KOF-UT1.resources.boc.local" "SSV-DB2KOF-UT1.resources.boc.local" "SSV-DB2KOF-UT1.resources.boc.local" ...
 
I've a hard time linking this with your left_join error above (in terms of observations names)
 
@Tensibai that's because the error happened at C level so R outputs the last R call, which was within a dplyr function. That's more confusing than helpful so we're planning to suggest some changes in R to avoid this.
 
But from the strs, I'd strip the par20 columns from one of the df
@lionel I mean the left_join reference x and y, and none of the columns here
 
@Tensibai I stripped eventid and machine name
par20 I need it though
and cdatetime
 
But it seems being the same in each df
 
2:11 PM
@Tensibai yeah because it's from a private dplyr function
 
if you left_join the second of first one, there's no need to keep duplicated messages in two df
 
Oh God, you are right!!!!
I feel stupid
 
R won't do a magic deduplication of memory pages because the content of both df is the same
 
So basically
this d1[d1$username %in% full_df$username,] will do just fine
 
That sounds like the higher ram consumer in both your df
 
2:13 PM
So I was merging on username and hence extra columns with par20 were created
 
It seems most likely the problem is that your ids are not unique so you end up with a cartesian style join that blows up your memory use. I can't see how you could use that much memory just by carying around a few extra cols. Or maybe that's not what tensibai means.
 
@lionel that's not exactly my problem, I've no idea of the df name at first so I can't be sure which one is x and wich is y here, and without the code preceding this left join I can't guess neither (and I really dislike guessing :p)
 
@Tensibai sorry. Top one is y and bottom one is x
When I remove par20 everything works fine
 
@BrodieG $ par20 : chr "%%1832 An account was successfully logged on. >> Subject: >> Security ID: S-1-5-21-1177238915-796845957-72534"| __truncated__ "%%1832 An account was successfully logged on. >> Subject: >> Security ID: S-1-5-21-1177238915-796845957-72534"| __truncated__ <- I don't know how long is the string here, but by 1M entry this makes a a bunch of MB, double it as it's into 2 data frame and R won't de-duplicate the entries in memory
so you start with a fair amount of datas already using ram before starting
 
It's about 100 words
120
So I guess that was also a problem
 
2:18 PM
If as I suspect those are long logs lines of dozens kB each it could easily eat up a bunch of ram creating the resulting df
 
Yup. It worked instantly when I removed that variable
 
@Tensibai good point.
 
I wonder If impala takes regex so I can query that column directly from the db
 
@Sotos sounds like a good use case of elastic search
 
@Tensibai Yes. We did go to elastic a couple of months ago so that would be totally possible
Impala too! I guess RLIKE is a thing
 
2:27 PM
@Sotos that sounds like you want a semi_join() not a left_join()
 
@hadley That also worked instantly without removing the huge par20 var. !!!
 
2:43 PM
sorry @AnuragChoudhury; please see the room-rules
 
Hi all
 
3:10 PM
hi Steven
 
3:36 PM
Hey @Frank - been ages, how's life ?
 
Last 30 minutes on a Friday seem like ages
 
@Sotos :)
I haven't written a line of code in the past 2 months :$
 
@StevenBeaupré You mean on the last 30mins on Fridays or in general? :p
 
In general
I am (was?) mostly an R hobbyist. No programming involved in my actual job. I merely use it as a tool when it's convenient for the task at hand (which in the past was quite a lot, but not so much these days it seems).
 
Ohh..ok...makes sense then
What is your actual job If you don't mind me asking?
 
3:41 PM
Corporate strategy / management consulting.
 
Corporate Strategy sounds interesting. You must be a good chess player :)
 
I'm pretty decent :P
 
@StevenBeaupré not bad :) you?
 
Great ! I'm allowing myself a "semi" day off today, going out of town for a long relaxing week-end :)
 
cool, sounds fun!
 
@Queen k
 
@Tensibai Now that I think about it more this actually doesn't make sense. If those are all character strings, they should only be stored once. The only thing that gets replicated are the 4 byte pointers (on 64 bit systems) to the strings in the R string pool. And at 1MM rows there is no way you can fill 160GBs with an additional 4MB (4bytes * 1MM rows) unless you were already perilously close to the limit.
Obviously the changes worked, but with 160GBs it feels like there is something funny going on.
 
4:17 PM
@BrodieG I'm pretty sure there's duplication going on somewhere
but that's too late for me to dig on the code today
 
@Tensibai The outcomes suggest that that's what's happening, but there shouldn't be, at least not of the actual strings, only the pointers to them.
 
@BrodieG I don't remember a copy on write per string (per vector yes, but not per entry of the vector)
 
It is the C++ allocator that fails so the R string pool wouldn't be used. Not sure what is being allocated, I'm not familiar with that part of dplyr.
 
@lionel I think I saw this behavior in base R
 
an std::bad_alloc in base R?
I don't think it's possible
 
4:21 PM
@lionel Nope, I mean the duplication of memory usage when copying a vector into two dataframes
 
ah gotcha
 
@Tensibai There two things going on, the duplication of vectors, and the duplication of the contents of the vectors. For almost every type of vector those are the same, except for character vectors were the contents of the vector are not actually stored in the vector.
 
Well that's the behavior when doing df <- df2[,1:5] IIRC, it copies the subset into new memory pages
 
@Tensibai yes, but with character vectors only the pointers to the strings are copied, not the actual strings.
And the big part of the data here is the strings. I can only explain what @sotos ran into right now if he was very close to the 160GB limit to begin with.
 
grmblblb I can't remember the .XX fucntion to see an object address
 
4:32 PM
@Tensibai lol, I always remember thinking to myself "that would be really useful to remember" and failing to do so every time.
But note that won't tell you anything about whether the string is duplicated or not. You'll need the memory address of the CHARSXP inside the STRSXP, which I don't think you can get directly from R.
 
I think I've worked with that to showcase copy on write once
 
@Tensibai A data frame is a vector of pointers so in this case only the pointers would be copied. If you subset rows new vectors would be created though.
 
@lionel in this case x$par20 and y$par20 are of different length
As such they are 2 different vectors
 
@lionel Yes but not new CHARSXPs, which are the big things here.
 
the question is: is x$par20[1] address the same as y$par20[1]
 
4:35 PM
@Tensibai I was replying to df <- df2[, 1:5]
 
@lionel yep indeed, misplaced my range for the exemple
 
Here is what I mean:
> library(inline)
> vec1 <- c("hello", "world", "goodbye", "world")
> vec2 <- vec1[c(2, 4)]
>
> f <- cfunction(sig=c(a='character'), {'
+   for(R_xlen_t i = 0; i < XLENGTH(a); ++i)
+     Rprintf("%p - %s\\n", CHAR(STRING_ELT(a, i)), CHAR(STRING_ELT(a, i)));
+   return(R_NilValue);
+ '})
>
> f(vec1)
0x7f9926bc0290 - hello
0x7f9926bc0230 - world
0x7f9926bc01d0 - goodbye
0x7f9926bc0230 - world
NULL
> f(vec2)
0x7f9926bc0230 - world
0x7f9926bc0230 - world
NULL
 
@BrodieG agree when you copy subset
now what about
 
putting vec inside a df?
 
vec1 <- c("hello", "world", "goodbye", "world")
vec2 <- c("hello","world")
I don't remember R being clever enough to find there's already a character existing when creating a new vector with same chars
(and that would sound pretty crazy to do)
 
4:43 PM
That's exactly what R does
> vec1 <- c("hello", "world", "goodbye", "world")
> vec2 <- c("hello", "world")
>
> f(vec1)
0x7f9926bc0290 - hello
0x7f9926bc0230 - world
0x7f9926bc01d0 - goodbye
0x7f9926bc0230 - world
NULL
> f(vec2)
0x7f9926bc0290 - hello
0x7f9926bc0230 - world
NULL
 
(But I must admit I wouldn't have bet in the same vector neither)
 
It hashes all the strings it encounters and stores them in a global string pool. If a new string comes around, it checks against the hash table, and if it exists it doesn't add it, just references it.
 
So could sotos entries hit a limit in size ?
(or maybe there's a slight difference somewhere like a line ending resulting in different hashes)
 
sotos' issue must be about something else, it's not the R allocator that fails but the C++ one
 
@lionel It fails because memory has been exhausted, so there's something using it somewhere
 
4:47 PM
@Tensibai I doubt it because both data frames and their contents are already in his sessions, so all the strings are already in the string pool.
@Tensibai It could be a bug in the left join, which I think is what @lionel is getting at.
We really need @sotos to show what his session memory usage is just before he does the join to just after, to see how much memory was taken up.
 
Hi all. I'm against making this room public. I'm absolutely fine with tags, renaming and a mention in the R room description (but don't feel strongly about that).
5
 
I think I missed the discussion about turning this room public (well publicly writable)
 
@Tensibai chat.stackoverflow.com/transcript/75819?m=40068200#40068200 not really proposed, just hadley asking about why it's not public
came up in the context of "what's the point of directing people from the R room here if this room is also read-only for most?"
 
 
2 hours later…
7:31 PM
@Frank and Hadley trying to figure out exactly what the point of this room is
 
7:42 PM
@hadley sure. re your proposal about linking the rooms, could you explain why you think it's a good idea? i guess little discussion will happen over the weekend; it tends to be pretty empty here
 
@hadley Fairly simple. This is the place to drop a question/answer that needs moderation and magic happens. A lot of people know their data.table stuff, regex... It's a self help group if you will. :)
 
@BrodieG I will rerun the session on Monday first thing and will post the relevant info here
The semi_join as per hadley's suggestion (which was the correct one) worked instantly
 
@RomanLuštrik so in that case, why isn't it an alternative to the R chat room?
 
@hadley It wasn't designed as an exclusive R chat room AFAIK. I don't see why it should be, either.
 
@Frank because it seems like the R chatroom would prefer people go elsewhere? ("Note that the room is so toxic that you may die within minutes") If you're interested in moderating R questions this seems like the room to do it?
@RomanLuštrik it doesn't have to be exclusive, but as an outsider it seems like the primary purpose of this room is to talk about R and related stuff
 
7:59 PM
@hadley I don't have strong opinions on where something should be done. If one chat room doesn't suit one's needs, move elsewhere or even better, create your own and do whatever you feel the purpose of the room should be.
By that I don't mean you, but anyone.
Odd, these chat rooms have been quite a side thing until a few days ago. What changed?
 
@hadley ok, good point. @Roman if you want to edit it in there, sounds like no one here objects, maybe "Other rooms for R chatter: r-public (help) gmts (SO moderation)" or something. not sure if that matches what hadley has in mind
we can change the description here to "Procrastination + R Moderation" or similar (chat.stackoverflow.com/transcript/message/40069456#40069456). David meant that as a joke, but i'd be fine with something along those lines... i'll edit it later if no one else does, probably
re the R room preferring people go elsewhere, indeed, verbosity is not the norm there, so not much moderation happens there compared to here
 
Not exactly a programming question:
0
Q: Generating the value of a predictor variable necessary for the probability of a response variable to be x in Regression

RasmusI've made a logistic regression model with a categorical response variable of two levels (YES/NO coded as 1/0) and a numerical predictor value (measured in ms). I need to figure out a way to write code that will allow me to generate the value of the predictor variable(ms) needed to give a probab...

(yet)
Too broad, looking for a tool. Take your pick. stackoverflow.com/questions/47357709/…
 
8:16 PM
yeah, doesn't seem likely to become one either. if the effect of the var is nonmonotonic, inverting the mapping from it to the point estimate/fitted value won't be possible. best shuttle them off to stats.se
 
@Frank I m not sure I m getting that correctly. Are they talking about the maximum likelihood estimates and solving for x?
 
@Sotos in pseudocode, i think they have y = fitted(x1, x2, x3) and want to have x1 = fitted^(-1)(y; x2, x3) where x2, x3 are held constant (or don't exist since they have only one regressor)
besides perhaps not being strictly monotone, the function might not have full domain over values of y in 0-1
(i'm not really thinking about the mechanics of logistic regression..)
 
@Frank Is that a problem? You might just run into numeric problems or get really small or large values.
 
I am against any changes to the rules of this room. No tags, keep rep limits, "goofing around" mandatory. Also, against "Procrastination + R Moderation", gives negative impression. Keep it as is, keep it light, keep it a kind place. This is not a "help me" place, this is also not a "moderation tool for R".
9
 
@Frank Oh ok, yes that sounds about right. I went straight to the mechanics I guess... the statistician within :)
 
8:27 PM
@RomanLuštrik well, for a general model of a binary outcome, the right answer might be NA... again, maybe logistic regression is an exception ... haven't used or looked at one in a while
@zx8754 so also no link from the other room here, yes?
 
Yes
I don't understand the need for change. R and R Public needs change, not this room.
 
9:21 PM
@RomanLuštrik they got a lot of public exposure so a lot of people learned about them for the first time
@zx8754 so what is it? I'm trying to understand.
 
@Verv, welcome, you now have write access
 
Thanks (:
 
yullo and welcome, Verv
 
9:38 PM
yullo?
 
tipo
 
mutation of "hello"
 
Gotcha. Those are like Javascript frameworks; I come across a new one every week but they all mean the same thing.
yullo to you (:
 
:)
 
@hadley No need to define. This room existed happily as is. Only came to light because of otheR rooms. Happens to be started by an R user (probably as a joke), so attracted more R users to have "General chat" and do (not only R) moderation. For me, R Public is, well, too public, R is too elite. This room I think is a happy medium. Again no need to compare as this room is not strictly about R.
5
 
10:02 PM
@zx8754 if you don't define what it is, expect people to continue to be confused.
 
@hadley Of course. Maybe this is a good thing? In any case let's hear more from other owners.
 
So far there have been no complaints. People have joined the room, observed what is going on and fit right in. Doesn't really matter what says in the description, I think.
Takes me back to good ol' IRC days. You joined a room, chatted up a few people and that was it.
Perhaps it's just an old fart in me that doesn't need to keep things so tidy (pun not intended)? :)
 
It just seems weird to me to not even mention R in the description, even though seems to be the predominant topic.
I guess I just like things to be clearly labelled
 
10:18 PM
To be frank (not @Frank), I don't think the description adds anything to this room in particular, because the search function in chat.stackoverflow.com is utterly useless.
That being said, I haven't tested any regular expression as a search term.
 
11:13 PM
@RomanLuštrik that presupposes that SO search is the way people find this room
@RomanLuštrik personally, I was taken aback to discover a room where there is a decent amount of discussion of tidyverse (unfortunately accompanied by some incorrect statements and flawed assumptions about my motivations). To me, it felt like the obscure name + lack of mention of R in the description was a deliberate attempt to conceal the room
 
11:27 PM
looks like I got drunk in the night...
https://stackoverflow.com/a/47360316/5977215
3
 
11:45 PM
@hadley The tidyverse discussions are a side topic here. Yes, the room leans somewhat anti-tidyverse (though not as much as the R chat room), and there is the occasional rant against it, but for the most part this room is just a bunch of people who like to moderate Q/As and chit chat. It does end up being principally about R, but I think the group that hangs out here like the status quo.
A bunch of folk that are friendly and productive 95% of the time and don't feel the need to bring much attention to themselves. I probably shouldn't have suggested we link to this room, but as someone else put it there it is a happy medium between the more intimidating R room and R public.
I think most people that request access to it are granted it.
Also, I think the recent high activity periods are not representative of the room at all.
Here is a good example of a typical day in the room. Note there is a lot of stuff that isn't really about R:
Just a bunch of folks that know each other well from hanging out together most days shooting the breeze.
 
@BrodieG ok, I don't understand the reasoning behind not updating the description to reflect that, but I've passed on my point of view and I won't bring it up again
I'm signing off for the weekend. Have a good evening everyone!
 
@hadley Have a good weekend.
 
Uwe
Dear GMTs, I've been silent during the turmoil of the past few days but I feel urged to put in my 2 cents. I'm grateful that I was invited and accepted to take part in the GMT chat room. I am deeply impressed about the friendliness, respect and the way people obviously feelc connected although they probably have never seen each other.
2
 

« first day (940 days earlier)      last day (2354 days later) »