« first day (554 days earlier)      last day (2621 days later) » 

hhh
6:14 AM
Error: invalid multibyte character in parser at line 1
Execution halted
Any idea what can cause this err?
 
check the format of your .R file for non-acii characters
 
hhh
how can I do that? Some one-liner in Vim?
Or one-liner to remove them?
 
I don't use vim. sorry.
 
hhh
$str !~ s/[^[:ascii:]]//g;
perl
 
 
4 hours later…
10:34 AM
Hello @BenBarnes, haven't seen you around.
 
 
1 hour later…
11:56 AM
@RomanLuštrik, I pop in and out to see if anything's happening, but often no one's around when I check.
 
What time zone are you in?
 
Hello. Central euro.
 
Ah, I see, you're from zi Germany. Well, a couple of us here are Euro and I suspect @mdsumner never sleeps. Business is slow lately, though.
 
Ah, and you're in Ljubljana. I've come across some very interesting work from the uni there.
Business is slow, unless you're padding strings with kittens.
 
12:16 PM
I see you're in epidemiology. Who do you know from our Uni?
 
"know" is a bit strong, but I've exchanged a couple emails with Maja Pohar Perme, in conjunction with her R package relsurv
 
Ha!
I listened to one of her lectures last Friday and I was attending her lab class last semester.
 
Ha!??
Ahh.
 
I was quite impressed by the survival analysis. Very complex topic, plenty of room for work. :)
 
She wrote a very nice article last summer that added a lot to the theory and practice of relative survival analysis.
OK, I'm gone for now. Nice to chat with you!
 
12:37 PM
Did I see @jorismeys briefly on SO?
 
hi everyone
I'm currently at the EGU conference in Vienna
anyone else present?
there are 10.000 people, so there might be a chance :)
 
12:52 PM
Whoa, 10k. I can't even imagine. :)
 
:)
it is actually like 20-30 conferences in the same building
so the individual sessions are much smaller than 10k
But is still large...
 
 
2 hours later…
2:42 PM
I attended the 2005 SFN conference back in my neuroscience days
now that was a meeting
of course attendance dropped by 10k in 2006, when I no longer attended. correlation? yup. causality? definitely.
 
Howdy everyone.
 
@gsk3, maybe people just don't like Athens, GA? Oops misread -- that's Atlanta ...
 
I certainly don't :-)
Would be cool to compile data on a bunch of conferences and see which cities cause attendance to drop
fixed effect for the conference and a non-linear time trend...would need a lot of data
hi @JoshuaUlrich
 
 
1 hour later…
3:55 PM
More food for thought: lambda-the-ultimate.org/node/4507
2
@andrie I forgot lack of concurrency when we were listing big R shortcomings.
 
@IanFellows My list of shortcomings:
Not supporting change by reference.
(i.e. copy on change, thus leading to all kinds of problems, including slower performance)
 
@Andrie environments support change by reference though.
Granted, they are not the normal paradigm. The proto package takes advantage of environment change-by-reference, if you're interested.
 
4:22 PM
@JoshuaUlrich I know that anything is possible...
... But my point is more about the "normal" paradigm.
 
@Andrie I realized that when I stopped to think... hence my "granted" sentence. :)
 
:-)
 
Out of curiosity, what do you think the proportion of all R users is for whom performance/speed of R is basically never going to be a limiting factor as long as they write "reasonable" code?
 
@andrie what about reference classes. they are part of the core language now.
 
@joran That's going to be really hard to determine. R users are so diverse, I don't think anyone knows enough about all the different disciplines to make anything more than a wild guess.
 
4:39 PM
Well, I wasn't hoping to be all scientific about it. I just think that there's a pretty simple explanation for "why R is so popular despite all its drawbacks": its drawbacks simply don't matter to a very large proportion of its users.
 
Wild guess: Most users are not speed limited, but are memory limited. I bet 10x more users run into "can not allocate vector of size..." than speed problems
 
4:58 PM
@IanFellows Being part of the base distribution doesn't mean its core to the language.
@IanFellows I agree that memory issues is also a problem. But again, only applicable to a subset of people.
I have never really run into a memory allocation problem. My data simply isn't big enough.
But poor performance and how to solve it with vectorization crops up on SO every day.
 
@andrie you are probably right
 
@Andrie In your own work do you routinely find that even with well written code R's performance (i.e. speed) isn't good enough, and you have to turn to other languages or supplements (like Rcpp or something)?
 
Hi all,
joran, if you want to start counting: I'm continuously hitting memory limits. Memory limits cause speed limits: how much I can parallelize is usually determined by available memory. In a few weeks you can hopefully mark me as real speed limitation as well. So far, everything could be done within a few nights....
 
5:20 PM
@cbeleites How much RAM is available on the machines you use; and are they 32 or 64 bit?
 
5:36 PM
@joran No, not really. Properly written R code is really quite fast. But it does mean I have had to jump through all kinds of hoops to replace plyr with base R code or data.table, and to rewrite a bunch of graphics in lattice where I had it in ggplot2 in the first place.
 
@Andrie That matches my experience as well: it's not that R can't be fast, its that making it fast isn't nearly as straightforward or convenient folks expect.
 
@joran Agreed. And some stuff is just slow. Like reading a large csv file. (150K rows takes 6 seconds, for example, but using native Linux tools this is pretty much instantaneous.) And I have not seen any way around this.
wtf?
0
Q: How do I locate values based on their pixel and line grid?

Sami YemeinI have 3 files A ,B ,C as binary files. C represents the values of tepm measured every 3 hours for one month so it has 674200 columns and 248 rows.values of A represent the correspond lines(these values could be within 1to360) and , while values of B represent the correspond pixel(1to720) for eac...

 
@Andrie how many columns, and what are their classes (mix of numeric/character)?
 
@JoshuaUlrich 18
(Not even that large, come to think of it.)
Mostly character classes, with some numeric thrown in.
 
Do you guys have any feel for what "common" tasks in R that are "too slow" look like? I'd love to get my hands on them!
 
5:55 PM
@Tommy: nice to see you in here. You may be interested in the Strange Loop conference. If you come, I'll buy you some beers. :)
@Andrie I see what you mean. foo <- read.csv("bar") takes quite awhile. It's ~3x faster if you specify colClasses and nrow though.
 
@JoshuaUlrich - Thanks, nice to be here :) That conference actually looks interesting, thanks! ...so I might hold you to your promise!
 
@JoshuaUlrich Strange Loop is a killer conference! Were you there this year?
 
@Justin I attended in 2011. Just registered for 2012.
 
I've yet to talk my boss into this year's conference
 
6:27 PM
@Andrie - so reading data is one common task that is "too slow". Are any other parts of your analysis "too slow"?
...I'm asking since I spend a lot of time trying to optimize things :)
...but I need good use cases!
 
@Tommy I regularly have to read in 4M line csv files and it is quite slow. The other slow step is writing out plots to files. (I'm on an i7 with a SSD...)
 
@Justin - so is the slow part generating the plots then? Or the actual writing to disk?
Is it like a scatter plot with millions of markers, or just a lot of stats to calculate before drawing a few markers?
 
6:43 PM
@Tommy ggplot2 used to be slow in general. The latest version might be quicker, I'm not sure.
But no - everything has the possibility of being slow, unless you are very careful.
In my case, I do a lot of survey analysis. A single question might only take 2-5 seconds to analyse, including analysis, creating plots and writing results to file (I wrote a memory caching system, so the file write isn't the bottle neck.)
But do that on just 25 qeustions with 4 cross-breaks each, and you're suddenly looking at report generation time of ~6 minutes.
So I had to write another caching system that checks whether things changed between the last run and this run, and cache the plots to disk, then compare the MD5 sum, then... you get the picture.
 
@Andrie Thanks, very informative!
 
OK, granted, R wasn't designed for this type of workflow.
But it works. I love it. I just wished it had some native way of being faster all the time.
 
@Andrie - so if some parts of the analysis were faster, it still wouldn't improve the overall use case performance that much then...
 
@Tommy I haven't profiled them specifically. But I'd be interested in Andrie's mem caching system for the file writes!
The plots vary widely, some are many points some are simple paths.
 
@josh
 
7:00 PM
@Tommy Well it would, if I can reduce time by a further 75%! It's already down from 6 seconds to 1.5. So why not another 75%?
 
@JoshuaUlrich: Unfortunately only 8 GB for 12 cores (but yes, of course they are 64 bit). But our imaging instrument can produce something like 250 GB of good quality spectral images a day... The 8 GB server was bought by computational phyicists: they need computational power but not that much memory. We usually have computationally rather easy problems, but quite larger data...
 
@Andrie - could you elaborate on the analysis step then?
@cbeleites - What kind of analysis do you do? Would less memory-intense algorithms be the solution for you then? - Or perhaps just buying more memory ;-)
 
7:36 PM
@Tommy, yes we're thinking how to convince burocracy that we can use some existing money and spend it on RAM...
But, honestly, less memory-intensive algorithms would make me use more algorithms on more data, I guess ;-)
(Which reminds me: I have to take a look at flashclust)
I mean, maybe I'm spoiled, but at my last unversity I had a desktop with 8 GB of RAM...
 
@cbeleites is the machine a rack server or a desktop?
 
The 12 core is a server
just, you know: it is another group who own it. And our project is incredibly picky about what money is to be spent on what... I tried to argue that an external HD for backups is a consumable, but no way...
 
That's a shame. You could build a desktop for $2-3k and it would be better than that server for your purposes.
 
On the whole, if I have, say 250.000 spectra x 400 data points (there could be more, but I don't take more - someone has to analyse all the data...) that amounts to 750 MB. I can comfortably live with that as long as my colleague stays away from the server with his data. Fortunately, our desks are side by side, so this kind of communication is easy: "Do you need the server for the next hours?" "Go ahead, you can have it"
 
I now have access to a lightly used cluster with 108 cores and a grip of memory. I don't know how i lived before...
 
7:49 PM
I mean, obviously the things you do with the data will depend on what resources you have. Fortunately the imaging spectrometer produces the raw data in files of 64 x 64 spatial pixels x typically ca. 4000 wavelengths, i.e. chunks of 128 MB. So I do a whole lot of data reduction pre-processing first with moderate parallelisation (say, 4 threads), and decide parameters on a small random sample of the data.
@Ian: I believe that...
 
@cbeleites - So you call hclust then? That is memory intensive for sure, but is that a bottleneck for you?
 
8:22 PM
I keep thinking that what I'd really need is a pre-processing that clustes together everything up to a given level of dissimilarity. But with other (for us, mainly k-means and fuzzy c-means) clustering, I've had the experience that I missed small but very different and interesting clusters, even though I was running the k-means a couple of times.
Actually, what I found powerful technique is a combination of HCA and then fuzzy clustering with the start centroids from the HCA: the dendrogram is very useful in judging the number of clusters.
however, I have to run now. good bye, all.
 

« first day (554 days earlier)      last day (2621 days later) »