last day (21 days later) » 

10:45 PM
2
A: Binning values in R with multiple files

r2evansThere are several ways to do it, I'll provide one method using base functions. (An alternative would be to use dplyr, also well suited for this. However, the base example should be simple enough.) Generate Data (This is here merely because we don't have any of your data.) n <- 10 for (ii in 1:...

 
Awesome I'll try implementing this now and will get back to you asap but thank you so much for this!
Apologies for taking time to get back to you about this, the files I have are very large and it's still processing allData <- unlist(sapply(fnames, read.table, sep = ',')) part
 
I'm not surprised, files with large amounts of data on a single line can be problematic. If the files are that large, perhaps you should consider an alternative way of reading them in (in lieu of read.table). I don't have enough experience in any one to make a recommendation, but there are several previous questions on the topic (e.g., 1727772 and 11782084).
(If it worked, please mark it as an accepted answer. If not, let me know how it failed.)
 
Sorry I got through the loading of the data, however I'm having problems with binning the data. I require all values to be binned inbetween 0.0005 so I set the binSize to that value, however I get the following error when performing the allBins part:- Error in seq.default(floor(allRange[1]/binSize) * binSize, ceiling(allRange[2]/binSize) * : 'by' argument is much too small
 
What are the values of allRange? That should only ever happen when you get ridiculously large (more than 2,147,483,647 on windows).
 
Hi mate
so when i look at allRange
I get
0 31322924
 
10:56 PM
k
Does that make sense to you? That's a fairly broad range for data, especially if you are trying to increment by 0.0005 ...
 
it is a broad range, unfortunately with the data that I'm working with
it comes from an instrument that has an error known to us of 0.0005
and so things that fall within that value range are like assumed to be the same like feature if that makes sense
is there any way of still being able to bin the values despite the large range
into bins of that size or is it just too large
 
I'm thinking it's just too large ... is there a way you can subset it, perhaps everything below (say) 30000000 is an error?
 
I guess it would depend on the value at which I can subset
I may be able to get away with subsetting the value at 30000000, but would that be enough for it to then work?
or would it have to be much lower
 
If both "0" and "31322924" are not erroneous readings, then my next method would involve reading in all the data, sorting it, and iterating the sequence manually.
(BTW: (31322924 - 30000000) / 0.0005 is 2645848000, which is still above .Machine$integer.max of 2147483647.
 
balls
hmm
sorry as you can probably already tell my knowledge of R is low
 
11:06 PM
what does fivenum(x) (replace x with your data) give you?
(fivenum is giving us the min, Q1, median, Q3, and max, and will help understand where most of the data is)
 
it gives
0, 3.16 e+02, 7.316 e+02, 1.564 e+03 and 3.132e+07
 
Alright, that tells us that 50% of your data is within 316 and 1564, very interesting.
 
so the vast majority of the data seems to be predominately in the lower part of the range
 
Can I ask what the measurement indicates?
 
it's mass to charge ratio, the values are of protein sequences
so peptides that make up proteins
it's a way of deducing a protein sequence if that makes sense
 
11:11 PM
okay, thank you ... is a value above (say) 1,000,000 still meaningful?
... or is there a point (high or low) at which it is not meaningful?
 
ideally you would want to retain all values, but I'm not 100% sure where to draw the line in terms of where the values stop being meaningful. I would have to speak to a colleague of mine to find that out I'm afraid
would it be possible for me to get back to you at some point tomorrow in regards to this?
really sorry about this
 
sure, but another thought or two if you have a moment
 
yeah sure go ahead
 
You said that the known error of the instrument is 0.0005 ... that's true for readings of 100 as well as 3e7?
It's a static error?
 
yeah it is, it's because the instrument is believed to give us 1ppm or as close to 1ppm reading of peptides
 
11:15 PM
ok ...
 
and 1ppm basically translates into a m/z value of 0.0005
so basically
that 0.0005 value
is instrument error
 
Got it, that makes sense
 
the error associated with the values it reports
 
you mentioned the median in your initial question; does your statistic need to be the median of each bin, or would it be meaningful to truncate precision beyond that error rate?
 
it would ahve to be the median of each bin, only because we assume that for all values the machine reports
the one that is in the middle
or so to speak the one that separates out the values into two equal halfs
would be represent the true value for all the other values that have been binned
personally I would prefer taking a mean
but that's just how our lab operates
 
11:19 PM
median/mean are equally difficult based on the sample makeup ...
 
so I'm assuming in terms of the range of data, if I was to subset the data, the highest value that it could obtain would be the machines integer max
 
... but if it's an error, would rounding to the nearest 0.0005 not suffice? Because of the error, any value within the error interval is likely, so rounding it should not introduce any, well, error, into your ensuing calculations
You have a couple of things on this one to consider:
(1) the number of bins cannot exceed the machine's integer max; I've never heard of this being even remotely a problem, since most machines can handle integers up to 2.14e9;
 
lol glad to be your first :p
 
(2) your data itself can far exceed that, but at that point you're getting into the floating point problem where you will eventually lose precision, though at very higher numbers with fractional components.
For example, try (.Machine$double.xmax - 1) == .Machine$double.xmax, which should obviously return FALSE, doesn't
The problem you're seeing with my code is from seq.default, meaning you have too many bins.
About how many data do you expect in each bin? 5? 5000?
 
well on the lower end of the values I'm expecting to see slightly more than that, upper end not so much
it's mainly values that are of 10,000 or below that tend to overlap
 
11:27 PM
slightly more than 5000?
 
with other values
within the 0.0005
range if that makes sense
I think it's more the case that some bins will have a large number of values within it
 
about how many observations do you have?
 
but others won't
 
the reason I asked about the number of data per bin is that the median/mean will was out with a sufficient n, and using the number rounded to the error would still suffice
 
erm observations I'm not 100% sure, but it's easily within the tens of thousands, if not more, unless there's a way of being able to tell from the use of R?
 
11:29 PM
length(x) assuming that x has all data
(if it's a vector, length(x); if it's a data.frame or matrix, nrow(x))
 
I have 5566784 values
 
brb
 
ok, thank you again for your help
 
alright, I'm back
5e7 isn't bad at all for data, so performance in other options would be quite reasonable
If we sort the data and then increment manually, you could make some headway.
wait ... 5.6e7 in 0.0005-wide bins only spans 2784 ...
 
but not all of those bins will have values in them
it will predominately be bins that are in the lower range of values that will be populated
but we sitll need to retain any values that are of high value as well
if that makes sense
 
11:38 PM
sure, that makes sense.
I think sorting and manual binning would work without exceeding the integer limitations. Are you willing to give me some sample data? If so, then just paste in sample(x, size=50), and I'll see what I can find out with it ...
 
sure will get that to you in just a second
RB2_idx_11.txt.V988121 RB2_idx_18.txt.V141881 RB2_idx_7.txt.V867432 RB2_idx_18.txt.V109672 RB2_idx_9.txt.V740132 RB2_idx_15.txt.V570522 RB2_idx_16.txt.V77481
1590.1141 232.6095 19197.0000 1323.0000 400.0000 2297.0000 226.1408
RB2_idx_12.txt.V625302 RB2_idx_4.txt.V979412 RB2_idx_18.txt.V112701 RB2_idx_24.txt.V380932 RB2_idx_8.txt.V363422 RB2_idx_7.txt.V463762 RB2_idx_25.txt.V435541
2396.0000 11743.0000 227.3829 1126.0000 1439.0000
the formatting of that is horrendous, sorry give me one second
 
I got it, stby
 
oki dok
 
performance may not be awesome, but I gotta tweak some things ...
 
I don't mind if it takes ages to process tbh, it'd be such a massive help
btw you should consider working in bioinformatics
your skills would be incredibly useful
 
11:46 PM
well, I'm a research analyst during the day and I teach undergraduate biostatistics (emphasis on statistics, not bio) at night, so ...
 
I take that back :p
 
oh, I think this will work quite elegantly, actually ...
I'm still assuming your data is x, so catch all of these references
 
oki dok
 
do you need to retain the labels (filenames) for each datum?
 
nope, as all the data for the files will effectively be collated into one
 
11:52 PM
Are you familiar (or willing to be) with the dplyr package? This makes it quite easy to read and maintain, and performs quite well for this type of data.
 
I'm willing to read through it and give it a go
 
I used your data to confirm things, and then create 5e7 of my own data, with x <- 5e7 * runif(5e7)
(It's fake, but it's okay-fake for the sake of this fun, I think)
 
yeah should be ok :p were you able to sort and manually bin?
 
I think so, I'm doing some basic benchmarking and validation ...
I just reduced my dataset to 5e5, and it took about 12 seconds to process, so I would expect 5e7 to be a lot longer, more than I want to test during this chat. However, testing with smaller datasets and confirming it does what you want will allow you to let it run unchaperoned.
I'd recommend you sample your data with something like samp <- sample(x, size = 1e5) and using samp on this code, to make sure that it is giving you what you want/need before wasting time.
library(dplyr)
binSize <- 0.0005
df <- data.frame(dat = sort(x))
df$bin <- floor(df$dat / binSize) * binSize
 

  last day (21 days later) »