last day (38 days later) » 

5:58 PM
hi
 
Wold it at all be possible to give an extract of your data file that is 5 lines long, just so we can work on this topic more clearly?
And hi :-)
 
sure
thanks for all your help
really helping for this project
 
Sure thing, no problem, I'm here for the challenging questions and yours was just the one I was looking for yesterday :)
 
10 1408 1409 T T T T T T T T T T T T T
T T T T T T T T T T T T T T T
T T T T T C/T T T T C/T T T C/T C/T T
T C/T T C/T T T T T T C/T T C/T C/T T T
C/T T T T C/T T C/T C/T C/T C/T T T C/T T T
hmm not well formatted but thats 5 rows
 
Just a sec, so I can load all this on my pc...
 
6:00 PM
sure
 
okay, I think I've got the formatting correct, let me generate that table first and then we can discuss how you'd like to process the output
 
thanks
 
okay, the first error you describe is fixed
I forgot to add the "enumerate" on that line
 
ah ok
 
that's where you got the "unpacking error" from
(edit made on relevant thread)
 
6:05 PM
ah ok
 
okay, I have a table generated, so now I have a layout of 2 columns, which are used for keeping track of the row numbers and such, then 5 columns with the frequencies of the alleles in A and then another 5 but for B
 
It looks something like this (I've modified numpy's print_options to keep it small enough):
[[ 1.0e+01, 1.4e+03, 0.0e+00, 0.0e+00, 1.0e+00, 0.0e+00,
0.0e+00, 0.0e+00, 0.0e+00, 1.0e+00, 0.0e+00, 0.0e+00],
[ 1.0e+01, 1.5e+03, 0.0e+00, 1.0e+00, 0.0e+00, 0.0e+00,
0.0e+00, 0.0e+00, 1.0e+00, 0.0e+00, 0.0e+00, 0.0e+00],
[ 1.0e+01, 1.6e+03, 0.0e+00, 0.0e+00, 0.0e+00, 9.1e-01,
8.7e-02, 0.0e+00, 0.0e+00, 0.0e+00, 9.0e-01, 1.0e-01],
[ 1.0e+01, 1.7e+03, 5.2e-01, 0.0e+00, 0.0e+00, 0.0e+00,
4.8e-01, 6.0e-01, 0.0e+00, 0.0e+00, 0.0e+00, 4.0e-01],
Now what do you mean by filtering it based on a threshold?
 
i mean that i dont want to consider rows where there is a high amount of missing data in either population. So where more than 10% of the values in a population are missing i.e. '.', then that line should be skipped
The frequencies for each population in the table above should always sum up to 1 right?
 
yes
(remark that with the current printing options, it might not look that way)
 
6:11 PM
i understand, thats fine
 
okay, so then let's look at your 5th row, because I think that one will show best what you want
 
in the table I've got in front of me, I see now that for popA, there's about 52.17% of 'G' and the rest is '.'
whereas for popB, there is 60%G and 40% '.'
 
right
 
so you'd like to ignore rows like these
 
6:13 PM
these arent the best example lines (my fault ) as there are cases where the difference is 90-100%.
yes exactly
 
then let me ask you this: do you want to ignore them if the unknowns-frequency (so the '.' ) is above 10% for either popA OR popB?
 
yes exactly
 
Okay, then you can easily work with a "mask"
for example (give me a sec)
 
in reality i will modify that frequency in the future, but im sure that can easily be changed in the code
ah a mask sounds like a good idea
never implemented it in python
 
mask = np.logical_or(results[:,6] > .1, results[:,-1] > .1)
for these 5 lines of data, this mask looks like this:
so the last two rows have a lot of dots in them
You'll want to combine this "mask" with the mask that was labelled "specials" in the stackoverflow thread
specials was defined like this:
specials = np.any(condition, axis=1)
Now I hope you see where I'm going with this... :)
 
6:20 PM
ah i see
 
You want to print those lines of the results table where mask has a False value AND specials has a True value
So how would you combine this? :)
 
lets see
hmm im used to using if statements, which dont seem necessary in this code
 
Well, you could try it with an if statement...
 
id do something llike if Mask = False: print results[specials, :2]
but that would need to iterate through all the lines again
i think there is a more straightforward way
 
indeed and that's not terribly efficient
(numpy was built with efficiency in mind)
let me give you one more hint
 
6:24 PM
thaks
thanks
 
the mask named "specials" that you would have gotten from the code I posted earlier, looks like this (okay, not spectacular for this example):
array([False, False, False, False, False], dtype=bool)
let's imagine for a moment that it looked like this:
array([False, True, False, False, True], dtype=bool)
That would mean the 2nd and last row of this example have a frequency difference between popA and popB that is out of the norm (=mean+2*std, as you wanted)
 
and we see that in the two True statements?
 
but unfortunately, the last True (so line 5) is not interesting, because our "mask" also has a True there
yes
Those 2 "True" values in "specials" indicate that those lines were special
 
do they each refer to a different nucleotide count?
oh wait i get it
 
yes, they refer to the 5 different lines and more specifically to the difference between the frequencies of the CATG. alleles in pop A and popB
 
6:30 PM
each False and True summarises a row of data,=
right got it
 
yes! :)
so now, you only wanted to take into account that even though some of these rows of data are marked as special, that they should not be printed if the count of the unknown '.' is too high
 
and and all the five examples have les than 10% missing data in either popn, so the Mask says they are False
yes
but also they should not be used to calculate the mean, or this will bias the data
 
actually, your last 2 rows in the example you gave me have a lot of '.' in them
that's why the variable I named "mask" here, has False False False True True
 
ah ok
 
let me have a look if the mean calculation considers the '.' columns too...
It does
so you'll want to modify this line here:
freq_diffs = np.abs(results[:,2:7] - results[:,7:])
freq_diffs = np.abs(results[:,2:6] - results[:,7:-1])
(which was mentioned in the answer as well)
 
6:35 PM
modify it to look like
freq_diffs = np.abs(results[:,2:7] - results[:,7:])
?
 
For the small example of 5 rows you gave, it does not influence the values of "specials" nor "mask" though
 
right, but i imagine it would when i have thousands of lines of data
 
No, it should look like the 2nd line:
freq_diffs = np.abs(results[:,2:6] - results[:,7:-1])
 
just so i understand, how does that line of code work to not include masked rows?
 
results[:,2:6] looks only at the frequency counts of C A T and G in population A
it does not yet: you had to do the final step which is:
worth_investigating = np.logical_and(specials, np.logical_not(mask))
 
6:38 PM
and np.abs finds the absolute difference between the two populations, so whichever frequency difference is greatest?
 
and then you could do, like in the last line:
print(results[worth_investigating, :2])
yes
so, e.g.:
 
ah ok , so you make a new class, worth investigating, that used logic to include rows where specials is true and mask it false
is*
 
indeed
 
then dont we want to modify the code so that the mean and std are only calculated on the worth_investigating data?
 
[.3, .4, .2, .1, 0, .2, .7, .1, 0, 0]
 
6:40 PM
looks good
 
ah but worth_investigating already depends on that calculation!
hmm
wait I see where you're going
okay, just a sec
 
might it not be easier just to remove rows with >10% missing data from the array?
as we dont need them for anything
 
to_ignore = np.logical_or(results[:,6] > .1, results[:,-1] > .1) # same thing as "mask", just renamed
to_consider = np.logical_not(to_ignore)

limited_results = results[to_consider,[0,1,2,3,4,5,7,8,9,10]
that's what the code above does for you ;-)
it removes the rows where any of the '.' in either popA or popB is too high
 
ok great
 
it also removes those '.' columns as they are now no longer important
 
6:45 PM
why the ,[0,1,2,3,4,5,7,8,9,10] ?
true
 
notice that the 6 and 11 are missing?
 
ah!
 
those are the indices of the columns '.' in popA (6) and in popB (11)
(looks like I'm missing a bracket by the way)
so finally you'd call all those "mean" and "std" functions on this limited_results table
I hope it's clear now
 
i must miss something, i thought there would be 4 indices for each popn now, one for the frequency of each nucleotide
 
yes, plus 2 for the labels of those rows
(the labels were the first 2 columns in your data)
 
6:49 PM
ah gotcha
 
so this new results table looks like this:
label1, label2, freqC_in_A,..., freqG_in_A, freqC_in_B, ..., freqG_in_B
 
great
 
I'll make one more update to my post to include some of the results that were discussed here, okay?
 
sounds great thank you
i have to head out (late in Germany) but will check back later and work to make sure the code operates. If there are any problems i cant solve I will write a comment tomorrow. Would be great if you have time to check back in at somepoint just encase.
Thanks a lot for your help, ive learned a lot that should help me handle data in python from now on
way more efficient than my for loops!
 
Glad to be of assistance. I've updated the post just now. If you have more questions... well stackoverflow is here and for sure I'll have a look back on that thread tomorrow. I'm also heading out, time for dinner I'd say :-) Greetings to Germany from Belgium!
 

  last day (38 days later) »