« first day (1828 days earlier)      last day (1385 days later) » 

8:46 AM
aaah just managed to get rid of about 150gb of junk on my computer
feels good
 
 
1 hour later…
9:53 AM
@flawr switched to linux?
 
10:38 AM
I ran into a problem while analyzing some data, and while I found a way to solve it, I would be glad to hear how others would've approached it.
I have a function that accepts two vectors, which are supposed to represent some continuous function. However, sometimes the X and Y are flipped, and I need to determine if this is the case. Here's an example where X,Y are flipped:
And here's an example where they are "ok":
(the self intersections of the curve are because the data isn't sorted in advance, and is only sorted by ascending X after the correct X and Y axes are determined)
If so - how would you distinguish these two cases programmatically?
(when I said that X,Y are flipped, it is more correct to say "rotated 90 degrees CCW")
this is an image processing task of sorts
 
11:02 AM
@Dev-iL Look at standard deviation of both components?
Subtracting the minimum or something
 
@AndrasDeak That's what I initially looked into, also IQR.. It's not always conclusive
 
Do inconclusive cases look different?
 
nope, compare the bottom image above to:
 
how many points on the baseline? Just a few?
 
11:08 AM
@AndrasDeak it was on my work-linux machine:)
 
You could naively look at both functions with sorted "x" and check monotonicity or something... but that's slow
@flawr docker perhaps?
 
@AndrasDeak nope, just the data that didn't support my hypothesis
 
@AndrasDeak it varies:
 
@Dev-iL so either the x-values or the y-values should be unique right?
 
@flawr that's right. The "real x" values should be unique
 
11:11 AM
should they have constant step size when sorted?
 
@flawr but floats...
 
@flawr that cannot be guaranteed
 
is the dependent variable always expected to make some bump like in your pictures?
 
Overkill solution: fitting a Gaussian on both
 
@flawr yeah, though sometimes it's much wider and looks more like a trapezoid
@AndrasDeak that's what I ended up doing
 
11:15 AM
That's reliable and even has a metric, if slow is not an issue
 
@AndrasDeak I agree... And I do have much more significant bottlenecks in the code...
Fortunately for me, I will only ever need to run this code once, to process ~200 such datasets, and even if it takes 45min it's not a problem at all
 
^ left is sorted by x, right is sorted by y
there should be a way to leverage this right?
 
@flawr yeah, I saw many such charts during my attempts :)
 
you could probably look at the total length
and then just take the way that gets you the shorter curve
 
@flawr monotonicity
 
11:23 AM
can you elaborate?:)
 
~O(1) if the data is reliable
@flawr check if y is monotonic if sorted by x
 
I don't think that works
 
(This seems like it should be a fairly simple problem, right?)
 
@Dev-iL Only when you know the solution:P
 
@Dev-iL yeah, just like computer vision ;)
 
11:25 AM
@flawr When a human is looking at it, the solution can be determined almost instantly
 
@Dev-iL that is what makes the really hard problems in programming:P
 
@flawr check until max
 
@AndrasDeak that's unreliable because there are tiny ripples in the flat parts
perhaps if you're considering only from mid-height to max-height
 
Larger integral when mean is shifted to 0?
 
:)
 
11:31 AM
yes, the length thing works if you normalize the ranges of each of the vectors to [0,1] first! (subtract min, then divide by max)
 
btw, it should be subtract min, divide by new max (old max-min)
cool solution nonetheless!
 
@flawr computation-wise you could just divide by the geometric mean of the peak-to-peak values along x and y
Less clear without comments but less work (for the CPU)
 
@AndrasDeak what does that give...?
 
11:47 AM
@AndrasDeak but don't you get negative peak-to-peak values?
@Dev-iL right:) I just tried it here: pastebin.com/zpV16XGB and it seems that the funciton has to do very wild things until it fails
 
@Dev-iL alternative to pre-normalization. Post-.
@flawr never
I'll continue from laptop later. On my way to the market :D
 
then I don't understand what these peak-to-peak values are:)
 
max - min I guess
 
@Dev-iL max - min. See np.ptp
 
that's what I meant (typo because words sound almost the same)
@flawr do you know about bounds? gives you min+max in 1 pass (I hope) disregard that, it's just some inefficient wrapper around both min and max
 
11:55 AM
there's also range (max-min) that also just uses the max/min functions
 
12:06 PM
@flawr so there's no good name for ptp. There's "amplitude" but that has quasiperiodic connotations and in any case ptp = 2 amplitude. There's "span" but that has a linear algebra meaning. There's "diameter" which would be technically correct but has confusing geometric connotations for most people.
That being said I don't like that it's called ptp. I'd also like a function that returns min and max in one sitting, but that's usually seen as not worth adding to the numpy namespace.
 
@AndrasDeak but I think "range" is what is usually used in maths:)
(AND matlab:)
ah no it seem "range" in maths refers to the a set
and is more used in school maths
 
range has been taken since python 1.0 or something ;)
 
@AndrasDeak I just remembered in electrical engineering you sometimes write Vpp for the "peak to peak" voltage
 
I guess that's why TWM chose "bounds" for this
 
Yeah, bounds is not bad, but I'd expect that to return min and max
 
12:11 PM
@AndrasDeak why not use ranged then:P
realrange
 
#notmyrange
 
max_minus_min()
 
@Dev-iL ah, it does return min and max. Where did you see that it's inefficient?
 
12:25 PM
@AndrasDeak look @ source code
 
I don't think I even have matlab installed here :) But then I get it
shame, it's a straightforward chance to halve computational cost
 
12:47 PM
When virtually the entire moderator team is opposed, "reduces burden on moderators" stops being a good argument in favor of the feature. We would know. — Cody Gray ♦ 8 hours ago
Outch, poor SE
path(path,'c:\Users\katie\Desktop\HU3\DATA\forKatie') I smell very good file management.
 
@Adriaan heh
Proof of the pudding is in the eating, @yaakov. I've seen this same mistake play out countless times over the years: everyone pulling together to clean up the mess is great, but if it just happens again in a few months then nothing really changed. FWIW, I said essentially the same thing back in February. Let's see how things look by October... — Shog9 yesterday
 
@AndrasDeak very true, but I doubt the company will listen in any way
 
1:03 PM
He's not talking to the company :)
 
 
2 hours later…
3:01 PM
Do we have any zookeepers here? I have a problem with pandas.
 
3:21 PM
Pandas problem solved:)
 

« first day (1828 days earlier)      last day (1385 days later) »