« first day (1426 days earlier)      last day (1785 days later) » 

6:39 AM
"I know this will be out of the website rules, but..." stackoverflow.com/questions/56319932/…
 
6:53 AM
@Adriaan The talk about you in the introduction: arxiv.org/abs/1901.07017
 
7:48 AM
@flawr I'm an entangled spatial problem?
Ah, I see. Lovely to read fan mail :)
I do miss the pictures of me, my smaller cousin and the penguin though
@CrisLuengo edited to remove the code and just include a nice picture of (dead?) pig. Brilliant improvement
 
 
4 hours later…
12:23 PM
@Adriaan I'm not sure I can answer that. Maybe you should ask your SO. :P
 
1:03 PM
@AndrasDeak I am trying to get a robust mean and standard deviation using sm.robust.scale.huber. However, 30 iterations is not enough, so I tried increasing it to a hundred, but this error popped up:
mean, std = sm.robust.scale.huber(data[indices,columns],maxiter=100)
TypeError: __call__() got an unexpected keyword argument 'maxiter'
I don't get why I get this error. Should I first call sm.robust.scale.huber(maxiter=100) and then mean, std = sm.robust.scale.huber(data[indices,columns])?
 
1:33 PM
@Adriaan that docs is for Huber with a capital H
 
@AndrasDeak but its example with huber with small h?
 
yeah, I can see that, weird
I mean, it's probably pulled into the robust namespace
 
Something goes on in the internal class
 
if a keyword doesn't work you probably have the wrong version
(good) docs should specify when keywords and params were added
Aaaah I see
 
yea, I found the thing lacking.
Should I call sm.robust.scale.Huber.maxiter=100 first?
 
1:35 PM
What you linked is a class called Huber, what you're calling is a convenience function huber that constucts an instance under the hood
@Adriaan no
I'd create an instance with Huber(data=..., maxiter=...) and then call the returned object to get an estimate
hmm, I don't see a lowercase huber in the code at a glance
gotcha!
huber = Huber()
OK, so you need to do huber = Huber(maxiter=100) then call it with your data
 
But that still says maxiter=30, so why can't I change it?
 
@Adriaan what does?
 
huber = Huber(maxiter=100)
mean, std = sm.robust.scale.huber(data[indices,columns])
@AndrasDeak sorry, missed the reply, I meant the Github thing you just linked
 
the github I linked instantiates a Huber instance with the default args
so if you use sm.robust.scale.huber you get that
sm.robust.scale.huber is ~ a function, sm.robust.scale.Huber is a class for which the __init__ takes maxiter as an arg
 
I think I understand, so the two lines I wrote in my penultimate reply would do the trick?
 
1:42 PM
I think so
 
To get it into terms understandable for me: it is like MATLAB's train function, in which you specify all your parameters, then call some fitting function with fitting_function(train,data) (except that this is a class, not a function)
 
I'm not sure, I'm not familiar with that. It's just that Huber is a class and its instances are essentially functions. There are similar things with some interpolators: you first construct an interpolator object using the input points, then you apply that object (which is again a function) onto the output x points to get the interpolated y values
 
huber = sm.robust.scale.Huber(maxiter=100)
mean, std = sm.robust.scale.huber(data[indices,columns])
That worked (except that I still have not enough iterations)
Hum, wait, no?
python remove_median_per_bin.py
/home/adriaan/anaconda3/lib/python3.7/site-packages/statsmodels/robust/scale.py:163: RuntimeWarning: invalid value encountered in sqrt
  / (n * self.gamma - (a.shape[axis] - card) * self.c**2))
/home/adriaan/anaconda3/lib/python3.7/site-packages/statsmodels/robust/scale.py:159: RuntimeWarning: invalid value encountered in less_equal
  subset = np.less_equal(np.fabs((a - mu)/scale), self.c)
Traceback (most recent call last):
  File "remove_median_per_bin.py", line 96, in <module>
Still "failed to converge in 30 iterations", so the code is syntactically correct, but still uses 30 iterations?
 
indeed
 
Basically huber = sm.robust.scale.Huber(maxiter=100) is ignored?
 
1:51 PM
11 mins ago, by Adriaan
huber = Huber(maxiter=100)
mean, std = sm.robust.scale.huber(data[indices,columns])
you forgot to use your own huber
but those RuntimeWarnings don't look so good
 
    huber = sm.robust.scale.Huber(maxiter=100)
    mean, std = huber(data[indices,columns])
 
yup ^
 
That works! Eskerrikasko!
This python thing is very difficult :(
 
Most python modules don't have (read: don't use) global state. You usually have to construct objects and configure those.
there are exceptions such as matplotlib's rcParams and whatnot, but that's a bit different, semantically speaking
@Adriaan it would've been easier with better docs
 
@AndrasDeak Definitely.
So if I set maxiter=100, tol=1e-1 and it still doesn't converge I have crap (almost random) data?
 
1:54 PM
there should at least be a mention that huber is a short-hand for Huber() in that example
@Adriaan I have no idea what you're doing but the warnings in the traceback suggest some divisions by zeros or sqrt(negative number)s. That's rarely a good sign.
odds are the data it's trying to converge is full of nans
 
@AndrasDeak I have ~40M datapoints distributed over the surface of Mars, try to bin them in equal areas (and equal number of measurement bins) and then get a mean and standard deviations. That all works. Problem arises when I try to get a robust, Huber, alternative for the mean and std
I might have had too small a test set, let's see what it does on the full data set
 
no idea what that entails
 
Me neither, but it's what the boss wants :P
 
Sock-puppet? q1 q2
(already flagged it just in case)
 
or classmate, but could be
Ah, exact same linked picture. Sock.
 
2:00 PM
@AndrasDeak As I understood it the mean is very un-robust (equal weights for each measurement), so if you have an outlier it basically goes wrong very soon, very fast. On the other hand the median is very robust; only if 50% of your data are outliers (and on the same side of the median) it starts to significantly alter the median. "A Huber location estimate" tries to find a common ground between the two: take more data into account than the median, but be more robust than the mean
 
I see. I don't know how that could lead to these warnings. Perhaps some input semantics are off, for instance the shape of your data or something
 
Well, if I only load the first 200k points, I might have bins where no satellite has flown yet, and thus is empty. I'm testing now with the full data set. Takes an awful long time, but at least I'll have ~1000 points in each bin, if not more
 
I wouldn't expect empty bins to matter
not that I claim to understand what's going on
I'm just saying it's possible your problem will prevail
 
@AndrasDeak empty bins = no data = possible division by 0 elements in a bin?
Nope, still fails, alas
@Dev-iL tags are awkwardly narrow btb; almost no-one uses , so I support the claim of them being socks.
 
this is really weird, Huber.__call__ uses an axis keyword that isn't documented
Is your data 1d?
you can try with a small dummy example you put together yourself (linear stuff with noise, see if it works if you try to use it in the same way)
 
2:16 PM
@AndrasDeak yes, but I have 3 columns of them. I'm looping over them now, instead of doing median(data,axis=0). That's fine for me.
I'll leave it at 100 iterations and a 12-3 tolerance (might even go to 1e-4, must discuss) and wrap that in a try/catch with a normal median and mad call in the catch part
 
@Adriaan can you give an example for when MAD is a useful metric? I never use it in my analyses, but perhaps I should.
 
Try passing axis=1 to huber and all 3 columns
Or 0...
 
@Dev-iL Again: not sure. The boss wants me to use robust statistics, so Robust I go. I think the same I just wrote for the mean/median/Huber also holds for std/MAD/Huber-scale. Why that's necessary to use in my particular problem as opposed to a naive mean or median estimate I do not know yet
 
Hmm, looks like I am using it after all, just not used to the name...
function stats = analyzeErr(err)
  stats = struct();
  colErr = err(:);
  absErr = abs(colErr);

  stats.MAX = max(absErr,[],'omitnan');
  stats.STD = std(colErr,'omitnan');
  stats.AVG = mean(absErr,'omitnan');
  stats.IQR = iqr(colErr);
  stats.NNN = nnz(isnan(err));
end
I guess that's what I call "avg"
 
2:24 PM
ah ok, wikipedia says that the M stands for "mean" and not "median" src
 
@Dev-iL oh, again: boss told me it was median. I'll read into it more before I get pesky reviewers on my article :P
 
3:04 PM
@Dev-iL I see your Wikipedia source and raise you another Wikipedia source.
I know the MAD as a way to estimate the standard deviation in a robust way using medians.
 
3:58 PM
    for columns in [4, 5, 6]:
        # mean, std = sm.robust.scale.huber(data[indices,columns],maxiter=100)
        try:
            huber = sm.robust.scale.Huber(maxiter=100, tol=1e-3)
            mean, std = huber(data[indices,columns])
            data[indices,columns] -= mean # subtract robust Huber location (='mean')
            data[indices,columns+3] = std # Store robust scale (= 'standard deviation')
        except ValueError:
            data[indices,columns] -= np.median(data[indices,columns])
So, at least I get a list of proper numbers now, but still the runtime warnings
oh well
 
you can suppress warnings if you want to
I'd use a context manager inside the try docs.python.org/3/library/…
 
@CrisLuengo haha.. I guess statisticians aren't that great at naming things either
 
 
1 hour later…
5:11 PM
This user has posted 5 questions so far in 24 hrs. He’s working towards a problem and posting increasingly refined questions as he’s figuring things out. That makes me think he’s posting before trying to solve it himself. They’re all unanswered so far. What to do with this?
 
downvote, close as too broad/no MCVE where appropriate
a few of them are definitely close-worthy
 
5:34 PM
I corrected thanks a lot — mohsen May 15 at 12:09
No, you edited. I would not call that a correction
facepalm
@CrisLuengo what @AndrasDeak suggested, and if you feel like being nice comment them to please try something first, as it looks like their answers are solving a bigger problem
 
5:48 PM
> i need to obtain an approximation of the derivative of the approximate signal,
I am tempted to just throw a free-hand red line out there as an approximation of an approximation :P
 
6:34 PM
What do you expect to happen with z=real(y);z=0? How is this different from A=1; A=2;? What will A be, 1 or 2, and does the second assignment bear any relation to the first? Second: All complex numbers with their real part equal to zero map to the imaginary axis, i.e. even with a correct code it should produce a line plot. — Adriaan 7 secs ago
unclear ^
I had a discussion about a similar question in the SOCVR last week; there it was another mathematical impossibility (they wanted to plot log(x)=0). The problem is that unless the question is purely about maths (Why can't you plot log(x)=0), all we can do is close as a dupe apparently...
Preferably a dupe outlining why it's mathematically not possible, as well as giving a workaround
Whoops, I failed. b=0.5 is a perfectly valid entry for a real-valued y. Revised my comments; code is still horrible, No MCVE/Too broad/Tutorial req is a better fit
He plots 1+z^2, where z is the complex number?!? This code makes so utterly little sense...
so a^2-b^2+2abi = real works for a==0 or b==0 basically?
 
6:52 PM
wow, much confusion
 
@AndrasDeak Sorry, I was thinking out-loud
 
no idea what they are trying to do
 
@AndrasDeak basically 1+z^2 is pure real on the coordinate axes, and pure imaginary 45 degrees rotated, otherwise it is complex with real and imaginary parts nonzero. Why they'd want to plot this, and how this would yield a 2D plot for the \pm 1 sloped lines is unclear to me :P
 
yup
 
I think the dude speaks A) no maths (both complex and real valued K would yield y=1+k^2`), and B) no MATLAB, with him reassigning everything and expecting a sensible output
Interesting; the questions they have on maths.SE aren't too low level
 
6:59 PM
code like that reminds me of blog.codinghorror.com/…
 
@AndrasDeak jup. First example though, if you have never programmed, you're more or less taught that = is both "assign to be equal to" and "logical operator to check whether it is equal to", which is easily remedied in their first programming lesson
Interesting read, that apparently people who do not grasp the meaning of a=10;b=20;a=b intuitively aren't capable (in general) of being taught what it means.
Is this why R uses <- as assignment operator?
 
7:31 PM
no idea, but if the (now-retracted) paper is right the syntax is irrelevant, because some people are lacking the kind of algorithmic/structured thinking that is necessary for programming
 
 
2 hours later…
9:31 PM
user image
2
 

« first day (1426 days earlier)      last day (1785 days later) »