« first day (1940 days earlier)      last day (1287 days later) » 

12:31 AM
@AnderBiguri awesome typo! Could be “who” or “how”, nobody knows... :)
 
 
9 hours later…
9:06 AM
@CrisLuengo my typoes are always awesome :D
 
 
4 hours later…
12:58 PM
I'm solving a 600k x 200 matrix using linear regression now. This is problematic, because I can't add much more coefficients (columns) due to RAM constraints. I got the tip to look into solving the normal equations, which would reduce my problem to 200x200. Does anyone here have a good reference on how to do that? (Theoretical/mathematical is good, no need for a MATLAB tutorial)
 
instead of solving Ax=b you solve (At*A)x=At*b
 
BOOM, Andered
2
 
not sure that is supposed to be positive or negative XD
 
absolutely positive <3
 
1:05 PM
@AnderBiguri right, sounds good. I'm thus looking for a way to set up At*b and At*A without forming A explicitly, right?
@AnderBiguri te vagy a legjobb!
(My Hungarian is getting very rusty)
 
to be fair I got this a bit rusty now. I remember using CGLS for this, as it solves the normal equations
when I used this thing is because indeed never wanted to form A explicitly
 
Makes sense. I'd like to incorporate a lot more coefficients, and my A matrix is already 1GB
 
and used algorithms that only required A(x) and At(b) without A
 
Solving time, with MATLAB's built-in Huber regression, is just one minute or so though
 
matlab has good solvers too, pcg does the job, you can input a function handle instead of A
GMRES and others are also good builtins, MATLAB has few solutions there
 
1:17 PM
@AnderBiguri That could work... I need to go to "only" degree and order 15 or so for the spherical harmonics. Not the 134 I needed in the previous problem :P
 
the good thing is it doesnt matter. if you input A as a function handle, it can be whatever, as long as it inputs outputs a particular size of matrices
if you write a generic form for b=A(x) fucntion, then you can try 5 or 5000 order polinomials :D
 
@AnderBiguri also if you have polynomials of order 150? Don't you run into horrible VPA problems then?
Sounds very useful. I'll go try that!
 
yeah, not sure how useful 150 order polinomials are
also in practice, like, wtf of a function do you need to fit? human DNA vs nose shape?
 
@AnderBiguri I'll upload the formula
Which has 2S_max 2P_max ((Nmax+1) ^2-1) coefficients
we'd like Smax = 4, Pmax = 4, Nmax = 15
So in this case the polynomial order isn't that high. (That was a few weeks ago, where I had the formula, but wanted it symbolic, rather than numeric for each of my 200M measurements)
 
yeah, just suprised about the complexity of that, you know your science, not doubting it. But yeah, shit is going to get ugly if you use numbers
the good thing is that that equation seems quite straightforward for A :)
albeit a bit non-linear, no?
 
1:26 PM
Basically Ynm are spherical harmonics (spatial patterns on a sphere), P modulates a daily frequency (makes the planet go round), S determines the season (Takes care of the ellipticity of Mars' orbit). We want \tilde \iota of course
@AnderBiguri Yup. We solve the real-valued problem, thus loads of cos/sin involved
 
and you can write that as a linear system?
 
B = - nabla V; I measure B(r,theta,phi,t), thus I have all the information to write out the sums and make a nice and big full matrix
 
and x?
of Ax=b? what are you solving there?
S,P and N
 
A is basically the entire RHS of that formula, except the iota. x = iota (what we solve for), V (actually B = -nabla V) the vector of observations b
 
well, Ihaven't done the math, but I believe you. It just looks so ugly to end up becoming a linear system XD
 
1:31 PM
So the Bs on the left are the vector of observations, we want iota, our unknown, and we know everything inbetween. We choose a certain set of p,n,m (and S if you multiply over that as well), know where the satellite was, i.e. we know r,theta,phi,t. That means that everything between the = and the iota can be calculated, leading to loads of columns.
 
ah, I think I may have mistaken P by having an exponent, instead of an index
well, good luck with that mosnter :D
 
Hehe, thanks.
It's actually not too bad, except that the derivatives of the associated Legendre polynomial aren't built-in MATLAB. Otherwise it's just a bunch of loops recursing over the Legendre polynomial and adding a few cos/sin terms ;)
 
2:35 PM
@Adriaan 10/10
 
 
1 hour later…
3:48 PM
@AndrasDeak but if you're already legjobb, can Ander be one as well?
 
@AnderBiguri but the poor condition :/
 
you pre condition it for that! small hugs, a warm cup of tea
improves its condition
 
hehe
what method would you suggest for preconditioning?
@Adriaan autograd ftw:)
 
warm cup of tea, indeed
 
oh actually if ram is a problem, could you maybe consider a stochastic method? @Adriaan
 
3:52 PM
even just an ordered subsets method may work (i.e, select chucks of A, but no need to be stochastic about it, just one after the other).
but all these depend on the problem itself I guess
 
btw does anyone know graph stuff? I have a question
 
@AnderBiguri Does it matter who gives the small hugs I'd like me missus to give me some, but rather not the boss. That'd be awkward.
 
@Adriaan depends on what you want to precondition
 
I mean, to the problem, not to you. You need to condition the matrix
if its not you, then it has to be the boss, the missus knows nothign of the matrix
@flawr ( ͡° ͜ʖ ͡°)
 
Oh, my bad
but warm tea on me laptop might not do it too good
 
 
4 hours later…
8:11 PM
@AnderBiguri I can't find it right now: But I've heard that that could sometimes cause issues where you end up moving in a circle, whereas with a little bit more randomness you can avoid that with a higher probability or something like that
 
 
3 hours later…
 
10:58 PM
@flawr yes its correct, but you also introduce unecesary statistics. OS-stuff is good when you want fast convergence to tthe solution, but indeed, it may have bias, or cycle trhough the solutions. Depends on the application I guess. In medical imaging, OS-algorithms are the norm on some machines
 

« first day (1940 days earlier)      last day (1287 days later) »