« first day (1674 days earlier)      last day (1552 days later) » 

8:43 AM
@AndrasDeak Hello! Are you available for a conceptual mathematical question?
 
 
2 hours later…
10:18 AM
@AnderBiguri @Adriaan ^ the above is also addressed to you guys, now that I think about it (the context is roughly PCA/SVD, inverse problems, dimensionality reduction)
 
haha yeah I am available :D
 
Great 😀 so I'll explain what I want
I'm trying to measure a phenomenon represented by 3 independent parameters. A typical measurement system produces 4 reading, on the basis of which, the 3 parameters may be estimated. Now I need to choose between 2 measurement systems. The difference between the systems is summarized by the first 4 singular values (SVs): the first system has 3 SVs that are large and 1 that is small, and the other system has 1 large SV and 3 small, such that the condition number of both systems is the same.
My intuition tells me that "there is more information" in the system that has 3 large SVs, but I need to explain it in more mathematical terms (such as spanned subspace size, error amplification or something like that)
 
There is a nice analisys on the decay of SVDs that may give you intuition too. I forgot the specifics (equation and so on), but when I was working on EIT, they would measure the number of good, independent measurements using the SVD. They would plot the SVD in decibels, andon the SVD decay plot, you can also plot the expected SNR, or noise level. All singular values below the expected noise in db where considered "not a valid measurement"
 
Now, in the context of PCA, one usually want to reduce the dimensions as much as possible, which appears to be the opposite of my goal. To my understanding, if a measurement system is reducible to just 1 dimension, it means most of my measurements are redundant, which is not good
 
ofc you can not do 1-1 measurementt-singular value, but you get the idea
@Dev-iL hum this is not always true. In your case, indeed, you do not want to reduce the amount of dimensions. PCA just gives you the tool to analise how good of a job you have done in keeping independent principal components big
PCA is only for dimensionality reduction when your objective is to reduce dimensions, but in other cases its just an analysis tool
 
10:26 AM
Is it correct to say that if I have no more than 1 "irrelevant" dimension (and thus the spanned subspace is of size >=3), it should make it possible to recover a 3d parameter vector?
 
(Sorry, I started writting without letting you finish, go on)
 
Is it correct to call them "insignificant dimensions"?
 
@Dev-iL another interpretation of this is the one I put up there for SVDs. Mathematically speaking, PCA will give you complete representation of the system, no "irrelevant" dimensions. That "irrelevance" is a subjective thing, only applicable when talkign about real life meaning of those numbers.
So, while the PCA tool will give you 100% reconstruction using those tiny dimensions, the real worry is that any small noise in the system will generate perturbations on that dimension bigger than the actual descriptor size, thus making them "irrelevant". But they are only "irrelevant" due to inherent noise in the measurement, they would not be irrelevant if the measurements where perfect,
therefore, they are not robust to noise
It shows you that your system is not robust to measurement noise, not that the dimensions are irrelevant.
does that make sense?
So, the system with higher number of SVD allows for much more noise before giving you results that are unreliable. That is what the values tell you. Similar to PCA dimnesion size (eigenvalues)
I've been trying to find the paper saying this with maths, but I dont have uni-level access to papers now, so its hard
 
@AnderBiguri (this is a statement I'd like to learn more about)
 
So first thing is: Lets do just math, forget any measurement in the world. PCA is just a decomposition that will return the data 100% to its original state if invertert. Its just another representation of the data. Thus, mathematically speaking, no dimension is irrelevant.
right?
Its like saying that hihg frequency sin-waves in the Fourier transform are irrelevant. Well, maybe in a practical human scenario they are, but really, mathematically speaking, all of them, to the infinity, are required for a perfect representation of the input fucntion
 
10:42 AM
sure
 
To chime in here; I always think of PCA in a 3D laser-scanning setting. LiDAR generates a 3D cloud of points of the environment. Now, what I used PCA for when dealing with these measurements was to find out how much energy was concentrated along 1, 2, or 3 PCs. If it was 1, the object was linear (lamp post, tree trunk etc), 2 PCs meant a surface (building facade, ground), 3 meant "scattered poo" (e.g. tree canopies).
Now, this does not mean that any of your initial measurement dimensions, X,Y,Z, are irrelevant. Because although 2 PCs carrying 99% of the signal means a surface, you need X, Y
 
If we continue with the example of the Fourier transform (we engineers tend to understand this one better): However, if you have measured a signal, with a noisy measurement, at some point, you will ignore higher frequency sin waves, because you know they are just noise. So you can say they are "irrelevant", because you know about the system
Its similar with SVD, however the ordering is different. The first SVDs will be the most important ones, and then they will decay. There is a point where those low value SVDs are just telling you that if you have noise, it will have higher impact in the result than the information that they provide. So they are "irrelevant", just because you know how much noise you will have.
In any case yes: This is a theoretical analysis of your system response. You still need to make all the measurements.
I dont know, I think im going in circles trying to explain.
 
@AnderBiguri :)
 
Ultimately the difference comes from knowing what are you studiyng. In your case, you are studying the response of a measurement system
Not trying to describe some data in a diferent axis
 
Does it matter?
(BTW, I am not attempting to do PCA, in case that was understood from what I said; I just mentioned it for philosophical context)
 
10:52 AM
Yes, because the conclusions are different. If you get a tiny eigenvector from data, you can say "this data does not require this dimension to be represented, thus I can ditch it and reduce memory", while if you do it for a matrix representing a system you just can say that you have less reconstruction capability due to noise-response. LEts put this same example in the fourier domain:
 
The fundamental question I'm trying to answer is whether the measurement system in question is able to recover the unknown values, and if so - to quantify it, so that different systems can be compared.
 
If you are doing the Fourier analysis of an image, or a camera detector, and it shows no high frequecy values, what is the difference? Well, in the image, you just now that this image has no high frequency values. But with the camera, you knwo that the camera can not capture such high frequency. It does not say that images have irrelevant high frequencies, just that your camera can not capture it. The conclusions are different
@Dev-iL So, you can SVD, and plot the values. The more there are in high values, the more proper "axis of freedom" to recover values your system has. More robust to inherent noise
Found the math: Plot log(sigma/sigma_o)
 
Hmm, alright.. So to conclude - "the composition of SVs is related to noise resistance"?
 
that is SVD in dBs. You can directly put your noise level there in dBs and any value below that is an independent axis you lost
A singular vector you can not recover, a "high frequency sin coefficient that you will not be able to obtain", thus loosing a bit of capability on getting the final result. You want maximum number of SVDs avobe that line
Electrical Impedance Tomography: Methods, History and Applications, page 16
 
k, lemme find it
is that fig 1.2 ?
 
11:01 AM
yes
they dont have the noise level
bu they show a couple of different singular vectors
in EIT they kinda look like high freq 2D fourier coefficients
They are "sensitivity" maps. They show that singular value, which data is measuring. That is the "axis you loose if noise is high"
its all very conceptual, but ultimatelly, you can say that thing about SVDs and noise level. You want lots, and high value
 
I have a problem visualizing this in the example of EIT tbh
 
its EIT, you can ignore it, just giving you a reference to cite if you want :P
 
I see, thanks :)
Also thank you @Adriaan
 
If you understand the Fourier example, its the same thing. At some point, your system will not be good enough to capture high frequency values of a measurement. For either system design or noise.
This is the same, just the "high order frequencies" ar enot frequencies, but singular vectors, which may have some arbitrary shape
 
@Dev-iL your welcome; sorry I couldn't be of more help
 
11:06 AM
I've never seen the terminology "mildly ill-posed"
 
In the PCA example, they are eigenvectors, but the same. You will not be able to capture the variance on those small axis. They are not irrelevant, just small and you loose all that info due to noise, which is bad
 
k
 
@Dev-iL hahaha, EIT is the most ill-posed thing I have ever seen, so not sure what is mild about it XD
 
So "the composition of singular values determines the degree of ill-posedness of a system"?
 
the condition number is, by definition, the ratio between the first and last SVD
so yes!
and in a normal matrix, the ration between the largest and smallest eigenvalues too. Closely related
 
11:13 AM
@AnderBiguri you see, I was trying to explain the shortcoming of using the condition number alone, as opposed to considering all intermediate SVs too
 
Ah, but then you use the SVD plot and noise level that I mentioned!
 
@AnderBiguri ill-posed != ill-conditioned
 
Say you compute all the SVDs. You plot log(sigma/sigma_0) and you get a straigth line. Now for a second system, you get 1 high value, and the rest a very low value.

Then, while they have the same condition number, the first is better, because any arbitrary horizontal line that you may put as "noise threshold" will cause the first one to have more singular values above that level, thus for the same condition number, the first one will get a better result
@Dev-iL indeed. An ill-posed problem will always cause an ill-conditioned description of it, but not the other way around.
 
@AnderBiguri That's a good explanation!
Why would noise be expressed "in units of" singular values, though?
 
Finally I hit the "oh I get that" spot then :D
@Dev-iL noise in dBs. dB is ultimately a ratio between the signal and the noise
so by doing log(sigma/sigma_0) you are putting the SVDs in the same ratio
 
11:21 AM
May 25 '17 at 13:05, by Ander Biguri
that is awesome. Good job Ander.
 
Ok but how is the fact you {choose to} express the ratio of singular values in [dB] related to the fact you {choose to} express noise in [dB]?
 
yes, otherwise, as you say, its not directly comparable
but in any case, you dont need to explicitly do it. If you know your noise level in dBs, yes, so you know exactly how your system behaves
 
I've never heard of comparison between dB of noise and dB of SVs... o_0
Interesting concept to be sure, but it sits uncomfortably with me
 
otherwise, by knowing you can do this comparison, you understand why its shape matters. Then the units are not too important, but the shape. There is some marginal improvement in understandabilty of the dB version, as ovbiously in log space it will have a stretched shape
@Dev-iL I have only seen it on the EIT field
but there, its supper common. Because noise is super high, and SVD decay super fast
everyone tests their new models by observing that decay and go "Yay, got 2 SVs above the line!"
 
hehe
I just hope this understanding stays with me (and not vanish as fast as it came 😛 )
 
11:27 AM
hahaha type it super fast
 
lol yeah... but I will surely remember having this discussion to refer to later
 
 
3 hours later…
2:34 PM
@Dev-iL sorry, had a crazy day, I'm just starting to read my emails for today...is it solved now?
skimming the transcript I probably wouldn't have been much help anyway, at least not today ;)
 
@AndrasDeak crazier than usual?
 
much
It's not usual that I can't read emails from 10 AM to 3 PM and have about a dozen relevant emails
 
Sounds tough
 
@AndrasDeak no worries
 
 
3 hours later…
5:23 PM
posted on January 30, 2020 by Cleve Moler

In an unpublished 1972 technical report "Conserving confluence curbs ill-condition," Velvel Kahan coined the descriptive term pejorative manifold. In case you don't use it in everyday conversation, pejorative means "expressing contempt or disapproval."... read more >>

 
 
3 hours later…
7:59 PM
posted on January 30, 2020 by Johanna Pingel

Every January, our company hosts an annual “kickoff”: an opportunity for sales, marketing and folks in customer facing roles to meet in Boston and learn about all the new and exciting features coming... read more >>

 

« first day (1674 days earlier)      last day (1552 days later) »