« first day (73 days earlier)      last day (3162 days later) » 
00:00 - 08:0008:00 - 22:00

12:04 AM
wait... I'm not the only one seeing this right: youtu.be/t9Kla_YFdfs?t=3m23s it multiplies a scalar by a matrix but the multiplication is wrong
 
 
1 hour later…
1:17 AM
@OneRaynyDay - What is the problem?
@OneRaynyDay - MATLAB always passes by value. Never by reference.
 
1:48 AM
@rayryeng 2*[x;y;z] example
it was [-1;-0.5;0;0.5;1]
and the answer it said was:
[-2,-1,0,1,1]
it should be [-2,-1,0,1,2]... unless I'm currently incapacitated by a foreign substance LOL
 
You're right, it's a mistake.
 
gotcha. I was flipping out :)
Anyways, matlab successfully download woot! Let's try it out :D
 
cool!
 
@rayryeng hey ray, you said you take the coursera course for machine learning every year as a refresher right?
 
not every year lol
 
2:02 AM
does that mean you'll be taking the one that starts in 3 days?
 
I took it in 2012. I took it again recently.
 
ohh gotcha :D
 
no I recently finished it.
I did it just the last session.
 
Ah I see, how often do they open up sessions?
 
every 2 months.
This is because if you fall behind, you can just wait for the next session.
they never used to do that.
It used to be an actual course where you wait each week for material.
Now they release all of the material, and you complete each assignment/quiz by the recommended due dates.
 
2:05 AM
ahh I see
Is it very hard for a beginner?
 
For someone who has no knowledge in math and programming yes.
But because you have good programming experience and some good Calculus knowledge, this should be totally fine.
There is some linear algebra you need to know, but you can learn that as you go.
 
awesome :) The linear algebra won't be too hard?
 
no for you I don't think so.
 
Haha alright thanks!
Also I think they release the material weekly right?
I only see the first week's material right now(but then again the class hasn't started yet)
 
Yes it's just previewing right now.
once the course open, all of the material gets released.
 
2:09 AM
oh sweet. I'll try and rush through everything since school will be starting soon :)
 
ok :)
It's possible to do it in two weeks or less if you have nothing better to do
I was balancing this with work.
 
forrealies? holy
 
yeah lol
I'd watch the videos during my lunch break sometimes.
 
stays in coffee shop for 8 hours everyday
 
do the assignments on the train home.
 
2:12 AM
ah - why train? do you live in a metropolitan area?
 
I commute into work most of the time. Traffic driving downtown sucks.
No I live in the suburbs. 20 minutes from Toronto.
the assignments don't take me long because well... I have a clear advantage :)
not so much the theory, but the MATLAB syntax.
 
Ah, I see :) I used to live in west side of canada actually!
 
oh really? Interesting.
 
I grew up til 3rd grade in china and then flew over to vancouver
 
Vancouver I'm assuming?
 
2:14 AM
where I then stayed until 8th grade
haha bingoooo
 
ahh, how long did you live there until moving to the Valley?
Vancouver is usually the destination.
 
8th grade! :) Or more like end of 8th grade
yeah truee, not a lot of people live in albeta/saskatchewan
(I forgot how to spell saskatchewan)
(is that the correct spelling oops)
I was in highschool or secondary school at 8th grade, but then when I moved to valley 8th grade was middle school
So I felt super out of place at first haha
education in vancouver(or at least where I lived) wasn't too good in comparison... I went to middle school in the valley not knowing what a quadratic equation was
and everyone ridiculed me - that was kind of when I started doubling up on STEM stuff
 
That's how you spell it!
ohhh... really? Are we that far behind in our education system?
8th grade was when I learned the quadratic equation. Standards must be getting worse.
 
@rayryeng oh pheww, still proud to be canadian haha
I think it's because, in vancouver
Things don't start to ramp up until IB classes
Also for some reason in my highschool/secondary, we learned math for only half a year
every class was half a year long. It was so strange
 
2:59 AM
@OneRaynyDay what? that's so weird!
 
3:15 AM
@rayryeng yeah... Its name was Semiahmoo, which is a weird name too haha
 
oook then lol
BTW, I'm taking two coursera courses or about to... now that we're on the topic
Audio Signal Processing for Music Applications and Mining Massive Datasets.
The second course is a natural extension to machine learning that I've always wanted to learn. How do you deal with big data right?
The first course is because of pure interest. I have the signal processing background, but I'd like to see where sound comes into play.
 
@rayryeng Audio Signal Processing?
That sounds interesting!
and also, mining massive datasets... what kind of datasets? Will you be learning R? I heard that language is one of the highest level languages out there :)
 
I know some R. Dealt with it in the past.
I actually don't know what language it will be... the course description was very vague lol
If it's R that would be great. I'd like a revamp on it
 
ah I see. I read in the machine learning course's FAQ
"Why don't we use R?"
"Because everything is already implemented in R. You don't need to perform any tasks"
something along those lines haha
 
hahahah
well technically the Statistics Toolbox in MATLAB also has everything implemented too.
I find R's syntax cryptic though
I could never get used to it
 
3:27 AM
ahh I see... never used R
but my mom who's also a programmer
dragged me to a big data seminar when I was like.. in 10th grade
and they were preaching like evangelists about R
Like "THIS IS GOING TO BE THE NEXT BIG THING GUYS"
Which I think is probably somewhat accurate :)
hθ(x)=θ0+θ1x
We give to hθ values for θ0 and θ1 to get our output 'y'. In other words, we are trying to create a function called hθ that is able to reliably map our input data (the x's) to our output data (the y's).

Example:

x (input)	y (output)
0	4
1	7
2	7
3	8
Now we can make a random guess about our hθ function: θ0=2 and θ1=2. The hypothesis function becomes hθ(x)=2+2x.

So for input of 1 to our hypothesis, y will be 4. This is off by 3.
Just a quick question, Why does it say it's "off by 3" ?
Shouldn't it be off by two... if hθ = 2 + 2x, and x = 0... 4-2 = 2...
so it should be off by 2 right... these mistakes make me think so unnecessarily hard at these problems haha
 
Because the expected value of y when x=1 is 7, but your proposed model gives you 4.
it's off by 3 because 7 - 4 = 3.
 
Oh. derp, I totally read that as x = 0
Sorry, that was a dumb mistake haha
 
no worries :)
R is used by statisticians. I prefer to stay away from it and use MATLAB
or Octave.
 
ahh okay. Can R be used to write an OOP program? or scriptbased?
 
Yes you can certainly use it to make classes.
I've only briefly looked at it but I've never written classes for it
 
3:33 AM
ahh okay. Interesting :o I don't think I'll be touching it for a while though
 
ah if you don't, you won't miss it :)
 
yeah haha
One second, getting out a notepad and doing some gradient descent derivations
 
oh :)
This Mathematics SE post may help
31
Q: Partial derivative in gradient descent for two variables

voithosI've started taking an online machine learning class, and the first learning algorithm that we are going to be using is a form of linear regression using gradient descent. I don't have much of a background in high level math, but here is what I understand so far. Given $m$ number of items in our...

 
ahh thanks for the link!
I think I'm going to try and discover this part out a bit by myself first though :) I always learn better when I write things down
 
yup, that's there if you want it. But yes, always good to derive it yourself.
 
3:42 AM
yupyup :) Hmm.. only thing is I'm not sure whether to derive from θ0 or θ1
 
gradient descent optimizes each parameter independently.
so you do derivatives with respect to each variable separately while leaving the other constants.
 
ah.. So one dJ/dθ0 and the other dJ/dθ1, leaving θ1 and θ0 constant respectively
gotcha
 
Correct. That's how partial derivatives work
differentiate with respect to the variable you want, leaving the other ones constant.
So for example, supposing we had a three variable function: F(x,y,z) = xyz
doing dF/dx gives us yz
doing dF/dy gives us xz
and you can guess what dF/dz is.
 
haha xy
I remember a bit of this :) Thanks for the refresher!
 
haha sure :)
 
3:45 AM
oh crap, I'll have to be back in maybe an hour :o I forgot to do something
this class was too interesting to stop LOL
Alright I'll be back! talk to you later!
 
no worries! ttyl
 
 
2 hours later…
5:20 AM
@rayryeng hey, back! :)
I went to workout at the local YMCA, got a schedule but forgot to go haha
Plus side on going late is I see different people, including a cute girl ;)
 
6:01 AM
hahaha sweet.
 
yeah haha, I'm reading over the linear algebra "refresher" for the machine learning coursera page
hey by the way, I have no idea what a pseudo-inverse is in linear algebra, what's the difference between that and inverse? :/
 
The inverse only exists for square matrices.
 
@rayryeng oh, if I don't reply, I'll most likely be in the shower
 
Pseudo-inverse is used in the context of least-squares minimizatoin.
 
Right - so it's a way to compensate for non-square matrix right
 
6:09 AM
the reason why it's called "pseudo" is because you can artificially make the problem so that you're solving the inverse of a square matrix.
 
but the formula looks... kind of obfuscated
 
It's actually not bad. Given that we have Ax = b, if you multiplied by A^T on both sides, we get:
 
and there's two versions from what I can see o_O
 
A^T*Ax = A^T*b
solving for x gives us x = (A^T*A)^-1 * (A^T) * b
 
oh.. so multiplying by the transverse gives us.. say a 3x2 instead of a 2x3, and when you multiply you get a 2x2 matrix
 
6:10 AM
3 x 3, but yes
This is called the "pseudo-inverse" because it finds the solution of x to the system with the least-squared error.
It does not give you the exact solution. Only an approximation.
When you have more rows than columns, this is what we call an overdetermined system.
 
Oh... Huh
Oh gosh so many terms thrown at me at once
 
So the solution of x gives you the best approximate that satisfies each constraint to the best of its ability.
 
I know the least-squared error is the (hypothesis(x) - y)^2
 
Let me find something that could explain it better.
 
thank you! Sorry I couldn't really understand - I think I'll go watch some khan academy videos tomorrow for linear algebra so I don't ask stupid questions
 
6:13 AM
oh lol.
ok, well let's start from the beginning.
Probably in high school, you had to solve a system of linear equations. Right? Something like this
You've frequently dealt with the simplest case where there are N equations with N unknowns.
 
yep!
 
Ok well that's cool... now let's go back to the case of fitting a straight line through 2 points.
well, going with the above picture, you can turn that into a matrix equation.
Ax = b. A is a N x N matrix and b is a N x 1 vector.
 
ah yes
so it would be if a is a
m x n matrix
and x is a vector...
 
Yes.
 
of n size, leading to the b vector
 
6:18 AM
Now, in the context of machine learning, you have a bunch of x and y points.
let's say you have like 50 points.
and you want to fit a line through as many points as possible that have the least amount of error.
Well... all you need are 2 points to make a line.
 
right!
 
So... which 2 points do you choose?... or should do something else?
You can formulate this into Ax = b quite nicely. Remember, the equation of a line is y = mx + b.
 
uhmm I don't think you should choose pre-existing points
 
certainly not! but we'll get there.
 
Right. Let me see...
 
6:20 AM
so we have x and y points... and we want to solve for m and b. You can think of m and b as x1 and x2 in the diagram.
 
ah I see... so b is multiplied by 1 while m is multiplied by x
 
Correct.
So we can actually put this into matrix form.
So we'd get the following matrix for A
I'm going to LaTeX this out. One moment.
 
so it's a n by 2(power of the regression +1) matrix multiplied by...
a vector of 2(power of the regression +1) by 1?
 
well, vectors are always [by 1]
Ahh I see that
OH. is the 1 the derivative
That's the only reason why there would be a 1... I'm only messing around in the dark here haha
 
6:23 AM
and x would be a vector such that x = [m b]
So if you worked it out, Ax = b gives you y = mx + b pairs for each pair of (x,y) values we have.
 
right! Yeah that's what I said above :)
 
So this is what we call an overdetermined linear system.
Overdetermined means that you have more equations than unknowns.
This means that there is more than one possible answer that solves this linear system.
however, each solution will give you some error because not all of the points will fit nicely for a given pair of m,b.
 
head explodes I think that's where I get confused
 
That's where the pseudo-inverse comes into play
 
We have n amount of equations, and 2*n amounts of unknown I guess?
 
6:25 AM
It solves this linear system in such a way where m and b are determined that will fit a line through the points with the least amount of errors.
no, just two unknowns. m and b.
Let me put it into a better form
m and b are unknown. A is a N x 2 matrix. The total number of columns in A are how many variables you are solving for.
the total number of rows is how many points / equations you have.
we want to find the best value of m and b that will make each equation in our system roughly equal on the left and right hand side.
With an overdetermined system, there's more than one possible m and b that can satisfy most of the equations exactly, but not all of them.
 
so you have n equations and 2 variables. I see
 
Which is why we turn to the pseudo-inverse. This finds the best value of m and b that will make most equations generally happy.
there will obviously be some errors, but these errors are for the most part as small as possible.
 
So it's like saying 3x+2 = 5, but 2x+3 = 10 as well, and a ton more... which means the x has multiple answers
 
correct.
so the pseudo-inverse finds the solution to x that will satisfy most equations.
 
Ahhh I see it now.
 
6:29 AM
because there will bound to be some equations where the chosen solution x does not satisfy the equation.
it's bound to happen. You have more equations than you have variables.
 
right.
 
so that's where the pseudo-inverse comes into play.
 
So would you rather get more equations exactly right, or would you get it so that the error of each equation is the minimum?
 
Correct.
 
wait.. so.. both?
 
6:30 AM
oh sorry. The second.
The first one will give you bad results. When you study machine learning, this is what is called overfitting
you'll know about that stuff later.
 
Ahh I've heard of it
gotcha.
Okay... so the pseudo-inverse turns the 3x2 into a 3x3 by multiplying the transposed version
and inversed because it's moving to the other side of the equation to find X
and in the process of doing the pseudo-inverse, I'm guessing it does a type of averaging out effect where
each number's stdev is slightly decreased?
So it's not referring to the original value, and then you're trying to find X with this not-so-precise-because-averaged matrix which is why it's not 100% accurate
brb shower!
 
Morning you happy people!
Hoppa! Reached the 1k rep!
 
6:47 AM
@Adriaan - yay!
 
Seeing the votebreakdown is rather useless though :P
might as well set it to default view imho
 
@OneRaynyDay - Yes. I won't prove it here, but the solution of x from the pseudo-inverse finds the parameters that best minimize the error of the overall equations.
 
@rayryeng mkay :) thank you ray!
 
Think of it this way. Suppose we have a parabola, and we want to fit a straight line through it. One way of thinking is to fit as many points as possible through the lines... so you'd think that drawing a line through the first half or second half of the parabola may be nice
but the errors on the other side of the parabola are bad.
if you try and draw a straight line from one side of the parabola to the other side... choosing perhaps the middle of the two sections, you could say this would be better at minimizing the error.
@Adriaan - I agree!
 
well, time to get the next 1000 rep. At least I can do something useful then
 
6:54 AM
@rayryeng Right, I understand that :)
Thank you so much by the way! Do you use venmo by the way?
 
Now... why would you want to use gradient descent, rather than the pseudo-inverse you may ask?
Well for problems that require 1000s of variables... storing the data in matrix form will prove to quickly run out of memory
gradient descent is more memory friendly.
 
Ahh I see
 
@OneRaynyDay - I don't actually.
 
But it takes a lot more steps to get to the bottom of the gradient slope though
 
Correct.
So you have to make a decision.
 
6:55 AM
so it takes longer time to compute but less memory taken - ah!
 
Do you want more data?... or do you want it to compute faster?
Things like this factor in when designing a ML system.
 
That just totally clicked in my head - I thought they were two completely different processes
 
Both of them are finding the right parameters of your line to minimize the error.
Gradient Descent is an iterative method to minimize the objective function
the pseudo-inverse does it as well, but does it in one step
Gradient descent requires iterations.
 
Thank you so much! Want to consider making a venmo so I can tip you a donation? haha I don't have a paypal since my friend used my gmail to buy things online
 
oh :D lol
that's really nice of you. Sure, let me see.
 
6:57 AM
Gotcha - I understand gradient descent much better than this linear algebra stuff, but
I kind of get the gist from your explanation now :)
 
haha well kind of is good
 
haha yeah, I'm the kind of learner that needs the click to put everything together
 
that's fine with me :)
 
yeah haha :) Well anyways, let me know when you make the account!
 
Sure :)
@Adriaan - Divakar answered lol
 
7:10 AM
that use of bsxfun is magic to me :P
 
@OneRaynyDay - Venmo not available in my area.
@Adriaan - It just takes practice :)
 
what?! Errrrm.. I'll think of some way :)
 
@OneRaynyDay send him an envelope with Rubels
 
@Adriaan I'll get a crane to fly over to Toronto with 3 zillion zimbabwe cash
 
which I will see ZERO of.
damn corrupt government.
 
7:13 AM
Ah, he appreciates that very much!
 
@Adriaan
Suggestion. You don't need to use find. Use logical operations and sum: C(ii,1) = sum(A > B(ii,1) & A < B(ii,2));. — rayryeng 8 mins ago
 
I implemented it?
 
I didn't see it!
oop see it now
it didn't refresh
 
I even mentioned your name
 
yes it didn't refresh lol
 
7:17 AM
LOL hey at least you can flaunt around zimbabwe dollars in public
 
money that I will never see. There's nothing to flaunt!
 
7:34 AM
haha trueee
don't worry ;) I'll find some way to send you usd
 
Brr, American money
 
7:50 AM
@rayryeng grmph, I don't like OPs accepting an answer and then asking additional requirements ...
 
hmm... interesting - so the mean square error function is a function 1 degree higher than the regression
In univariate linear regression you'd get a parabolic function for the error bound
cool! :)
(Or at least fom my speculation)
 
@Adriaan - Yes. It's a sneaky way for them to entice you to do more work.
@OneRaynyDay - Yes that's correct.
 
ah - so for a 3rd degree regression, you'd get a mean square error function of a x^4 function...
 
at least it was an easy fix. Not sure how to edit the second solution though, except for adding another forloop indexing C = [1 C]; and then using that to collect A
 
but then it would have 2 minimums...?
 
00:00 - 08:0008:00 - 22:00

« first day (73 days earlier)      last day (3162 days later) »