currently working on the fifth and final lab of ML and was wondering if anyone knows what to do with the l-parameter in the k-function. At the moment my K is not a positive definite matrix which means that Cholesky does not work.
Also, is x_star simply the mean of x? I can not find it anywhere.
x-star is the test input. I is a diagonal matrix, so a matrix with ones on the diagonal (and the rest is zeros).
ReplyDeleteThanks for the x-star, seems logical enough. But I do not mean the identity matrix I but the length parameter l in the kernel function :)
ReplyDeletehey,
ReplyDeletecheck out the slides of the 12th lecture, i.e. Parzen Estimators. It defines how sensitive you wanna have your estimation wrt the data points.
Anyone else getting variances that are way smaller than they should be?
ReplyDeleteAt the moment my variance is zero which is clearly wrong :)
ReplyDeleteah, it is actually -6.5802e-26
ReplyDeleteah, sorry. The only thing that is told about that is that it has to be small, but not zero. So we experimented with different values for it. (0.1, 0.001, etc.) to see how it performed.
ReplyDeleteSo it turns out the statement about using extremely small sigma^2 is utter nonsense. Sigma is, obviously, the prior target noise variance, and so you can look at the data and see that it's somewhere between 1 and 20. We found through leave-one-out cross-validation that 5 is about optimal.
ReplyDeleteMartijn told us this is correct, but didn't mail you guys the correction so I thought I'd post it..
So is it still necessary to include the theta parameter in the k-function? or can we leave that one out?
ReplyDeleteAlso, does exp apply only to the (-1/2*l) or to the entire function?
The theta parameter is not needed for fixing the variance issue (in fact it can't fix it..), but is still a useful parameter.
ReplyDeleteThe exp is over the rest of the formula; it's like a Gaussian.
hey,
ReplyDeletehere's our report, though not a perfect one.
https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B-BenD7gC8qdYzk2OGU5OGMtN2Y3ZC00YjY1LWExY2UtYzY5YmVhNjllMmI1&hl=en_US
There are not enough datapoints, so if you dont increase the weight of your kernel with theta the 0 prior weighs in too much. You need the theta to make the influence of the kernels stronger. So yeah, you need theta :P.
ReplyDeleteJam: Because the prior of your distribution is equal to the 0 function, it is better to zero-mean the target functions before you do the regression. For the plots you can add the mean after. This improves the results by quite a bit :)
ReplyDeleteAlso: your last plot says A = 0.01 two times. To get good predictions it is also wise to choose a sigma between 1 and 20 (this was a big mistake on the part of the teacher and the TA...) And a theta of around 60-80 (kernel * theta) as Gwenn stated on blackboard.
ReplyDeleteJust a few tips for if you still want to change some stuff XD.
T
Added our and Jammin's report to the first post <3
ReplyDelete