back propagation in a neural network in order to compute derivatives.
can be a little bit tricky to implement.
have subtle bugs in back prop.
some other optimizational algorithm, it could actually look like it's working.
J of theta may end up decreasing on every iteration of gradient descent.
bug in your implementation of back prop.
error than you would with a bug free implementation.
that eliminates almost all of these problems.
any other reasonably complex model, I always implement gradient checking.
your implementation of four prop and back prop or whatever is 100% correct.
with a sort of a buggy implementation as a back prop.
faith that those actually do compute the gradients of the cost function.
does indeed, is indeed computing the derivative of the cross function J.
So here's the idea, consider the following example.
and for this example gonna assume that theta is just a real number.
and so the derivative is equal to the slope of that tangent one.
rather here's a procedure for numerically approximating the derivative.
I'm going to compute theta plus epsilon, so now we move it to the right.
the slope of that little red line as my approximation to the derivative.
Which is, the true derivative is the slope of that blue line over there.
So, you know it seems like it would be a pretty good approximation.
divided by this horizontal width.
So this point on top is the J of (Theta plus Epsilon).
of theta minus epsilon and this horizontal distance is just 2 epsilon.
J of theta plus epsilon minus J of theta minus epsilon over 2 epsilon.
epsilon, expect epsilon to be maybe on the order of 10 to the minus 4.
There's usually a large range of different values for epsilon that work just fine.
this term here, actually mathematically, it becomes the derivative.
It becomes exactly the slope of the function at this point.
too small, because then you might run into numerical problems.
So I usually use epsilon around ten to the minus four.
s meeting the derivative which is this formula.
whereas the formula on the left, that's called a two-sided difference.
I usually use that, rather than this one sided difference estimate.
J of theta plus epsilon minus J of theta minus epsilon divided by 2 times epsilon.
And this will give you a numerical estimate of the gradient at that point.
And in this example it seems like it's a pretty good estimate.
we considered the case of when theta was a rolled number.
let's say theta is an R n.
And it might be an unrolled version of the parameters of our neural network.
So theta is a vector that has n elements, theta 1 up to theta n.
use a similar idea to approximate all the partial derivative terms.
parameter, theta one, that can be obtained by taking J and increasing theta one.
So you have J of theta one plus epsilon and so on.
Minus J of this theta one minus epsilon and divide it by two epsilon.
and here you're decreasing theta two by epsilon and so on down to the derivative.
derivative of J with respect to any one of your parameters theta i.
Completely, what you implement is therefore the following.
We implement the following in octave to numerically compute the derivatives.
i = 1:n, where n is the dimension of our parameter of vector theta.
And I usually do this with the unrolled version of the parameter.
So theta is just a long list of all of my parameters in my neural network, say.
then increase thetaPlus of the (i) element by epsilon.
thetaPlus(i) which is now incremented by epsilon.
Epsilon, so theta plus is equal to, write theta 1, theta 2 and so on.
Then theta I has epsilon added to it and then we go down to theta N.
So this is what theta plus is.
that this instead of theta I plus Epsilon, this now becomes theta I minus Epsilon.
respect of theta i of J of theta.
we can then take the gradient that we got from backprop.
So DVec was the derivative we got from backprop.
was a relatively efficient way to compute a derivative or a partial derivative.
Of a cost function with respect to all our parameters.
that is this gradApprox that we just had from up here.
small values of numerical round up, that it's pretty close.
So the DVec that I got from backprop.
then I'm much more confident that my implementation of backprop is correct.
hopefully my code will run correctly and do a good job optimizing J of theta.
tell you how to implement this numerical gradient checking.
Here's what I usually do.
First thing I do is implement back propagation to compute DVec.
to compute DVec which may be our unrolled version of these matrices.
is implement a numerical gradient checking to compute gradApprox.
So this is what I described earlier in this video and in the previous slide.
you know let's say up to a few decimal places.
the numerical derivative formulas that we talked about earlier in this video.
that's a very slow way to try to approximate the derivative.
that is the thing we talked about earlier for computing.
You know, D1, D2, D3 for Dvec.
correct, you should turn off gradient checking and just stop using that.
in order to train your classifier.
every single iteration of gradient descent.
then your code would be very slow.
you remember, we were computing delta(4), delta(3), delta(2), and so on.
That was the backpropagation algorithm.
That is a much faster way to compute derivates than gradient checking.
while you train your algorithm, or else you code could run very slowly.
tha implementation of back propagation is correct.
this really helps me make sure that my code is correct.