@ Override public void applyUpdater ( INDArray gradient , int iteration , int epoch ) { if ( historicalGradient == null ) throw new IllegalStateException ( "Updater has not been initialized with view state" ) ; double learningRate = config . getLearningRate ( iteration , epoch ) ; double epsilon = config . getEpsilon ( ) ; historicalGradient . addi ( gradient . mul ( gradient ) ) ; INDArray sqrtHistory = sqrt ( historicalGradient . dup ( gradientReshapeOrder ) , false ) . addi ( epsilon ) ; // lr * gradient / (sqrt(sumSquaredGradients) + epsilon) gradient . muli ( sqrtHistory . rdivi ( learningRate ) ) ; }
Gets feature specific learning rates Adagrad keeps a history of gradients being passed in . Note that each gradient passed in becomes adapted over time hence the opName adagrad