2
$\begingroup$

I read on Wikipedia:

AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011. Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse.

What do they mean by parameter sparsity? I read that AdaGrad averages / accumulates gradients over time. Does parameter sparsity refer to how often (dense?) these gradients are accumulated? or something else?

$\endgroup$
2
$\begingroup$

Yes by sparse parameters they mean that they do not receive often/large updates. This naming is a little bit confusing, because this often happens when the data is also sparse, so that some features appear only a few times.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.