Results 81 to 90 of about 1,424,164 (189)

Certain Systems Arising In Stochastic Gradient Descent [PDF]

, 2019
Stochastic approximations is a rich branch of probability theory and has a wide range of application. Here we study stochastic approximations from the perspective of gradient descent.
Karatapanis, Konstantinos
core  

Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent [PDF]

, 2022
Gess B, Gvalani R, Konarovskyi V. Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent.
Gvalani, Rishabh, Gvalani, Rishabh S., Gess, Benjamin, Konarovskyi, Vitalii   +3 more
core  

Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions [PDF]

, 2020
Fehrman B, Gess B, Jentzen A. Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions. Journal of Machine Learning Research (JMLR) .
Gess, B., Gess, Benjamin, Fehrman, B., Fehrman, Benjamin, Jentzen, A., Jentzen, Arnulf   +5 more
core   +1 more source

A Bootstrap Perspective on Stochastic Gradient Descent

CoRR
Machine learning models trained with \emph{stochastic} gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the statistical bootstrap: SGD uses gradient variability under batch sampling as a proxy for solution variability under
Hongjian Lan, Yucong Liu, Florian Schäfer 0001   +2 more
openaire   +2 more sources

Semi-Cyclic Stochastic Gradient Descent

CoRR, 2019
We consider convex SGD updates with a block-cyclic structure, i.e. where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different ...
Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, Kunal Talwar   +4 more
openaire   +3 more sources

On the Hyperparameters in Stochastic Gradient Descent with Momentum

CoRR, 2021
34 pages, 4 figures.
openaire   +3 more sources

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

We prove, under mild conditions, the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model, both in batch gradient descent and stochastic gradient descent.
Wang, Bao, Whiting, Wes, Xin, Jack
core   +1 more source

A note on diffusion limits for stochastic gradient descent [PDF]

In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in stochastic gradient algorithms, has ...
Lanconelli, Alberto, Lauria, Christopher S. A.   +1 more
core   +1 more source

Adam Algorithm with Step Adaptation

Algorithms
Adam (Adaptive Moment Estimation) is a well-known algorithm for the first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
Vladimir Krutikov, Elena Tovbis, Lev Kazakovtsev   +2 more
doaj   +1 more source

Convergence of Stochastic Gradient Descent for PCA

CoRR, 2015
We consider the problem of principal component analysis (PCA) in a streaming stochastic setting, where our goal is to find a direction of approximate maximal variance, based on a stream of i.i.d. data points in $\reals^d$. A simple and computationally cheap algorithm for this is stochastic gradient descent (SGD), which incrementally updates its ...
openaire   +3 more sources