Results 61 to 70 of about 1,273 (111)
On the Hyperparameters in Stochastic Gradient Descent with Momentum
CoRR, 2021 34 pages, 4 figures.
openaire +3 more sources
Adam Algorithm with Step Adaptation
AlgorithmsAdam (Adaptive Moment Estimation) is a well-known algorithm for the first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
Vladimir Krutikov +2 more
doaj +1 more source
Semi-Cyclic Stochastic Gradient Descent
CoRR, 2019 We consider convex SGD updates with a block-cyclic structure, i.e. where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different ...
Hubert Eichner +4 more
openaire +3 more sources
A Bootstrap Perspective on Stochastic Gradient Descent
CoRRMachine learning models trained with \emph{stochastic} gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the statistical bootstrap: SGD uses gradient variability under batch sampling as a proxy for solution variability under
Hongjian Lan +2 more
openaire +2 more sources
Convergence of Stochastic Gradient Descent for PCA
CoRR, 2015 We consider the problem of principal component analysis (PCA) in a streaming stochastic setting, where our goal is to find a direction of approximate maximal variance, based on a stream of i.i.d. data points in $\reals^d$. A simple and computationally cheap algorithm for this is stochastic gradient descent (SGD), which incrementally updates its ...
openaire +3 more sources
Emergent universal long-range structure in random-organizing systems
Nature CommunicationsSelf-organization through noisy interactions is ubiquitous across physics, mathematics, and machine learning, yet how long-range structure emerges from local noisy dynamics remains poorly understood.
Satyam Anand +2 more
doaj +1 more source
Stochastic Gradient Descent for Kernel-Based Maximum Correntropy Criterion
EntropyMaximum correntropy criterion (MCC) has been an important method in machine learning and signal processing communities since it was successfully applied in various non-Gaussian noise scenarios.
Tiankai Li +3 more
doaj +1 more source
Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
Applied Sciences, 2019 Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of ...
Hyeyoung Park, Kwanyong Lee
doaj +1 more source
Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods
MathematicsMini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies—Perceptron-Inspired SGD (PISGD), Pruning SGD ...
Roberto Cavoretto +2 more
doaj +1 more source
Mixing of Stochastic Accelerated Gradient Descent
CoRR, 2019 We study the mixing properties for stochastic accelerated gradient descent (SAGD) on least-squares regression. First, we show that stochastic gradient descent (SGD) and SAGD are simulating the same invariant distribution. Motivated by this, we then establish mixing rate for SAGD-iterates and compare it with those of SGD-iterates.
Peiyuan Zhang +2 more
openaire +3 more sources