Results 91 to 100 of about 1,424,164 (189)
On the regularizing property of stochastic gradient descent
Inverse Problems, 2018 Stochastic gradient descent is one of the most successful approaches for solving large-scale problems, especially in machine learning and statistics. At each iteration, it employs an unbiased estimator of the full gradient computed from one single randomly selected data point.
Bangti Jin, Xiliang Lu
openaire +3 more sources
A note on diffusion limits for stochastic gradient descent
, 2022 In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in stochastic gradient algorithms, has ...
Lanconelli, Alberto +1 more
core
Emergent universal long-range structure in random-organizing systems
Nature CommunicationsSelf-organization through noisy interactions is ubiquitous across physics, mathematics, and machine learning, yet how long-range structure emerges from local noisy dynamics remains poorly understood.
Satyam Anand +2 more
doaj +1 more source
Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
Applied Sciences, 2019 Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of ...
Hyeyoung Park, Kwanyong Lee
doaj +1 more source
Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks [PDF]
Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing ...
Stephan, Ludovic +4 more
core
Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods
MathematicsMini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies—Perceptron-Inspired SGD (PISGD), Pruning SGD ...
Roberto Cavoretto +2 more
doaj +1 more source
Mixing of Stochastic Accelerated Gradient Descent
CoRR, 2019 We study the mixing properties for stochastic accelerated gradient descent (SAGD) on least-squares regression. First, we show that stochastic gradient descent (SGD) and SAGD are simulating the same invariant distribution. Motivated by this, we then establish mixing rate for SAGD-iterates and compare it with those of SGD-iterates.
Peiyuan Zhang +2 more
openaire +3 more sources
Free-energy optimization based on stochastic gradient descent to minimize prediction error.
, 2017 Free-energy optimization based on stochastic gradient descent to minimize prediction error.
Alexandre Pitti (438059) +2 more
core +1 more source
Parallelized stochastic gradient descent
, 2010 With the increase in available data parallel machine learning has become an increasingly pressing problem. In this paper we present the first parallel stochastic gradient descent algorithm including a detailed analysis and experimental evidence.
Martin A Zinkevich +3 more
core
SGDR: Stochastic Gradient Descent with Warm Restarts
CoRR, 2016 ICLR 2017 conference ...
Ilya Loshchilov, Frank Hutter
openaire +4 more sources