Results 11 to 20 of about 28,952 (292)

Semi-Stochastic Gradient Descent Methods [PDF]

Frontiers in Applied Mathematics and Statistics, 2017
In this paper we study the problem of minimizing the average of a large number of smooth convex loss functions. We propose a new method, S2GD (Semi-Stochastic Gradient Descent), which runs for one or several epochs in each of which a single full gradient
Jakub Konečný, Peter Richtárik
doaj   +5 more sources

Byzantine Stochastic Gradient Descent

CoRR, 2018
This paper studies the problem of distributed stochastic optimization in an adversarial setting where, out of the $m$ machines which allegedly compute stochastic gradients every iteration, an $α$-fraction are Byzantine, and can behave arbitrarily and adversarially.
Alistarh, Dan-Adrian, Allen-Zhu, Zeyuan, Li, Jerry   +2 more
core   +6 more sources

Analysis of stochastic gradient descent in continuous time [PDF]

Statistics and Computing, 2021
AbstractStochastic gradient descent is an optimisation method that combines classical gradient descent with random subsampling within the target functional. In this work, we introduce the stochastic gradient process as a continuous-time representation of stochastic gradient descent.
openaire   +7 more sources

Stochastic Modified Flows for Riemannian Stochastic Gradient Descent [PDF]

SIAM Journal on Control and Optimization
We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry we show that, in the small learning rate regime, RSGD can be approximated by the ...
Benjamin Gess, Sebastian Kassing, Nimit Rana   +2 more
core   +6 more sources

Conjugate Directions for Stochastic Gradient Descent

, 2002
The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from conjugate gradient in the stochastic (online) setting, using fast Hessian-gradient products to set up low ...
Nicol N. Schraudolph, Thore Graepel
openaire   +2 more sources

Dual Stochastic Natural Gradient Descent

CoRR, 2020
[EN] Although theoretically appealing, Stochastic Natural Gradient Descent (SNGD) is computationally expensive, it has been shown to be highly sensitive to the learning rate, and it is not guaranteed to be convergent. Convergent Stochastic Natural Gradient Descent (CSNGD) aims at solving the last two problems.
Sánchez-López, Borja, Cerquides, Jesús   +1 more
openaire   +3 more sources

Stochastic Reweighted Gradient Descent

CoRR, 2021
Despite the strong theoretical guarantees that variance-reduced finite-sum optimization algorithms enjoy, their applicability remains limited to cases where the memory overhead they introduce (SAG/SAGA), or the periodic full gradient computation they require (SVRG/SARAH) are manageable.
Ayoub El Hanchi, David A. Stephens
openaire   +2 more sources

Almost sure convergence of randomised‐difference descent algorithm for stochastic convex optimisation

IET Control Theory & Applications, 2021
Stochastic gradient descent algorithm is a classical and useful method for stochastic optimisation. While stochastic gradient descent has been theoretically investigated for decades and successfully applied in machine learning such as training of deep ...
Xiaoxue Geng, Gao Huang, Wenxiao Zhao
doaj   +1 more source

Training a Two-Layer ReLU Network Analytically

Sensors, 2023
Neural networks are usually trained with different variants of gradient descent-based optimization algorithms such as the stochastic gradient descent or the Adam optimizer.
Adrian Barbu
doaj   +1 more source

Stochastic gradient descent on GPUs [PDF]

Proceedings of the 8th Workshop on General Purpose Processing using GPUs, 2015
Irregular algorithms such as Stochastic Gradient Descent (SGD) can benefit from the massive parallelism available on GPUs. However, unlike in data-parallel algorithms, synchronization patterns in SGD are quite complex. Furthermore, scheduling for scale-free graphs is challenging.
Rashid Kaleem, Sreepathi Pai, Keshav Pingali   +2 more
openaire   +1 more source