Enhancing Sharpness-Aware Optimization Through Variance Suppression [PDF]
Neural Information Processing Systems, 2023 Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation.
Bingcong Li, G. Giannakis
semanticscholar +1 more source
Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization [PDF]
Neural Information Processing Systems, 2023 Despite extensive studies, the underlying reason as to why overparameterized neural networks can generalize remains elusive. Existing theory shows that common stochastic optimizers prefer flatter minimizers of the training loss, and thus a natural ...
Kaiyue Wen, Tengyu Ma, Zhiyuan Li
semanticscholar +1 more source
Robust Generalization Against Photon-Limited Corruptions via Worst-Case Sharpness Minimization [PDF]
Computer Vision and Pattern Recognition, 2023 Robust generalization aims to tackle the most challenging data distributions which are rare in the training set and contain severe noises, i.e., photon-limited corruptions.
Zhuo Huang +8 more
semanticscholar +1 more source
Sharpness-Aware Minimization Leads to Low-Rank Features [PDF]
Neural Information Processing Systems, 2023 Sharpness-aware minimization (SAM) is a recently proposed method that minimizes the sharpness of the training loss of a neural network. While its generalization improvement is well-known and is the primary motivation, we uncover an additional intriguing ...
Maksym Andriushchenko +3 more
semanticscholar +1 more source
Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach [PDF]
Neural Information Processing Systems, 2022 Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of ...
Peng Mi +6 more
semanticscholar +1 more source
Fisher SAM: Information Geometry and Sharpness Aware Minimisation [PDF]
International Conference on Machine Learning, 2022 Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood ...
Minyoung Kim +3 more
semanticscholar +1 more source
Normalization Layers Are All That Sharpness-Aware Minimization Needs [PDF]
Neural Information Processing Systems, 2023 Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings.
Maximilian Mueller +3 more
semanticscholar +1 more source
Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning [PDF]
International Conference on Machine Learning, 2022 Model-agnostic meta learning (MAML) is currently one of the dominating approaches for few-shot meta-learning. Albeit its effectiveness, the optimization of MAML can be challenging due to the innate bilevel problem structure.
Momin Abbas +4 more
semanticscholar +1 more source
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction [PDF]
Neural Information Processing Systems, 2022 Normalization layers (e.g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets, but they clearly also help generalization, even in not-so-deep nets.
Kaifeng Lyu, Zhiyuan Li, Sanjeev Arora
semanticscholar +1 more source
Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term [PDF]
Knowledge Discovery and Data Mining, 2023 Deep Neural Networks (DNNs) generalization is known to be closely related to the flatness of minima, leading to the development of Sharpness-Aware Minimization (SAM) for seeking flatter minima and better generalization. In this paper, we revisit the loss
Yun Yue +5 more
semanticscholar +1 more source