Results 271 to 280 of about 13,997,719 (325)

DoRA: Weight-Decomposed Low-Rank Adaptation

International Conference on Machine Learning
Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen   +6 more
semanticscholar   +1 more source

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

International Conference on Machine Learning
Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix ...
Jiawei Zhao, Zhenyu (Allen) Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian   +5 more
semanticscholar   +1 more source

Learning Tensor Low-Rank Representation for Hyperspectral Anomaly Detection

IEEE Transactions on Cybernetics, 2022
Recently, low-rank representation (LRR) methods have been widely applied for hyperspectral anomaly detection, due to their potentials in separating the backgrounds and anomalies.
Minghua Wang, Qiang Wang, D. Hong, S. K. Roy, J. Chanussot   +4 more
semanticscholar   +1 more source

Low-Rank and Sparse Representation for Hyperspectral Image Processing: A review

IEEE Geoscience and Remote Sensing Magazine, 2022
Combining rich spectral and spatial information, a hyperspectral image (HSI) can provide a more comprehensive characterization of the Earth’s surface. To better exploit HSIs, a large number of algorithms have been developed during the past few decades ...
Jiangtao Peng, Weiwei Sun, Hengchao Li, Wei Li, Xiangchao Meng, Chiru Ge, Qian Du   +6 more
semanticscholar   +1 more source

LoRA+: Efficient Low Rank Adaptation of Large Models

International Conference on Machine Learning
In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated
Soufiane Hayou, Nikhil Ghosh, Bin Yu
semanticscholar   +1 more source

The low-rank hypothesis of complex systems

Nature Physics, 2022
Complex systems are high-dimensional nonlinear dynamical systems with heterogeneous interactions among their constituents. To make interpretable predictions about their large-scale behaviour, it is typically assumed that these dynamics can be reduced to ...
Vincent Thibeault, Antoine Allard, Patrick Desrosiers   +2 more
semanticscholar   +1 more source

Low-Rank Multilinear Filtering

Digital Signal Processing
Published by Elsevier Digital Signal Processing. ; International audience ; Linear filtering methods are well-known and have been successfully applied to system identification and equalization problems. However, when high-dimensional systems are modeled, these methods often perform unsatisfactorily due to their slow convergence and to the high number ...
Maryam Dehghan, J. Henrique de M. Goulart, André L.F. de Almeida   +2 more
openaire   +2 more sources

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications

International Conference on Machine Learning
Large language models (LLMs) show inherent brittleness in their safety mechanisms, as evidenced by their susceptibility to jailbreaking and even non-malicious fine-tuning. This study explores this brittleness of safety alignment by leveraging pruning and
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, Peter Henderson   +8 more
semanticscholar   +1 more source

Hyperspectral Image Denoising Using Factor Group Sparsity-Regularized Nonconvex Low-Rank Approximation

IEEE Transactions on Geoscience and Remote Sensing, 2022
Hyperspectral image (HSI) mixed noise removal is a fundamental problem and an important preprocessing step in remote sensing fields. The low-rank approximation-based methods have been verified effective to encode the global spectral correlation for HSI ...
Yong Chen, Tingzhu Huang, Wei He, Xile Zhao, Hongyan Zhang, Jinshan Zeng   +5 more
semanticscholar   +1 more source

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

arXiv.org
Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges.
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, Song Han   +9 more
semanticscholar   +1 more source