Results 251 to 260 of about 14,259,947 (309)

LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

International Conference on Learning Representations, 2023
We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component ...
Han Guo, Philip Greengard, Eric P. Xing, Yoon Kim   +3 more
semanticscholar   +1 more source

DoRA: Weight-Decomposed Low-Rank Adaptation

International Conference on Machine Learning
Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen   +6 more
semanticscholar   +1 more source

Robust Low-Rank Latent Feature Analysis for Spatiotemporal Signal Recovery

IEEE Transactions on Neural Networks and Learning Systems, 2023
Wireless sensor network (WSN) is an emerging and promising developing area in the intelligent sensing field. Due to various factors like sudden sensors breakdown or saving energy by deliberately shutting down partial nodes, there are always massive ...
Di Wu, Zechao Li, Zhikai Yu, Yi He, Xin Luo   +4 more
semanticscholar   +1 more source

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

International Conference on Machine Learning
Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix ...
Jiawei Zhao, Zhenyu (Allen) Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian   +5 more
semanticscholar   +1 more source

Low-Rank Tensor Based Proximity Learning for Multi-View Clustering

IEEE Transactions on Knowledge and Data Engineering, 2023
Graph-oriented multi-view clustering methods have achieved impressive performances by employing relationships and complex structures hidden in multi-view data. However, most of them still suffer from the following two common problems.
Mansheng Chen, Changdong Wang, J. Lai
semanticscholar   +1 more source

Low Rank Solution of Lyapunov Equations

SIAM Journal on Matrix Analysis and Applications, 2002
The Cholesky factor-alternating direction implicit algorithm is presented to compute a low rank approximation to the solution \(X\) of the Lyapunov equation \(AX+XA^T=-BB^T\) with large matrix \(A\) and right hand side of low rank. The algorithm requires only matrix-vector products and linear solvers.
Li, Jing-Rebecca, White, Jacob
openaire   +2 more sources

LoRA+: Efficient Low Rank Adaptation of Large Models

International Conference on Machine Learning
In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated
Soufiane Hayou, Nikhil Ghosh, Bin Yu
semanticscholar   +1 more source

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications

International Conference on Machine Learning
Large language models (LLMs) show inherent brittleness in their safety mechanisms, as evidenced by their susceptibility to jailbreaking and even non-malicious fine-tuning. This study explores this brittleness of safety alignment by leveraging pruning and
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, Peter Henderson   +8 more
semanticscholar   +1 more source

Low-Rank High-Order Tensor Completion With Applications in Visual Data

IEEE Transactions on Image Processing, 2022
Recently, tensor Singular Value Decomposition (t-SVD)-based low-rank tensor completion (LRTC) has achieved unprecedented success in addressing various pattern analysis issues. However, existing studies mostly focus on third-order tensors while order- $d$
Wenjin Qin, Hailin Wang, Feng Zhang, Jianjun Wang, Xin Luo, Tingwen Huang   +5 more
semanticscholar   +1 more source

Low rank tensor deconvolution

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015
In this paper, we propose a low-rank tensor deconvolution problem which seeks multiway replicative patterns and corresponding activating tensors of rank-1. An alternating least squares (ALS) algorithm has been derived for the model to sequentially update loading components and the patterns.
Anh Huy Phan 0001, Petr Tichavský, Andrzej Cichocki   +2 more
openaire   +1 more source