Results 11 to 20 of about 9,649,421 (248)

LoRA+: Efficient Low Rank Adaptation of Large Models [PDF]

CoRR
27 ...
Soufiane Hayou, Nikhil Ghosh, Bin Yu 0001   +2 more
core   +5 more sources

Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation [PDF]

CoRR
64 Pages, 9 Algorithms, 22 Theorems, 10 Lemmas, 2 Figures, 3 ...
Igor Sokolov 0001, Abdurakhmon Sadiev, Yury Demidovich, Fawaz S. Al-Qahtani, Peter Richtárik   +4 more
openaire   +3 more sources

PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation

CoRR
Accepted at T4V ...
Injoon Hwang, Haewon Park, Youngwan Lee, Jooyoung Yang, SunJae Maeng   +4 more
openaire   +4 more sources

Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [PDF]

CoRR
36 pages, 4 figures, 2 ...
Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, Peter Richtárik   +6 more
core   +7 more sources

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices [PDF]

, 2023
In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices ...
Wang, Jianan, Wong, Kam-Fai, Qi, Xianbiao, Wang, Lingzhi, Zhang, Lei, Zi, Bojia   +5 more
core   +1 more source

Sparse Low-rank Adaptation of Pre-trained Language Models [PDF]

, 2023
Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is
Lv, Xingtai, Ding, Ning, Liu, Zhiyuan, Zhou, Bowen, Sun, Maosong, Wang, Qiaosen, Chen, Yulin   +6 more
core   +1 more source

TDG-LoRA: Token-Level Dynamic Gating for Mitigating Catastrophic Forgetting

IEEE Access
Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), is widely used to adapt large language models (LLMs) to specialized downstream domains.
Shushan Zhu, Shangqing Huang, Xingyuan Chen   +2 more
doaj   +1 more source

Null-LoRA: Low-Rank Adaptation on Null Space

CoRR
Parameter-efficient fine-tuning methods have gained considerable popularity for adapting large-scale models to downstream tasks, particularly LoRA and its variants. Existing methods perform low-rank adaptation over the full parameter space. However, fine-tuning within a subspace can achieve comparable effectiveness. Inspired by the observation that pre-
Yi Zhang, Yulei Kang, Haoxuan Chen, Jinxuan Li, Jian-Fang Hu   +4 more
openaire   +2 more sources

A Study on Text Classification in the Age of Large Language Models

Machine Learning and Knowledge Extraction
Large language models (LLMs) have recently made significant advances, excelling in tasks like question answering, summarization, and machine translation.
Paul Trust, Rosane Minghim
doaj   +1 more source

Activation-Guided Low-Rank Parameter Adaptation for Efficient Model Fine-Tuning

IEEE Access
Fine-tuning large language models is computationally expensive, and while existing parameter-efficient methods like Low-Rank Adaptation (LoRA) reduce computational costs, they are limited by suboptimal initialization strategies.
Qingchen Wang, Shengyu Shen
doaj   +1 more source