LoRA: Low-Rank Adaptation of Large Language Models

Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

Published June 17, 2021

Read the paper: arXiv 2106.09685 · PDF

Why it matters

LoRA turned fine-tuning from a data-center exercise into something individual developers can do on a 3080. Combined with 4-bit quantization (QLoRA), it's the practical ceiling for hobbyist / small-team model customization. The entire Hugging Face PEFT library is organized around this idea.

If you're considering fine-tuning in 2026, your default is LoRA, not full fine-tuning. Full fine-tuning makes sense only for infra teams at scale or when adapter-based approaches aren't enough.

How to use it

When prompt engineering plateaus on a narrow task with lots of examples, LoRA-train an adapter on 5-10k labeled examples. Swap in at inference time via PEFT. Many concurrent LoRAs can coexist on one base model -- a nice multi-tenant story.

Abstract

We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.