Fine-Tuning LLMs on a Budget: A Practical Guide

Fine-tuning a large language model used to mean renting a cluster of A100s and burning through a research budget in a weekend. That's no longer true. With parameter-efficient fine-tuning (PEFT) techniques like LoRA and 4-bit quantization, you can adapt a 7B-parameter model on a single consumer GPU.

This post walks through the practical setup I use, the trade-offs that actually matter, and the code to get you from a raw dataset to a fine-tuned model.

Why parameter-efficient fine-tuning?

Full fine-tuning updates every weight in the model. For a 7B model that's 7 billion parameters, each needing gradients and optimizer states, which is easily 80+ GB of memory. LoRA (Low-Rank Adaptation) instead freezes the base model and injects small trainable matrices into each attention layer. You end up training well under 1% of the parameters.

The result is comparable quality on domain-specific tasks, at a fraction of the memory and compute.

Setting up the environment

Start with a clean environment and the core libraries:

pip install transformers peft bitsandbytes accelerate datasets

The key players:

transformers for model loading and the training loop
peft for LoRA and other PEFT methods
bitsandbytes for 4-bit and 8-bit quantization
accelerate for device placement and mixed precision

Loading a quantized base model

Quantization is what makes this fit in consumer memory. Here we load the base model in 4-bit precision:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
 
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto",
)

The nf4 (normalized float 4) quant type is calibrated for the normal distribution of neural network weights, and double quantization shaves off a little more memory by quantizing the quantization constants themselves.

Attaching LoRA adapters

Now wrap the frozen base model with trainable LoRA adapters:

from peft import LoraConfig, get_peft_model
 
lora_config = LoraConfig(
    r=16,                      # rank of the update matrices
    lora_alpha=32,             # scaling factor
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
 
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6.8M || all params: 7.24B || trainable%: 0.09

That last line is the whole point. Only 0.09% of the parameters are trainable.

The trade-offs that actually matter

A few things I've learned the hard way:

Rank (r) is a lever, not a magic number. Higher rank means more capacity but more memory and overfitting risk. Start at 8 to 16 and only go higher if the task genuinely needs it.
Target the right modules. Adapting only q_proj and v_proj is a good default, but for some tasks adding the MLP layers helps more than raising the rank.
Your dataset matters more than your hyperparameters. A few hundred high-quality examples beat tens of thousands of noisy ones.

Wrapping up

PEFT has genuinely democratized model adaptation. The gap between "I have an idea" and "I have a fine-tuned model serving it" is now an afternoon and a single GPU, not a grant proposal.

If you want to go deeper, the PEFT documentation is excellent, and the QLoRA paper is worth a careful read.

Have a question or want to talk shop? Get in touch. I'm always happy to compare notes on applied ML.