QLoRA Fine-Tuning in Python: Train a Custom LLM on One GPU (2026)

You have a task where a general model keeps missing: it answers in the wrong tone, ignores your internal terminology, or produces JSON that is almost your schema. You have a few thousand examples of what correct looks like. Full fine-tuning would fix it, but a 7B model in 16-bit needs roughly 140GB of GPU memory for weights, gradients, and optimizer state — that is a multi-GPU cluster, not your workstation.

QLoRA collapses that requirement to something a gaming GPU can handle. The base model stays frozen and quantized to 4 bits; you train only a small set of low-rank adapter weights in 16-bit. This guide walks through the whole path: loading a 4-bit base, formatting data, choosing LoRA hyperparameters, training with TRL, merging the adapter, and exporting to GGUF so you can run it in Ollama.

1. When Fine-Tuning Is Actually Worth It

Fine-tuning is the most over-reached-for tool in applied LLM work. Try these first, in order:

• Prompting with few-shot examples. Costs nothing, takes an afternoon. Fixes most tone and formatting issues.
• Retrieval. If the model lacks facts — your product catalog, last quarter's tickets, a policy document — fine-tuning is the wrong fix. It will memorize a fraction of it and hallucinate the rest with more confidence. Build retrieval instead; our LlamaIndex RAG guide covers it.
• Fine-tuning. Use it when you need consistent behavior: a fixed output shape, a classification decision boundary, a domain-specific style, or a small model that must match a larger one on one narrow task.

The clearest signal that fine-tuning will win: you can write down the correct output for a given input, and a human would agree your examples are all correct. If you cannot define correctness, you do not have training data — you have opinions, and no amount of training will converge on them.

A practical threshold: below roughly 500 high-quality examples, few-shot prompting usually beats a fine-tune. Between 1,000 and 10,000 well-formed examples, QLoRA reliably produces something noticeably better than prompting on a narrow task.

2. LoRA vs QLoRA vs Full Fine-Tuning

LoRA works by freezing the base weight matrix W and learning a low-rank decomposition ΔW = B·A alongside it, where A is r × k and B is d × r with r far smaller than either dimension. Instead of updating 4096×4096 values you update two thin matrices. QLoRA adds a 4-bit NormalFloat quantization of the frozen base on top of that, plus a paged optimizer that spills to host RAM during memory spikes.

The quality gap between LoRA and QLoRA is small on most tasks. The gap between QLoRA and full fine-tuning is real but narrower than the VRAM difference suggests — start with QLoRA and only escalate if you can measure that you need more.

3. What You Need on Your Machine

Rough working numbers for QLoRA on a 7B model with 4-bit base, gradient checkpointing, and 2048-token sequences:

• 16 GB VRAM (RTX 4080/4090, 5070 Ti/5080) — comfortable. Room for batch size 4–8 with packing.
• 12 GB VRAM (RTX 3060/4070) — workable with batch size 1–2, gradient accumulation 8–16, shorter sequences. Slower but it finishes.
• 8 GB VRAM — tight. Drop to a 3B model or use Unsloth; expect to fight with sequence length.
• Apple Silicon — note that bitsandbytes 4-bit quantization is CUDA-only. On M-series Macs, use full-precision LoRA through the MPS backend, or use the MLX ecosystem which has its own LoRA tooling. Do not expect the code below to run unchanged on a Mac.
• CPU only — technically possible, practically not. Rent a GPU.

Also budget disk: a 7B model in 4-bit is around 4 GB, but the HF cache, checkpoints, and merged 16-bit output together want 40–60 GB of free space. Running out mid-merge is a genuinely annoying way to lose an hour.

4. Load a 4-Bit Base Model

Install the stack. Pin your versions — this ecosystem breaks across minor releases more than most:

pip install "torch==2.5.*" --index-url https://download.pytorch.org/whl/cu121
pip install "transformers==4.46.*" "peft==0.14.*" "trl==0.13.*" \
            "bitsandbytes==0.45.*" "accelerate==1.1.*" datasets

Now load the base in 4-bit. The configuration object is where most broken setups go wrong:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"   # or mistralai/Mistral-7B-Instruct-v0.3

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # nf4 beats fp4 for normally-distributed weights
    bnb_4bit_use_double_quant=True,       # quantize the quantization constants: ~0.4GB saved
    bnb_4bit_compute_dtype=torch.bfloat16 # compute dtype, NOT storage dtype
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb,
    device_map="auto",
    attn_implementation="sdpa",           # or "flash_attention_2" if installed
)
model.config.use_cache = False            # must be False during training
model = prepare_model_for_kbit_training(model)   # from peft

tok = AutoTokenizer.from_pretrained(MODEL_ID)
tok.pad_token = tok.eos_token             # Llama ships without a pad token
tok.padding_side = "right"                # right padding for training

Two lines here cause the most confusion. compute_dtype is the dtype used for the forward pass, not the storage precision — keep it bfloat16 on Ampere and newer, fall back to float16 on older cards. And prepare_model_for_kbit_training is not optional: it enables gradient computation on the quantized layers, casts LayerNorms to fp32 for stability, and without it you get either an error or silently frozen weights.

Many base models also ship with use_cache=True for fast generation. During training that cache conflicts with gradient checkpointing and produces a warning that people ignore and then debug for two hours. Set it to False. You turn it back on for inference.

5. Formatting Your Dataset

Dataset quality dominates every hyperparameter in this guide. A thousand clean, diverse, correctly-formatted examples beat ten thousand noisy ones.

For instruction tuning, the simplest robust route is a JSONL file with prompt/completion pairs, which TRL's SFTTrainer understands natively:

{"prompt": "Classify the support ticket.\n\nTicket: I was charged twice for the Pro plan.\n\nCategory:", "completion": "billing"}
{"prompt": "Classify the support ticket.\n\nTicket: The app crashes when I export to PDF.\n\nCategory:", "completion": "bug"}

from datasets import load_dataset

raw = load_dataset("json", data_files={"train": "tickets.jsonl"})["train"]
raw = raw.train_test_split(test_size=0.1, seed=42)   # always hold out
print(len(raw["train"]), len(raw["test"]))

Three formatting rules that matter more than they look:

• Match your inference prompt exactly. If you train on "Classify: {text}" but serve "Please classify the following: {text}", you have trained the model on a distribution it will never see. Write the serving prompt first, generate the dataset from it.
• Use the model's chat template for chat tasks. For conversational fine-tuning, apply tok.apply_chat_template() rather than hand-rolling delimiters, or you will train on tokens that differ from what the tokenizer expects at inference.
• Include the stop condition. For completions that should end, make sure your EOS token is appended. Otherwise the fine-tuned model generates your answer and then keeps going into a second invented question.

If your target is structured output, it is worth reading our structured LLM output guide first — combining a fine-tune with constrained decoding gives you reliability either method alone will not reach.

6. LoRA Parameters That Matter

from peft import LoraConfig, get_peft_model

lora = LoraConfig(
    r=16,                     # rank: capacity of the adapter
    lora_alpha=32,            # scaling; alpha/r is the effective learning-rate multiplier
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[          # attention + MLP projections
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5195

What each knob actually does in practice:

• r (rank) — start at 16. Raise to 32 or 64 if the model underfits (training loss plateaus high and outputs stay generic); lower to 8 if you have few examples and it overfits fast. Larger rank is not free: it grows the adapter and increases overfitting risk.
• lora_alpha — the convention alpha = 2 × r is a good default. What matters is the ratio; many practitioners fix alpha at 32 and tune r only, which keeps the effective scaling interpretable.
• target_modules — including the MLP projections (gate_proj, up_proj, down_proj) alongside attention noticeably improves results on tasks requiring new knowledge or format changes. Attention-only is cheaper and sometimes enough for style adaptation.
• lora_dropout — 0.05 is a mild regularizer. Raise toward 0.1 if you have under ~2,000 examples.

7. Training with TRL SFTTrainer

from trl import SFTTrainer, SFTConfig
from transformers import TrainerCallback

args = SFTConfig(
    output_dir="./out-qlora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,     # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_seq_length=2048,
    packing=True,                      # concatenate short examples: big throughput win
    bf16=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="paged_adamw_8bit",          # falls back to "adamw_torch_fused" if unsupported
    logging_steps=10,
    save_strategy="epoch",
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=raw["train"],
    eval_dataset=raw["test"],
    processing_class=tok,
)
trainer.train()

Notes on the choices that are not obvious:

• packing=True concatenates multiple short examples into one sequence instead of padding each to max_seq_length. For short prompt/completion pairs this can be a 3–5× speedup. Disable it for chat-style data where examples must not bleed into each other, or use packing with position-ids-aware implementations.
• use_reentrant=False is required for gradient checkpointing to work with PEFT in current versions. Skipping it gives you a confusing error about tensors not requiring grad.
• Learning rate 2e-4 is the standard QLoRA starting point, roughly an order of magnitude above full fine-tuning because you are training far fewer parameters. If loss oscillates wildly, halve it.
• Epochs: 2–3 for 1k–10k examples. Watch the eval loss, not the training loss. When eval loss flattens or rises while training loss keeps dropping, you are done — take the earlier checkpoint.

On an RTX 4090, 3,000 examples at 2048 tokens with packing runs in roughly 20–40 minutes. If your first epoch takes three hours, check that bf16 is actually on and that you did not accidentally load the model in 8-bit.

For roughly 2× faster training with lower memory, Unsloth is a drop-in alternative that fuses the LoRA kernels. It is worth adopting once your pipeline works, not before — debugging two abstractions at once is slower.

8. Testing and Merging the Adapter

Test before you merge. Merging is not easily reversible, and a bad adapter is much cheaper to discard than a bad 16 GB model:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, quantization_config=bnb, device_map="auto"
)
ft = PeftModel.from_pretrained(base, "./out-qlora/checkpoint-XXX")

tok = AutoTokenizer.from_pretrained(MODEL_ID)
prompt = "Classify the support ticket.\n\nTicket: My invoice says 2025 but I paid in 2026.\n\nCategory:"
ids = tok(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    out = ft.generate(**ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Run the same prompt through the untouched base and compare. If you cannot tell the difference, the fine-tune did not take — usually a learning rate that was too low, target_modules that matched nothing, or a dataset where the completion field was empty.

When you are happy, merge for deployment. Merge in 16-bit on CPU or GPU with enough headroom:

# Merge in fp16/bf16, NOT 4-bit — merging into a quantized base degrades output
base = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto"
)
merged = PeftModel.from_pretrained(base, "./out-qlora/checkpoint-XXX").merge_and_unload()
merged.config.use_cache = True          # re-enable for generation
merged.save_pretrained("./merged-7b", safe_serialization=True)
tok.save_pretrained("./merged-7b")

The adapter alone is only 50–200 MB, which is the real operational advantage of this approach: you can ship ten task-specific adapters alongside one shared base and swap them at load time in milliseconds.

9. Exporting to GGUF for Ollama

To run the result locally through Ollama or llama.cpp, convert the merged model to GGUF and quantize it:

git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
pip install -r requirements.txt

python convert_hf_to_gguf.py ../merged-7b --outfile model-f16.gguf --outtype f16
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

cat > Modelfile <<'EOF'
FROM ./model-Q4_K_M.gguf
PARAMETER temperature 0.2
PARAMETER stop "<|eot_id|>"
TEMPLATE """{{ .Prompt }}"""
EOF

ollama create my-ticket-classifier -f Modelfile
ollama run my-ticket-classifier

Use Q4_K_M as the default quantization — it is the usual quality/size sweet spot. Q5_K_M if you have the RAM and the task is sensitive to small wording differences; Q3_K_M only if you are genuinely constrained. Background on running local models is in our Ollama local LLM guide, and if you want a chat UI over it, see the Open WebUI setup walkthrough.

10. Common Mistakes

Mistake 1: Fine-Tuning When You Needed Retrieval

If the model is missing facts, fine-tuning teaches it to guess facts confidently. That is worse than "I don't know." Facts go in a vector store; behavior goes in a fine-tune.

Mistake 2: No Held-Out Set

Training loss always decreases. Without an eval split you have no signal for overfitting, and you will ship an adapter that memorized your examples and generalized to nothing. Split before you train, even if it is only 100 examples.

Mistake 3: Mismatched Prompts Between Train and Serve

The single most common cause of "it worked in the notebook." Your serving prompt must byte-match the training prompt, including the trailing newline in the completion separator. Template the prompt string once and import it in both places.

Mistake 4: Training Too Long on Small Data

With 800 examples and 5 epochs you will get a model that reproduces training examples verbatim. Two or three epochs max, and pick the checkpoint by eval loss — often the first one.

Mistake 5: Merging Into the 4-Bit Base

Merge in bf16/fp16 from a fresh full-precision load. Merging into the quantized model you trained against produces a degraded result that is hard to diagnose because it still runs.

Mistake 6: Forgetting use_cache = False

Generation caching and gradient checkpointing do not coexist. Set it to False before training, back to True before deployment.

Mistake 7: No Versioning of Data and Config

An adapter file with no record of which dataset, base model, and hyperparameters produced it is useless in three months. Save the config, the dataset hash, and the eval scores next to the adapter. Treat it like a model artifact, because it is one.

Bottom Line

QLoRA turned fine-tuning from a cluster project into a Saturday afternoon. A 7B model, 4-bit base, a 40 MB adapter, and a few thousand clean examples gets you a model that behaves consistently on one narrow task — which is what most real applications actually need.

Work in this order: get the base model loading in 4-bit and generating (30 minutes). Build 500–1,000 examples and split off a test set before writing any training code (this is the part that determines your result). Train at r=16, 2–3 epochs, lr 2e-4. Evaluate against the base model on held-out prompts. Merge only after the comparison shows a real improvement, then export to GGUF and run it locally.