Two years ago, training a large language model meant renting time on an A100 cluster and writing a grant proposal to justify it. Today, a $400 used RTX 3060 can produce a working fine-tune of a 7B model in an afternoon. The techniques that made this possible — LoRA, QLoRA, and a maturing ecosystem of libraries — are no longer exotic research artifacts. They're documented, benchmarked, and packaged into tools you can install with pip.
This guide walks through the entire process: picking hardware, choosing a base model, preparing data, configuring training, and evaluating results. It's written for someone with basic Python skills and a consumer GPU who wants to move from "I've used ChatGPT" to "I have my own fine-tuned model."
Key Takeaway: You don't need a data center. A 24GB consumer GPU can fine-tune a 7B model on a few thousand examples in 1–3 hours using QLoRA.
The single biggest constraint on fine-tuning is VRAM. Everything else — speed, cost, convenience — is secondary to whether your model and training state fit in memory.
NVIDIA RTX 3090 / 4090 (24GB VRAM). These are the workhorses of the local LLM community. The 3090 is often available used for $700–900 and offers 24GB of GDDR6X. The 4090 costs roughly $1,600–2,000 new and delivers around 82 TFLOPS FP32 with faster memory bandwidth. Either can fine-tune 7B and 13B models comfortably with QLoRA.
NVIDIA RTX 3060 (12GB VRAM). The 12GB variant is the budget entry point, often available for under $300. It handles 7B models with QLoRA if you keep batch sizes small and sequence lengths reasonable. Phi-2 (2.7B) trains comfortably.
Apple Silicon (M1/M2/M3, 16–64GB unified memory). Apple's unified memory architecture lets the GPU access system RAM directly. A MacBook Pro with 32GB or 64GB can fine-tune 7B models using MLX or PyTorch with the MPS backend. It's slower than an NVIDIA card with comparable memory, but it works — and the machine is already on your desk.
AMD GPUs. ROCm support exists but remains rougher than CUDA. If you already own a 7900 XTX (24GB), it's worth trying, but expect more debugging.
Here's the practical picture for QLoRA fine-tuning (4-bit base model, LoRA adapters in bf16):
| Model size | Minimum VRAM | Comfortable VRAM |
|---|---|---|
| 2.7B (Phi-2) | 6GB | 8–12GB |
| 7B (Mistral, Llama 3) | 10GB | 16–24GB |
| 13B | 16GB | 24GB |
| 34B | 24GB (tight) | 48GB |
These numbers assume a sequence length of around 512–1024 tokens and a batch size of 1–4 with gradient accumulation. Longer sequences and larger batches scale memory roughly linearly.
If your local hardware falls short, three options cover most cases:
Key Takeaway: 12GB VRAM is the realistic floor for 7B QLoRA. Below that, drop to a 2.7B model like Phi-2 or use cloud GPUs.
The reason any of this works on consumer hardware is that we stopped updating all the model's weights.
Full fine-tuning of a 7B model updates all 7 billion parameters. That requires storing gradients and optimizer states for every one of them — roughly 80–100GB of memory in fp32. Not happening on a 3090.
LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2021, takes a different approach. It freezes the original weights and injects small trainable matrices into each transformer layer. Instead of updating a 4096×4096 weight matrix, you train two matrices of shape 4096×r and r×4096, where r (the "rank") is typically 8, 16, or 32.
The original LoRA paper reported a 10,000x reduction in trainable parameters and a 3x reduction in GPU memory for GPT-3 175B, with performance matching full fine-tuning on several benchmarks.
In practice, for a 7B model with rank 16, you're training about 0.1–0.5% of the parameters. The optimizer state shrinks accordingly.
LoRA alone still requires the base model in fp16, which for 7B is ~14GB just for weights. QLoRA, from Dettmers et al. (2023), quantizes the frozen base model to 4-bit using a format called NormalFloat4 (NF4), then trains LoRA adapters in bf16.
The result: a 7B model's weights drop to about 4GB. Add LoRA adapters, activations, and optimizer state, and you land around 10–16GB total. The paper demonstrated fine-tuning a 65B model on a single 48GB GPU while matching 16-bit fine-tuning performance.
Important distinction: QLoRA is not quantization-aware training. The base model stays quantized and frozen; only the adapters train in full precision. You're not teaching the quantized weights anything — you're learning a correction on top of them.
LoRA and QLoRA dominate, but a few alternatives exist:
For most projects, start with LoRA or QLoRA. Reach for alternatives only if you have a specific reason.
Three techniques stack on top of QLoRA to fit training into limited VRAM:
Gradient checkpointing — instead of storing all intermediate activations for backprop, recompute them during the backward pass. Trades ~20% more compute for a large memory reduction. Enable with gradient_checkpointing=True.
Gradient accumulation — run several forward/backward passes before updating weights. Simulates a larger batch size without the memory cost. If you want an effective batch of 16 but can only fit 2, use gradient_accumulation_steps=8.
Mixed precision — train in bf16 or fp16 instead of fp32. Halves activation memory and speeds up math on modern GPUs. bf16 is preferred on Ampere and newer (RTX 30/40 series) because it has a wider dynamic range.
Key Takeaway: QLoRA quantizes the base model to 4-bit and trains small LoRA adapters in bf16. Combined with gradient checkpointing and accumulation, a 7B model fits in 10–16GB.
The model you start from determines your ceiling. For single-GPU work in 2024, the strongest options are:
If you're new to this, start with Mistral 7B. The license is permissive, the tooling is mature, and there's no shortage of tutorials and community fine-tunes to compare against.
Dataset quality matters more than size. A few hundred well-curated examples often beat tens of thousands of scraped ones.
Format. Most fine-tuning uses instruction-response pairs. The common structure is:
{
"instruction": "Summarize this customer complaint in one sentence.",
"input": "I've been waiting three weeks for my order...",
"output": "Customer reports a three-week shipping delay without resolution."
}
Many modern models use chat templates instead. For Mistral and Llama 3, use the model's built-in chat template via the tokenizer.
Size. For consumer-GPU scenarios, 500–5,000 examples is typical. Below 500, you risk overfitting. Above 10,000, training time grows and returns diminish unless your task is genuinely diverse.
Quality checks. Deduplicate. Remove examples with truncated or malformed outputs. Check for label noise — if 5% of your outputs are wrong, the model learns the wrong pattern. For a deeper look, tools like cleanlab can flag suspect examples.
Splitting. Hold out 5–10% as a validation set. You'll need it to detect overfitting.
The standard stack:
pip install transformers datasets peft trl bitsandbytes accelerate
For faster training, add Unsloth (covered in the next section):
pip install unsloth
bitsandbytes provides the 4-bit quantization. peft implements LoRA. trl handles the supervised fine-tuning trainer. accelerate manages device placement.
If you're on Apple Silicon, bitsandbytes support is limited. Use MLX instead, or fall back to fp16 LoRA without quantization.
Here's a working starting point for 7B QLoRA on a 24GB GPU:
from transformers import TrainingArguments
training_args = TrainingArguments(
output_dir="./results",
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # effective batch = 16
num_train_epochs=3,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
bf16=True,
gradient_checkpointing=True,
logging_steps=10,
save_strategy="epoch",
evaluation_strategy="epoch",
optim="paged_adamw_8bit", # memory-efficient optimizer
)
And the LoRA configuration:
from peft import LoraConfig
lora_config = LoraConfig(
r=16, # rank
lora_alpha=32, # scaling factor, usually 2*r
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
Learning rate. QLoRA tolerates higher rates than full fine-tuning because fewer parameters are being updated. 2e-4 is a common default. Drop to 1e-4 if training loss oscillates.
LoRA rank. Use rank 8 for simple style transfer, 16 for general instruction tuning, and 32–64 for tasks requiring new knowledge. Higher rank means more parameters and more memory.
Target modules. Applying LoRA to all attention projections (q, k, v, o) is standard. Adding MLP layers (gate_proj, up_proj, down_proj) improves quality at the cost of more trainable parameters.
A minimal training script:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-v0.1",
quantization_config=bnb_config,
device_map="auto",
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
)
trainer.train()
What to watch. Training loss should decrease steadily. Validation loss should decrease, then flatten. If validation loss starts rising while training loss keeps falling, you're overfitting — stop or reduce epochs.
If you see loss spiking or NaN, check your learning rate and make sure bf16 is enabled (fp16 with QLoRA can overflow on some setups).
Key Takeaway: Start with rank 16, learning rate 2e-4, 3 epochs, and effective batch size 16. Adjust from there based on loss curves.
Unsloth is a library that rewrites the attention and MLP kernels for Llama and Mistral architectures using hand-optimized Triton code. According to the project's benchmarks, the result is 2–5x faster training and 50–80% less memory usage compared to standard Hugging Face QLoRA.
The speedup comes from two sources: eliminating the need for gradient checkpointing (by recomputing activations more efficiently) and using faster attention implementations. Memory savings come from more aggressive quantization and better activation management.
pip install unsloth
The API mirrors Hugging Face closely:
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/mistral-7b-v0.2",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=32,
lora_dropout=0.05,
)
Then train with the standard SFTTrainer. Unsloth patches the model in place.
When to use it. If you're training Mistral, Llama, or Gemma, use Unsloth. It's a drop-in replacement with no quality tradeoff. If you're training an architecture it doesn't support (Phi-2, Qwen), fall back to standard Hugging Face.
Training loss tells you the model is learning something. It doesn't tell you the model is useful.
Perplexity — measures how well the model predicts held-out text. Lower is better. Useful as a sanity check but doesn't capture task performance. A model can have low perplexity and still give wrong answers.
Task-specific accuracy — if your task has a right answer (classification, extraction, structured output), measure exact match or F1 on a held-out test set. This is the most informative metric.
Human evaluation — for generation tasks (writing, dialogue), have a person rate outputs on relevance, coherence, and correctness. Slow but irreplaceable.
EleutherAI's lm-evaluation-harness runs standardized benchmarks (MMLU, HellaSwag, GSM8K, etc.) against any Hugging Face model:
pip install lm-eval
lm_eval --model hf \
--model_args pretrained=./my-finetuned-model \
--tasks mmlu,hellaswag \
--batch_size 8
Run this on both the base model and your fine-tune. If your fine-tune regresses on general benchmarks while improving on your target task, that's expected — you've specialized the model. If it regresses everywhere, something went wrong.
With small datasets, overfitting is the default outcome, not an edge case. Three defenses:
lora_dropout=0.05 to 0.1. Higher dropout for smaller datasets.Key Takeaway: Evaluate on a held-out set with task-specific metrics, not just training loss. If validation loss rises, stop.
A small SaaS company fine-tunes Mistral 7B on 2,000 historical support tickets, formatted as instruction-response pairs. They use QLoRA with rank 16, batch size 2 with gradient accumulation 8, and train for 2 epochs. Total time: about 2 hours on a used 3090. The resulting model handles 70% of common queries without escalation, up from 40% with the base model prompted carefully.
A researcher fine-tunes Llama 3 8B on a curated set of 5,000 medical question-answer pairs from peer-reviewed sources. Using 4-bit quantization, gradient checkpointing, and sequence length 1024, training fits in 24GB. Three epochs take roughly 4 hours. The model improves on domain-specific accuracy benchmarks but must be paired with disclaimers — fine-tuning doesn't make a model clinically reliable.
An indie developer uses a MacBook Pro M2 Max (64GB) with MLX to fine-tune Mistral 7B on 800 examples of their own writing. Training takes about 6 hours — slower than an NVIDIA card, but no hardware purchase required. The resulting model mimics their tone well enough for first drafts.
A solo developer fine-tunes Phi-2 (2.7B) on 500 examples for a niche chatbot. With rank 8, batch size 4, and 3 epochs, training completes in under an hour on a 12GB RTX 3060. The small model is fast enough to run locally at interactive speeds.
Open-weight does not mean unrestricted. Every model comes with a license that governs what you can do with derivatives.
Apache 2.0 (Mistral 7B, Qwen 2.5) — permissive. Commercial use allowed, attribution required, no restrictions on derivatives.
MIT (Phi-2) — similarly permissive.
Llama 2 / Llama 3 Community License — allows commercial use below 700 million monthly active users. Above that threshold, you need a separate agreement with Meta. Derivatives must carry the same license and include attribution.
Gemma Terms of Use — permits commercial use with use-case restrictions (no weapons, no deceptive content, etc.).
Non-commercial licenses — some research models (older LLaMA variants, certain academic releases) prohibit commercial use entirely. Check before building a product.
What this means for fine-tunes. Your fine-tuned model inherits the base model's license. If you fine-tune Llama 3, you must comply with Meta's terms — including the naming convention (your model must include "Llama" in the name) and the acceptable use policy. If you fine-tune Mistral 7B, you're bound by Apache 2.0, which is far simpler.
Compliance checklist: - Read the base model's license before training. - Include required attribution in your model card. - Don't use restricted models for prohibited applications. - If you plan to commercialize, prefer Apache 2.0 or MIT base models.
Three trends are converging to make this easier and cheaper.
Smaller models, better quality. Phi-2 at 2.7B outperforms older 7B models on many benchmarks. Gemma 2 2B is competitive with first-generation 7B models. As small models improve, the hardware bar drops.
Quantization advances. 4-bit NF4 is now standard. 2-bit and 3-bit methods (like AQLM and QuIP#) are maturing. If they become practical for training, 13B models will fit on 8GB cards.
Tooling consolidation. A year ago, fine-tuning required stitching together five libraries and debugging version conflicts. Today, Unsloth, Axolotl, and LLaMA-Factory offer end-to-end pipelines with config files. Expect this to keep improving.
What to expect in the next year: 3B models that match today's 7B, training libraries that handle more of the configuration automatically, and broader support for non-NVIDIA hardware. The gap between "I have a gaming PC" and "I can fine-tune a useful model" is closing.
Can I fine-tune a 7B parameter LLM on a GPU with 8GB VRAM? Not comfortably. QLoRA for a 7B model typically needs 10–16GB. You can squeeze it down with very small batch sizes, short sequences, and aggressive gradient checkpointing, but training will be slow and unstable. Use Phi-2 (2.7B) or a cloud GPU instead.
What is the minimum VRAM required for fine-tuning a 7B model? About 10GB with QLoRA, batch size 1, sequence length 512, and gradient checkpointing. 16GB gives you room to breathe. 24GB lets you use larger batches and longer sequences.
How long does fine-tuning take on a single consumer GPU? Roughly 1–3 hours for a 7B model on 1,000 examples with an RTX 3090 or 4090. Multiply by 2–3x for a 3060. Apple Silicon is slower — expect 4–8 hours for the same workload.
Do I need an NVIDIA GPU, or can I use AMD or Apple Silicon? NVIDIA is the path of least resistance. Apple Silicon works via MLX or MPS, with some library limitations. AMD via ROCm is possible but expect more debugging and fewer pre-built wheels.
What datasets can I use for fine-tuning? Any instruction-response dataset that matches your task. Public options include Alpaca, Dolly, OpenAssistant, and thousands of task-specific datasets on Hugging Face. For best results, curate your own from domain data — a few hundred high-quality examples often beat large generic datasets.
Is full fine-tuning possible on a consumer GPU? For small models (under 1B), maybe. For 7B, no — full fine-tuning needs 80–100GB of optimizer and gradient memory. PEFT methods are the only realistic path on consumer hardware.
What are the best open-weight models for single-GPU fine-tuning? Mistral 7B, Llama 3 8B, Qwen 2.5 7B, Gemma 2 7B, and Phi-2 (2.7B). Mistral and Qwen have the most permissive licenses; Llama 3 has the strongest ecosystem.
How do I evaluate my fine-tuned model?
Use a held-out test set with task-specific metrics (accuracy, F1, exact match). For generation tasks, human evaluation. Run lm-evaluation-harness to check for regressions on general benchmarks.
Can I merge LoRA adapters back into the base model?
Yes. peft provides a merge_and_unload() method that folds the LoRA weights into the base model, producing a standalone model you can load without PEFT. This is useful for deployment.
What are the licensing implications of fine-tuning open-weight models? Your fine-tune inherits the base model's license. Apache 2.0 and MIT models are safe for commercial use. Llama models have a community license with a 700M MAU threshold. Always check the base model's terms before commercializing.
Ready to fine-tune your own LLM? Start by picking a model and dataset, then follow the step-by-step guide above. Share your results and join the community discussion — the r/LocalLLaMA subreddit and Hugging Face forums are full of people who've done exactly this on hardware like yours.