Two years ago, running a capable language model on your own machine meant settling for something noticeably worse than what you'd get from a cloud API. That gap has closed. In 2026, a mid-range GPU and a weekend of tinkering will get you a model that handles coding help, document summarization, and research assistance—all without sending a single byte to someone else's server.
Linux sits at the center of this shift. The tooling matured here first—llama.cpp, Ollama, vLLM—and the driver situation for both NVIDIA and AMD hardware is more predictable than it's ever been. If you're comfortable with a terminal, you're already most of the way there.
Three developments converged to make this possible.
First, quantization got good. A 4-bit version of a 70B model now retains roughly 95% of the original's quality while fitting in memory that a high-end consumer card can actually address.
Second, the model ecosystem diversified. Llama 3.1, Mistral, Gemma 2, Phi-3, and Qwen 2.5 all ship with permissive licenses and strong quantized builds.
Third, the runners became boring. Installing Ollama and pulling a model is a two-command operation. Boring is exactly what you want from infrastructure.
This guide covers hardware sizing, the software stack that actually works on Linux, quantization formats explained without hand-waving, a hands-on walkthrough, performance tuning, model recommendations, and the pitfalls that trip up first-timers.
If you want the short version: install Ollama, run ollama run llama3, and come back when you want to go deeper.
Key Takeaway: Local LLMs on Linux are no longer a hobbyist compromise. For many workloads, they're a practical default.
The single most common question is "what do I need?" The answer depends almost entirely on model size and quantization level.
These figures assume 4-bit quantization, which is the sensible default for most people:
| Model size | RAM/VRAM needed | Notes |
|---|---|---|
| 7B–8B | 5–8 GB | Runs on most modern laptops |
| 13B | 10–16 GB | Comfortable on 12GB+ GPUs |
| 30B | 18–24 GB | RTX 4090 territory |
| 70B | 35–48 GB | Multi-GPU or high-VRAM workstation |
For reference: Llama 3 8B in 4-bit quantization needs roughly 5–6 GB of VRAM, Mistral 7B fits in about 6 GB, and a 70B model in 4-bit wants 35–40 GB. An RTX 4090 with 24GB can comfortably run 30B models at 4-bit.
NVIDIA (CUDA) remains the path of least resistance. CUDA support in llama.cpp, Ollama, and vLLM is mature, and driver installation on Ubuntu and Fedora is well-documented.
AMD (ROCm) has improved substantially. RDNA 2 and RDNA 3 cards work with llama.cpp and Ollama, though you'll occasionally need to build from source or track down the right ROCm version for your distro. Expect more friction than NVIDIA, but less than you'd have had in 2023.
Intel (oneAPI/SYCL) covers Arc GPUs and integrated graphics. llama.cpp supports SYCL backends, and for Arc owners it's genuinely usable. Integrated Intel graphics will run small models, albeit slowly.
You can run LLMs on CPU alone. A modern 8-core desktop will generate maybe 5–10 tokens per second on a 7B 4-bit model—usable for chat, painful for long document processing. A laptop will be slower.
CPU inference makes sense when you have plenty of RAM (32GB+), your workload is batch-oriented rather than interactive, or you're running a small model for a specific task. It's also the fallback when your GPU doesn't have enough VRAM and you're doing hybrid CPU/GPU offloading.
Key Takeaway: Match VRAM to model size. A 12GB card runs 13B models well; a 24GB card opens up 30B. Don't buy more GPU than your workload needs.
Ollama is the entry point most people should start with. It handles model downloads, quantization selection, and serving through a clean CLI and REST API. The project has crossed 100,000 GitHub stars, and the ecosystem around it—Python libraries, Open WebUI, LangChain integrations—is substantial.
ollama run llama3
That's it. The model downloads, quantizes if needed, and drops you into a chat prompt.
This is the foundation under much of the ecosystem. llama.cpp supports over 100 model architectures, runs on CPU and GPU (CUDA, ROCm, Metal, SYCL), and is aggressively optimized. If Ollama doesn't support a model or you want fine-grained control over offloading, llama.cpp is the tool.
A web interface that supports multiple backends (llama.cpp, ExLlama, Transformers). It's useful if you want to compare models, manage multiple configurations, and run everything through a browser. Heavier than Ollama, but more flexible.
Download one binary, run it, get a web UI. KoboldCpp is popular for creative writing and roleplay use cases, with strong GGUF support and a built-in interface. No dependency wrangling required.
LM Studio is polished and beginner-friendly, but it's primarily a Windows/macOS product. Linux support exists via AppImage, though native integration remains limited. If you want a GUI-first experience on Linux, text-generation-webui or Open WebUI is usually the better choice.
When you're serving multiple users or running batch jobs, vLLM and Text Generation Inference (TGI) are the tools. They implement paged attention and continuous batching, delivering throughput that single-user runners can't match. This is what you'd deploy on a Linux server.
CUDA dependencies, Python versions, and ROCm builds can conflict. Docker containers—including the official Ollama and vLLM images—let you isolate environments and reproduce setups across machines.
Key Takeaway: Start with Ollama. Move to llama.cpp when you need control, vLLM when you need throughput, and Docker when you need reproducibility.
Quantization reduces the numerical precision of a model's weights. Instead of storing each weight as a 16-bit float, you store it as a 4-bit integer. The model gets smaller and faster, and modern techniques lose surprisingly little quality.
GGUF (GPT-Generated Unified Format) is the successor to GGML and the format llama.cpp and Ollama use. Its key advantage: you can split a model between GPU and CPU. If your GPU has 8GB but the model needs 12GB, GGUF lets you offload some layers to system RAM. Slower, but it runs.
GPTQ is a post-training quantization method producing models that run entirely on GPU. It's faster than GGUF when everything fits in VRAM, but there's no CPU fallback. Popular with ExLlama and vLLM.
Activation-aware Weight Quantization preserves the weights most important for accuracy. In practice, AWQ models often edge out GPTQ on quality at the same bit width, with similar speed.
| Format | Bits | Quality | Use case |
|---|---|---|---|
| FP16 | 16 | Baseline | Reference, fine-tuning |
| Q8_0 | 8 | ~99% | When you have headroom |
| Q5_K_M | 5 | ~97% | Good balance |
| Q4_K_M | 4 | ~95% | Default recommendation |
| Q3_K_M | 3 | ~90% | Tight memory |
| Q2_K | 2 | ~80% | Last resort |
The "K" in Q4_K_M refers to k-quants, a mixed-precision scheme that keeps important layers at higher precision. Q4_K_M is the sweet spot for most people.
The Hugging Face Hub hosts thousands of GGUF and GPTQ models. Search for the base model name plus "GGUF" (e.g., "Llama-3.1-8B-Instruct-GGUF") and pick a reputable uploader—TheBloke, bartowski, and the official model orgs are reliable.
Key Takeaway: Q4_K_M in GGUF format is the right default. It fits most hardware and retains roughly 95% of full-precision quality.
Ubuntu/Debian:
curl -fsSL https://ollama.com/install.sh | sh
Arch (AUR):
yay -S ollama
# or
paru -S ollama
Fedora:
curl -fsSL https://ollama.com/install.sh | sh
The install script handles the systemd service and GPU driver detection. After installation, verify with ollama --version.
ollama run llama3
First run downloads the model (about 4.7GB for the 4-bit build), then opens an interactive prompt. Ask it something. You're now running a local LLM.
To use it as an API:
curl http://localhost:11434/api/generate -d '{
"model": "llama3",
"prompt": "Explain TCP handshakes in two sentences."
}'
Clone and build:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # or -DGGML_HIP=ON for ROCm
cmake --build build --config Release
Download a GGUF model from Hugging Face, then run:
./build/bin/llama-cli -m models/llama-2-7b.Q4_K_M.gguf -p "Hello" -n 128
The -ngl flag controls how many layers go to GPU. Set it high if you have VRAM.
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
./start_linux.sh
The installer sets up a Python environment and dependencies. Once running, open http://localhost:7860 in a browser, pick a model, and load it. The Model tab lets you download GGUF and GPTQ models directly.
Download the latest release binary from the KoboldCpp GitHub releases page, extract it, and run:
./koboldcpp --model models/mistral-7b.Q4_K_M.gguf --port 5001
Open http://localhost:5001 for the web UI. No dependencies, no build step.
Key Takeaway: Ollama gets you running in under five minutes. llama.cpp gives you control. Everything else builds on these foundations.
For llama.cpp, the -ngl (number of GPU layers) flag is the biggest lever. Setting it to 99 offloads everything if VRAM allows. For Ollama, GPU usage is automatic—check ollama ps to confirm the model is on GPU.
For ROCm, set HSA_OVERRIDE_GFX_VERSION if your card isn't officially supported. Community threads for your specific GPU model are the fastest path here.
If you have VRAM headroom, move up from Q4_K_M to Q5_K_M or Q6_K. The quality difference is small but measurable on reasoning-heavy tasks. If you're tight on memory, Q4_K_M is the floor—going below Q3 noticeably degrades coherence.
For throughput, vLLM is the answer. It implements paged attention and continuous batching, and can serve dozens of concurrent requests on a single GPU. Setup:
pip install vllm
python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.3
You get an OpenAI-compatible API endpoint. Point your tools at it.
Key Takeaway: The
-nglflag and quantization level are your two biggest performance levers. Everything else is fine-tuning.
Meta's workhorse. Strong general reasoning, good instruction following, permissive license. The default choice for most people. Runs comfortably in 5–6GB VRAM at 4-bit.
Mistral 7B punches above its weight class, and NeMo 12B (built with NVIDIA) is a strong mid-size option. Both quantize well and have active fine-tune communities.
Google's open model. Excellent at structured tasks and benchmarks well against larger models. Slightly different personality from Llama—worth trying if you find Llama's outputs too bland.
Microsoft's small models (3.8B and 14B variants) are trained heavily on textbook-quality data. Phi-3 Mini runs on almost anything and handles reasoning tasks surprisingly well for its size.
Alibaba's family spans 0.5B to 72B. The mid-size variants (7B, 14B, 32B) are competitive with Llama and Mistral, and the multilingual performance is genuinely better.
Key Takeaway: Llama 3.1 8B is the safe default. Qwen 2.5 and Gemma 2 are worth trying if you want different strengths.
LoRA (Low-Rank Adaptation) lets you fine-tune a model by training a small set of additional weights instead of the full model. QLoRA extends this to quantized base models, cutting VRAM requirements dramatically. Tools like Unsloth and Axolotl make this accessible—a 7B model can be fine-tuned on a single 24GB GPU.
For production-style serving, vLLM and TGI are the standards. Both support continuous batching, which means new requests don't wait for in-flight ones to finish. Throughput improvements over naive serving can be 10x or more.
Retrieval-augmented generation (RAG) pairs a local LLM with a vector database (Chroma, Qdrant, pgvector). The LLM answers questions grounded in your documents. Since everything runs locally, this is viable for sensitive data. LangChain and LlamaIndex both support Ollama and llama.cpp backends.
The official Ollama and vLLM Docker images bundle CUDA dependencies. For a workstation that needs to stay reproducible, containerizing is worth the initial setup:
docker run --gpus all -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama
Key Takeaway: Fine-tuning is accessible via QLoRA on consumer hardware. For serving, vLLM is the throughput king.
CPU-only inference works. It's slow, but for batch jobs or light use, a 32GB RAM machine runs 7B models fine. Hybrid CPU/GPU offloading in llama.cpp extends this further.
A 7B model on an RTX 4090 generates tokens faster than most cloud APIs respond. Network latency disappears. For interactive use, local can feel faster.
Modern 4-bit quantization retains roughly 95% of full-precision quality. For most tasks, you won't notice. The exception is highly precise reasoning or math—there, higher precision helps.
Ollama is two commands. text-generation-webui is a single script. The barrier is genuinely low now.
Hardware costs money, and so does electricity. A 4090 pulling 350W for hours adds up. Local wins on privacy, control, and per-token cost—but the upfront investment is real.
Key Takeaway: The barriers are lower than they were, but local LLMs still involve trade-offs. Know which ones matter for your use case.
Smaller models trained on better data are outperforming larger models from two years ago. Quantization research continues to push the quality floor down. Edge AI—running models on phones and single-board computers—is becoming practical for narrow tasks. And the tooling is consolidating around a few standards (Ollama API, GGUF format, OpenAI-compatible endpoints).
The Hugging Face Hub now hosts hundreds of thousands of quantized models. Subreddits like r/LocalLLaMA and Discord communities around llama.cpp and Ollama are active and helpful. The pace of improvement hasn't slowed.
Key Takeaway: The trend line is clear—local models keep getting better, smaller, and easier to run. Investing time in this ecosystem pays off.
What hardware do I need to run LLMs locally on Linux? Minimum: 16GB RAM for 7B models at 4-bit. Recommended: a GPU with 8–12GB VRAM for comfortable 7B–13B inference. For 70B models, 48GB+ of combined VRAM/RAM.
Which Linux distribution is best for running LLMs? Ubuntu has the best driver support and the most community documentation. Arch and Fedora work well if you're comfortable with those ecosystems. Debian is fine but sometimes lags on newer CUDA versions.
Can I run LLMs on CPU only? Yes. Expect 5–10 tokens per second on a modern 8-core CPU with a 7B 4-bit model. Usable for chat, slow for long-form generation.
What is the difference between GGUF and GPTQ? GGUF supports CPU/GPU hybrid inference and is the format llama.cpp and Ollama use. GPTQ is GPU-only and generally faster when the model fits entirely in VRAM.
How do I install Ollama on Linux?
Run curl -fsSL https://ollama.com/install.sh | sh. The script handles installation and systemd service setup for most distributions.
Is running LLMs locally free? The software is free and open source. You pay for hardware and electricity. If you already have a capable machine, marginal cost is near zero.
Can I fine-tune LLMs locally on Linux? Yes. QLoRA lets you fine-tune 7B models on a single 24GB GPU. Tools like Unsloth and Axolotl simplify the process.
What are the best models to run locally in 2026? Llama 3.1 8B, Mistral 7B, Gemma 2 9B, Phi-3, and Qwen 2.5. Pick based on task and hardware.
How do I speed up local LLM inference?
Use GPU offloading (-ngl in llama.cpp), pick a smaller quantization (Q4_K_M), reduce context length, and consider vLLM for batch workloads.
Is local LLM inference private? Yes, if you disable telemetry and don't route through external services. Everything stays on your machine.
Ready to take control of your AI? Install Ollama today and run your first local LLM in minutes. Share your setup and tips in the comments below.