WP Copilot
Search… /
Running Local Models

vLLM and llama.cpp as Alternatives

⏱ 14 min

Ollama is the easiest way to get a local model running, and for a single-site, moderate-traffic WordPress feature it’s often the right choice to stop looking further. But it’s not the only option, and it’s not always the best-suited one. vLLM and llama.cpp cover the two ends Ollama sits between: vLLM for serving real production throughput on GPU hardware, llama.cpp for running on modest or CPU-only hardware with a minimal footprint. Both expose the same OpenAI-compatible shape this course’s custom provider already knows how to speak.

What you'll learn in this lesson
When vLLM is the right choice over Ollama
Higher-throughput GPU serving, continuous batching, and production inference workloads.
When llama.cpp is the right choice over Ollama
CPU-only or resource-constrained hardware, quantized models, minimal dependencies.
Real setup commands for both
vllm serve and llama-server, with their default ports and OpenAI-compatible routes.
Why the custom provider from Lesson 4 doesn't care which one you pick
All three speak the same request shape, only the base URL changes.
Prerequisites

Lesson 2’s Ollama setup, so you have a working comparison point. This lesson doesn’t require switching away from Ollama, only understanding when you’d want to.

Step 1: When vLLM is the better fit

vLLM is built for serving LLMs at real production throughput on GPU hardware. Its scheduler batches concurrent requests far more efficiently than a general-purpose tool like Ollama, which matters once a self-hosted feature has to handle multiple simultaneous requests rather than a single developer testing from a terminal. If your self-hosting motivation includes serving genuine traffic volume, not just a low-traffic internal tool, vLLM is worth the extra setup complexity over Ollama.

# File: terminal
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000

The OpenAI-compatible endpoint is then reachable at http://localhost:8000/v1, same route shapes as Ollama: /v1/chat/completions, /v1/models.

vLLM's --api-key flag is a real security control, unlike Ollama's

Unlike Ollama, vLLM’s server actually validates an Authorization: Bearer header if you start it with --api-key <your-key> (or set the VLLM_API_KEY environment variable). That makes the API-key authentication the PHP AI Client SDK always sends genuinely meaningful for vLLM, not just an ignored formality. Still don’t treat it as your only protection, vLLM’s documentation is explicit that other endpoints on the same server can remain unauthenticated, so pair it with network-level restrictions for anything beyond a local development machine.

Step 2: When llama.cpp is the better fit

llama.cpp (and its bundled llama-server) is the right choice when the hardware is the constraint: no GPU, limited RAM, an edge device, or simply a preference for a minimal, dependency-light C++ inference engine over a full Python serving stack. It runs quantized GGUF-format models, trading some precision for a dramatically smaller memory footprint, which is often the difference between a model running at all on modest hardware and not running.

# File: terminal
llama-server -m models/llama-3.1-8b-instruct.Q4_K_M.gguf --port 8080 --api-key your-local-key

llama-server listens on port 8080 by default and exposes a fully OpenAI-compatible /v1/chat/completions endpoint, including support for streaming and tool calling. Its --api-key flag works the same way as vLLM’s: real validation, not Ollama’s ignored header, and you can supply multiple valid keys or load them from a file with --api-key-file for anything beyond a single-developer setup.

Step 3: The comparison that actually matters

Choosing between the three
Ollama
Easiest model management and setup, best for development and low-to-moderate traffic. No real request authentication.
vLLM
Best raw throughput on GPU hardware via continuous batching, the right choice for genuine production request volume. Real, optional API-key authentication.
llama.cpp
Best for CPU-only or resource-constrained hardware and quantized models, minimal footprint. Real, optional API-key authentication.

None of this changes what Lesson 4 builds. Every one of these three tools answers /v1/chat/completions in the same OpenAI-compatible shape, so the custom provider class this course builds only needs its base URL (and, for vLLM and llama.cpp, a real API key) changed to point at whichever one you’re running.

Don't switch tools mid-project without re-testing latency and output quality

The three tools can serve the exact same model weights and still behave differently under load, quantization settings, batching configuration, and context length limits all affect real-world latency and, for quantized llama.cpp models specifically, output quality. Treat a switch between them as a change worth re-testing, not a drop-in swap.

Test it: confirm whichever one you pick with the same curl pattern

# File: terminal - adjust host/port for vLLM (8000) or llama-server (8080)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-local-key" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      { "role": "user", "content": "Reply with exactly the word: connected" }
    ]
  }'

Confirm you get the same choices[0].message.content shape back that Lesson 2 confirmed for Ollama, before moving on to wiring either one into the PHP AI Client SDK.

Recap

Ollama, vLLM, and llama.cpp all expose an OpenAI-compatible chat-completions endpoint, which is what makes a single custom provider implementation in Lesson 4 work against any of them. Pick Ollama for the easiest local development experience, vLLM for genuine production throughput on GPU hardware, and llama.cpp for CPU-only or resource-constrained deployments. Unlike Ollama, both vLLM and llama.cpp can enforce a real API key, worth using if the server is reachable from anywhere beyond your own development machine.

Resources & further reading

← Running Ollama Behind an OpenAI-Compatible Endpoint Building a Custom Provider for the PHP AI Client SDK →