Model Selection and Serving

Decision framework for choosing the right model for a task and the right way to serve it. Consult this before defaulting to the largest model your budget allows. The right model is often smaller, faster, and cheaper than you might expect.

If you'd like to work with the model/provider pairs I tested with this guide, review the Model-Provider Matrix companion to this page.

Ecosystem layers

These categories get mixed together constantly in vendor docs and community posts. Keeping the layers separate will save you a lot of confusion.

A common confusion: Gemini API vs. Vertex AI

Google exposes Gemini through two different product surfaces, and it is easy to blur them together if you are new to the ecosystem.

• Gemini API: the simpler developer-facing API used with Google AI Studio and API keys. This is the surface the curriculum means when it says "Gemini" in most runnable lesson examples.
• Vertex AI: Google Cloud's platform surface for Gemini. This is the place where project/location setup, IAM, service accounts, and Google Cloud governance enter the picture.

For the curriculum, the practical distinction is:

• If you want the fastest start for hosted inference examples, the Gemini API is usually the simpler path.
• If you need hosted Gemini tuning, Google currently supports that on Vertex AI, not on the Gemini Developer API.

If you want to dive deeper into the official product docs behind that distinction, start here:

• Gemini API and Vertex AI comparison
• Gemini API tuning status
• Vertex AI supervised tuning for Gemini

Model families

Three model families serve different purposes. These are introduced in LLM Mental Models and used throughout the curriculum.

Parameters and model sizing

A model's parameter count is the number of learned weights in the network. It's the single most commonly cited measure of model size, and it directly determines how much memory the model needs.

What parameter count tells you: More parameters generally means more capability, and the model can represent more complex patterns. But it also means more memory, more compute, more cost, and more latency.

What parameter count doesn't tell you: Training data quality, architecture efficiency, and post-training alignment matter as much as size. A well-trained 7B model often outperforms a poorly trained 13B model on specific tasks. Don't choose models by parameter count alone. Choose by benchmark performance on your task type.

The practical question: "Can I run this model on my hardware?" See the Hardware and Model Size Guide for VRAM requirements and GPU recommendations.

Quantization

Full-precision models store each parameter as a 16-bit (FP16) or 32-bit (FP32) floating-point number. Quantization reduces the precision of those numbers (typically to 8-bit, 4-bit, or even 2-bit integers) to shrink memory usage and increase inference speed.

Why quantization matters

A 7B parameter model at full precision (FP16) needs ~14 GB of VRAM. Quantized to 4-bit, that same model fits in ~4 GB. This is the difference between needing a $1,000 GPU and running on a laptop.

Common quantization formats

Practical guidance

• For curriculum exercises with Ollama: Ollama automatically downloads quantized models. When you run ollama run qwen2.5:7b, you get a quantized version that fits in reasonable VRAM. You don't need to choose a quantization format manually.
• For production serving: Start with the highest precision your hardware supports. Only quantize further if you need to reduce memory or improve throughput, and measure the quality impact on your specific task with your eval harness.
• For fine-tuning: QLoRA (Module 8) uses 4-bit quantization during training to reduce VRAM requirements. The trained LoRA adapter is applied on top of the quantized base model.

The tradeoff in one sentence

Quantization trades precision for speed and memory. The question isn't "should I quantize?" but "how much precision can I give up before my evals show a problem?"

Choosing by task type

Hosted vs. self-hosted

Decision rule: Start with hosted APIs. Move to self-hosted when one of these is true:

• Privacy requirements prevent sending data to providers
• Per-token costs exceed the hardware cost of running locally
• You need a fine-tuned or distilled model not available as a hosted endpoint
• Latency requirements demand local inference

Self-hosted serving options

Ollama

Best for: Development, prototyping, running models during curriculum exercises.

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run qwen2.5:7b

# Serve via REST API
curl http://localhost:11434/api/chat -d '{
  "model": "qwen2.5:7b",
  "messages": [{"role": "user", "content": "Hello"}]
}'

Ollama downloads, quantizes, and serves models with minimal configuration. It's the right choice for individual development and curriculum work.

vLLM

Best for: Production serving with high throughput and concurrent users.

pip install vllm

# Serve a model with OpenAI-compatible API
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000

vLLM provides an OpenAI-compatible API endpoint, so switching from a hosted API to a local model requires changing only the base URL and model name. Its PagedAttention mechanism enables efficient memory management for concurrent requests.

When to use vLLM over Ollama: When serving models to multiple users, when you need precise control over batching and throughput, or when running in a production environment.

llama.cpp

Best for: Maximum control over quantization, running on CPU, or resource-constrained environments.

# Build from source
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build

# Run a GGUF model
./build/bin/llama-cli -m model.gguf -p "Hello"

llama.cpp supports CPU inference (slower but no GPU required) and fine-grained quantization options. Use it when you need to run models without a GPU or when you need specific quantization configurations.

Model selection by curriculum module

Cross-references

• Hardware and Model Size Guide — VRAM requirements, GPUs, and cloud options
• Model-Provider Matrix — the model and provider combinations that actually held the lesson contracts during validation
• LLM Mental Models — introduces the three model families
• Cost and Reliability Patterns — model routing pattern for cost optimization
• Choosing a Provider — provider setup and API key configuration