Local AI cheat sheet

Ollama

Everyday commands

ollama pull llama3.2          # download a model
ollama run llama3.2           # chat (type /bye to leave)
ollama run llama3.2 "Hi"      # one question, then exit
ollama list                   # downloaded models
ollama show llama3.2          # details of a model
ollama ps                     # models loaded in memory
ollama stop llama3.2          # unload from memory
ollama rm llama3.2            # delete a model
ollama cp llama3.2 my-copy    # copy / rename a model
ollama serve                  # start the server by hand

Create a model from a Modelfile

# file named Modelfile
FROM llama3.2
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """You are a concise assistant. Answer in plain language."""

ollama create my-helper -f Modelfile
ollama run my-helper

Lower temperature = steadier answers; num_ctx = context window in tokens (more uses more memory).

Environment variables

OLLAMA_HOST=127.0.0.1:11500 ollama serve        # Linux / macOS
OLLAMA_KEEP_ALIVE=30m ollama serve

API on http://localhost:11434

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}'

curl http://localhost:11434/api/tags      # list local models

llama.cpp

Runs GGUF model files. Binary names: llama-cli (terminal chat), llama-server (HTTP server with OpenAI-style API).

llama-cli -m model.gguf -p "Write a haiku" -n 128
llama-cli -m model.gguf -c 8192 -ngl 99 -t 8
llama-server -m model.gguf --port 8080 -c 8192 -ngl 99
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

Quantize a model

llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Takes a full-precision GGUF and writes a smaller one. Converting original model files to GGUF uses the project's convert_hf_to_gguf.py script.

LocalAI

Self-hosted server, OpenAI-compatible API. Image names and tags change: copy the current command from localai.io.

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_NAME","messages":[{"role":"user","content":"Hello"}]}'

Models live in a models folder (mount it into the container); each model can have a YAML file:

name: my-model
parameters:
  model: model.gguf
context_size: 4096

The exact YAML keys depend on the backend and version: check the LocalAI docs.