Models

Everything model lives in one place: the Models page. Open it from the 🤖 icon in the activity bar (bottom of the rail, below Brain), from View › Models in the menu bar, or with ⌘⇧M. It is a full-page tab — the sidebar steps aside while you’re in it — with three sections: Library, Find models, and API.

LatchAI thinks with two kinds of models, and the Library treats them as one list:

• On this Mac — models you download from the in-app catalog. They run entirely on your machine: nothing you say leaves the computer, there is nothing to sign up for, and they keep working offline.
• Connected — models behind an API: OpenAI-compatible endpoints (Ollama, LM Studio, a local vLLM server, or OpenAI itself) and Anthropic via the official SDK.

With neither configured, LatchAI falls back to an offline demo model so everything stays clickable — and says so whenever that model is the one answering.

Note

There is no “Settings → Models” anymore. ⌘, opens Settings, which starts at Agents; external providers moved to the Models page, where local and connected models sit side by side.

The Library

The Library is where you pick the default model — one radio button that spans both kinds of row. The default powers chat and agent nodes unless a conversation, node, or agent definition picks another; a model running on this Mac is as valid a default as a hosted one.

On this Mac

Each downloaded model is a row with Load / Unload, a delete button, and — while loaded — live stats: how much memory it is holding and, when it is generating, its current tokens per second. One model is loaded at a time (loading one unloads the previous one), and you rarely need to press Load at all: the first chat turn or workflow run that names a local model loads it automatically.

In-flight downloads appear in the same list with a progress bar, transfer speed, and a pause button; a paused or interrupted download keeps its partial files and offers Resume — even across an engine restart.

Local models show up in every model picker as local:<name> — for example local:Qwen3-4B-GGUF — with no configuration: downloading a model is what registers it.

Connected

External models are rows in the same Library: a name, a type (OpenAI-compatible or Anthropic), a base URL where the type needs one, a model name, and an optional API key. Each row has Test connection, which sends one real, tiny completion and reports either ✓ Connected — <model> replied in <n>ms or a specific failure (see Troubleshooting).

API keys you enter here are stored in the macOS Keychain via the secret store — never in a config file. Saving hot-swaps the providers in the running engine; no restart.

Connected models in the Library; keys go to the Keychain and are never echoed back.

OpenAI-compatible endpoints

Anything that speaks the OpenAI chat-completions dialect — Ollama, LM Studio, a local vLLM server, or OpenAI itself — connects with a base URL, a model name, and (optionally) a key. For Ollama that looks like base URL http://localhost:11434/v1, model qwen3:8b. The base URL is the prefix LatchAI appends /chat/completions to, so it includes the /v1. Leave the key blank for a server that doesn’t want one — nothing is sent when it’s empty.

Anthropic

Anthropic is a first-class type with no base URL. Credentials resolve the way they do for any Anthropic SDK application: an explicit API key first, then ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN, then an ant auth login OAuth profile (browser sign-in, no static key on disk).

Adaptive thinking is enabled on plain turns. Tool turns run without it — echoing thinking blocks back through LatchAI’s provider-neutral history isn’t wired yet, and Claude requires them unchanged inside a tool loop.

Caution

Claude Pro/Max subscription billing is not available to third-party API callers — that path is reserved for Claude Code and the Agent SDK. LatchAI’s Claude usage bills to your Anthropic platform organisation.

Find models

Find models searches a public catalog of open models (the Hugging Face Hub) and only shows what this app can actually run. A hand-picked ★ featured shelf opens the page — models known to run well — followed by the highest-rated compatible ones. Search by a model’s plain name (qwen, llama, gpt-oss), type Publisher/ to browse one publisher, and use the ↗ on any row to read about the model on Hugging Face.

Opening a result lists its available download sizes: smaller ones use less memory and disk at a small cost in quality. Each size carries a fit badge — ✓ fits, ⚠ tight, or ✗ won't fit — checked against this machine’s memory. The badges are explicitly estimates, and a “won’t fit” download warns rather than forbids.

Downloads are resumable (a restart offers to pick up where it left off), verified on completion, and visible from anywhere in the app via a status bar chip. Models are never bundled with LatchAI — each one is fetched from Hugging Face at your request and carries its own license, surfaced on its Hugging Face page.

Under the hood

The local runtime is llama.cpp’s llama-server (MIT-licensed; attribution ships in the repo’s THIRD-PARTY-LICENSES.md), running GGUF-format models. Each loaded model runs as a separate, supervised process bound to loopback only, guarded by an API key minted fresh for that process — so nothing else on the machine (or a website in your browser) can talk to it. Context size is fitted to the machine automatically: the server starts from the model’s training context and shrinks it until model plus context fit memory. Set models.contextTokens in latchai.config.json to pin an explicit size instead.

Running from a clone

Running LatchAI from a cloned repo, local hosting needs the llama-server binary — the packaged desktop app ships it, the repo does not. The engine looks for it in this order:

1. The LATCHAI_LLAMA_SERVER environment variable — an absolute path to any llama-server binary, e.g. the one from brew install llama.cpp.
2. vendor/llama-cpp/*/llama-server under the repo root. vendor/ is gitignored: download a macOS arm64 release archive from the llama.cpp releases page and unpack it there.

Without the binary, loading a local model fails with a message naming both options — everything else keeps working, and connected models don’t need it at all.

The API

The API section runs an OpenAI-compatible gateway: one endpoint — default 127.0.0.1:7778/v1 — serving /v1/models and /v1/chat/completions (with streaming) over every model in your Library. Requests route by model name: a local:<name> model is loaded on demand, a connected provider is proxied through. Point any OpenAI-compatible app at it instead of configuring each one separately:

curl -s 127.0.0.1:7778/v1/models
curl -s 127.0.0.1:7778/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "local:Qwen3-4B-GGUF", "messages": [{"role": "user", "content": "hi"}]}'

On this machine, no key is needed. An Expose to network toggle widens the listener to your LAN — and generates a bearer key that is required for every non-localhost request; the network listener never starts without one. Before enabling it, the UI spells out what exposure means: requests routed to paid providers spend on your keys, local-model requests occupy your machine’s memory and GPU, and prompts transit the LAN as plaintext HTTP.

The gateway serves completions and nothing else. Tool definitions pass through to the model, but tool execution stays with the caller — LatchAI’s workflows, files, and shell are not reachable through it, and LatchAI’s own app API stays localhost-only either way.

Choosing a model per chat, node, or agent

The Library’s default is just that — a default. Three places override it:

• Per chat — the model dropdown in the chat panel lists every model in the Library, local (local:<name>) and connected; picking one applies to the next turn.
• Per node — an agent node’s config.provider takes a model name. An unknown name silently falls back to the default rather than failing the run, so check your spelling if a node seems to ignore it.
• Per agent — an agent definition’s model: frontmatter names one too. The seeded assistant ships with model: default.

Demo mode

With no model configured — or a local default still downloading — chat runs against an offline mock model, and says so: a banner above the composer marks replies as canned examples, not a real AI, with a link to the Models page (and, while a download is in flight, its live progress). The demo model exists so the app is explorable with zero setup; the banner exists so it can never masquerade as a real one.

The status bar

Two chips in the status bar keep models visible from every surface, and both click through to the Models page:

• 🤖 the loaded model — with live tokens per second while it is generating.
• ↓ an active download — with its percentage.

Under the hood: latchai.config.json

The Models page writes latchai.config.json in your LatchAI home — connected models live under providers, keyed by display name, with a defaultProvider:

{
  "providers": {
    "ollama": {
      "type": "openai",
      "baseUrl": "http://localhost:11434/v1",
      "model": "qwen3:8b"
    },
    "claude": {
      "type": "anthropic",
      "model": "claude-opus-4-8",
      "apiKey": "{{secret:provider.claude.apiKey}}"
    }
  },
  "defaultProvider": "ollama"
}

Downloaded local models never appear in this file — they register themselves — but local:<name> is a valid defaultProvider. Keys are {{secret:…}} references into the secret store, named for the row (provider.<name>.apiKey); keys you don’t touch on a later save carry over, so the UI never has to display a key back to you. A legacy single provider block is normalized into providers.default the first time the config is saved from the UI.

Streaming is used for plain and tool turns alike. A server that mangles streamed tool-call fragments (the least interoperable corner of the OpenAI dialect) can set streamTools: false on its provider entry to fall back to a single non-streaming request for tool turns.

Embeddings

Semantic code search (search_code) needs an embeddings endpoint, configured separately in the same file. It is any OpenAI-compatible /v1/embeddings service — LatchAI is developed against a local embedding server running gte-base-en-v1.5 at 768 dimensions.

{
  "embeddings": {
    "baseUrl": "http://localhost:8000/v1",
    "model": "gte-base-en-v1.5"
  }
}

The index persists to runs/semantic-index.json keyed by file mtime, so re-indexing is incremental. POST /api/reindex rebuilds it across the workspace and every mount. With no embeddings configured, search_code returns a clear “not configured” message instead of failing the run.

Note

embeddings is read when the engine starts. Add it to a config the daemon is already running against and restart the engine before expecting search_code to work.

Checking what’s live

/api/status reports the active model along with the registered tools and MCP server health:

curl -s localhost:7777/api/status

{
  "provider": "http://localhost:11434/v1 (qwen3:8b)",
  "providers": ["ollama", "claude", "local:Qwen3-4B-GGUF"],
  "tools": ["read_file", "write_file", "edit_file", "list_dir", "run_shell", "…"],
  "mcp": {},
  "embeddings": true,
  "running": [],
  "dashboardsRunning": []
}

provider is the resolved default’s display name — the base URL and model for an OpenAI-compatible provider, anthropic (<model>) for Claude, local (<repo>) for a model on this Mac, and a bare mock when nothing is configured. From the terminal, npx tsx packages/engine/src/cli.ts tools lists the whole registry, built-ins and MCP tools together.

Troubleshooting

Everything “works” but the answers are canned. A provider spec missing baseUrl or model does not fail — it falls back to the mock provider, loudly on the console:

[latchai] provider spec is missing baseUrl/model — falling back to MOCK; completions will be canned until latchai.config.json is fixed

That is deliberate (an unattended pipeline must not die at 3am over a typo), but it means /api/status reporting "provider": "mock" when you configured a real one is the symptom to look for. In chat, the same situation shows the demo banner.

Test connection tells you which of the three things is wrong. It sends one real completion and translates the failure: could not reach … (base URL, or the server is down), authentication failed — check the API key, endpoint reached, but model “x” was not found (the model isn’t pulled or the name is wrong), or no response within 15s.