Local models & semantic search

Independent local-first paths:

• Chat and utility work on a local LLM via Ollama — optional. A loopback

server is desktop-only; mobile utility work can use a LAN or remote endpoint.

• Semantic search on an on-device embedding model — every platform, on by default, no external runtime.
• On-device GPU chat inside Obsidian — a downloaded, quantized model answers

in the existing chat panel. Requires WebGPU, float16 support, and a persistent browser cache in Obsidian's runtime. No helper app or server.

You don't need the first to get the second.

On-device GPU chat

1. In *Settings → Companion for Claude → Choose how I chat*, choose On-device GPU

as the chat backend.

1. Start with SmolLM2 360M (about 275 MB, the smaller default), or select

Qwen3 0.6B (about 579 MB). These are pinned 4-bit ONNX exports. Download sizes exclude runtime assets and do not represent peak memory use.

1. Click Download model. The plugin checks GPU support inside Obsidian

before downloading from Hugging Face and the ONNX runtime CDN.

1. Open Companion chat and send a text request. The reply streams into the

conversation; no app switch is involved.

Chat uses only cached assets and never retries on a cloud provider. If the OS evicts model files, download explicitly again. The cache belongs to this device and does not sync with vault settings. Delete model removes the selected export without deleting embedding models.

This backend supports text chat, up to 2048 input tokens and 256 output tokens. Agent tools and media are unavailable. Longer context is rejected rather than silently truncated. Custom system instructions and project context still apply; the large HTML design prompt is omitted to leave room for the conversation. Utility tasks continue to use their separately configured backend.

Models never load on startup. Each foreground request owns one worker, which is released on completion, failure, cancellation, or backgrounding. Mobile embedding inference pauses when chat loads its model; use Rebuild index to resume it afterward. A smaller export reduces weight storage but cannot guarantee immunity from iOS memory termination.

Run Companion: Check on-device GPU support to diagnose the actual Obsidian runtime. Safari support alone does not establish support in an embedded WebView. The earlier native MLX result callback remains available for pending requests; the plugin no longer offers the command that launches the helper app.

Never lose functionality: the Auto backend

*Settings → Companion for Claude → Choose how I chat → Chat backend*

Fallback triggers on network loss, rate limits and usage caps, rejected credentials, and server errors (429/401/403/5xx, plus message-level signals like fetch failed, quota, overloaded, ECONNREFUSED, timeout).

It does not trigger on a genuine 400 bad request — that's your prompt or parameters, and a different model won't fix it. A request that already ran locally is never retried locally.

When a fallback happens you get a short reason rather than a silent model swap: *"Claude is rate-limited or out of usage"*, *"Claude credential rejected"*, *"No connection to Claude"*, *"Claude service error"*. A live connectivity indicator shows which backend is active.

!Auto backend showing a completed response generated by the configured local model

Setting up Ollama

A localhost Ollama server is desktop-only. Mobile utility work can reach an Ollama server exposed at a valid LAN or remote HTTP(S) endpoint; do not expose an unauthenticated model server to the public internet.

1. Install Ollama and start it.
2. Pull a model: ollama pull llama3.1 (any chat model works).
3. In Obsidian, open *Settings → Companion for Claude → Local models (Ollama & endpoints)*.
4. Check Ollama host — default http://localhost:11434.
5. Pick a Local chat model. The field becomes a dropdown populated from your Ollama server once models are detected, and stays free text until then.
6. Click Test local connection.

Capability badges

Companion reads each detected model's metadata (/api/show) and badges it in settings:

• tools ✓ — the model can drive agent mode locally (llama3.1, qwen3,

and similar tool-trained families). Without it the agent switches off for that model with a notice, and chat continues as plain completions.

• thinking ✓ — the model reasons before answering; the composer's

reasoning indicator lights up whenever the active backend thinks (this flag on a local model, or extended thinking toggled on for Claude).

Utility work (summaries, tagging, enrichment) always runs with thinking disabled and a structured-JSON request, so a reasoning model can't burn the reply budget on hidden reasoning — that failure mode used to surface as "reply was not valid JSON" during inbox ingestion.

Routing cheap work locally

Companion routes by task role, not just by a global switch:

• `chat` — your primary provider, per the Chat backend above.
• `utility` — summaries, auto-tagging, and ingestion.

Utility tasks backend — Claude, Ollama (local), or your OpenAI-compatible endpoint — sends utility work somewhere other than chat. That's the setting that saves the most tokens for the least quality cost: bulk summarizing and tagging don't need a frontier model, and a small local model is fine at them. It defaults to Claude.

Structured utility work (source enrichment) requests JSON against a field-derived schema and turns thinking off, so reasoning models like qwen3 answer with the fields instead of spending the token budget on hidden reasoning.

Routing checks that an Ollama host is configured, not that it's answering — so if you enable this and then stop Ollama, utility tasks will fail rather than silently re-route to Claude. Use Test local connection after changing anything. (The chat-backend fallback described above *does* probe reachability.)

On mobile, Companion evaluates the utility destination before sending note content. Loopback and wildcard hosts—including localhost, 127.0.0.1, *.localhost, and IPv6 loopback aliases—cannot reach the desktop from a phone. For those endpoints, Companion asks whether to use Claude for utility work for the current plugin session. Choosing Don't send, changing the endpoint or credential, closing the plugin, or lacking an Anthropic credential aborts the operation without a provider call or vault write. A configured LAN or remote HTTP(S) endpoint is used directly, and the Source Inbox displays the backend selected at runtime.

Semantic search

*Settings → Companion for Claude → Capture and find knowledge*

On by default. Builds a local vector index so the vault is searchable by meaning, not just keywords — powering the "Search vault" context chip, ask-your-vault, vault_search for the agent and the MCP bridge, and related-note suggestions. Private and on-device.

Search is hybrid: keyword hits and semantic hits are fused by reciprocal rank fusion, deduped per note, each note keeping its best snippet. If the index is empty or the embedder is unavailable, it degrades to keyword search rather than returning nothing.

Three embedding engines

Built-in (recommended, default) — runs Snowflake/snowflake-arctic-embed-xs (384-dimension vectors) inside Obsidian via transformers.js. Mobile uses single-threaded WASM; desktop uses WebGPU where available and WASM otherwise.

It needs one explicit download: ~24 MB of q8 weights and tokenizer/config files from huggingface.co plus ~23 MB of ONNX runtime from cdn.jsdelivr.net. Nothing downloads until you click Download; afterwards it's cached and runs fully on-device. A Clear button removes the cached model.

Ollama — uses your local Ollama server for embeddings instead (nomic-embed-text by default). On mobile, use a reachable LAN or remote server; localhost points to the phone.

OpenAI-compatible endpoint — embeds against the endpoint configured under *Local models*, using its embedding model.

The built-in embedding worker uses WASM on mobile to limit peak memory use, and WebGPU or WASM on desktop. On-device chat uses WebGPU with quantized ONNX exports; neither worker uses native MLX or Android LiteRT. Arctic XS is the mobile recommendation because it has the smallest download and vector dimensions in this catalog. Download size is not peak runtime memory, and retrieval quality depends on the vault.

Mobile indexes are limited to 8 MiB including vectors, note text, and metadata. If the next note would exceed that budget, indexing stops and retains the previous saved index. Keyword search remains available. A smaller embedding model reduces vector storage; large vaults may still exceed the limit.

Each download uses an immutable publisher revision. All three use CLS pooling, normalized vectors, Snowflake's search instruction for queries, and unprefixed documents. The tokenizer caps each input at 512 tokens, including the query instruction, on desktop and mobile; M-long's longer native context is deliberately bounded here to limit inference allocations. The index records the repository, revision, dtype, pooling, dimensions, token ceiling, and both prefixes.

Upgrading from an index without this encoding contract requires an explicit Download of the pinned assets and Rebuild index. Existing floating-revision cache entries cannot satisfy the new download check. This does not edit your notes or automatically download weights; keyword search remains available while the semantic index is unavailable.

One pinned default on every platform means one index format, so a desktop-built index that syncs to mobile is usable there until the phone builds its own, as long as it fits the mobile memory budget.

Building the index

On mobile, each launch starts with embedding inference paused. Search remains keyword-only until you explicitly choose Download or Rebuild index in embedding settings. Cached weights do not trigger startup indexing. Disabling semantic search or unloading Companion stops the worker and cancels pending indexing. An engine failure pauses inference until an explicit retry.

Mobile indexing caps the persisted index at 8 MiB and retained chunks at 400. A phone saves its own index file (semantic-index-mobile.json) beside the desktop one (semantic-index.json), so neither overwrites the other through vault sync. It checks a file's size before reading it; an index over the budget is never read, and the phone starts an empty index that Rebuild index fills. PDF extraction is limited to 200 pages and 256,000 characters, in addition to the input file limits. These are conservative allocation bounds, not a measured guarantee for every device. Use keyword search or desktop indexing when a limit is reached.

Indexing traverses the vault, chunks each note, embeds the chunks, and stores the vectors locally. Use Rebuild index after switching engines or models — vectors from different models aren't comparable, which is why the built-in model's index key includes its encoding contract, so it cannot collide with an Ollama model name or silently reuse vectors encoded with different model settings.

PDFs are indexed too (the Index PDF text toggle, on by default): text is extracted with pdf.js, chunked without ever crossing a page boundary, and every chunk carries its page number — so semantic results and evidence citations point at the page, not just the file. Encrypted or unreadable PDFs skip quietly and never abort a build.

Related-note lookups handle a not-yet-indexed note by live-embedding its first chunk, so a brand-new note isn't invisible while the index catches up.

What runs where

On mobile the settings tab collapses the desktop-only items into one *🖥 Desktop-only features* note.

Cost note

Running utility work locally and letting Auto fall back are the two levers that keep a heavy vault cheap. Prompt caching does the rest — see auth.md.