Compare
Compare measures ready models against the same prompts on this Mac and keeps every output, so you judge quality from what the models produced and speed from what was measured. Open Studies opens the separate controlled studies workspace in the Lab sidebar group.
When the selected mode has no results, setup comes first, followed by compact history and empty-state guidance. Existing results remain above setup when a run is available.
Hosted and mixed comparisons
The Compare source picker keeps Local models and Hosted & mixed setups separate. Switching preserves the existing local setup and results. Finish or cancel an active comparison before changing source.
For hosted comparisons, first add and refresh connections in Library's Hosted models source. Choose Hosted & mixed, add up to four models and select Text or ComfyUI images. Selecting models never starts inference.
Text
Text comparisons accept hosted models and ready native chat models in the same run. Choose an existing chat prompt set, then Run comparison. Each lane runs sequentially. Requests use temperature zero and the recorded per-prompt output-token limits, capped by Settings. Templates, active context and other host defaults are not overridden. Unsupported model operations fail visibly rather than being replaced with another model.
Native lanes use an owned temporary server and the existing receipt-based cleanup boundary. External lanes send requests to the selected host without starting or stopping its server. Before each hosted lane, the saved connection and available model identity are rechecked; a changed recorded digest blocks inference. Missing digests remain unknown and matching names do not establish identical weights.
The same inventory read rechecks the selected model against the comparison's task rules. A model now reported as embedding-only or decision-only cannot start a text lane; ComfyUI image lanes still require a reported checkpoint. Ollama also refuses a lane when it reports running weights with the selected tag's name but a different known digest from the installed tag. Resolve that mismatch in Ollama, then refresh and select the model again. These checks send no additional requests and never unload models automatically.
Unknown capability or running-weight metadata does not add a new requirement to load or inspect a model. Known compatible models may start unloaded. A rejected lane records its error without generating or submitting a workflow; later compatible lanes can still run. Checks describe the inventory read before that lane, not an atomic guarantee throughout generation.
Results preserve full outputs, requested prompts and limits, selected source snapshots and pre-inference host metadata. Latency includes transport and any model loading. Token speeds can differ because of tokenizer, host configuration and streaming behavior; estimated metrics remain labeled. Quality ratings stay separate from speed. These records do not enter local recommendation, promotion or reclaim evidence.
Recorded sources and settings → Conditions before each lane retains estimated system headroom, the reserve used for model-fit estimates, host activity, reported loaded models, unknown residency and inventory warnings. Memory, inventory and activity keep separate capture times. Native lanes record system headroom only; their existing sampled memory results remain separate. The host observations reuse the inventory read before that lane, without additional polling or automatic unloading.
Activity, other loaded models and headroom below the recorded reserve can produce a timing caveat. These snapshots do not establish the cause of a timing difference, continuous host activity or per-model GPU allocation. Missing readings remain unknown and do not block inference. Older comparisons show Not recorded; cancelled or partial runs retain observations only for lanes reached. Conditions and fresh source identity must be saved before a lane can start.
ComfyUI images
ComfyUI lanes retain the dated Reported device memory snapshot from the inventory read before inference. Expand it within the lane's conditions for each device's reported free and total memory. MPS/CPU readings overlap shared system headroom; devices are never added together. Missing historical readings stay unknown, and refreshing a live host does not change saved conditions. These readings do not establish model allocations or memory released by cleanup.
Choose checkpoint models, then Import API workflow… using ComfyUI's API-format JSON export. Bind a checkpoint loader, the positive CLIPTextEncode input and a SaveImage output. Review the graph and enter the image prompt before running.
The supported graph uses standard local image nodes. Custom nodes and cloud/API nodes are refused. The host's advertised node definitions and checkpoint choices are validated before submission. Only the bound checkpoint and positive text change; seeds, negative text, samplers and remaining settings are preserved. Any referenced LoRAs, input images and other assets remain host-owned; their file contents are not captured.
The accepted prompt ID and submitted graph are saved before observing that job's history. Outputs are retrieved from the selected output node only after the owned job reports success. Compare retains a PNG or JPEG output for image inspection. Elapsed time includes queue wait, loading and execution; cached nodes can affect it, and per-step inference speed is not inferred.
Cancel stops local observation and skips later work. A submitted ComfyUI job may continue on the host. Its recorded prompt ID remains available under Recorded sources and settings; no global interrupt, queue clear or server stop is issued. Runs interrupted by an app restart retain their records and are marked interrupted. Failed native server cleanup must be retried before another run.
Run a comparison
- Choose a mode. The picker is segmented when it fits and a menu otherwise.
- Fill up to four model slots. Each slot lists only ready models whose task type and runtime support the mode.
- Choose a prompt set.
- Run comparison. A setup message names whatever blocks the run: a pending prompt-set save, a missing or empty prompt set, no model selected, or the first unavailable slot.
Models run one at a time. Chat serves each model on a loopback server, warms it up, replays the prompts and stops it before the next model; warm-up output is excluded from results. Other modes run each model per prompt through the engine's convert describe, transcribe, speak, music, generate or video commands.
Local chat requests use the selected model's path, including its exact Hugging Face snapshot directory. Readiness checks prefer the server's lightweight health endpoint, falling back to the model endpoint when no health endpoint exists. MLX-LM's cache inventory does not have to finish listing before the comparison can proceed.
From Library, Select for Compare opens the model's supported mode with that model in lane A. Lane B waits for your selection; Run comparison starts the work. Vision-language models open Chat by default; choose Vision or Video understanding for media input. Strata is text-only and is excluded from both media-input modes. If a comparison, save, editor or profile verification is in progress, finish it and choose Use selected Library model to apply the pending selection.
Cancel and recover cleanup
Cancel comparison stops the run's owned chat request/server or media subprocess group, retains recorded samples and skips remaining prompts and models. Run stays blocked until the worker and cleanup return. Server startup may need to finish before its receipt can be stopped. Failed server cleanup offers Retry cleanup.
Activity also shows the active comparison's prompt set, mode and progress, with the same Cancel comparison or Retry cleanup action. Open Compare returns to its setup and recorded results. Unavailable saved comparison state appears there with its storage error; settled comparisons leave no empty card.
Confirmed chat-server ownership is saved before warm-up or measured prompts. Unresolved cleanup survives restart and blocks new comparisons until explicit Retry cleanup checks the original server's receipt and process identity. Changed ownership is not stopped; an already-exited original needs no signal. The cleanup record clears only after a successful durable save. Unreadable comparison state blocks new runs, and failed ownership saves stop before inference.
Cancelled runs and runs interrupted by an app restart remain explicit in history and do not supply completed-run recommendations, champions or winner promotion. An interrupted run with no recorded completion time keeps that time unavailable.
Modes
| Mode | Accepts | Input | Output | Lane metric |
|---|---|---|---|---|
| Chat | Text LLM, vision-language | Text | Text | Decode speed (tok/s) |
| Vision | Vision-language | Image | Text | Generation speed (tok/s) |
| Video understanding | Vision-language | Video | Text | Generation speed (tok/s) |
| Speech to text | Speech-to-text | Audio | Text, scored by word error rate | Real-time factor |
| Text to speech | Text-to-speech | Text | Audio | Real-time factor |
| Music generation | Music generation | Caption, lyrics, duration, steps, seed | Audio | Real-time factor |
| Image generation | Image generation | Prompt, size, steps, seed | Image | Seconds per step |
| Video generation | Video generation | Prompt, size, frames, frame rate, steps, seed | Video | Seconds per frame |
Prompts can carry an expected-words check (all must appear, case-insensitive; a|b accepts either). Speech to text scores against a reference transcript. Built-in media inputs are generated at run time: images and video with CoreGraphics and AVFoundation, speech with say. The repository holds no binary fixtures.
Results
Chat results use a model-by-prompt outcome matrix beside a reading inspector. Each cell records Finished, Token limit, Incomplete, Blocked, Unknown, Not recorded or Waiting from the saved evidence. Missing finish reasons remain unknown even when text is available. Median decode speed stays secondary and includes recorded incomplete outputs; completion does not establish answer quality. A model’s information button retains runtime, startup, prefill estimates and profile actions.
On wider windows, the matrix grows with the available space and recorded prompt count. Model names and prompt headings gain wider columns while the reading inspector keeps a minimum readable width. Large prompt sets scroll horizontally instead of shrinking their columns; narrow windows retain the collapsible matrix and single-output reader.
Select a prompt heading or an outcome cell to inspect its recorded outputs. Selecting another model replaces the first inspected model; a model already in the pair keeps its position. Each output heading can change its model, swapping the pair when necessary to keep distinct identities. The inspector separates the current prompt outcome from whole-run coverage and retains recorded lane letters, full text, original-output copying and bounded line differences. Previous/Next changes only the inspected prompt. Wide windows stack both outputs; narrow windows collapse the matrix behind Models in this run and offer Read left / Read right for one readable output. Quality ratings remain human judgments across the entire run.
Fenced code uses monospaced formatting. Reasoning, including recorded inline <think> sections, and available measurements sit behind Reasoning and measurements. Copy copies the original text and differences compare that original text. Reading, selecting or switching never rewrites a saved run.
Reasoning and measurements retains per-prompt speed, first token, answer-start timing, duration, tokens and tool-call counts when recorded. First token may include reasoning. Answer-start timing is unavailable when inline reasoning makes that boundary ambiguous. Missing completion and timing evidence stays unknown. Tool-only prompts require a recorded tool-call finish and valid arguments for every emitted call to count as finished.
Media results retain one lettered lane per model, with the mode’s primary metric and a relative bar, followed by one row per prompt and one column per lane for text, images, audio or video.
New streamed inline reasoning is separated before recording. Streaming examines each incoming fragment without reparsing earlier reasoning; fragmented thinking tags do not count as the start of an answer. An incomplete opening tag leaves answer-start timing unavailable. Each request closes its own HTTP stream after completion, failure or cancellation, leaving the shared connection session available. Decode speed is omitted when tokens arrive in a single burst, since arrival timing cannot establish generation speed.
Inspect outputs side by side
| Completed run | Inspector |
|---|---|
| Chat | Visible A/B reader with prompt navigation, full recorded responses and line differences |
| Vision, video understanding, speech to text | Compare text · A/B: read two models' recorded responses for one prompt, or highlight line differences |
| Music, text to speech | A/B listening: one player, pause and scrub; switching models keeps the elapsed time |
| Image generation | Inspect images: two outputs with linked zoom and pan; arrow keys pan the focused image |
| Video generation | Inspect videos: two clips on one elapsed-time control; audio off by default, one clip at a time when on |
Shorter clips hold at their end. Missing outputs are excluded and labeled.
Rate quality
Each lane has a Rate quality menu with a 1–5 task-outcome rating; music lanes carry a 1–5 listening rating instead. Ratings are human judgments, stored separately from speed and automated checks. A new rating needs a completed run with every output available; saved ratings stay visible after media is pruned and can be cleared.
A quality champion needs every model in the run reviewed under the same rubric; ties are shared. A current champion also needs matching model identities and environment.
Prompt sets
The prompt-set picker opens a searchable list grouped into temporary setups, saved sets and built-ins, with prompt counts. Searching matches names, prompt text and tool names and never changes the selection. The menu beside it holds:
- New set… and Edit…: per-prompt cards with the mode's settings, Add prompt, Duplicate, Move up and Move down, Revert prompt and a confirmed remove. A warning inside each card names invalid values, and Fix prompt N jumps to the first one. Save stays disabled until the draft is valid and changed.
- Customize copy…: a new, independent set from any built-in or saved set.
- Rename… and Remove…: saved user sets only. Removal changes no past run.
- Import OpenCode prompts: imported prompts become selectable once the set is saved.
Cancel on an edited draft offers Keep editing or Discard changes. Saving copies picked input files into the set's own storage, so a saved set survives output pruning and never changes your original files.
Reuse setup on a completed run copies its prompts, settings and model selections into a temporary editor. Use setup restores them to Compare; Save as prompt set… saves an independent copy. Models no longer available stay selected and block the run until replaced or removed.
Storage and retention
Native and hosted comparisons load saved history in the background. Run remains unavailable until history and pending server ownership are restored. Serial history writes finish before inference or cleanup advances; failed initial writes start no model. Each run and its history entry are saved together in a transactional archive, so an update does not rewrite every run. History retains metrics, cohort identity and reviews; full outputs and submitted graphs load when selected, with visible loading and error states. Original JSON arrays remain untouched after migration. Quality ratings publish after their save succeeds.
Switching runs cancels queued output-detail reads. A read already in progress may finish, but cancelled results and errors are not displayed. Durable writes and server-ownership checks still finish.
Outputs are saved to ~/Library/Application Support/mlx-workbench/comparison-outputs/<run-id>/. New media runs first save one copy of each input. When a media run starts, the output folders of all but the 10 newest media runs are removed; run records are kept and a missing file shows as output pruned. History is a searchable popover grouped by mode.
Library music tasks generate audio with one model and retain their inputs and outputs in the engine's task store. Their Recent tasks list is separate from comparison history and its ten-run media retention. Task completion supplies no listening rating or comparison winner; use Compare to judge models against matching prompts.
Act on the result
- Use model A… (using the eligible model’s lane letter) requires a completed chat run, a positive finite speed and a finished outcome for every recorded prompt in the selected model’s cohort. Legacy runs without recorded prompt coverage cannot qualify. The action saves the winner as the preferred model for its use case and can enable or swap the always-on endpoint, keeping verified gating. Wiring and reclaim keep their own previews.
- Measured chat speed feeds the recommendation engine as local benchmark evidence. Media runs do not.
- Measurements count as current only when the selected prompt set still matches the recorded prompts exactly.
Workflow evidence
Import workflow report… reads WorkflowReport JSON files (schema 1) recorded by an external harness. Imports preview new and duplicate observations and model and environment status before you confirm. Charts above the setup show runtime, timing breakdown, task quality and recorded peak memory for comparable observations; a Quality vs runtime scatter pairs score and runtime from the same report. Missing timings show as Breakdown unknown, never inferred.
From the charts, Compare these models loads the cohort's ready models into the current mode without starting a run, and Use model… opens a review of the role preference and optional endpoint change. Exports carry the measurements and machine context for an agent; they are advisory.
Strata lanes
A prepared Strata model can run in two lanes with different speculation, expert-memory and context settings, and Find best settings… prepares a four-lane settings search. See Strata.