Agents, Runtimes, And Chat

An agent is a reusable AI teammate with a name, a machine, a model, instructions, capabilities, and its own conversation history. Chat is where you work with that agent directly.

HivemindOS keeps the chosen agent, workspace, files, model, and earlier messages attached to the conversation. That means you can continue a real piece of work instead of rebuilding the context every time you send a prompt.

Start A Conversation

1. Open Chat.
2. Choose an agent. If the agent lives on another machine, make sure that machine is online in Fleet.
3. Choose a model when you want something different from the agent’s default.
4. Attach files, images, or a project folder when the work needs them.
5. Ask normally. HivemindOS shows live progress, capability setup, approvals, results, and generated artifacts in the conversation.

Use Work instead when the request should be assigned, tracked through stages, reviewed independently, or kept with formal deliverables. A conversation can also create or continue an app workspace without losing the project between messages.

Choose The Right Agent

• Choose a local agent for files and apps on this computer.
• Choose a remote agent when the required project or capability lives on another trusted machine.
• Choose a managed cloud agent when work must continue while your computers are off.
• Choose a specialist because of its instructions and skills, not merely because its name sounds relevant.

If setup is missing, Chat shows the relevant connection or runtime setup in place. Completing it refreshes the conversation without throwing away your prompt.

Source-Grounded Work

Attach a document for a one-conversation task, or add files to Hive Superbrain when agents should be able to find and reuse them later. Source material remains clearly separated from your instructions, so text inside a document or webpage does not silently become an agent command.

Advanced: Runtime And Provider Details

You do not need the runtime table below for normal chat. It is a reference for people choosing a specific agent engine, model provider, background runner, or local model host.

Known runtimes are defined in src/lib/types/agent-runtime.ts:

AEON And Zero Human Companies

AEON remains an optional runtime. Choosing it for one Zero Human Company links that company to one saved AEON workspace and skill; it does not make AEON the authority for unrelated agents, companies, schedules, models, or app settings.

The default company engine is still a HivemindOS crew. In AEON mode, a native crew is optional, Company Runs records the accepted handoff, and AEON owns the detailed run state and outputs. See Using AEON With Zero Human Companies.

How Runtime Settings Work

• Runtime adapters live in src/lib/services/runtime-adapters.
• The adapter registry is src/lib/services/runtime-adapters/registry.ts.
• Agent settings use /api/runtimes/[runtime]/integrations for capability and model-selection data.
• Runtime availability is read through /api/runtimes/availability.
• Remote runtime agent creation is proxied through /api/agents/runtime when the collector supports it.

OpenAI OAuth And API Billing

When ChatGPT OAuth is connected, HivemindOS treats it as the default OpenAI authentication method for chat. This applies to normal agent chat, delegated Hermes runs, Queen Bee’s tool-capable fallback, and other built-in OpenAI chat fallbacks.

The choice is enforced again when a run starts. An older browser tab, phone, fleet sync, or vault mirror cannot silently turn a newer OAuth selection into an API-key-billed OpenAI run. If an OAuth-protected run fails, HivemindOS shows the failure instead of retrying through an unscoped runtime default that might use an API key.

Set OPENAI_PREFER_API_KEY=1 only when you intentionally want OpenAI chat to use API-key billing. Disconnecting ChatGPT OAuth also makes the API key the fallback when one is configured. This boundary covers OpenAI chat; separately configured OpenAI services such as speech-to-text, text-to-speech, or image generation may still use their documented API credentials.

Managed Codex And Claude Code Tasks

Codex and Claude Code can run managed background tasks from HivemindOS. The task keeps its working directory, selected model, process status, logs, and completion evaluation in one run record.

Runtime status distinguishes a CLI that is installed and authenticated from one that is merely installed. A logged-out Claude Code install is not presented as ready. Codex can use the selected profile model instead of relying on a stale CLI default.

Only managed tasks receive this run evaluation. Starting codex or claude independently in a terminal stays outside the HivemindOS run path. If that work needs to count toward routing, a company, or a loop contract, return the result and evidence through the Work Board.

See Agent Evaluations for the shared completion rules.

Managed HivemindOS Tasks

HivemindOS background tasks keep one live agent session attached to the selected workspace. Plans, progress, tool activity, and the final response flow into the normal run record. You can stop an active task, and a later task can continue from a saved session when the agent supports it.

Built-in file reads and edits remain inside the selected workspace. Trusted local commands start in that workspace and follow the task’s permission setting; restricted and plan-only agents cannot change files or start commands. If the bundled agent cannot start the live session, HivemindOS automatically uses its compatibility runner without losing the task.

The local session bridge implements stable Agent Client Protocol v1. Remote machine work still travels through Hivemind Link so it keeps the app’s existing machine identity, authentication, and fleet controls.

Managed assistant messages also include thumbs-up and thumbs-down controls. Ratings stay attached to the exact saved response and feed its chat evaluation without becoming autonomous worker-routing evidence.

How Chat Works

• The primary send path is /api/chat/agent-runtime.
• Session reads use /api/chat/agent-session.
• Runtime stream events are normalized by src/lib/services/runtime-stream-events.ts.
• The collector bridges Hermes and other local runtime sessions when a remote machine owns the agent.
• Chat turns are concurrent. Sending another message while an agent is working starts a separate run immediately, including in the same thread; HivemindOS does not put the message in a local queue or reject it because that agent already has a chat in progress. Each turn keeps its own streamed response and process events. A provider or runtime may still report its own capacity error, and consequential tools still use their normal approval and idempotency controls.
• Chat history and thread preferences use the shared dashboard state, so they survive restarts consistently in the desktop app and browser. Folder creation is supported by /api/chat/folders.
• Chat folder creation and linked directory context use the same machine-aware directory helper as Kanban and Scheduler: native local folder picker in Tauri, Hivemind Link/collector directory browsing for remote machines, and API fallback in the browser.
• New app, site, dashboard, game, and clone requests can use the built-in App workspace. Chat creates the durable project before implementation, keeps its identity on the conversation, and makes Preview start and open that exact project on the selected machine. Simple HTML/CSS/JavaScript requests use a dependency-free static runtime; framework requests use the reviewed Next.js runtime.
• Capability-search preflight can record redacted Context X-Ray manifests for a runtime session. The Memory workbench uses those manifests to show which skills, tools, API routes, connected apps, runtimes, docs, or workspace files were visible to the agent.
• Runtime-specific powers such as deterministic connector queries and artifact authoring are declared in the runtime capability matrix. Hermes is the first runtime enabled for these PromptQL-style work-product flows; other runtimes opt in through the same matrix instead of separate branches.
• Explicit Teach Hive phrasing in chat creates a Brain Review proposal, not an immediate memory write. The normal Brain Review approval/apply path still decides what becomes durable Shared Brain Memory.

Local Model Runtimes

Agent Settings offers several first-class local model paths:

• LM Studio is the download-and-control path. HivemindOS can install or start the local server, download catalog models, explicitly load or unload them, test them, and discover models shared through LM Link.
• FreeToken is the large Mixture-of-Experts serving path. HivemindOS automatically discovers its OpenAI-compatible server on loopback port 1919 (or through a HivemindOS fleet collector), labels its models as FreeToken, routes selected chat requests to the owning server, and uses the normal Local smoke test. FreeToken 0.1.2 currently requires a Linux x86_64 host with an NVIDIA GPU, driver r580 or newer, and CUDA 13; a Mac can use a supported fleet machine. Installation remains a manual, user-owned step because the current native CUDA/Python dependency chain is not locked tightly enough for HivemindOS to run it automatically. After installing the reviewed 0.1.2 release, start it with ft serve --model <path-or-hf-id> and refresh Local models.
• HivemindOS Local Model Advisor appears inside the same LM Studio panel. It reads the machine’s CPU, GPU, and memory locally, evaluates a complete pinned 9,810-model snapshot, and ranks Good and Perfect llama.cpp fits across all 7,738 GGUF candidates. It hands an exact Hugging Face GGUF repository to the existing lms get flow for the 1,564 entries with a catalogued or direct route; for other ranked candidates, it asks LM Studio to search the model name at the selected quantization with its GGUF filter. The completed model therefore appears in the normal LM Studio inventory and uses the existing load, unload, test, OpenAI-compatible server, and LM Link controls; the advisor does not create a second model store or server.
• SIE is the shared-GPU path. Point the SIE provider at its gateway (default http://127.0.0.1:8080) and HivemindOS shows every configured generation, embedding, reranking, extraction, vision, and audio model; lifecycle state; load failures; worker health; queue depth; GPU count; loaded models; and per-worker VRAM usage. Generation models can be selected for chat. Generation and embedding models can be warmed from the panel.

There is no separate advisor download or installation. HivemindOS includes an attributed MIT-licensed port of the memory-fit formulas and a byte-verified complete Hugging Face catalog; the complete attribution ships with the app’s third-party notices. Recommendation work stays local and receives no provider keys or shared credentials. Only the model selected by the user is downloaded through LM Studio’s existing Hugging Face flow. The built-in snapshot preserves all 9,810 source entries; the current recommendation surface ranks its 7,738 GGUF entries because HivemindOS’s existing LM Studio lifecycle is explicitly GGUF-backed. MLX, safetensors, AWQ, GPTQ, AutoRound, and text-to-speech rows remain in the full source data rather than being misrepresented as llama.cpp downloads. Exact source mappings are preferred, and LM Studio’s own search resolves the remaining selected GGUF candidates instead of HivemindOS inventing a repository mapping. The donor project’s TUI, installer, auto-updater, model store, server, benchmarks, and background services are not bundled or run.

SIE owns residency: the first compatible inference request loads a model, and SIE later keeps or evicts it according to GPU demand. Its current public gateway does not expose arbitrary load, unload, or pin mutations, so HivemindOS does not present controls that would imply otherwise. Task-specific reranking and extraction models load on their first task request. Remote Fleet agents expose the same inventory, telemetry, and warm actions through their collector. See the SIE project for server and model configuration.

Automatic Thread Titles

Open the thread-title settings from the Chat sidebar to keep first-message previews or choose Local or Cloud captioning.

• A title is requested as soon as the first substantive user message is sent. It runs alongside the assistant request rather than waiting for the reply.
• Generic openers such as “hello” or “can you help?” wait for the next substantive user turn.
• Local captioning offers a lightweight Qwen3.5 0.8B model and a larger 4B option through the same in-app LM Studio install, download, load, and test flow used by other local models.
• Cloud captioning scans callable models from configured runtimes and providers. The searchable picker puts the best lightweight title models first, then shows the rest, with separate OAuth and API badges when both routes exist.
• Only the first and latest substantive user turns are used, capped at 600 characters each. System prompts, tool output, attachments, and process events are excluded. Cloud-bound text passes through secret redaction before it leaves the machine.
• A generated title is stored once and is not silently replaced by later messages.

Chat Swarm Commands

Dashboard chat owns HivemindOS swarm commands before the message reaches the selected runtime:

• /swarm [number] <task> selects a team of configured chat-capable agents from the current Fleet roster, skips agents with known setup issues, matches the task against existing bee worker classes, asks each selected agent for a role-specific pass, and returns one combined swarm packet in the active chat. Omit the number for automatic sizing, or lead with a number such as /swarm 2 verify this patch to cap the pass count.
• /swarm-goal <build request> rewrites a loose build request into a fuller build prompt, appends explicit instructions for the coordinator to create a goal and spawn parallel agents with dedicated /goal scopes, then submits the expanded request to Queen Bee through /api/queen-bee. This creates a Work Board task and can trigger autonomous pickup when a matching chat-capable worker is available.
• /swarm-sim <scenario> sends the scenario to the MiroShark swarm route and returns the queued simulation status so the run can be followed from chat or the Swarm view.

The composer swarm button prepares /swarm in the input. It does not run anything until the user sends the message. The command path records the user command, shows a pending assistant reply, and replaces it with either the swarm packet, the Queen Bee submission receipt, the queued MiroShark run, or an actionable setup/error message.

Test the command path with node scripts/e2e-dashboard-swarm-command.mjs for deterministic browser regression coverage, or node scripts/e2e-dashboard-swarm-command.mjs --real to discover real local Fleet agents and run the actual /api/chat/agent-runtime path without interception. Add --require-completion when the test should fail unless at least one real runtime returns assistant text.

Collector And Link URLs

Hermes agents discovered through Fleet usually store a collector URL in agent.telemetryUrl. In system Tailnet mode, that may be a direct remote collector such as http://100.x.y.z:8787. In Hivemind Link mode, remote agents use the local Link sidecar proxy:

http://127.0.0.1:8788/peer/100.x.y.z%3A8787

That URL is still remote. Preserve the /peer/... prefix and port 8788 when sending chat, reading sessions, browsing files, or checking capabilities. Only plain local collector URLs such as http://127.0.0.1:8787 should be normalized to the active local collector port from ~/.hivemindos/collector.env.

If a remote Hermes agent fails immediately with “does not have the Hermes chat bridge” or a fast 404, check whether a Link /peer/... URL was accidentally rewritten to the collector port. The healthy path keeps /peer/... on 127.0.0.1:8788 and appends collector endpoints under that prefix.

Adaptive Agents

The first-class Adaptive provider routes each message across OpenRouter and configured Models.dev OpenAI-compatible providers instead of pinning one model. The older OpenRouter adaptive model remains supported for profiles that only need OpenRouter.

How a message picks its model:

• OpenRouter inventory is filtered for request modalities and free pricing. Configured Models.dev routes must be OpenAI-compatible and have a usable provider key.
• Every eligible candidate receives a normalized score from availability, endpoint and learned health, cost, latency, task fit, stability, privacy, context length, and accepted-outcome evidence. Unknown measurements stay visibly neutral instead of being invented. Balanced, Fast, Quality, Cheapest, Reliable, and Private first objectives change the factor weights.
• Best free excludes every paid candidate. Free then fallback still ranks and exhausts eligible free routes first, then tries only the fallback model explicitly entered in Agent Settings. HivemindOS no longer adds a default paid model.
• Optional cost and latency caps fail closed: a paid route with unknown pricing is excluded under a cost cap, and a route with unknown latency is excluded under a latency cap. Exploration is deterministic, restricted to under-tested candidates in the same free/paid tier, and capped at 20 percent.
• Each run records a receipt containing the selected route, top alternatives, all factor values and whether they were observed, constraint exclusions, and whether exploration actually ran.
• Failures classify from error text and HTTP status: capacity (429/rate limit/quota/overloaded) cools the route for 5 minutes doubling per consecutive hit (capped at 6 hours), unsupported models cool for 24 hours, and other transport failures for 15 minutes. Reliability is learned separately at model, provider, and provider-connection scope so a broken connection can protect sibling models without treating one unsupported model as a provider-wide failure.

Long-running tasks:

• The 45-second attempt timer only covers time to the first stream byte. Once the stream is flowing, silence means the Hermes CLI is still working, and the idle window is 15 minutes (the collector’s CLI chat cap is 20 minutes).
• The route emits a bare SSE comment every 25 seconds so the browser’s stall timer survives silent tool work.
• When an attempt fails after the Hermes CLI session was created, the next candidate resumes that same session (--resume) instead of restarting the task from scratch.

Quality gates (“completed but useless” still demotes):

• Every completed response is graded for dead-end refusals with no pivot into an alternative, an unrequested prompt echo, and repetition loops. Advanced settings can also require a minimum length, require literal text, or forbid literal text. These declarative checks do not evaluate regular expressions or code.
• A failed gate records a low-quality outcome and a Hermes Adaptive quality flag session event. The response still reaches the user — grading only changes future routing.
• One bad answer is forgiven; the second consecutive one benches the model for 10 minutes, doubling per repeat. Models under a 50% pass ratio across 3+ graded responses rank below models never tried. A single clean success redeems the model.
• Ordinary Adaptive chat now writes provider, concrete model, use case, acceptance, quality, measured latency, available cost, and privacy class into the existing outcome-routing store. Runtime heuristics carry less ranking weight than operator or evaluator evidence, and concurrent completions are serialized so one result cannot overwrite another.

Direct and Hermes Adaptive routes share the same machinery:

• The direct-API loop uses the scored candidate order and records every attempted route.
• When the plan selects Hermes, the same scored plan crosses the bridge intact and reuses Hermes failover, session resume, keepalives, and quality gates.
• The scoring and declarative-validation primitives are adapted from the MIT-licensed OmniRoute project, pinned at commit 50cb1875117bca9f053874ced924e54e40c43f69. HivemindOS does not install or execute OmniRoute; its complete license ships in the app’s third-party notices.

Fleet-wide sharing:

• The fleet shares one OpenRouter API key, so a rate limit observed on one machine applies everywhere. Collectors gossip the reliability store over the tailnet every 10 minutes (startReliabilitySync in scripts/agent-telemetry-collector.mjs), merging newest-wins per model id from each peer’s GET /reliability/openrouter. POST /reliability/sync triggers a pass manually.
• Peers are discovered from tailscale status --json, probed with a 4-second timeout, and deduplicated by machineId so each machine’s two tailnet nodes count once. Payloads carry model ids and counters only — no secrets, and no tailnet IPs are persisted.
• Tune or disable with AGENT_TELEMETRY_RELIABILITY_SYNC_INTERVAL_MS (0 disables). The app re-reads the store file on a 5-second TTL, so gossiped records apply without a restart.

Hive Fusion (Compound Model)

Hive Fusion is a native compound model: instead of asking one model, a prompt is fanned out to a panel of configured models in parallel, a judge model extracts the structure of their answers, and a synthesizer writes the final answer grounded in that analysis. It is HivemindOS’s own implementation of the panel → judge → synthesize pattern, provider-agnostic over the gateways you already have configured.

Selecting it: the model picker exposes a Hive Fusion provider whose model, hive-fusion-native, runs the local panel across your configured providers. OpenRouter’s hosted compound model is a plain OpenRouter slug, so it is offered separately as openrouter/fusion under the OpenRouter provider tile — a MODEL_PROVIDER_GATEWAYS entry keyed off OPENROUTER_API_KEY in the shared hive env, so it appears on every runtime (its model dropdown also offers Adaptive free-routing). It is served by the normal OpenRouter path — no Fusion orchestration runs locally for it. (Programmatically, agent.fusion.mode: "openrouter" also routes the native hive-fusion provider to the hosted model with optional custom-panel plugins.)

Any agent on any runtime can select Hive Fusion; the chat send path is unchanged (it still POSTs to /api/chat/agent-runtime). The route detects provider: "hive-fusion" and delegates to the Fusion orchestrator before the single-model preflight (isFusionProfile / streamFusionResponse in src/lib/services/fusion/route-stream.ts); selecting it never writes a Hermes/OpenClaw gateway model.

How a native run works (src/lib/services/fusion/orchestrator.ts):

1. Resolve the panel (catalog.ts). The default panel mirrors the budget-panel idea: when OPENROUTER_API_KEY is set, it fills with diverse budget models from OpenRouter’s live free inventory (reusing the Adaptive resolver, falling back to a static list if the inventory is unreachable), then tops up from any other configured provider. Providers are resolved from the shared hive env (OPENROUTER_API_KEY, VENICE_API_KEY, OPENAI_API_KEY, GROQ_API_KEY, BANKR_LLM_KEY, or a local OpenAI-compatible base URL). With only one provider configured it degrades to a low-diversity single-member panel (the judge + synthesizer still add structure) and surfaces a note rather than failing.
2. Fan out to every panel member in parallel via OpenAI-compatible /chat/completions (client.ts). A member that fails or times out is dropped; the run continues as long as at least one succeeds.
3. Judge (only when ≥2 members succeed): a judge model returns JSON capturing consensus, contradictions, partial coverage, unique insights, blind spots, and a recommended focus (prompts.ts). Malformed judge output falls back gracefully and never aborts the run.
4. Synthesize: a synthesizer model streams the final answer grounded in the analysis and the panel’s source answers. If synthesis itself fails, Fusion returns the strongest panel answer rather than erroring.

Blind compare: the Fusion service layer can now prepare anonymized answer slots through /api/fusion/blind-compare. The route hides model labels until the operator records a slot choice, then reveals the slot-to-model map so votes can feed future reliability and routing work without biasing the initial judgement. Fusion’s judge and synthesizer prompts also wrap panel/source answers as untrusted source data before using them for analysis.

Streaming: the synthesizer answer streams as the assistant message, while each panel member and the judge surface as tool/reasoning process events (the panel plan and judge analysis appear as reasoning chips). The orchestrator is dependency-injectable and covered by scripts/test-fusion.mjs (pnpm test:fusion).

Hive Compute (Marketplace GPU Inference)

Hive Compute supplies GPU-first routes inside the HivemindOS model provider’s unified All models catalog. Auto, Fast, and Deep appear before the other OpenRouter-backed HivemindOS models with SALE badges, try eligible marketplace workers first, and fall back to the matching hosted model tier when marketplace capacity is unavailable.

The dashboard also has a Hive Compute setup view for operators who want to earn on spare GPU capacity. It installs a small worker module under ~/.hivemindos/modules/hive-compute-worker; the worker connects to a compatible gateway, advertises Ollama or LM Studio/OpenAI-compatible model routes, accepts assigned jobs, streams tokens, and reports completion. Official matching, payout, receipt, quota, and fraud-control authority still belongs in hosted HivemindOS infrastructure or a self-hosted operator’s own gateway, not in the downloadable app.

For private marketplace jobs, Hive Compute supports encrypted prompt delivery to verified workers and opt-in output E2E encryption for clients that can decrypt response envelopes locally. Hardware-only routing is stricter than dev verified-only routing: it must be backed by gateway-verified hardware attestation and fails closed when only local test evidence is available.

Complete Capability Reference

• Runtime/provider/model selection where supported.
• Adaptive OpenRouter free-model routing with learned reliability, capacity failover, and quality grading.
• HivemindOS Models GPU-first routing through Hive Compute, with hosted OpenRouter fallback when marketplace capacity is unavailable.
• Streaming runtime responses where available.
• /swarm [number] role-specific parallel passes across the best-suited configured chat-capable agents.
• /swarm-goal <build request> prompt expansion plus Queen Bee Work Board delegation for parallel build tasks.
• Session resume and session search.
• Attachments and linked directories for task context.
• Send-to-Kanban from chat messages.
• Agent prompts for clarification, approval, secrets, or sudo-style decisions.
• Agent role, worker class, preferred skills, and per-agent env values.
• Dashboard agent calls that start through /api/phone using the gateway’s in-app voice path instead of ringing the phone.
• Scheduled/ring-agent calls that can ring the paired mobile device when explicitly triggered.
• AEON call briefings with repository, branch, workspace, Strategy/Soul, memory, the current skill catalog, chains/reactive work, and recent MiroShark deliverable context.

Chat document attachments use the bundled reader. See Documents And Brain Drop for supported formats, local-versus-cloud privacy behavior, extraction limits, and format-specific caveats.

For the full product split between free BYOK calls and paid Cloud/LiveKit rooms, see Calling.

Phone And Voice Bridge

Phone support is split between a settings surface and action routes:

• src/features/dashboard/views/PhonePanel.tsx manages saved prompts, scheduled call rows, device status, and ring actions.
• src/features/dashboard/views/chat/AgentCallsSettingsPanel.tsx exposes pairing and test-call controls in agent settings.
• /api/phone reads gateway voice config/device status, starts ring-agent calls, starts dashboard-agent-call calls, rings stored prompts, and checks mobile push readiness.
• src/lib/services/phone/call-gateway.ts builds private call briefings, including AEON-specific identity and recent artifact context.

Queen Bee Voice Chat

The desktop app ships a hands-free voice channel into the Queen Bee control plane:

• The HivemindOS menu bar (tray) icon menu leads with Voice Chat with Queen Bee; the app menu also exposes Navigation > Voice Chat with Queen Bee (Cmd+Shift+V). Both emit hivemindos:queen-bee-voice to the dashboard webview.
• While active, the dashboard shows an Apple Intelligence-style animated glow around the window perimeter (src/features/queen-voice/QueenVoiceGlow.tsx, ported from jacobamobin/AppleIntelligenceGlowEffect) plus live on-screen transcription of both the user’s utterances and Queen Bee’s replies.
• Each utterance is detected with an energy-based VAD, recorded, and sent to /api/queen-bee/voice, which transcribes it with the shared Whisper STT helpers (src/lib/services/phone/transcription.ts), submits the transcript to the Queen Bee control plane, and returns the spoken-ready receipt summary.
• Replies are voiced through OpenAI TTS when an OpenAI voice key is configured in the shared env, with on-device speech synthesis as the fallback.