Architecture

vmux has two local, cooperating pipelines. The terminal pipeline selects one provider (tmux or one explicit Herdr session), captures its endpoints, and publishes PaneState. The tmux-only agent-context pipeline observes supported runtime logs and publishes structured, resumable session state. Terminal state remains the fallback and the source of live prompt verification.

tmux ───────┐
            ├── TerminalProvider ──▶ poller ── detectors ──▶ PaneState ── REST + /ws
Herdr ──────┘         ▲                                      │
 optional events ─ wake only                                 │
                     │                                       │
                     └── exact route + prompt guards ◀── POST /api/input

Codex / Claude logs (tmux only) ──▶ AgentService ──▶ SQLite ── REST + /ws/agents

Runtime components

CLI and configuration

vmux.__main__ parses CLI overrides, loads YAML plus the JSON overlay, validates the bind/token boundary and selected terminal provider, and starts Uvicorn. vmux disables tmux automatic-rename by default unless configuration opts out. Herdr startup probes but never starts, reloads, or otherwise owns the configured session/server.

Terminal providers

terminals/base.py defines normalized endpoint, hierarchy, native state, capability, health, capture, and guarded-action contracts. TmuxProvider wraps the existing safe tmux.py module and preserves %N ids, target strings, legacy action fallbacks, and command ordering.

HerdrProvider pins one executable and one explicit session at startup. Every session-scoped argv ends with --session <name>. Discovery validates one atomic snapshot and exact workspace/tab/pane/terminal/agent relationships. Opaque public handles map only to current private routes; labels are display-only. A failed pass is non-authoritative, so the poller retains the last snapshot as stale/read-only instead of inferring deletion. Herdr 0.8.2 route revisions do not advance for terminal output, so captures are never cached by revision; vmux requests at least 200 rows and trims locally.

TerminalActionService keeps old routes tmux-only. Guarded Herdr input resolves one current registry entry, compares client guards, reads a fresh detection buffer between two exact route snapshots, reparses the prompt/options, and then issues the requested text and/or keys under a per-endpoint lock. Idempotency is reserved before I/O and uncertain sends are never replayed. Herdr's protocol has no atomic compare-and-send; see the guarded-input contract for operation guards, delivery outcomes, and the remaining local-client race.

HerdrEventSubscriber checks the configured session socket's identity, ownership, type, and write permissions, then subscribes to native status changes. An event can only wake the normal poller; it never patches state or triggers input. Polling remains active through all event failures.

tmux adapter

tmux.py invokes the tmux executable with argument lists. It lists panes, captures joined scrollback, checks pane identifiers, sends allow-listed named keys, and sends user text in literal mode.

Poller and detector

Hub.poll_once() discovers provider endpoints and captures them concurrently. For each included pane it:

1. hashes captured text and records whether it changed
2. classifies the process as claude-code, codex, grok, opencode, antigravity, generic, or shell
3. detects needs_input, error, working, or idle
4. resolves a display name and manual override
5. builds a PaneState

Herdr native status and hierarchy enrich the compatibility result without becoming action authority. Native blocked maps to needs_input; terminal questions/errors retain precedence; native working maps to working; native idle/done map to compatibility idle while the original status remains visible. Configured targets that are absent become offline states. The loop runs every poll_interval, 0.7 seconds by default, and an action wakes it for an immediate new pass.

Detector matching is bounded. Claude Code has a dedicated selection-box path. Codex has a dedicated questionnaire parser that separates aligned labels and descriptions; legacy approvals and other agents retain the conservative generic numbered-menu and regex paths.

Agent context subsystem

AgentService is deliberately separate from the pane state model. The poller passes it small live pane observations through a latest-wins queue, so log extraction cannot delay terminal polling. Runtime observers discover and tail Codex and Claude Code JSONL sessions read-only. They normalize an allowlist of user-visible messages, explicit plans/tasks, explicit structured questions, and lifecycle events; hidden reasoning, tool arguments/results, arbitrary events, and captured terminal output are discarded at the adapter boundary.

The projector builds a canonical context and a semantic delta. AgentStore commits context, new snapshots, visible messages, decision candidates, and resolution events atomically to a local SQLite database. Historical data is pruned after 30 days by default; current context and unresolved decisions are kept. The legacy SessionVisit and v2 SessionReview records are separate server-side monotonic baselines. Review reads are non-mutating; only an acknowledgement naming an exact displayed snapshot advances SessionReview.

AgentService.review_payload() joins deterministic SQLite groups to privacy-minimized current pane references. It ranks explicit urgent decisions, errors, oldest decisions, new blockers, then other semantic changes. A singleton review-settings row provides the optional persisted timer. Due windows are claimed transactionally before one generic APNs digest and one review_due invalidation are published.

Association and control are distinct. A probable or ambiguous log/pane match is readable but cannot receive input. Chat requires a confirmed current pane incarnation, matching binding revision, unchanged prompt fingerprint, and a live idle prompt. Decision replies additionally require a structured question whose options still match the live menu. The server revalidates immediately before a literal tmux action and uses client idempotency keys to prevent replay.

/ws/agents carries small invalidation events with a resumable cursor; agent clients rehydrate authoritative objects over REST. It is separate from /ws, so a parser or agent-protocol failure cannot make terminal monitoring appear offline.

Server and PWA

FastAPI exposes authenticated REST actions and a WebSocket. Every WebSocket tick sends a full state snapshot, not a patch. The server then mounts vmux/web/ at the root for same-origin static delivery.

The additive POST /api/images path is separate from pane state and static delivery. ImageStore streams authenticated raw bodies into private partial files under ~/.vmux/uploads, enforces signature/type/size/quota boundaries, then atomically renames a valid image. Startup, hourly, and pre-quota cleanup remove files after 24 hours. The response exposes an absolute path and a shell-quoted form for the existing literal-text composers; there is no image download route and no new WebSocket frame.

The PWA keeps the no-build model but is no longer one HTML implementation file. index.html is a minimal shell; styles.css owns the visual system; and native ES modules divide runtime primitives, state/transport, shared UI, usage, settings, image upload, and application composition. React, ReactDOM, and htm remain vendored same-origin files, and the Lucide subset is a vendored SVG sprite with its license notice.

The client first reads /api/config and /api/state, evaluates compatibility, normalizes the snapshot, and only then adopts the workspace. Successive full snapshots reuse unchanged pane objects so terminal scroll state and unaffected cards survive updates. Unknown statuses and agent kinds receive neutral generic presentation instead of being omitted.

Connection recovery has explicit states: Connecting, Live, Updating via REST, Offline, Unauthorized, and Incompatible. A live socket is considered silent after max(3 seconds, poll_interval × 2 + 1 second). REST fallback runs every 2 seconds while needed. WebSocket retries start at 0.5 seconds, double to an 8-second cap, and add up to 0.3 seconds of jitter. A failure must persist for 10 seconds before the UI changes from Connecting to Offline. The last snapshot stays available for read-only inspection throughout recovery.

Layout depends only on viewport width:

• below 820 pixels, Queue, Active, All, and Stats use a four-item bottom dock; pane detail and the tree use focus-managed full-height sheets
• from 820 through 1199 pixels, a 360-pixel master column sits beside pane detail; Stats replaces the split workspace and the tree is a leading drawer
• at 1200 pixels and above, the navigator, attention queue, and inspector form a three-column workspace; Stats retains the navigator and spans the other two columns

Browser preferences, snippets, shortcuts, terminal wrapping, and the token are stored locally in that browser profile. Plan Review additionally persists only server/decision/option ids, revisions, and opaque prompt/options fingerprints; authoritative refreshes purge resolved, missing, or changed drafts. Pane snapshots, terminal output, agent responses, prompt and option copy, action state, and other pending work stay in browser memory only. The server-side normalized agent history is the separate SQLite store described above.

The service worker has an explicit application-shell allowlist. It never handles API or WebSocket traffic and never caches query-bearing or authorized requests. Navigation can fall back to the canonical cached shell after a short network timeout; missing modules and images never receive the HTML fallback. A waiting worker activates only after the visible update prompt asks it to do so.

Optional subsystems

• usage.py invokes an explicitly configured tokscale command and normalizes quota/history output. The PWA renders that data as quota meters, summary cards, local SVG charts, and an equivalent table. Before each serialized report pass it performs a bounded, best-effort Antigravity sync; quota refresh remains independent. It is disabled by default.
• push.py stores registered device tokens locally and can send APNs alerts when optional dependencies and credentials are present. Pane and agent decision alerts use generic copy; agent decision routing carries only opaque identifiers and a revision. Scheduled Review digests are also generic and contain only an event type and opaque server id.
• naming.py supplies local naming heuristics and an opt-in AI naming layer.
• creation.py owns canonical root authorization, bounded directory browsing, runtime availability, serialized automatic naming, and detached tmux target creation. Clients select only an allowed runtime ID.

Failures in these optional paths are designed not to terminate the pane polling loop.

Configuration model

The Settings API reads and writes only a validated subset of fields. It persists the complete editable subset to a JSON overlay:

built-in defaults < YAML < JSON overlay < CLI overrides

The overlay never rewrites YAML. Bind, token, terminal provider/Herdr authority, tmux auto-rename, APNs credentials, usage.command, and AI backend settings stay YAML/CLI-only. Creation roots and runtime arrays are also YAML-only.

Trust boundaries

terminal boundary

An authenticated client can intentionally cause input to be sent to a pane. That pane may be a shell or an agent capable of running commands as the vmux OS user. The bearer token therefore authorizes a high-impact capability. When tmux creation is enabled, it additionally authorizes in-root directory browsing and detached process creation as that same OS user. Herdr creation, deletion, focus, move, rename, resize, broadcast, agent start, and all server/session lifecycle operations are absent from product code.

network boundary

vmux serves plain HTTP. Localhost is the default; Tailscale is the recommended remote route. Non-loopback binds require a token. Public access additionally requires HTTPS/WSS termination while the vmux listener remains private.

The WebSocket token is a query parameter and can enter proxy logs. Reverse proxies must suppress or redact it.

content boundary

Captured pane output may contain secrets or private source. It is sent to every authenticated client and stored in browser memory. Selected recent lines can leave the host through explicitly enabled AI naming. Pane and agent-decision APNs copy is generic; decision titles, descriptions, prompts, and options remain on the vmux server. Those network features are off or unconfigured by default.

Supported runtime logs can contain the same sensitive material plus internal runtime records. Agent observers read those files with the vmux OS user's permissions, but persist only the normalized allowlist described above. The database and its containing directory are permission-restricted where the host supports POSIX modes. The authenticated agent APIs expose normalized state to every bearer-token holder; there is no per-session authorization layer.

Terminal output is untrusted text. The PWA interpolates it into React text nodes and never uses innerHTML or HTML injection for captured content. Link tools extract only HTTP(S) strings from that text; they do not make terminal output executable markup.

Security invariants

Changes must preserve all of these:

• subprocesses receive argument lists; terminal actions never construct a shell command
• every Herdr command is routed to the one configured session; no ambient focus or label resolves an action
• Herdr legacy mutation routes fail closed; guarded input requires current opaque identity plus fresh prompt/options verification
• Herdr events are wake-only, malformed discovery retains stale read-only state, and uncertain input is never replayed
• Herdr literal text rejects terminal control bytes so submission remains a separate Enter operation
• live pane ids and configured targets are format-checked before tmux use
• named keys are allow-listed
• text uses tmux send-keys -l -- so it remains literal
• agent control requires a current confirmed pane incarnation and matching binding revision; decisions also require a matching structured prompt fingerprint and revision
• Review GET/open/history actions never acknowledge or answer work; review acknowledgements advance an exact displayed snapshot monotonically
• staged decision choices contain metadata only and are refetched, compared, and submitted sequentially through individual guarded replies
• terminal-review responses and scheduled digest payloads exclude prompt, transcript, path, menu, preview, and terminal-capture content
• unverified decision candidates and runtime log paths never enter public API responses
• REST and WebSocket bearer comparisons use hmac.compare_digest
• a non-loopback bind with an empty token fails before serving
• public documentation never presents bare HTTP as safe for public exposure
• custom detector regex input and execution time are bounded
• API config edits cannot replace the bearer token, executable usage command, APNs credentials, or AI backend configuration
• overlays, naming caches, and push registries never rewrite config.yaml
• the token is never included in an API response
• vendored PWA assets remain same-origin
• an incoming ?token= is removed from browser history immediately after local persistence; all other query parameters remain intact
• sign-out clears browser credentials and purges legacy credential-bearing cache entries
• the service worker never caches API, WebSocket, authorized, or query-bearing requests, and never persists pane data
• captured terminal content remains plain text; React style properties remain objects rather than strings

Compatibility surface

The documented CLI flags and exit categories, YAML/overlay semantics, REST and WebSocket contract, PaneState shape, and the security invariants above form the public v0.x compatibility surface. See Compatibility and versioning before changing one.