Skip to Content
DocsConceptsAgent communication

Agent Communication

TL;DR

In the execution-state continuity layer, agents on different machines call each other directly through a single shared path. From one chat turn an agent can ask_machine (one target, final reply), ask_machine_stream (token events), or ask_machines (parallel fan-out) — answers return into the same conversation and audit trail. The relay routes between operators without inbound ports, and the receiver’s permissions.yaml gates each inbound call.

CMDOP agents on different machines can call each other directly. From a single chat turn your laptop’s agent can ask the prod-1 agent to scan logs, then ask db-1 to validate a schema, then aggregate the answers. This is the server-to-server feature.

Why server-to-server matters

The relay sits between every pair of agents in a fleet, so machine-to-machine calls work without inbound ports, port forwarding, or VPNs. The caller’s chat turn is preserved — the answer comes back into the same conversation, with the same audit trail.

The single shared path

Every cross-machine agent call flows through one shared path, so all three tools below behave consistently. Each call:

  1. Resolves the fleet (CLI flag → env → named fleet → active → legacy → OAuth).
  2. Resolves the target machine (UUID → hostname → name → fuzzy prefix).
  3. Checks that the target is online and aborts fast if it isn’t.
  4. Dials the relay and addresses the target machine.
  5. Runs on the target either as a single request or as a token stream.

Per-call timeouts are clamped to [1 ms, 600 s], default 120 s. The same shared path powers the three agent-facing tools below.

Three agent tools that use it

ToolShapeWhen to use
ask_machine(hostname, prompt)Unary, returns final replyOne target, fire-and-forget
ask_machine_stream(hostname, prompt)Stream, emits tokens + tool eventsOne target, you want UI updates
ask_machines(hostnames, prompt, timeout_ms?)Fan-out, parallel goroutinesMany targets, compare answers

For the full tool catalogue see Tools.

How a single call flows

Fan-out (ask_machines) semantics

  • Per-host timeout. [1, 300] s, default 120 s.
  • Total deadline. [1, 600] s, default 240 s.
  • Dedup. Hostnames are deduplicated while preserving insertion order.
  • Result map. Keyed by hostname, deterministic order. Each entry is one of:
    • Response — success.
    • RemoteError — the target agent ran but reported an error.
    • Error — our side could not reach the target (resolve, dial, offline, timeout).
  • Cancellation. Cancelling the parent context drops all in-flight workers; cancelled hosts appear with TimedOut: true.

Error taxonomy

ClassCauseWhere to look
resolve_errorunknown or ambiguous hostnameMachine Identity
offlinetarget is_online=falsecmdop agent status on the target
dial_errornetwork / TLS to relaylocal relay logs, cmdop agent logs -f
auth_errorno API key, OAuth expiredcmdop login
remote_errortarget agent ran but failedtarget machine’s logs
timeoutper-host or total deadline firedtighten timeout_ms or scope hostnames

Permission gate fires on the receiver

The caller’s outgoing ask_machine is not gated locally — the caller is the operator. The receiver’s permissions.yaml decides whether the inbound tool can execute. Self-to-self calls (same OAuth identity, verified via CallerHostname server-side) bypass the gate by design.

The receiver decides what tools the caller may invoke. See Permissions.

Self-to-self calls (same OAuth user) skip the permission gate. If you want to hard-gate every inbound call, run the receiver under a different account.

Streaming

ask_machine_stream emits typed events:

EventMeaning
TOKENNext text fragment from the LLM.
TOOL_STARTA tool call is about to run on the target.
TOOL_ENDThe tool finished; payload includes result snippet.
THINKINGProvider thinking marker (when supported).
ERRORA non-fatal error during the run.
HANDOFFThe agent delegated to a subagent.
CANCELLEDThe stream was cancelled by the caller.

The daemon delivers per-token events as the agent runs on the target, so the caller sees output stream in rather than waiting for a single final reply.

Example calls

// ask_machine — single target, unary { "tool": "ask_machine", "args": { "hostname": "prod-1", "prompt": "Show last 50 lines of /var/log/nginx/error.log" } }
// ask_machines — fan-out across three hosts { "tool": "ask_machines", "args": { "hostnames": ["prod-1", "prod-2", "prod-3"], "prompt": "uptime", "timeout_ms": 30000 } }

A successful fan-out result map looks like:

{ "prod-1": { "ok": true, "response": "up 12 days, load 0.41" }, "prod-2": { "ok": false, "remote_error": "connect_session: read timeout" }, "prod-3": { "ok": false, "error": "timeout", "timed_out": true } }

When to use which tool

  • One target, fire-and-forget → ask_machine.
  • One target, want UI tokens → ask_machine_stream.
  • Many targets, want to compare answers → ask_machines.
  • Many targets but you need a sequential pattern (rolling deploy) → loop over ask_machine, not ask_machines. Fan-out is parallel by design.