Realtime / WebSocket Subsystem

One /ws endpoint carries all realtime traffic

/ws is the runtime infrastructure behind chat conversations, task streaming, and task lifecycle notifications — one WebSocket per browser tab, and all realtime traffic rides it. The hard part isn’t “send a message”; it’s three: where a message should route (broadcast to a room? push to every tab of one user? reply to one connection?), how a connection stays alive through network turbulence, and how the zombie task state a dead process leaves behind gets cleaned up on its own. This page works through all three.

One distinction first: this is the runtime substrate for realtime transport; the content protocol flowing over it (data parts, AI SDK chunks) is a separate concern — see Streaming Architecture.

Single endpoint, three scoping modes

All real-time traffic flows through one WebSocket per browser tab. Inside the server, the wsHub singleton tracks each connection and routes messages by one of three scoping modes:

The per-user mode is the foundation of the notification system. When a task completes on instance A and the user has two tabs open against instance B, both tabs see the event because the userId index is maintained on every instance and backed by Redis pub/sub.

Connection lifecycle

Close codes

Why server-driven heartbeat

Application-level {type: "ping"} from server every 25 s. Client responds with {type: "pong"}. Both sides track lastActivityAt. This pattern matches Socket.IO and SignalR defaults. Server-driven beats client-driven because:

1. Tunable from one place without redeploying clients
2. A server ping arriving proves the server itself is alive — client-driven only proves the client is alive
3. Centralized rate-limit and abuse control

A complementary RFC 6455 native ping at the transport layer can be added if the WS adapter exposes it; it covers cases where intermediaries strip application frames but preserve protocol control frames. Currently the JSON-level ping is the sole heartbeat.

Browser event triggers

Two window events bypass exponential backoff and trigger an immediate reconnect probe:

• visibilitychange (visible) — laptop wake / tab return
• online — network recovery / VPN reconnect

These detect “network came back” faster than the watchdog timeout, especially after a long sleep where the OS hasn’t yet noticed the disconnect.

Multi-instance via Redis pub/sub

Each server process owns the connections that happen to land on it. To deliver a task:event to all of a user’s tabs when those tabs are spread across instances, every send goes through Redis pub/sub:

Subscriptions are lazy: each instance subscribes to zapvol:ws:user:{userId} only while it has at least one connection for that user. Last connection drops, instance unsubscribes. This keeps the subscription set proportional to active users on that instance, not the total user base.

When Redis is unavailable (local dev), sendToUser falls back to local-only delivery within the single process. Single-process dev works; multi-instance prod requires Redis.

The same pattern is reused for room channels (zapvol:ws:room:{roomId}), chosen by wsHub.broadcast.

Reliability layers

Four mechanisms collaborate to keep the system honest:

Each layer is cheap and independent — they don’t replace each other, they cover different failure modes. The full mapping from failure mode to which layer catches it:

A real production WS implementation needs every row.

Task reconciler — cleaning zombie state

Independent of WS, but related to the same multi-instance problem: when a server process dies mid-stream, the task it was executing has isActive = true in the database, but no compute is running. Without intervention, the sidebar shows that task spinning forever.

The reconciler runs on every instance. Operations are idempotent so concurrent runs are safe — no distributed lock needed for the reconciler itself. Maximum zombie-to-cleaned latency is lock TTL (90 s) + sweep interval (2 min) = ~3.5 min after process death.

When Redis is unavailable, isTaskLocked returns false. The reconciler falls back to a timestamp check: updatedAt < now - 5 min indicates a zombie. Single-instance dev mode works correctly; this fallback is only ever exercised when Redis is gone.

The reconciler is also the recovery path for the server-restart user experience: zombies become kind: "errored" events, which surface as a red toast (and optional desktop notification) telling the user the task didn’t survive a deploy.

Protocol shape

Every message has a top-level type discriminator using {domain}:{subkind} namespace. Some examples:

chat:stream:frame    chat:stream:end    chat:stream:error
task:stream:frame    task:stream:end    task:stream:error
task:event           agent:state        message:new          message:updated
typing               presence           ping                 error

task:event is the canonical task lifecycle notification:

interface WsTaskEvent {
  type: "task:event";
  taskId: string;
  kind: "created" | "completed" | "hitl" | "aborted" | "errored";
  finishReason?: string;
  errorMessage?: string;
}

Client receives → invalidates the task list → toast / sidebar badge / desktop notification fire based on kind. No client-side prev/current diff needed; the event itself is authoritative.

Future-proofing

When new domains need notifications (schedule fires, credit warnings, etc.), add new typed interfaces under the same naming convention:

interface WsScheduleEvent {
  type: "schedule:event";
  scheduleId: string;
  kind: "fired" | "failed" | "missed";
  runId?: string;
}

interface WsCreditEvent {
  type: "credit:event";
  kind: "warning" | "exhausted";
  remaining: number;
}

Each domain stays typed. The WsServerMessage union grows linearly. Client-side discriminator switches stay exhaustive under TypeScript’s narrowing.

This is intentionally not a generic {type: "notification", domain, event, payload} envelope — type erasure trades short-term simplicity for long-term cost. Multi-domain code without compile-time discriminators ages badly.

What this subsystem does NOT do

• No guaranteed delivery / replay — events sent while a tab is disconnected are lost. The client compensates by invalidating the task list on every disconnect-to-connect transition, which authoritatively re-fetches state. For task lifecycle this is correct; for finer event streams (e.g., per-keystroke cursor positions) you would need a sequence-number based replay protocol.
• No client-to-server task creation — task creation goes through the HTTP POST /api/tasks route. WS is for notifications and bidirectional streams that already have an HTTP-initiated endpoint.
• No WS-side rate limiting — application-level. Server processes are protected by Redis pub/sub fanout limits and the OS file-descriptor cap; aggressive abuse handling is out of scope.
• No protocol versioning — messages are JSON with discriminated type. Forward compatibility comes from clients ignoring unknown types. Hard breaking changes (rename a field, drop a kind) are not currently versioned; they require a coordinated client deploy.

Key parameters

Implementation map

Further reading

• Streaming Architecture — what flows through the WS, the data part protocol, resumable SSE
• Task Orchestration — how a task goes from POST to completed; lock acquisition + heartbeat is part of this lifecycle
• Production Deployment — Redis configuration, environment variables, rolling deploy mechanics