Node Runtime Health

What this doc answers

A health check answers one thing: can the process still respond (alive / dead, binary). It cannot answer the more insidious question — the process is alive, but Node is sick: memory leaking, GC thrashing (frequent, long pauses), the event loop dragged down, handles leaking. In all of these /health still returns 200 while the service degrades, until it OOMs or the timeouts cascade.

This doc covers runtime health — detecting “does Node have a problem”, not just “is it alive”. The approach is platform-agnostic; it applies to both Railway and self-hosted EC2, and it rides your existing pino → Alloy → Loki → Grafana pipeline rather than standing up a new one.

Two layers: health is not runtime state

Layer	Question	Mechanism	Catches a memory leak?
Liveness	Is the process alive	`/health` probe + container healthcheck	No
Runtime state	Is Node degrading	Runtime metrics + diagnostic report + profiling	Yes

/health currently returns just { ok: true } — on purpose: a liveness probe must be featherweight and not touch the DB, else a brief DB hiccup makes the load balancer declare every instance dead and restart them all. Detecting degradation is a separate mechanism, below.

Metric → problem table

“Does Node have a problem” isn’t a boolean — it’s a few trend lines. Each maps to a concrete failure:

Metric	Abnormal shape	The problem it signals
`heapUsed` / `rss`	Rises monotonically across restarts, never falls	Memory leak
GC pause duration / frequency	Climbing	GC pressure (the agent-loop’s known risk)
Event loop lag	p99 > 50–100ms sustained	Something blocks the event loop (sync IO / huge JSON / tight loop)
`activeHandles` / `activeRequests`	Monotonic rise	Handle / connection leak (sockets, files, timers not released)
`rss` nearing `mem_limit`, then exit 137	Sudden restart	OOM (killed by the kernel on cgroup limit)
Container restart count	> 0	Crash loop — pair with `uncaughtException` in the logs

Modeling rule: treat these as the primary signals of runtime state; everything else is a drill-down around them.

Layered approach

Ordered by benefit / cost for “detecting Node problems”:

Tier 0 — container healthcheck + memory bounds

Wired — docker-compose.yaml has the server (/health) and worker (heartbeat-file) healthchecks; memory bounds live in docker-compose.prod.yaml (mem_limit + NODE_OPTIONS=--max-old-space-size, starting points for a t3.large — tune per instance).

Add a healthcheck to server (hitting the existing /health) so docker restarts a process that’s wedged but not crashed (restart: unless-stopped only covers crashes, not hangs). Set a mem_limit per service and align Node’s --max-old-space-size just below it — V8 GCs before the cgroup OOM-kills, turning a memory problem from “mysterious restart” into “visible GC pressure in the logs”.

The worker has no HTTP endpoint, so its healthcheck uses a heartbeat file: each loop touches a file, and the healthcheck checks its mtime is fresh.

The alpine image has no curl — use wget -qO- ... or node -e "..." in the healthcheck command.

Tier 1 — runtime heartbeat log (no new infra)

Wired — apps/server/src/lib/runtime-heartbeat.ts, mounted in both server and worker; on by default in prod, opt in for local dev with RUNTIME_HEARTBEAT=true. The worker’s heartbeat also stamps a liveness file for the Tier 0 healthcheck.

A small module emitting a runtime.heartbeat every 30s (default) with { rssMb, heapUsedMb, heapTotalMb, eventLoopLagP99Ms, activeResources, uptimeS } (process.memoryUsage() + perf_hooks.monitorEventLoopDelay()). It flows through your existing pino → Loki pipeline; in Grafana, {event="runtime.heartbeat"} | json | unwrap heapUsed charts memory / lag trends. event is a low-cardinality label and the numbers are fields — exactly matching the cardinality rule. Immediate memory-trend and event-loop-lag visibility at near-zero cost.

Tier 2 — prom-client default metrics (the workhorse)

Not yet wired (next upgrade) — add when you want real time-series + declarative alerting; Tiers 0/1/3 cover day-to-day detection.

prom-client’s collectDefaultMetrics() gives you the whole table above out of the box — nodejs_heap_size_used_bytes, nodejs_gc_duration_seconds, nodejs_eventloop_lag_seconds, nodejs_active_handles, process_resident_memory_bytes, process_cpu_seconds_total. The app exposes /metrics; Alloy scrapes it via prometheus.scrape and pushes to Grafana Cloud metrics (free 10K series). This is the core answer to “does Node have a problem” — it lands the leak / GC / event-loop / handle signals in one shot, and it’s exactly the upgrade path the overview doc already describes: no Alloy swap needed.

// append to config.alloy
prometheus.scrape "zapvol" {
  targets    = [{ __address__ = "server:8001", __metrics_path__ = "/metrics" }]
  forward_to = [prometheus.remote_write.cloud.receiver]
}

prometheus.remote_write "cloud" {
  endpoint {
    url = "https://prometheus-prod-XX.grafana.net/api/prom/push"
    basic_auth {
      username = env("GRAFANA_CLOUD_PROM_USER")
      password = env("GRAFANA_CLOUD_PROM_TOKEN")
    }
  }
}

Tier 3 — process.report (post-mortem, built in)

Wired — the compose command already carries --report-on-fatalerror --report-on-signal --report-directory=/app/reports; reports write to the reports_data volume.

Node ships a diagnostic report: start with --report-on-fatalerror --report-on-signal --report-directory=/app/reports. On a fatal error, or on SIGUSR2, it auto-dumps a JSON — full heap state, all active handles, libuv event-loop state, native stack, environment. It answers “what did Node look like inside at the moment it broke” without attaching a debugger. Zero steady-state overhead (writes only when triggered). Mount report-dir to a volume and retrieve reports after an incident.

Optional deepening

Pyroscope (Grafana’s, self-hostable) — metrics tell you “memory is rising”, Pyroscope’s continuous flame graphs tell you which function it’s rising in. Use it to locate a leak / CPU hotspot. For ad-hoc work, --heap-prof / --cpu-prof produce a one-off snapshot.
Sentry — your uncaughtException goes to Loki today, but without grouping / alerting / source-mapped stacks. Sentry (generous free tier, self-hostable) fills that gap, complementing Loki rather than replacing it.

Performance cost

The five recommended always-on items all cost effectively nothing — they’re production standard. The only one to rein in is continuous profiling.

Approach	Steady-state cost	Notes
Container healthcheck	Negligible	Spawns a probe process every ~30s; on alpine use `wget` / `node -e` instead of `curl`
`mem_limit` + `--max-old-space-size`	Negligible (slightly more GC)	V8 GCs earlier near the limit — that’s the point: controlled GC instead of uncontrolled OOM
`runtime.heartbeat` log	Negligible	One `memoryUsage()` + one log line per 30–60s; lag measurement is a libuv C++ timer
prom-client default metrics	< 1% CPU / a few MB	Low-frequency sampling + a few KB serialized per scrape; GC via `PerformanceObserver`. Standard in production everywhere
`process.report`	0 (only when triggered)	Dumps only on fatal error / signal; zero steady state
Pyroscope continuous profiling	1–5% CPU	The only non-trivial cost; tune the sample rate, or enable only while investigating
Sentry	Errors ≈ 0; tracing scales with sample rate	Error-only mode is negligible; keep `tracesSampleRate` low

So on selection: Tiers 0–3 are safe to run always; profiling is enabled on demand.

Example alert rules

Once you have metrics, let degradation notify you instead of you watching a dashboard (Grafana Alerting over Prometheus metrics):

Alert	Expression (illustrative)	The problem
Heap nearing the ceiling	`nodejs_heap_size_used_bytes / nodejs_heap_size_total_bytes > 0.9` (for 10min)	Leak / pressure
Event loop stalling	`nodejs_eventloop_lag_p99_seconds > 0.1` (for 5min)	Loop blocked
Handle leak	`deriv(nodejs_active_handles[30m]) > 0` staying positive	Handle leak
Crash loop	`increase(process_start_time_seconds[15m])` changing > 2 times	Repeated restarts

Before Prometheus, the first two can be approximated with LogQL over runtime.heartbeat (unwrap + threshold).

When to add which tier

Following the overview’s upgrade philosophy:

Start — Tier 0 (healthcheck + memory bounds) + Tier 3 (process.report) — pure config, covers hangs and forensics first.
Want memory trends — add Tier 1 heartbeat log — no infra, same-day.
Want real time-series + declarative alerting — go Tier 2 prom-client + Alloy scrape — the complete “does Node have a problem” answer.
Need function-level locating — then add Pyroscope.