← Back to docs index

The 07:05 Europe/Brussels daily press-review automation was interrupted by a gateway crash. At 07:07:35 V8 reported JavaScript heap exhaustion around the default 2 GiB old-space limit. Systemd rest…

Daily brief interruption — 16 November 2026

Confirmed cause

The 07:05 Europe/Brussels daily press-review automation was interrupted by a gateway crash. At 07:07:35 V8 reported JavaScript heap exhaustion around the default 2 GiB old-space limit. Systemd restarted the gateway at 07:09:36; the scheduler recorded the daily run as interrupted at 07:10:39 and retained the next regular run for Tuesday 17 November at 07:05. The failure notification also encountered agent database startup recovery. The independent 08:15 watchdog correctly reported missing artifacts rather than success.

Gateway, Discord and configuration checks subsequently passed. The press-review database passed integrity verification during startup recovery. OpenAI API embedding credit exhaustion is a separate issue; editorial and cover generation use subscription OAuth.

Remediation

A scoped user-service override at /root/.config/systemd/user/openclaw-gateway.service.d/20-runtime-resilience.conf sets the gateway Node process to --max-old-space-size=3872, TimeoutStopSec=330, and KillMode=mixed. This follows the installed gateway heap recommendation for this 7745 MiB host and allows graceful shutdown rather than killing pipeline children immediately. systemd-analyze --user verify passed and daemon-reload staged the configuration without interrupting recovery. A restart is required to activate the new heap setting; staging alone is not activation.

Recovery runs the canonical dated pipeline, verifies the language editions and cover before deployment, and checks receipts before any resend. Recovery evidence is retained in logs/recovery-2026-11-16.log. Do not rerun the all-channel publisher after partial success; reconcile each platform receipt and retry only missing outputs. Newsletter idempotency keys and recipient receipts must be preserved.

Verification required before calling recovery complete

Recovery outcome (verified 09:06)

The canonical publish pipeline finished at 08:37:22. The dated quality gate passes for EN/FR/NL (12 source links each). Saved live checks confirm HTTP 200 and correct edition for all three article URLs and dated cover. Newsletter provider accepted 13/13 sends (EN 8, FR 4, NL 1); the send-only Resend key cannot verify eventual mailbox delivery. Twitter and Nostr receipts were read back, LinkedIn confirmed sent with the correct dated image and no error, and the publisher recorded the Discord message receipt. Evidence: logs/recovery-live-checks-2026-11-16.json, logs/recovery-social-delivery-2026-11-16.json, logs/recovery-linkedin-delivery-2026-11-16.json, and logs/publish-2026-11-16.log. No all-channel rerun is needed.

BX1 connection recovery used IPv4; its collector now bounds connect time to 8 seconds and retries transient failures once. bash -n scripts/fetch-press.sh passes.