The first time my agent crashed at 3 AM, I found out at 9. Error log full of cryptic failures. No diagnosis. No recovery. Just a dead process and a morning wasted debugging.
That was April. By May, the same agent hit 196 errors in a single week — timeouts cascading into corrupted sessions, corrupted sessions cascading into failed jobs. The whole system felt fragile in a way that contradicted everything I was building toward.
Here's what I realized: giving an agent autonomy without giving it resilience is like giving someone car keys without brakes.
The Autonomy Paradox
We want agents to run unattended — background jobs at midnight, scheduled digests at dawn, multi-step research while we sleep. That's the promise. But every new capability introduces a new failure mode. More tools. More integrations. More ways to break.
The standard playbook is to add monitoring. Alerts. Dashboards. Someone watching.
But that defeats the purpose. If I have to babysit my autonomous agent, it isn't autonomous.
I needed something different: an agent that could detect its own failures, understand why they happened, and fix itself — with my permission, not my constant attention.
Three Levels of Self-Healing
I built AutoHeal as an escalation system. Not every failure deserves the same response. A network blip doesn't need root cause analysis. A recurring crash does.
Level 1 — Recover. Process crashed? Restart. API timed out? Retry with backoff. Context window full? Compact and continue. No analysis, no delay — just get back on your feet. This handles 80% of failures because most are transient.
Level 2 — Diagnose. When the same failure keeps recurring, recovery isn't enough. Once a week, AutoHeal invokes a separate Claude session that reads the structured error log, traces causal chains, and produces a diagnosis: "Here's the root cause. Here's why it cascades. Here's what would fix it." Plain language. Proposed patch. Waiting for my approval.
Level 3 — Repair. Apply the fix. Git backup first. Patch the code. Restart. Verify. If the health check fails — automatic rollback, no questions asked. If it passes — commit, notify me, move on.
The escalation mirrors trust: recover without asking, diagnose and present findings, repair only with explicit consent.
The Insight That Changed Everything
The first time AutoHeal ran its diagnosis, I expected it to say: "You had 98 timeout errors. Fix timeouts."
It didn't. Instead:
"One root cause — max_iterations hitting the tool loop limit on complex tasks — is generating orphaned tool_use blocks. Those orphaned blocks corrupt subsequent sessions. Corrupted sessions fail scheduled jobs. Fix the iteration budgeting and the orphaned block cleanup, and 70-80% of downstream errors disappear."
One cause. Dozens of symptoms.
That's the key: causal reasoning, not frequency counting. A naive system counts errors and fixes the most common one. But symptoms recur until their cause is addressed. AutoHeal traces the chain backward:
Root Cause (1x)
└─► Immediate Effect (2-3x)
└─► Cascade Failures (5-10x)
└─► Symptoms You See in the Log (20-50x)Fix the root, and the cascade vanishes. Not by patching 50 symptoms. By understanding one cause.
The Results
| Week | Errors | What Happened |
|---|---|---|
| Apr 17 | 98 | First diagnosis — 4 fixes proposed |
| Apr 24 | 196 | Worst week. Max iterations cascading. |
| May 1 | 142 | Root causes identified. Fixes applied. |
| May 8 | 0 | Silence. |
| May 29 | 0 | 21-day streak. |
| Today | 0 | 42+ days. Still counting. |
Eleven fixes total. That's it. Eleven patches turned a system that broke weekly into one that hasn't produced a single error in over six weeks.
The system survived my two-and-a-half week absence (international trip — no monitoring, no intervention). It survived new tools being added, new integrations connected, usage patterns swinging from heavy to zero and back. It just kept running.
How It Runs
No cloud infrastructure. No Kubernetes. No observability platform.
One MacBook. One process manager (pm2 handles Level 1 restarts). One weekly cron job that invokes Claude Code CLI for the diagnosis. A structured error log (JSONL — every error typed, timestamped, with context about what the user was doing). A folder of JSON reports and fix records.
That's the entire stack for self-healing on personal hardware.
The error log is the critical piece. Every failure gets classified: root cause or symptom. Every error captures what triggered it, what session it affected, what recovery was attempted. Without structured logging, diagnosis is guesswork. With it, Claude can trace the causal chain in seconds.
The Consent Architecture
AutoHeal never turns itself on. The escalation of capability requires escalation of trust:
After five significant errors accumulate, the agent asks — once — "Want me to start analyzing error patterns weekly?" Yes or no. Answer stored, never asked again.
After the first diagnosis produces proposed fixes, it asks again: "Want me to apply approved fixes automatically?" Only then does Level 3 activate.
And at any point: "disable self-surgery." Immediate. No argument. No persistence.
I think this matters more than the technical architecture. Autonomy without consent isn't a feature. It's a liability.
What I'd Build Differently
The diagnosis runs weekly. That's too slow for a system with acute failures — by the time Friday rolls around, I might have already debugged the issue manually. A trigger-based diagnosis (run immediately after N recurring failures of the same type) would close that gap.
The causal chain analysis is good but not perfect. It relies on temporal proximity in the log — "root cause happened at T, symptoms appeared at T+1 through T+5." That works when causality is temporal. It fails when the cause is latent (a config change that only manifests under load three days later).
And the repair level needs more sophisticated verification. Right now, "health check passes" means "the gateway starts and responds." It doesn't mean "the specific failure scenario that triggered the fix no longer reproduces." Regression tests per fix would be the real signal.
The Broader Pattern
Every agent system that runs unattended will need something like this. The specifics differ — your error log format, your diagnosis engine, your deployment model. But the pattern is universal:
Escalate response proportionally to failure severity. Trace causes, not symptoms. Earn trust before acting. Roll back when uncertain.
The agent that heals itself is the agent you can actually trust to run alone.
And that's the whole point of autonomy, isn't it?