What survives the restart?
Saving an agent’s work is the easy half. The hard half is who wakes it up when nobody has the tab open.
Suppose you ask an agent to generate a mushroom texture. It accepts the job, starts a tool call, and the process disappears before the result comes back.
You reopen the conversation. Your request is still there. That tells you what you asked for. It doesn’t tell you whether the texture was made, whether trying again would make it twice, or whether anything will happen if you leave the tab closed.
Those are separate problems. The retained Watchdog history makes that painfully concrete.
The watcher got stuck too
The old system had a separate service watching the agent over a WebSocket. It sent pings, watched for silence and tried to get stalled work moving again. That was WatchdogDO, the sidecar.
The retained postmortem describes a reconnect path that set its own closed
flag, then called code that refused to reconnect while closed. Asking it to
watch again didn’t clear the flag. The thing watching for stuck agents could get
stuck itself.
The replacement put recovery inside AgentDO, alongside the run state it needed to inspect. Current Watchdog stores jobs beside the owner’s other records. It asks the host whether a running job is still alive. A confirmed no lets it retry within the job’s recovery allowance, or mark the job interrupted. Silence alone isn’t that answer.
For our texture example, that gives the restarted agent a job to inspect. It still needs to check what the tool actually did. A tool could have saved the image before its reply was lost. Marking a job complete cannot make that external action happen exactly once. The tool needs a way to recognize the same request and return the saved result.
Who wakes Shiitake?
This is the part we went back to the source for. Removing the sidecar did not remove the alarm.
Retained AgentDO had an AlarmClock connected to Cloudflare’s persistent alarm storage. Work and mail supplied their next wake times alongside cron. The clock picked the earliest, and AgentDO’s alarm handler called back into the owners to check their work. The wake didn’t depend on your browser staying open.
The Cloudflare adapter now commits its SQL changes and persistent alarm together. Watchdog still decides what needs attention. The host makes sure it gets another turn.
We tested that locally by killing the process, then sending zero requests to the target. The alarm woke a fresh instance and queued work ran. Work that had been running in the vanished process was marked interrupted.
So the wake survived that restart without anyone nudging it. Hosted rollout is still separate work. So is safely retrying a tool that might already have made your texture before its reply disappeared.