Skip to main content

How subagent sessions cascaded into a three-and-a-half-hour, 3,486-firing wiki-update pileup

Updated 10 min readMarkdown
How subagent sessions cascaded into a three-and-a-half-hour, 3,486-firing wiki-update pileup

Every time I close a Claude Code session, a hook fires. It reads the session's transcript and decides whether anything in it is worth keeping. If so, it spawns a headless Claude process that distils it into my wiki: decisions extracted, gotchas filed. The result cross-links with everything else I've learned. I built it so I'd stop losing the reasoning behind a fix three weeks after I'd made it.

For most of a day in May it worked exactly like that. Then, starting late on 2026-05-17 and running past midnight into 2026-05-18, a client project that leans heavily on the Agent tool sent it into a three-and-a-half-hour tailspin, and until I charted it I'd assumed that meant a roughly steady stream of firings the whole time. It didn't. hook-trace.log rotated into hook-trace.log.1 about ten days later and has sat untouched since, so I could go back and reread the raw record instead of working from memory. Two sharp spikes bookend a long, quiet middle: 694 firings in the first ten minutes (23:50 to 23:59), then a two-hour-plus stretch (00:00 to 02:05) firing only about once a minute, 128 firings in total, before a second, heavier spike of 2,616 firings across the next forty-odd minutes (02:10 to 02:54).

Bar chart of subagent hook firings per 5-minute bucket from 2026-05-17 23:45 to 2026-05-18 03:15 UTC, showing a sharp spike around 23:50-23:59, a long low stretch through until roughly 02:05, then a second, larger spike from about 02:10 to 02:54

hook-errors.log lines up with all of it: one unbroken block of 56,184 lines reading "You're out of extra usage · resets 10am (Australia/Sydney)", starting right where the firings start and running through spikes and lull alike. 3,486 firings in total, and though the rate swings by two orders of magnitude between the quiet stretch and either spike, no gap between individual firings ever widened past 6:13, the longest internal gap in the whole window. That's why the semaphore stayed pinned and the API stayed rate-limited across the entire three and a half hours, spikes and lull alike. The busiest single minute, inside the second spike, saw 75 fire. Every one of those subprocesses was dying on arrival, until the quota reset at 10am.

I'd drafted an earlier version of this post from memory, before I went back to the log, and remembered it as a single sharp spike: a few dozen processes in under a minute. It wasn't that, and it wasn't a smooth continuous burst either. The real shape, two spikes with a real lull between them, is something memory wouldn't have reconstructed, and a summary that only reports the total count and duration wouldn't have shown it either.

What the hook was built to assume

The hook is registered on Claude Code's SessionEnd event, not Stop. I'd already learned that lesson the hard way: Stop fires once per response turn, not once per session, so a hook wired to it fires dozens of times before a session is anywhere near finished. SessionEnd was the fix. It fires once, when a session actually closes.

Once per session was the assumption the rest of the design leaned on. The hook reads session_id and transcript_path off stdin, checks a semaphore, and if nothing is already running, routes by working directory and spawns claude -p in the background with a wiki-update prompt pointed at that transcript. Correct, as long as "a session" means what I thought it meant.

It doesn't. The Agent tool spawns subagents to handle narrow pieces of a larger task, and each subagent gets its own session: its own .jsonl transcript, its own SessionEnd firing when it finishes. From the hook's side, a subagent that ran for four seconds and wrote four lines looks exactly like a session I sat down and worked in for an hour. The payload has the same shape either way. There is nothing in it that marks one as a subagent and the other as mine.

The client project that leans heavily on the Agent tool is what turned that gap into an incident. Working in that project's directory overnight from 2026-05-17 into 2026-05-18 produced 3,486 subagent SessionEnd firings across three hours and twenty-nine minutes, in the two-spike, one-lull pattern the chart above shows. Each queued behind the semaphore and spawned a wiki-update subprocess, which hit the API's rate limit and died within seconds.

The semaphore did its job, and that wasn't the problem

The instinct, looking at "duplicate processes queuing up," is to blame the queue. I did too, at first. And the queue did have a real bug: it was a plain file, checked with [[ -f "$SEMAPHORE" ]] and claimed with touch. Those are two separate syscalls, so two hook firings landing close enough together could both pass the existence check before either had written the file, and both would proceed. I fixed that the same day by switching to mkdir, which is atomic on a POSIX filesystem: it either creates the directory and succeeds, or finds it already there and fails, with no window in between.

That fix was real and worth having. It just wasn't the fix for the cascade. The semaphore was never designed to ask "should this session produce a wiki update at all." It only ever answered "is another wiki-update subprocess already running." A perfectly correct semaphore still let every one of those 3,486 firings through, one at a time. Each queued behind the last and spawned anyway, dying the moment it started because there was nothing worth distilling and no quota left to distil it with anyway. Fixing the race stopped two firings landing at the same instant from both slipping past the check together. It did nothing about the total number of firings still queued up behind it.

The fix that actually worked: ask before you spend anything

What stopped the cascade is a single stat call before any of the expensive machinery runs:

# Skip subagent/trivial sessions — anything under 100KB is almost certainly a
# subagent spawn or a failed/empty run.
TRANSCRIPT_SIZE=$(stat -f %z "$TRANSCRIPT_PATH" 2>/dev/null || echo 0)
if [[ "$TRANSCRIPT_SIZE" -lt 102400 ]]; then
  echo "$(date -u +"%Y-%m-%dT%H:%M:%SZ") hook skipped — transcript too small (${TRANSCRIPT_SIZE}B, session=$SESSION_ID)" >> "$TRACE_LOG"
  exit 0
fi

That's the script as it runs today, still at exactly 102,400 bytes: 100KB. Under that, the hook exits before touching the routing logic or spawning anything at all. There's no positive signal for "this is a subagent" to check against, so the fix doesn't try to detect subagents directly. It uses the one thing reliably true of them without needing to be checked: a subagent doing one narrow task produces a small transcript, and a session where I've actually been working produces a large one.

The session that diagnosed this cascade was, by my notes from that day, 455KB; the transcript itself has since rotated out of ~/.claude/projects/, so I can't re-run that number, only report it as I recorded it at the time. A subagent transcript, by comparison, rarely clears a few kilobytes. 100KB sits comfortably in the gap between the two, so the check doesn't need to be clever. It just needs to be cheap enough to run on every firing without caring how many there are.

This is also where reading the current script instead of trusting my own notes on it paid off. Both my architecture notes and the original decision entry describe this as a size check that runs before the semaphore is touched at all, "deliberately upstream," specifically so subagent transcripts never need to queue. That is not what the script does. The mkdir on the semaphore happens first; the size check comes after, and a trap set right after the mkdir releases the semaphore again if the transcript turns out to be too small to bother with. So a tiny subagent transcript still touches the lock, just for as long as an mkdir/rmdir pair takes, which is close enough to free that it doesn't matter in practice. But the design notes describe an ordering the code doesn't have, and I only caught it because writing this post meant opening on-session-end.sh again instead of paraphrasing what I'd written about it three months ago.

What this optimises for, and what it gives up

The 100KB cut buys a lot for one stat call. It doesn't need Claude Code to expose anything it currently doesn't. And it doesn't need to know how many subagents fire, or how fast; the check costs the same whether it runs once or thousands of times in a night.

What it gives up is precision. Small transcripts happen to correlate with subagent sessions in practice; the hook has no way to check the actual definition, only the correlation it can observe. If I ever have a genuinely short real session, a one-line fix, a quick question answered and closed, it falls under the same threshold and gets skipped too. Not entirely silently: it does log a line to hook-trace.log. But nothing surfaces that line to me unless I go looking, so in effect it behaves as if it were silent. I've decided that's the right trade. A missed two-minute session costs me nothing I'd notice; a cascade like this one cost quota I actually needed that day.

It's also worth being honest about what this fix didn't cover, because the same semaphore came back to bite me two months later in a different shape. The size filter and the atomic mkdir both guard against firings arriving at the same time. Neither has anything to say about a headless wiki-update subprocess whose own SessionEnd firing spawns another wiki-update subprocess in turn, one after another, each one clearing the semaphore just before the next one checks it. That's a different bug with a different fix (an explicit environment marker that short-circuits a meta-session's own hook firing, added to the script in July), and it's the reason I'd now describe this part of the hook as handling concurrent noise, not every way a session can spawn more sessions. Guarding against firings that arrive together is a narrower guarantee than covering every way a session can spawn another one. I'd written the first one down as though it covered the second.

Would I do it again

Yes, and I'd reach for the size check sooner. The alternative I actually had running before this fix, a correct, non-racing, serialised queue, is the kind of "solving" this that feels like progress and isn't: every firing still eventually gets a turn and spawns a subprocess. That subprocess still burns whatever it costs before it fails. A queue makes a burst orderly, but it doesn't make a single item in it any cheaper.

What I'd change is the paperwork around the fix, not the fix itself. The notes I wrote the same day I shipped it describe an ordering the code has never actually had, as far as I can tell without a git history old enough to check. That's a small thing until you're the one three months later, drafting a post about your own fix, reading your own architecture page like a report from a colleague you trust, and getting the mechanism wrong right up until you open the script.

It's also a small thing until you're the one drafting a post about a "clean" incident that turns out, on rereading the actual log instead of your own memory of it, not to have been clean at all.

About the author

  • I build LLM-powered systems and write about the engineering that keeps them safe in production — scoping, deterministic verification, and governance.

No comments yet. Be the first to comment.