Note · AI Implementation Strategies

The scream test hears onlywhat still points at it

David He, FounderSeptember 15, 20264 min read

Pruning 325 agent skills by waiting for errors missed the quiet failures; three counts from the logs found them.

Your coding agent reads every skill description you have installed before it reads your first word. I had 325 of them. On Sunday I moved out all of them to find out which ones I use, and the answer came back in two halves: the half that screamed within a day, and the half that only the logs could see.

Why I moved out all of them

The trigger was a 27B model running on my laptop that could not take a two-letter prompt. My coding tool had loaded 1,500 skills, every agent's copy of the same set, plus the tool definitions for 16 MCP servers, ahead of the two letters I typed. The request was over the context window before my prompt arrived.

I wrote about that the same night and ended with a promise: from now on, only the skills I call. Then I tried to write the list and found I could not name them.

So at 11:23 on Sunday I moved the whole tree into a quarantine folder my tools do not scan. It stayed in git. Nothing was deleted. Then I waited to see what screamed.

The scream test

Sysadmins have a name for this. Power off the server, wait, and see who calls. If nobody calls in a month, nobody needed it.

Over the next 33 hours, 142 skills came back, one at a time. Each time a session reported it could not find something, I moved that one skill back by hand. The harness did not heal itself; I read an error and acted. That matters later. By Monday night I committed the result: 142 kept, 173 out.

That felt like a clean answer. It was not, for two reasons the scream test cannot see.

The first thing it got wrong: the hooks did not scream

At 4:48pm on Sunday a session noticed that two of my registered hooks had broken with the move. They are the scripts that run at session start and session stop, and one of them injects the list of other live sessions working the same repository, so agents do not land on top of each other.

They did not error. They stopped running. Every session that afternoon started without that list, and nothing anywhere reported it. Nobody screamed. Somebody read.

That is the general shape, and it is why the fast restores earlier were a human loop, not a property of the system. A skill is loud only while something still points at it: a running task, a hook, a slash command, an instruction file. Remove it and the reference errors. But a skill whose only job is to shape how work gets done, the voice rules, the checklist, the house conventions, has no reference to break. The agent does the job without it, a little worse, and tells you nothing.

The second thing it got wrong: the logs already knew

On Monday night I checked the thing I should have checked on Sunday. This machine holds 4,988 agent session transcripts from the last 30 days. One line of shell counted the skill calls inside them.

The count came back at 101 distinct skills called in a month. Three of them were still sitting in the quarantine pile, called four times between them, and had not screamed. And of the 163 skills I had kept or restored, 75 had not been called once in 30 days.

So the scream test found the loud ones. The logs found the quiet ones in both directions: the quarantined skills that were still in use, and the kept skills that were not. I had the logs the whole time.

The rule I am keeping

Move, do not delete. Keep it in git so a mistake costs a minute instead of a rebuild.

Then run three counts before you trust the screaming. What still references each skill: grep your hooks, commands and instruction files for its path. What actually called it: count the invocations in your transcripts. What loads before your first word: read the context meter, because that is the bill you pay every session whether or not you use any of it. On my machine the skill descriptions alone summed to about 80,000 characters, and every session read all of them before I typed a letter. That number never showed up anywhere I was looking, because a cloud model with a large context window absorbs it and a flat-rate plan hides it. The small local model was the first thing that refused to.

The first count is the one that would have caught my hooks. The second is the one that found the 75. The third is the one that started all this.

Two things you can do tomorrow

If you use Claude Code, type /context and read what loads before you type anything. Then count your own calls:

grep -rho '"skill":"[^"]*"' ~/.claude/projects | sort | uniq -c | sort -rn

Mine took about a minute and told me more than a day of watching for errors had.

What have you installed and never once called?

More notesnewest first

Working on something like this?

Bring the app or the process to a free 15-minute call. I will tell you what I would look at first, and whether I am the right person for it.