Dan Neff
← All writing

huh, that's weird.

debuggingsystemstech-debtinfrastructurePAI

1

A weed is a plant out of place. — Jim Thompson, The Killer Inside Me

My bug report was one sentence long: "The logo on the dashboard returns a 404". I filed it myself, against my own system, after clicking past that broken dashboard icon for weeks. It lived on the home page of the personal status board I peruse most mornings between coffee and my TODO list.

The logo had never loaded. I knew it had never loaded. The header component references /pai-logo.png. The dashboard serves static files from its own public directory, and the file was not in it. Grepping around, turns out it had existed the whole time one directory tree away, in the installer's asset folder. The install pipeline was supposed to copy it over and never did. The favicon was missing for the same reason.

I copied both files, hit F5, and watched them load. The whole fix took less than ten minutes. That's where this story usually ends. Thanks for swinging by!

Looking with fresh eyes, the page I was reloading was saying other things, and once I was actually looking, I couldn't stop seeing them. There was a banner across the top: "You're looking at template content. Run /interview." I had run that interview three weeks earlier. It took most of an evening and it was good work. The banner was keyed to a marker file that the interview was supposed to delete on completion, except the cleanup step wasn't broken so much as it had never been written. I had been dismissing that banner almost daily, the way you ignore clutter in the garage.

Below the banner, my mission statement read "live well and stay curious." That is a fine mission! (I never wrote it.) Every value on the page was sample data, because the hook that was supposed to fetch my real data had been stubbed during development and never finished. Its entire body returned the fallback fixture. It had never called the API once.

So I wired the hook to the API, and the API returned empty arrays. It turns out the parser expected my goals to be formatted as markdown headings, but my files use bold-prefixed bullets, and the mismatch failed silently. Eleven other fields turned out to be hard-coded to null, a template that never connected to anything.

Then I restarted the dashboard process to pick up the changes, and it refused to boot! Two packages it imports were never declared as dependencies. It had only ever been running because of leftover module-resolution state from some earlier install, state that was never going to survive a restart. The process was one reboot away from being dead the entire time, and nothing anywhere would have said why.

One API endpoint was returning a 503 whose error message said, in full:

user-index.json not generated — run bun Pulse/modules/user-index.ts

The generator existed. The install never ran it. The API returned the error message for weeks, politely explaining exactly how to fix it, waiting for someone to read it.

And underneath all of it, the dashboard process itself had no service definition on this machine. The launcher script was written entirely for macOS and I run this box on Linux. The process was alive because at some point, months ago, I had started it by hand with nohup and forgotten. It was a service with no service behind it.

Six hours after a one-sentence bug report, I had found nine independent gaps, fixed eight of them, written a real systemd unit, added the missing cleanup step to the interview, and rewired the dashboard to live data for the first time.

2

From the inside, however, the abnormal is pretty normal, and making trade-offs in the direction of greater efficiency is nothing unusual. … If it was followed by a safe landing, apparently it must have been a safe decision. — Sidney Dekker, Drift Into Failure

Every one of the nine problems had the same anatomy: the visible symptom was downstream of an invisible omission: a copy step nobody added, a dependency nobody declared, a bootstrap nobody ran, a cleanup nobody wrote.

Every omission was reasonable on the day it happened. You defer the asset-copy step because you're iterating on layout. You skip the dependency declaration because it resolves fine on your machine today. Sidney Dekker wrote a whole book, Drift Into Failure, about this exact shape: systems rarely fail because a part breaks, they drift past their safety boundaries through an accumulation of decisions that were each locally rational. Each deferral's savings were measurable, but the borrowed safety wasn't. Nothing in my stack was broken. The aggregate was a Linux box confidently serving a stranger's sample life.

3

The most common form of process abuse is neglect, and the effect of that is that intelligent, conscientious developers find themselves working inefficiently and at cross-purposes when there's no need for them to work that way. — Steve McConnell, Rapid Development

Most writing about tech debt is about the invisible kind: the slow test suite, the brittle integration, the migration everyone defers. This was the other kind. Every one of my nine symptoms was visible in a single screenshot of the home page: the 404'd logo, the stale banner, the sample mission, the empty goal cards. Finding them required no log file, no debugger, and no tracing. The dashboard had been telling on itself the entire time, and the only missing component was someone willing to look at it.

That failure mode deserves its own name, and it already has one.

The 1982 "Broken Windows" essay, by James Q. Wilson and George Kelling, gets remembered as a theory about petty crime, but its real argument was about signaling. A window left broken signals that nobody is watching, and the next violation becomes easier for everyone, including the people who live there. You can argue about how well the theory held up in policing. In software, it holds up uncomfortably well. A 404 on your own home page, left for a month, signals that nobody is watching this system. The second broken window costs less than the first. By the ninth you aren't seeing windows anymore.

4

These [prep] cards were not checked, but if Chef Smith saw you in the weeds, he'd ask to see your cards. He could usually point to the spot on the card, the error or omission, that put you in the weeds. — Michael Ruhlman, The Making of a Chef

At work, dysfunction is ambient and ownership is diffuse. Most stale dashboards are backlog items in some other team's queue. This dashboard runs in my garage. Every deferral in that pipeline was mine. The banner I dismissed thirty-odd times was asking me to finish my own work, and I treated it like weather. Corporate dysfunction has someone else's name on it. This had mine, and I looked past it anyway, which tells me apathy is not a corporate disease—it is a human one. The only reason it felt different at home is that there was no one else to blame.

"huh, that's weird," is the beginning of the work instead of an interruption to it.

Every layer in that six-hour dig came from refusing a workaround. Copy the logo and stop, and I keep the banner. Dismiss the banner and stop, and I keep the fake data. Each symptom was the thread-end of the one below it. Working around a symptom is how the drift accumulated in the first place, which makes every workaround a small deposit on the next failure.

There were three distinct levels of fix available that day. Copying the logo was a workaround. Fixing the install pipeline so it copies assets, declares dependencies, and runs the bootstrap was the root cause. Writing down why the pipeline had nine holes, so the next installer I build doesn't inherit the same blind spots, was growth. We often stop at level one, because level one makes the symptom disappear and the symptom was the only thing anyone complained about. This kind of work is akin to gardening. Pulling one weed is not the job. The job is maintaining conditions where weeds don't accumulate faster than you notice them.

5

Discipline, in essence, is consistency of action: consistency with values, consistency with long-term goals, consistency with performance standards, consistency of method, consistency over time. — Jim Collins and Morten T. Hansen, Great by Choice

I'd like to tell you I now live at level three. Last week my reading tracker, part of this same system, failed silently for nine consecutive nights. Every night at 9:30 PM it wrote the same error to its log, "browser/extension unavailable," and every morning I read the briefing that same system produces and noticed nothing, because the briefing still arrived and the dashboard still rendered. The window was broken for nine days in a house I had just finished writing this essay about. This failure wasn't about apathy, and the solution wasn't about vigilance. What caught it was a routine verification of something else that happened to read the log.

So the practice I can actually recommend is the one that works without vigilance. Once a month, walk your own home page the way a stranger would, and say out loud what you see. Wire your checks so a quiet failure becomes a loud one, because the 503 that explains its own fix is only useful if you read it.

And when you hear yourself say "that's weird," stop treating it as noise. It is the most reliable bug report you will ever receive, and it is always addressed to you.

Weed on, my friends.