10 Mistakes I Made Building a Homelab Second Brain (And How Each Got Fixed)

What went wrong — and why — as I built a canonical Git vault, AI agent skills, write locks, and projection consumers: silent data loss, hanging CLIs, restore gaps, and the one mistake I made twice.

On this page
  1. 1. The stdin hang — misdiagnosed as a slow network mount
  2. 2. The --stage argument that silently staged nothing
  3. 3. The manifest-only commit that left consumers lying about their revision
  4. 4. Orphaned pages in the wiki that nobody deleted
  5. 5. Tools with ENOENT and cryptic failures instead of usage messages
  6. 6. Not testing the restore path until almost too late
  7. 7. SIGINT vs SIGTERM — the wrong signal leaves the lock held
  8. 8. Split-brain: writing to a clone instead of the canonical
  9. 9. A flat hand-maintained manifest that grew unmanageable
  10. 10. Assuming a build-passed because CI said green
  11. The pattern behind all of these

If you’ve been following this series, you’ve seen the polished version: clean architecture diagrams, tidy terminal output, a system that works. What you haven’t seen is the path I took to get there — the CLI that blocked forever, the write that committed nothing, the restore path that had never been tested.

This is that post. Every mistake here was real, cost real time, and produced a real fix that made the system more robust. Read it before you build, or read it after you hit the same wall. Either way, these are the traps that aren’t obvious from the happy-path documentation.


1. The stdin hang — misdiagnosed as a slow network mount

What happened. wiki-audit.mjs is the validation script that runs before every write — it scans the staged pages for broken links, stale paths, and anything secret-shaped. For weeks I was running it as:

The wrong invocation

node scripts/wiki-audit.mjs .

Sometimes it finished in a few seconds. Sometimes it sat there doing nothing for minutes. I blamed the SSHFS mount (the vault lives on a Proxmox LXC, mounted over SSH on my laptop). I tuned mount options. I checked for network drops. I restarted the connection. None of it helped.

The actual cause. The script reads the file list from standard input, not a positional argument. Invoked bare, it was blocking forever waiting for stdin that never came. The tool had no timeout, no usage message, and no detection for “I’m running interactively with no pipe.” It just sat.

The fix. The correct invocation pipes the file list in explicitly:

The correct invocation

git ls-files -s | node scripts/wiki-audit.mjs .

And the tool itself was hardened to detect a TTY or empty stdin and print a usage message with a non-zero exit rather than blocking. This one change would have saved me hours — if a CLI tool does nothing when you invoke it without arguments, that is a bug in the tool, not a mystery to diagnose in the environment.

Lesson 1

Build validation CLIs to fail fast with usage when they get no input. A tool that hangs silently trains you to distrust it and look for problems in the wrong layer. Always feed file-list arguments to audit tools explicitly via a pipe — never rely on them reading from a positional.


2. The --stage argument that silently staged nothing

What happened. After moving to the streamlined vault.mjs write path, I was passing the stage list like this:

Space-separated stage list (wrong)

node tools/canonical-vault/vault.mjs write --message 'feat: add restore runbook' --stage 'runbooks/claude/restore-vault.md runbooks/claude/canonical-write-procedure.md'

The command returned a success JSON: {ok: true, commit1: "abc1234", ...}. Everything looked fine. The write appeared to have landed.

The actual cause. The --stage parser split on commas only. A space-separated list collapsed into one bogus pathspec. git add silently ignored a path it couldn’t resolve — and then committed only whatever happened to already be staged from the preflight step (the auto-regenerated manifest). The content files I intended to commit were not in the commit. The system reported HEAD, but the canonical content wasn’t there.

I only caught this during a restore drill, when I realized the recovered state didn’t include the file I’d just written.

The fix. The parser now splits on commas or whitespace, and validates each path has a pending change before taking the lock. A path with no pending change produces {ok: false, error: "stage_unresolved"} and exits non-zero — no silent partial commits.

Correct invocation (comma-separated or space-separated, both work now)

# Use the exact stage_list output from preflight — don't type it by hand
STAGE=$(node tools/canonical-vault/vault.mjs preflight | jq -r '.stage_list')
node tools/canonical-vault/vault.mjs write --message 'feat: ...' --stage "$STAGE"
Lesson 2

Never hand-author your --stage list. Use the exact output from preflight. And build writes that validate each path has a pending change before committing — a {ok: true} that staged nothing is worse than a hard failure.


3. The manifest-only commit that left consumers lying about their revision

What happened. I added a new page to the vault. The wiki projection and embeddings index both reported the new HEAD commit afterward. I assumed they were up to date.

The actual cause. The importer’s rebuild-vs-relabel logic was keyed off whether any published page files changed in the commit range. The vault’s publishable-manifest file (wiki-publishable-manifest.tsv) — which tracks the list of pages — is not itself a published page. So a commit that only added a row to the manifest (a new page, a removed page) produced an empty changed-set, and the importer took the fast “relabel” path: stamp the new revision onto the stale consumers without rebuilding them.

The result: the new page was missing from the wiki. A removed page lingered in the index. Both consumers claimed to be at HEAD. The revision label was honest; the content was not.

The fix. The rebuild trigger now includes “did the manifest file itself change?” A manifest change forces a full reconcile regardless of whether any page files moved.

Lesson 3

A revision label tells you when a consumer was last rebuilt, not whether the rebuild was complete. If the rebuild logic has blind spots, the label is misleading. Verify that every consumer rebuild path covers the cases you care about — adding pages, removing pages, and changing the manifest — not just modifying content of existing pages.


4. Orphaned pages in the wiki that nobody deleted

What happened. I deleted a page from the vault. The canonical Git history showed it removed. But it kept appearing in the Wiki.js site.

The actual cause. The initial importer only did create and update — it never deleted. An orphaned page in the wiki had no mechanism to be removed. Once published, it lived forever until manually hunted down.

The fix. The importer now runs a delete pass: for every published page absent from the current manifest, it calls pages.delete. Then it asserts exact parity — the published set must equal the manifest set exactly. If the numbers don’t match, the import fails loudly.

Lesson 4

If your projection importer doesn’t cascade deletes, every removal from the source becomes a permanent orphan in the projection. Always assert exact parity — not just “all source pages are published” but “published set = source set.” That second direction catches the orphans.


5. Tools with ENOENT and cryptic failures instead of usage messages

What happened. I ran build-topic-map.mjs and wiki-publishable-manifest.mjs several times with slightly wrong invocations — a stray --write without a root, a non-directory path, an unknown flag. Instead of a clear “here’s how to use this tool” message, I got Error: ENOENT: no such file or directory or a hanging process.

The actual cause. The scripts didn’t validate their own arguments. They parsed flags in a loose way that let stray arguments silently misparse. A bare --write without a root positional got misinterpreted as the root path, then failed to stat --write as a directory.

The fix. All three core CLIs now validate their arguments on startup: if the root positional is missing or isn’t a directory, they print usage and exit 2. Unknown flags produce an error. -h/--help works. A failed invocation tells you exactly what it wants.

Lesson 5

Every tool in a write path will eventually be run wrong — at midnight, after a long session, by someone following a runbook slightly out of date. Build them to fail with usage, not to fail with a stack trace or hang silently. A clear error message is a form of documentation that’s always current.


6. Not testing the restore path until almost too late

What happened. The restore runbook existed. It was documented, reviewed, and linked from the right places. I’d never actually run it destructively against a real system. I had high confidence in it.

The actual cause. Untested runbooks drift. Assumptions baked in when you write the procedure aren’t always valid when the environment has changed. I only ran the full restore drill (destructive, against a scratch target) months after writing the runbook — and found two issues:

  • The “vault directory lost, CT 106 alive” path worked correctly.
  • The bundle-then-restore path from the backup mirror worked — but the embeddings index came up at 85/86 pages (one missing), which is exactly the manifest-only-commit bug from Lesson 3.
  • The SIGINT vs SIGTERM distinction for the writer lock was documented but had never been tested — SIGTERM does not trigger the finally cleanup; SIGINT does. This would leave the lock held after a crash if you used the wrong signal.

The fix. Ran the full restore drill. Fixed the bugs it surfaced. Added the drill to the regular operations schedule.

Lesson 6

An untested runbook is a hypothesis. A restore runbook that’s never been run is a liability that gives you false confidence. Drill it destructively on a scratch target before you need it in production. The act of drilling will find bugs that reading never will — including bugs in dependencies the runbook assumes are correct.


7. SIGINT vs SIGTERM — the wrong signal leaves the lock held

What happened. I interrupted a writer-lock process to abort a write. The lock file didn’t get cleaned up. Every subsequent write attempt blocked, waiting for a lock that had no owner.

The actual cause. canonical-writer-lock.py uses a try/finally to release the lock. Python’s finally runs on KeyboardInterrupt (which SIGINT raises) but not on SIGTERM (which kills the process immediately without running cleanup). I’d been sending SIGTERM (the default for kill).

The fix. Send SIGINT to abort — not SIGTERM. Have a way to confirm the lock is free afterward (e.g., check if the lock file exists). If the lock is stuck, delete it manually only after confirming no live process holds it.

Abort a lock process correctly

# Wrong — leaves the lock held:
kill <pid>         # sends SIGTERM by default

# Right — triggers cleanup in the finally block:
kill -INT <pid>    # sends SIGINT = KeyboardInterrupt = cleanup runs
Lesson 7

Any lock that doesn’t self-release on abort is a latent outage. Know what signal triggers your cleanup path. For Python try/finally, that’s SIGINT, not SIGTERM. Document this in the runbook and in any error message that mentions how to abort.


8. Split-brain: writing to a clone instead of the canonical

What happened. Early on, before the SSHFS mount and the canonical-writer pattern were established, I was editing vault pages directly on the Proxmox node via SSH, editing the same pages on my laptop’s local copy, and occasionally editing through the Wiki.js interface. All three were “the brain.”

The actual cause. No single authority. The three copies drifted apart. Reconciliation required a manual three-way merge that lost some history and forced a judgment call on every conflict. The hardest part was figuring out which version was correct — and accepting that no version was.

The fix. Single canonical authority: /wiki-data/git on CT 106. One write path. Every other consumer (laptop, wiki, agents) is read-only unless going through the controlled write. This is the architectural decision that makes everything else in this series possible — and it’s the one I wish I’d made on day one instead of three months in.

Lesson 8

Multiple writable copies of a knowledge base isn’t “distributed” — it’s a split-brain waiting to happen. Decide which copy is the canonical authority and make everything else derive from it. The earlier you make this decision, the less reconciliation debt you accumulate.


9. A flat hand-maintained manifest that grew unmanageable

What happened. The agent context file started as a hand-maintained list of “important pages for AI systems to read.” Every time I added a page, I updated the manifest. Entries drifted out of date. The file grew to the point where loading it consumed a meaningful chunk of an agent’s context window.

The actual cause. A hand-maintained index always drifts behind the content it describes. The more pages you add, the more maintenance it requires, and the more likely some entries are stale.

The fix. Replace the hand-maintained manifest with a generated context digest: a single file produced by a script that reads every page’s frontmatter and emits a compact, current table — region, path, last-updated, description — plus guardrails and the RESUME pointer. Regenerated automatically on every /end. An agent reads one file to get a map of the whole vault.

Lesson 9

If your agent’s startup context requires a file that humans maintain by hand, it will drift. Generate it from the ground truth instead. The script that generates it is always more current than anyone’s memory of what needs to be in the index.


10. Assuming a build-passed because CI said green

What happened. Draft pages are excluded from the Astro build. Blog posts with draft: true in their frontmatter are not validated, not rendered, and not visible in astro dev — the build completes successfully as if they didn’t exist.

The actual cause. I trusted “build passed, 0 errors” as “my content is correct.” It wasn’t — the validation simply never ran on the draft files.

The fix. Before publishing, temporarily set draft: false, run the build, verify every page renders without errors, then restore the draft state or just publish. Never trust a “build passed” that silently skipped half the content.

Lesson 10

A green build that didn’t validate your content is worse than a red build — it gives false confidence. Know what your build tool excludes, and explicitly test those excluded cases before shipping.


The pattern behind all of these

Looking back, almost every mistake here shares a shape: silent success. The write committed nothing and said {ok: true}. The restore restored the wrong content and labeled itself as HEAD. The build passed and skipped the content it was supposed to check. The tool hung and showed nothing.

Noisy failures are easy to fix. Silent successes are the traps — they give you confidence that collapses the moment you actually need the system to work.

Build every component to fail loud, verify its own outputs, and return something meaningful when it does nothing. Then test the failure paths — not just the happy path — before you trust the system in production.


This is the 13th post in the Build a Second Brain for Your Homelab series. The rest of the series covers the architecture this post describes from the inside out:

Comments

Comments are powered by GitHub Discussions — sign in with a GitHub account to join the conversation.