Anatomy of a silently broken release

Grafema ships native binaries — Haskell resolvers, Rust indexers — alongside the npm package. On May 25, three of them stopped building. We found out on June 10.

This is a postmortem. Not a self-flagellation exercise — the product worked fine on macOS and linux-arm64 throughout, and the npm install rate didn’t move. But grafema-resolve, grafema-rust-resolve, and haskell-resolve on linux-x64 were missing from the release for sixteen days, and nobody noticed, because the release still said 89/92 assets and the analysis still ran — it just silently dropped to structural-only mode.

Three things went wrong, in layers. Here’s what they were.

Layer 1: a path that worked for root and nobody else

Our linux-x64 CI jobs run on a self-hosted runner, ci-runner-1, with labels [self-hosted, linux, x64, haskell]. The runner process runs as user runner.

GHC was installed via ghcup — in /root/.ghcup. The directory /root has permissions 0700. The runner user cannot read /root. So every Haskell build step silently exited with EACCES and the jobs failed at the compilation stage, not at the upload stage. The release still got created with the Darwin and Windows assets, and the 89 out of 92 number didn’t trigger any alert, because there was no alert.

This had been true since the runner was set up. The previous release build (April 2) had passed; what exactly changed on the box between April and May we can’t fully reconstruct — a plausible suspect is a fresh ghcup installation under /root during maintenance. What’s certain is that the May 25 release build hit EACCES on every Haskell job, and no linux-x64 Haskell binary shipped after that.

Fix: moved the entire ghcup installation to /opt/ghcup (a+rX), relinked /usr/local/bin/ghc and /usr/local/bin/cabal, and patched the nine ghc wrapper scripts that had /root/.ghcup hardcoded. Left a symlink /root/.ghcup → /opt/ghcup for anything that might probe that path. Verified with sudo -u runner ghc --version → 9.8.4 ✓.

Then we triggered a rebuild (run 26420872312).

Layer 2: the rebuilt binaries crashed on slim images

With ghcup accessible, the rebuild completed. The three missing linux-x64 binaries appeared in the release. We ran gate-0 verification: npm install grafema on a clean node:22-slim container, then grafema analyze on express.

/root/.grafema/bin/grafema-resolve: error while loading shared libraries: libyaml-0.so.2: cannot open shared object file: No such file or directory
build-index request failed: Broken pipe
Analysis failed with exit code 1

The freshly rebuilt binaries linked libyaml dynamically. The rebuild on ci-runner-1 happened after we installed libyaml-dev to satisfy the Cabal dependencies — and Cabal happily used the system library rather than vendoring it. On node:22-slim, which is exactly what a lot of CI environments and Docker-based grafema users are running, libyaml-0.so.2 doesn’t exist.

So we had fixed the build and introduced a new crash at runtime. The analysis phase that produces the semantic graph — cross-file resolution, dataflow edges, type inference — would exit 1, and grafema would surface that as a full failure rather than a graceful degradation.

Fix: four cabal.project files updated to use the static libyaml flag (commit dd5d9b5d). No dynamic dependency, no runtime surprise. Tagged binaries-v0.3.30 and pushed a new full matrix build (run 27287400398).

The UX issue — failing loudly instead of downgrading gracefully with an explanation — is tracked separately as a product requirement. The correct behavior: if resolution is unavailable, say so, continue with the structural graph, and mark the graph level in grafema stats output. Crashing the whole analyze run is worse than producing a partial result with an honest warning.

Layer 3: the release race condition

During the binaries-v0.3.30 build, three darwin-arm64 jobs failed at the Upload step. When you push a binaries-v* tag, all matrix jobs start simultaneously. Each job’s final step is “Upload to Release.” The release doesn’t exist yet — the first push creates it. The second and third concurrent pushes hit a 422 from the GitHub API.

This was a known edge case that nobody had hit with this many parallel jobs before. The fix is a single prepare-release job at the start of the workflow that creates the release idempotently (gh release create --verify-tag || true), with all build jobs having needs: prepare-release. Three lines of YAML, one rerun of the failed jobs while the fix lands (tracked as REG-1175, now Done).

What the final verification looked like

After the static-linking fix and the second full rebuild: node:22-slim, npm install grafema, grafema analyze on the express repository.

Analyzed 141 files
67,398 nodes / 141,574 edges, errors=0
Resolution: 141/141 files, 16,504 edges
exit 0

ldd grafema-resolve-linux-x64 shows no libyaml dependency. The three layers are gone. Existing installations of 0.3.29 pick up the fixed binaries automatically, because grafema’s lazy-download logic fetches the latest binaries-v* release rather than pinning to the version the CLI was installed from.

What we changed systemically

Beyond the immediate fixes:

Runner access hygiene. Any tool that needs to run as a non-root user should be installed where non-root users can read it. /opt, not /root. This is obvious in retrospect; we’re documenting it now.

Static linking as the default for distributed binaries. If a binary ships in a release tarball and runs on arbitrary Linux environments, it should not assume the host has any particular shared library installed. Dynamic linking is convenient during development; it’s a deployment risk for distributed tooling.

Smoke test gate before declaring a build done. A node:22-slim Docker smoke is now part of our gate-0 checklist: npm install grafema && grafema analyze on a small repo, verify the output includes resolution edges, verify ldd shows no unexpected runtime dependencies. If this smoke had existed before May 25, we would have caught the EACCES failure on the first missed build, not sixteen days later.

Graceful degradation with explicit signaling. When a component is missing or fails, grafema should tell you what’s unavailable and why, continue with what it has, and mark the graph level in its output. Silent partial results are worse than honest failures.

The release process is a stacked system: the CI runner, the build flags, the release workflow, the smoke test. Each layer made a plausible local choice — ghcup works great as root, system libyaml is fine for local builds, parallel uploads are fast. None of those choices survived contact with the actual deployment environment. The fix for each was small. The lesson is that the smoke test should have been there first, not after we needed it.