Internal tools tend to rot the moment they leave the founding team. We rebuilt our admin from scratch around three principles: every action should be auditable, nothing should require a tribal-knowledge runbook, and there should be exactly one button that resolves the on-call page.
There is no faster way to grow as an engineer than spending an evening with a debugger and a project you depend on every day. Here are the libraries that taught us the most and the exact commits worth bookmarking.
Most database migrations fail not because the SQL is wrong but because nobody tested them against a realistic dataset. We share the rollout checklist we use for every schema change, including the awkward question of when to lock a table and when to live with the inconsistency.
Five years ago this would have been a controversial post. Today we have battle-tested operators, decent storage classes and a community that has finally agreed on backup tooling. Here is the stack we landed on for a fleet of around forty clusters.
After two years of running a custom message broker we finally pulled the plug and migrated everything to NATS JetStream. The migration took six weeks of careful planning, three weekends of late-night cutovers and one production incident that we are still apologising for.
Every team eventually discovers the joy of shipping behind a flag. Fewer teams discover the misery of cleaning up flags two years later when the original author has long since left the company. A short rant and a longer guide.