Section 8Ship without breaking prod

Passing gates earns a merge, not a release. Shipping safely to millions is its own discipline — the agent writes the code, but the pipeline protects the users. Build it once; every change flows through it.

Section 7 ended with every gate green: the salon's holiday-blocking feature has passed its behaviour tests, cleared the definition of done, and merged. It is still invisible to every customer — and that gap is deliberate. Merging means the code is good; releasing means real people now depend on it. While the change sat on a branch, a mistake cost nothing but a red check and an agent's time. The moment it reaches the booking page, the same mistake costs a double-booked chair, a missed appointment, and a little of the trust the salon's whole business runs on.

This chapter is about crossing that gap on purpose. The instinct that doesn't work is being more careful — reading the change one more time, deploying late at night, hoping. What works is a pipeline: one fixed, automated path from merge to users that every change travels, built once and then trusted. Agentic coding raises the stakes in both directions at once: agents produce more changes, faster, than any team of humans could — and precisely because no human can personally shepherd each one, the pipeline has to do the shepherding. The four layers below are that pipeline's skeleton; the rest of the chapter adds the disciplines around it — database changes, release notes, load, redundancy, observability and being your own on-call, cost, and the playbooks for the day something breaks anyway.

The four safety layers between merge and customer

Each layer answers a different question. Staging: does it work somewhere real before customers see it? Feature flags: can code sit in production without anyone seeing the feature? Gradual rollout: if it's broken anyway, how few people find out? Rollback: how fast can we undo it? Together they turn a release from one bet into a sequence of small, reversible steps.

The rule extends past the code to the infrastructure underneath it. The salon app's servers, database, DNS records, and queues should themselves be defined as code — infrastructure as code (IaC) — not assembled by hand in a cloud console and then remembered by nobody. That buys the same properties this guide asks of everything else: an infra change arrives as a reviewable diff on a branch instead of a mystery click, staging and production stay genuinely alike because both are built from the same definition, and a broken environment can be rebuilt from scratch — the property the ransomware playbook later in this chapter quietly depends on. The agent is fluent in the major IaC formats; the walkthrough of setting this up lives in the companion page Setting up production.

The pipeline in one story — holiday blocking ships

Run the salon's holiday-blocking feature through all four layers. Its pull request got a preview deploy — a private copy of the app at its own URL, where you clicked through the feature exactly as you'd watched it in the Section 6 demo reel. It merged dark, behind a flag named holiday_blocking, so the code reached production with the feature invisible to everyone. You turned the flag on for one pilot salon and watched the error tracker and the booking numbers for a day. Then you ramped it to everyone — knowing the whole time that a single flag flip would take it away again in seconds if anything looked wrong. Four layers, and at no point was the entire customer base exposed to an untested change.

One mechanical detail makes all of this invisible to customers: a deploy must never drop the requests already in flight. The new version starts, passes a health check, and only then receives traffic; the old instances are drained — allowed to finish the requests they're serving, including the customer halfway through paying for a Saturday appointment — rather than killed mid-request. Most platforms do this for you, but only if a health-check endpoint exists and long-running work has been moved off the request path; both are one-line asks in the ticket that sets up the pipeline.

Operate through tools, not through the database

The "no ad-hoc queries against live" rule above raises an honest question: then how do you refund a customer, fix a double-booked chair, or merge the two accounts one regular created by typo-ing her own email? The answer is a minimal admin surface — a small internal page (or a handful of audited scripts) for exactly the operations the business actually needs: issue a refund, move or cancel a booking on a customer's behalf, merge duplicate customers, execute the GDPR export and delete that Section 9 promises, and view the app as a specific customer sees it when debugging their complaint. Every one of those actions writes an audit log entry — who did what, to whose data, when — and the impersonation view especially is worthless without one.

The agent builds this like any other feature, and it ships like any other feature: behind login and RBAC, through the same tests, review, and pipeline. That's the point. An admin page that went through the gates is a tool you can trust at 11 p.m. with a customer on the phone; a raw database session at the same hour is how a refund becomes a data-loss incident. Build the admin surface the first time you catch yourself wanting to "just quickly fix it in the database" — that urge is the requirements document.

Database changes need special care

You can roll back code instantly; you cannot un-drop a column. Use expand-contract migrations: first add the new shape and write to both (expand), deploy, backfill, then remove the old shape later (contract). Never let an agent write a destructive, one-shot migration into a release.

The same discipline applies the moment any other code depends on yours — a public API, or just your own frontend evolving independently of your backend. Version it, keep the old version working through a stated deprecation window instead of a breaking change landing the instant new code merges, and keep a plain-language changelog so a business customer can see what changed without reading a diff.

Release notes — let the pipeline write the history

That changelog deserves better than being someone's chore. On a slow-moving project, a human writes a "what's new" page once a quarter and it's fine. An agentic pipeline can ship several releases a week, and by Friday nobody — including you — reliably remembers what went out on Tuesday. The fix is not more discipline; it's the same move this guide makes everywhere else: take the task away from memory and give it to the machinery. Release notes should be generated, per release, by the same CI run that cuts the release — never reconstructed afterwards from recollection.

Generation only works if the raw material is structured, and that's what Conventional Commits provides — a small, machine-readable convention for the messages attached to each change in git: a commit that fixes a bug starts with fix:, a new feature with feat:, and anything that breaks existing behaviour carries a BREAKING CHANGE marker. The spec maps those labels straight onto version numbers: a fix bumps the patch number, a feature the minor, a breaking change the major. Humans are famously sloppy about commit-message discipline; agents are the opposite — state the convention once in AGENTS.md and every commit the agent writes follows it exactly. This is one of the places where agents make an old best practice cheaper to follow than to skip.

The tooling for this is mature, and the pipeline-triggered flow is standard practice, not exotic. Release Please (an open-source tool from Google's GitHub organisation) watches your merged conventional commits and maintains a running "release PR" that accumulates the pending changelog — merging that PR is the release, and the tool updates the changelog file and version number itself. semantic-release goes one step more hands-off: it runs in CI after each merge to the release branch, computes the next version from the commit messages, generates the release notes, and publishes, unattended. And GitHub's built-in auto-generated release notes list every merged pull request and contributor for a release, sorted into categories you configure in a small .github/release.yml file. Which one you pick matters far less than the property all three share: the notes are produced by the pipeline, as a side effect of releasing — so a release without notes becomes structurally impossible, instead of merely frowned upon.

Then put the log where people can actually read it. A changelog file in the repository serves developers; your docs site (Section 13) should carry the human version — a living release-log page in the HTML docs, newest entry first, one entry per release: version, date, what was developed, what changed, anything to watch. One small pipeline step that appends each generated entry to that page on release buys you two things. Future-you — or a fresh agent session picking up a handoff — answering "when did the reminder emails change?" gets a dated, searchable answer instead of an archaeology dig. And the deprecation windows promised above stop being informal: the release-log entry is where "the old booking API keeps working until March" becomes public, dated, and linkable.

Know your limits before your users do

"It works" and "it works under load" are different claims. Test both on purpose, before real growth forces the answer on you:

• Load test — simulate the traffic you actually expect (a Saturday-morning booking rush) and confirm it holds.
• Stress test — keep increasing load past that until something actually breaks, so you know the real ceiling instead of guessing.
• Soak test — realistic load for hours, not minutes. This is what catches a slow memory leak or a connection pool that never releases — a five-minute load test never will.
• Spike test — a sudden burst (a press mention, a marketing push) rather than a ramp. A different failure mode from steady growth.

None of this needs a load-testing team. Describe the shape of your real traffic to the agent — "Saturday, 9 a.m., three hundred customers hitting the booking page within twenty minutes, most of them on phones" — and have it script that scenario with the load-testing tools from Section 6's toolbox, run it against staging (never production), and report where the first thing bent. Do it before the first marketing push, not after: the cheapest time to learn where your ceiling is, is while nobody is standing on it.

Watch the same four signals an SRE would — the "golden signals": latency, traffic, error rate, and saturation (how full a resource, CPU, memory, a connection pool, a queue, actually is). Alert on the leading indicator, not the lagging one: a warning at 70% of a defined capacity ceiling gives you time to act; an alert at 100% is already an incident.

Climb this one rung at a time, only when a real, measured signal says the current rung is the actual bottleneck — not speculatively. Jumping straight to rung 4 for a product with a hundred users is effort spent on the wrong risk.

Two parts of the system deserve their own load story, because they fail quietly rather than loudly:

Redundancy, automatic failover & proving your SLA

"What if AWS goes down" is really two separate questions: what if one availability zone fails (routine — your provider handles it if you've actually configured multi-AZ; confirm you have, don't assume), and what if the whole provider or region does (rare, and expensive to fully insure against). For almost every small or mid-size product, true multi-cloud active-active — fully redundant infrastructure on two providers running at once — costs far more engineering effort than the risk justifies. The pragmatic middle ground is multi-region within one provider, with automatic failover: DNS-based health checks (AWS Route 53 or Cloudflare Load Balancing) that detect an unhealthy origin and redirect traffic to a healthy one or a warm standby, typically within one to three minutes — not instant, but automatic and unattended.

Whatever you deliberately don't build, write down as an accepted risk in the register below, with the threshold that would make you revisit it — a revenue figure, a customer count, a specific incident. An intentional decision beats an accidental gap discovered during an outage.

If you sign an SLA — one you offer a business customer, or one a vendor offers you — track it independently. Your own dashboard isn't evidence in a dispute; a third-party uptime/status-page service (current options include Better Stack, Hyperping, Uptime.com, or the lighter UptimeRobot) gives you an outside, timestamped record of what was actually up. Use it both ways: to show customers you met your commitment, and to notice when a vendor missed theirs — most teams never claim the service credits they're owed simply because nobody was tracking it.

See what production is doing

Ship observability with the feature: error tracking, and real-user performance monitoring (Core Web Vitals from real sessions). Set an error budget — a threshold that, when breached, automatically pauses rollout. When something slips through, run a short, blameless postmortem — what happened, why, and the concrete action items that prevent a repeat — then feed those items back as new tickets into the loop (Section 5). The postmortem, not just the fix, is what keeps the same incident from happening twice.

Concretely, observability is the difference between two ways of learning about the same bug. Without it, a broken confirmation email ships on Friday and you find out on Monday from an angry phone call — your customers are your monitoring. With it, the error tracker flags the first failed send within minutes, the error budget pauses the rollout automatically while only a small slice of users is exposed, and the flag is off before the second customer is affected. The agent will wire up error tracking and real-user monitoring in an afternoon if asked; like the production-grade auth in Section 9, it reliably won't happen unless you ask.

The three pillars — logs, metrics, traces

Error tracking and real-user vitals are the starter kit; the full discipline is observability — instrumenting the system well enough that you can answer questions you didn't think to ask in advance. "Why was booking slow for one customer at 9:14 on Saturday?" is not a question any pre-built dashboard answers; it's a question you interrogate the telemetry for. The industry organises that telemetry into three pillars, and each one earns its keep on a different kind of question:

The tooling here is unusually kind to small teams, as of mid-2026. Sentry covers error tracking with a free single-developer tier, and is the piece to wire up first — it's the one that pages you. OpenTelemetry is the vendor-neutral open standard (under the CNCF) for emitting all three pillars: instrument the code once against it, and you can point the data at nearly any backend later without re-instrumenting — the same lock-in hedge this guide recommends everywhere else. Grafana Cloud is a common place to point it, with a free tier that comfortably covers a salon-sized product's metrics, logs, and traces. Which backend matters far less than the standard: ask the agent for "OpenTelemetry instrumentation" and the exporter is a config detail.

And the instrumentation follows the rule every other invisible quality in this guide follows: the agent writes it in an afternoon, and it reliably won't happen unless a ticket asks for it. Put "instrumented — errors tracked, key operations logged with correlation IDs, business metric emitted" into the definition of done, and observability stops being a retrofit you perform during your first bad incident.

Before wiring alerts to all this telemetry, decide what "healthy" means on purpose: pick an availability target — an SLO — and derive everything else from it. The target comes first because it sets the budget: promising 99.5% availability means accepting up to about 3.6 hours of downtime per month, which is honest and achievable for one person with a good pipeline; 99.99% means under five minutes a month, which is a paged, staffed, multi-region commitment no solo operator can truthfully sign. The gap between your target and perfection is exactly the error budget the rollout machinery above already spends: while the month's budget is healthy, you ship freely; when an incident eats most of it, rollouts pause and the remaining budget is spent carefully. One number, chosen deliberately, turns "how careful should I be this week?" from a mood into arithmetic.

You are the on-call

A company has an on-call rotation; you have you. Pretending otherwise is how solo products die of a Saturday outage nobody saw until Monday — so build the honest version. First, one notification channel that actually wakes you: not email, which you'll sleep through, but a real pager path. As of mid-2026, PagerDuty's free plan (up to five users, with push, SMS, and phone-call alerts) is the established option, and ntfy — an open-source push service you can self-host and trigger with a plain HTTP request — is the minimalist one. Wire the error tracker and the uptime monitor into it, and nothing else.

Then a two-tier severity policy, written down: Sev-1 wakes you — customers cannot book, payments are failing, data loss is in progress. Sev-2 waits for morning — one reminder email bounced, latency is elevated, a background job needs a retry. Configure quiet hours so only Sev-1 pages at 3 a.m., and hold the line on alert quality: every alert must be actionable and must link to the playbook for handling it. An alert you've ignored twice is either miscalibrated or decorative — fix its threshold or delete it, because alert fatigue is how the real Sev-1 gets slept through. Vacations and illness get the same honesty: before you're unreachable, freeze risky rollouts, check the error budget is healthy, and either line up a human who can execute your playbooks or accept — in the risk register, in writing — that a bad week off the grid may cost you customers.

The last on-call duty is talking to customers while it's broken. The status page you already run for SLA evidence is also your incident comms channel — and 3 a.m. is not when to compose prose. Pre-write the template now:

[INVESTIGATING] Booking is currently unavailable for some customers.
We're aware and actively working on it. Existing appointments are not
affected and nothing is needed from you. Started: {time}.
Next update here by: {time + 30 min}.

Post it within minutes, update on the promised schedule even if the update is "still working on it," and close with what happened and what you changed. A named incident with honest updates costs you less trust than fifteen minutes of silent downtime — customers forgive outages; they don't forgive dark windows.

Measure the product, not just the system

Everything above tells you the booking page is up and fast. None of it tells you whether anyone completes a booking. That second question is product analytics, and it deserves the same "consent-compliant by design" bar as the rest of the guide: as of mid-2026, Plausible (cookie-free, collects no personal data, and positions itself as usable without a consent banner — though banner rules vary by jurisdiction, so confirm locally) and self-hostable Matomo (configurable to run cookie-free) are the standard privacy-first options. Start with a single north-star funnel, not a dashboard zoo: visit → pick a service → pick a slot → confirm. The step where the drop-off happens is your most valuable ticket — a fast page that loses half its visitors at slot-picking has a product problem no golden signal will ever show.

Your feature flags quietly double as an experiment platform: roll a change out to half your users and compare funnel completion between the halves. Use it with the honest caveat that at solo-product traffic, a real statistical answer can take weeks — treat small-sample results as a directional hint plus a conversation with customers, not as proof. That limitation is fine; the flag habit is what matters, because it means every product bet ships reversible and measured instead of permanent and assumed.

Close the loop with support: pick one channel (a support email address is plenty), publish the response time you actually intend to honour — "we reply within one business day" beats an unkept "24/7" — and route every ticket into the Section 3 feedback board so five complaints about slot-picking collapse into one prioritised signal instead of five forgotten emails. The user-facing help pages that deflect half those tickets come from the same docs pipeline as everything else in Section 13; "customers keep asking this" is a ticket, and the agent drafts the help page like any other feature.

Cost is a budget, like performance

A performance budget stops a slow page from shipping; a cost budget stops a surprise invoice from shipping, and it's enforced the same way — as a gate, not a hope. Know your rough unit economics: what one active customer (or one hundred bookings) costs you in hosting, email sends, payment fees, and AI-agent usage. It doesn't need to be precise; it needs to exist, because "this feature doubles our email volume" only registers as a decision if you know what email volume costs. Then set billing alerts on every paid service — the cloud account, the email provider, the model provider — at two thresholds: one at expected spend (drift warning), one well above it (something is broken or being abused: a runaway retry loop or an agent stuck in a loop can burn a month's budget in a night). Review actual spend against the budget in the same rhythm as the risk register below, and let cost vote on the scaling ladder: multi-region isn't just rung 4 in effort, it's roughly doubled infrastructure spend forever — one more reason the ladder is climbed on evidence, not ambition.

Risk register & disaster-recovery playbooks

Name your risks on purpose instead of discovering them during an incident. A risk register is a short, living list: the risk, how likely and how bad, what you've already done about it, and what's genuinely left over — the residual risk. For a real product, that usually includes at minimum: data loss, a security breach, a critical vendor disappearing, a payment failure, a reputational incident, and one people skip — key-person dependency (the "bus factor": if you personally were unreachable for a month, could anyone else run this?).

For every genuine single point of failure on that list — a single payment provider is the classic example — ask the aerospace question on purpose rather than reflexively: is redundancy actually worth it here? Aerospace engineering has a real name for this discipline, FMEA (Failure Mode and Effects Analysis) — estimate how likely the failure is and how bad it would be, then weigh that against what redundancy genuinely costs. That cost is usually bigger than "sign up with a second provider": it's an ongoing integration to keep working, test data to keep current, and a failover path that itself needs rehearsing (the same game-day discipline as above) or it's false security, not real security — a backup payment provider nobody has ever actually routed a transaction through is a backup in name only. Sometimes the answer is genuinely yes — a second payment processor is common practice once payment volume matters enough that a day of outage would hurt. Often, for an early product, the honest answer is no, not yet: record it as an accepted risk with its revisit threshold, the same pattern already used for the multi-cloud decision above, rather than either ignoring the risk or over-building for a scale you're not at yet.

For each real risk, write an actual playbook — not "we have backups," but the literal steps: who does what, in what order, how you verify it worked, and what you tell customers. Take data loss as the worked example: identify the last good backup → restore it to a clean environment, never over the live database → run your test suite against the restored data → verify a sample of real records by hand → only then cut over → post a status update. That sequence, written down in advance, is the difference between a calm hour and a panicked afternoon.

The part most teams get wrong: a playbook nobody has run is a guess. Rehearse it — a scheduled "game day" where you deliberately trigger the scenario (in staging, or production if you're confident) and follow the playbook for real, the same discipline this guide already asks of a backup restore and a rollback. Keep playbooks versioned in the repo next to the infrastructure they describe, and treat one nobody has touched or tested in a quarter as stale — the same staleness you'd flag in an untested backup.

Ransomware is the sharpest version of "data loss" worth planning for by name — a competent attack tries to encrypt or delete your backups too, not just production, if it can reach them. Ordinary credential isolation (the protocol's backup-isolation gate) isn't enough on its own; at least one backup copy needs to be genuinely immutable — write-once, unable to be altered or deleted even by a compromised admin account, for a set retention window. Recovery is also faster and safer if you never have to "clean" a compromised server at all: with infrastructure defined as code, restoring means rebuilding fresh infrastructure from that code and restoring data into it — the same deploy pipeline that ships features, run to reconstruct an environment instead. Treat this as the one playbook worth actually automating end to end (a scripted "rebuild clean, restore data, run smoke tests, cut over" sequence) rather than a document you'd follow by hand under pressure. A breach like this also starts the same 72-hour GDPR clock as any other (Section 9), and cyber insurance — genuinely common now, worth pricing even for a small product — can cover both the technical recovery cost and the legal obligations that follow.

Every "test it by hand" needs its own playbook

This guide and its companion protocol ask for a lot of manual checks — the IDOR test in Section 9, a restore rehearsal, tabbing through the keyboard flow in Section 7, a load test, a game day. Each one is worthless as a repeatable gate if "test it by hand" is the entire instruction: what one person checks in five minutes, another skips in thirty seconds, and neither produces the same result twice. Every manual-verification gate needs the same treatment as a disaster-recovery playbook above — a written, literal, step-by-step procedure, not a reminder to "be thorough."

A playbook worth trusting has five parts: the preconditions (which test accounts or seed data it needs), the exact numbered steps (click here, enter this, not "verify access control works"), the expected result at each step, how to reset whatever the test touched afterward, and a visible last-verified date and owner. Take the IDOR check as the template: 1) log in as seeded test user A; 2) note the URL of one of A's own bookings; 3) log in as seeded test user B in a second session; 4) paste A's booking URL into B's session; 5) expected result — a 403 or 404, never the booking; 6) delete both seed accounts' test data; 7) update the date at the top of this file. Seven lines, and now anyone — a new hire, a fresh agent session, you in six months — can run it identically.

Store these playbooks in the repo, one file per manual gate, next to the disaster-recovery ones (Section 13's docs site is where they end up readable). Treat a playbook whose last-verified date is older than a quarter the same way this guide already treats an untested backup or a stale runbook: due for a rerun before you trust the gate it backs.

One gap remains, and it's the quiet one. Everything in this chapter protects users from a bad release — the bug that slipped through, the migration that went wrong, the server that fell over under Saturday's rush. None of it protects them from their data quietly ending up where it shouldn't. That is a different discipline, with different instincts, and it's Section 9.