What a failed launch taught us about load testingWhat a failed launch taughtus about load testing

The launch went out on a Tuesday morning and the checkout page timed out for forty minutes. We had load tested. We had just tested the wrong traffic.

This postmortem is blameless by design. The goal is the checklist at the end, which every launch now goes through.

#What we modeled, and what we missed

Our test ramped steadily to the expected peak. Real traffic arrived as a spike: an email went out at 9:00 and ten thousand people clicked in the same minute.

The database held. The connection pool in front of it did not, because every request opened a new connection during the first seconds of the spike.

• Model spikes, not ramps: step load from near zero to peak in under a minute
• Test with cold caches and cold connection pools
• Include the third-party calls, or stub them with realistic latency

#The checklist we use now

Every launch has a named owner, a load profile written before the test, and a rollback we have rehearsed. If the test does not match the marketing plan, the test is wrong.

We also agreed on a kill switch for any non-essential feature, so under pressure the answer to a struggling dependency is to turn it off, not to debug it live.

import http from 'k6/http'

export const options = {
  scenarios: {
    email_spike: {
      executor: 'ramping-arrival-rate',
      startRate: 5,
      timeUnit: '1s',
      preAllocatedVUs: 500,
      stages: [
        { target: 400, duration: '30s' }, // the 9:00 email lands
        { target: 400, duration: '5m' },
        { target: 20, duration: '1m' },
      ],
    },
  },
  thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(95)<800'] },
}

export default function () {
  http.get(`${__ENV.BASE_URL}/checkout`)
}