ServersTooling

Debugging in the fog

No debugger, no repro, one angry graph. How to narrow down a production problem with logs, timing and a written list of guesses.

Alex Rivera

· 6 min read

Clouds rolling over a dark mountain ridge

Write the guesses down

The first ten minutes of an incident decide the next two hours. Before changing anything, write down what you know, what you think, and what would prove each guess wrong. It feels slow; it is the fastest way to stop chasing the same idea twice.

A guess you can't disprove isn't a lead. It's a mood.

Logs you can query

A log line is only useful in an incident if you can filter on it. Log events as objects with a request id, a duration and the few fields you will want to group by later.

ts
const started = performance.now();try {  return await handler(request);} finally {  log.info({    event: "request",    id: request.headers.get("x-request-id"),    route: new URL(request.url).pathname,    ms: Math.round(performance.now() - started),  });}

Bisect in time

Most production problems start at a moment. Line the graph up with deploys, config changes and traffic, and the list of suspects usually shrinks to one or two. If it doesn't, the cause is probably outside your code: a dependency, a certificate, a quota.

  • When did it start, to the minute?
  • What shipped or changed within an hour before?
  • Is it every request, one route, or one customer?

Keep reading