What we monitor in production: An opinionated OpenTelemetry and alerting guide
The quickest way to destroy engineering morale is to configure alerts that wake up on-call developers at 3:00 AM because an auxiliary server had a brief CPU spike to 91%. Inevitably, the engineer wakes up, checks the site, sees users are unaffected, silences the alert, and goes back to sleep. Within three months, genuine outages get ignored because everyone is suffering from alert fatigue.
Instrumenting with vendor-neutral OpenTelemetry
Never lock your application code into proprietary APM SDKs. We instrument our applications exclusively with OpenTelemetry (OTel):
- Unified tracing, metrics, and logs across distributed services - Automatic W3C trace context propagation across HTTP headers - Vendor agnostic: we can switch our exporter from Datadog to Grafana Tempo or Honeycomb by editing one environment variable without rewriting code
Alert on symptoms, not causes
A user does not care if server CPU is at 95% if pages load in 200ms. A user does care if pages are failing with HTTP 500 or taking 15 seconds to load.
We structure alerts strictly around Service Level Objectives (SLOs):
- Latency: P95 response time exceeding 500ms over a 5-minute rolling window - Availability: HTTP 5xx error rate exceeding 1% of total traffic - Ingestion: Background queue processing lag growing faster than drain rate
Distributed tracing saves hours
When a checkout fails, looking at isolated server logs is useless. Distributed tracing connects the browser button click, the Next.js server action, the auth microservice, the Stripe call, and the database query under a single Trace ID. You can see in a single waterfall timeline that the database took 12ms, but a third-party fraud detection API took 4,200ms.
Have a similar challenge?
We build production-grade software for companies that need it done right.
Let's Talk
