Back to Insights
Cloud & DevOpsYesterdayJustin Pennington

Alerts Nobody Reads: Observability a Three-Person Team Will Actually Use

Alerts Nobody Reads: Observability a Three-Person Team Will Actually Use

Most lean teams have one of two monitoring problems. Either nothing is watching the systems that run the business, and you find out about an outage when a customer emails. Or something is watching everything, firing dozens of notifications a day into a Slack channel that everyone has muted. Both end the same way: an incident nobody caught until it was expensive.

Observability doesn't require a platform team or a six-figure tooling contract. It requires deciding what actually matters, wiring up a small number of honest signals, and making sure each one points to a human and a next step. Here's how to build that without hiring anyone.

Watch Symptoms, Not Causes

The most common mistake is alerting on infrastructure conditions: CPU above 80 percent, disk at 70 percent, memory climbing. Those are interesting on a dashboard and terrible as alerts. CPU spikes all the time without anyone noticing. Meanwhile your checkout can be broken while every server looks perfectly healthy.

Alert on what a customer or an employee would notice:

  • Can people log in, and how long does it take?
  • Are orders, quotes, or form submissions still flowing at the rate you'd expect for this hour?
  • Are the integrations between your ERP, CRM, and website completing successfully?
  • Are error rates on your most important pages or endpoints rising?
  • Are nightly jobs finishing, and finishing on time?

That's most of what a small business needs. Five or six symptom-level checks catch the overwhelming majority of incidents that actually cost money. Resource metrics still belong on a dashboard you review weekly, just not in the channel that wakes someone at 2 a.m.

Make Silence Meaningful

A missing signal is a signal. If your order sync usually processes records every fifteen minutes and it has processed zero for two hours at midday, that's an incident — even though nothing errored. Failures that show up as absence are the ones that hide longest, because error dashboards stay clean and everybody assumes the quiet means things are fine.

Build heartbeat checks for anything that runs on a schedule: nightly ERP batches, inventory pushes, payment reconciliation, backup jobs. The pattern is simple. The job reports in when it completes; if the report doesn't arrive inside the expected window, something alerts. This single habit catches the class of failures that turns into a week of data cleanup.

One Alert, One Owner, One Action

An alert that arrives without an owner becomes everyone's problem and therefore nobody's. Before you turn on any notification, answer three questions in writing: who receives this, what do they do first, and what happens if they don't respond in fifteen minutes.

Attach a short runbook to each alert — even three lines is enough. Where to look, what usually causes this, how to confirm it's resolved. On a small team, the person on call at 2 a.m. is often not the person who built the system, and the runbook is the difference between a ten-minute fix and a two-hour archaeology dig.

Then be ruthless about pruning. If an alert has fired more than a few times without anyone taking action, it isn't an alert. It's noise. Either fix the underlying condition, adjust the threshold, or demote it to a dashboard. Alert fatigue is not a discipline problem; it's a design problem.

Structure Your Logs Before You Need Them

When something breaks across an integration, you'll want to trace a single record — one order, one customer, one invoice — from the website through the middleware into the ERP. That's only possible if each system logs a shared identifier and each log line is machine-readable rather than a paragraph of free text.

Deciding this in advance costs almost nothing. Deciding it during an outage costs hours. Pick an ID that travels with the transaction, log it at every hop, and make sure the logs from your different systems land somewhere you can search together. This is where AI-assisted analysis genuinely helps: give a model structured logs and it will summarize a failure pattern far faster than a human scrolling. Give it unstructured mush and it'll guess.

Keep the Telemetry Bill Honest

Observability tooling prices on volume, and it is easy to end up paying more to watch a system than to run it. Sample high-volume debug logs, keep full fidelity on errors and transactions, and set retention deliberately — most teams need thirty days of searchable detail and longer retention only for compliance-relevant records. Review what you're ingesting quarterly, the same way you'd review any recurring vendor spend.

The Weekly Ten Minutes

The habit that separates teams that stay stable from teams that lurch between fires is a short recurring review. Once a week, look at what alerted, what got ignored, what almost broke, and which trend lines are drifting the wrong way. Fix one thing. That's it.

This is the unglamorous work that keeps growing companies from discovering their systems' limits at the worst possible moment. If your operations depend on integrations and jobs nobody is actively watching, we'd be glad to look at what's running blind and help you close the gaps — reach out and let's talk through it.