Who Should Get the Alert When an Automation Fails Overnight?

Who should get the alert when an automation fails overnight?

Nobody, unless somebody can and should act before morning. Most automation failures can wait, and waking a person for one that could have waited is how alerting stops working. Decide the severity first, then decide who hears about it and when.

Working from Bengaluru with clients in other time zones, my overnight is somebody else's working day and their overnight is mine. That makes this question concrete rather than theoretical, and it forced me to be precise about it earlier than most people are.

The answer is a small set of rules, written down once, that hold whether the system is run by one person or ten.

Why is alerting everything the same as alerting nothing?

Because attention is the scarce resource. A channel that fires for every retry, every warning, and every successful run trains the reader to ignore it, and the one alert that mattered arrives in a stream nobody reads any more.

I have seen this in every stack I have inherited. The alert channel has thousands of messages, everybody has muted it, and the failure that caused the client to call went unnoticed for two days in plain sight.

So the design goal is not coverage. It is signal. Fewer alerts, each one meaning something specific, is strictly better than complete coverage nobody reads.

What severity levels do you actually need?

Three. Something is broken and money or a customer is affected right now. Something failed and can wait until morning. Something looks unusual and belongs in a weekly review. That is enough granularity for any small operation.

The first tier is the only one allowed to interrupt a person outside working hours, and it should be rare enough that receiving one is genuinely alarming. If tier one fires weekly, it is not tier one.

The third tier should not be an alert at all. It is a report you read on a schedule, and moving things into it is the main way to keep the other two meaningful.

Which failures actually deserve a wake up?

Ones where the damage grows with time and a human can stop it. A sync writing bad data into a CRM, a payment process that has stalled, a workflow sending customer emails in a loop. Each of those gets worse every hour and can be halted by one person with access.

Compare that with a nightly report that did not generate. Nothing degrades. Nobody is affected before the morning. Running it at eight is indistinguishable from running it at two.

The test I use is whether the person woken can do something now that they could not do at nine. If the honest answer is no, the alert waits.

What must an alert contain to be useful at 3am?

What failed, what it affects, whether it is still running, and the single most likely action. A message that says an error occurred forces the reader to open three tools before they can think, and at 3am that is when mistakes get made.

I write these as a sentence a tired person can act on. Name the workflow, name the system it writes to, say whether it stopped or is retrying, and link straight to the run.

Include the blast radius if you can. Twelve records affected is a different decision from twelve thousand, and the alert is the only place that number is cheap to include.

Who is the right recipient in a one person practice?

You, for tier one only, and with a different sound from everything else. The honest limitation of solo work is that there is no rotation, so the discipline has to come from the severity rules rather than from a rota.

What makes this survivable is being strict about tier one. If only genuine damage reaches you overnight, you can afford to be reachable. If everything reaches you, you will turn notifications off entirely, and then nothing reaches you.

For client work, agree this explicitly before anything goes live. Who is called, on which number, for what class of failure, and what they are allowed to do about it. That belongs in the handoff documentation you write for clients rather than in somebody's memory.

What if the client's team should respond instead?

Then route to a role rather than a person, and make sure the role has a real owner. An alert addressed to a shared inbox with no named owner is a message to nobody, dressed up as process.

The rule I apply is that the recipient must have both the access and the authority to act. Alerting a person who then has to wake somebody else adds a step and a delay to a situation whose entire justification was urgency. If the only person with credentials is asleep and unreachable, the honest answer is that this failure is tier two no matter how it feels.

Write down the second person too. What happens when the first does not respond within a stated time is the part everybody skips, and it is the part that matters on the night it is needed.

How do you stop repeated alerts from one failure?

Deduplicate and suppress. Send one alert per distinct failure rather than one per retry, and follow it up only when the state actually changes. A workflow that retries twenty times before giving up should produce a single message, not twenty identical ones competing for the same attention.

Add a recovery message as well. Knowing that something failed and then healed itself at four is genuinely useful, and its absence is what makes people investigate problems that resolved hours ago.

If a failure repeats every night, it is no longer an incident, it is a defect. Move it out of alerting and onto your list of things to fix, because an alert you receive daily has already stopped being an alert.

How often should you review the alerts you received?

Weekly, and briefly, looking for exactly two things: alerts that fired without mattering, and problems you discovered some other way. The first list tells you what to downgrade or delete. The second list tells you what your alerting does not yet cover, which is the more valuable of the two.

That second one is the important half. Any incident a client reported to you before your system did is a gap, and it is worth more than any number of successful alerts.

Keep the review in the same session as the rest of your automation hygiene, next to checking whether an automation is still working at all. They answer the same question from opposite directions.

What should you do next?

Take your automations and sort each one into the three tiers. Most will land in tier two, which means they should never reach anyone overnight, and that single change usually quietens a channel enough to make it readable again.

Then rewrite your tier one alert message so a tired person can act on it without opening anything else. Name the workflow, the affected system, the state, and the first action.

If you want help setting the tiers for a stack you did not build, reach out. It is usually an afternoon of work and it is the difference between a system you trust and one you check anxiously. Let's chat.