What Is Your Rollback Plan When an Automation Goes Wrong?

What is your rollback plan when an automation goes wrong?

If you cannot answer that in one sentence, you do not have one, and the time to discover that is not while three hundred wrong emails are going out. A rollback plan is a decision you make before you build, because it changes what you build. Added afterwards, it is usually not possible at all.

The uncomfortable truth about automation is that the thing making it valuable, which is that it acts without you, is the same thing that makes it dangerous. A person making a mistake makes one mistake. A workflow making a mistake makes the same mistake at machine speed until somebody stops it.

I maintain automations in production, including the Airtable and WhaleSync work behind Ajust and the HubSpot and Zapier automation I run for Kismet Health. The build is the easy part. Knowing what you would do if it went wrong is the part that decides whether you should have built it.

Why can most automations not actually be rolled back?

Because they do things that cannot be undone. An email that was sent has been read. A record deleted from a system that keeps no history is gone. A webhook that fired into somebody else's platform has left your control entirely. Reversibility is not a feature you can add later, it is a property of the actions you chose.

There is a useful distinction between actions that change your own data and actions that reach the outside world. Internal changes can usually be repaired, given records of what changed. External actions cannot, because the recipient already saw it, and no amount of cleanup reaches them.

So the first design question is not how to undo things. It is which steps are irreversible and whether those steps need to be automatic at all. The most valuable rollback plan is usually a decision, made early, to put a human between the workflow and anything it cannot take back.

What is the first thing to do when something goes wrong?

Stop it, before you understand it. The instinct to diagnose first is natural and wrong, because every minute spent understanding is a minute the workflow keeps running. Turn it off, then investigate, and accept that you might be turning off something that was fine.

This only works if turning it off is quick, which means knowing where the switch is before you need it. For every automation you depend on, you should be able to name the exact place you would go to pause it and be confident you could do that from a phone. If pausing requires finding a login you do not have, you have a plan that fails under the only conditions it exists for.

Turning it off also includes the upstream trigger where possible. Pausing the processing step while the queue keeps filling means you have delayed the problem rather than stopped it, and the backlog will run the moment you restart. Stop the source, not just the consumer.

How do you work out what it already did?

By reading the run history before you change anything. The temptation is to start fixing records immediately, and doing that destroys the evidence of what happened, because your repairs and the original damage become indistinguishable. Export or screenshot the log first, then work from that copy rather than from the live system.

What you are establishing is the blast radius. Which records were touched, in what window, and what exactly was changed on each. That list is the thing everything else depends on, and assembling it carefully is worth the twenty minutes even when it feels like time you do not have.

This is much easier when records carry their own history, which is the argument for writing state into the data rather than inferring it from run logs. I made the same point about designing handoffs so every record carries its own state, and incident response is where that decision pays for itself.

What does a real recovery look like?

Targeted repair of the affected records, using the list you just built, rather than a general re-run. Re-running a workflow to fix its own mistake is the most common way a small incident becomes a large one, because the second run acts on data the first run already changed.

Work in small batches and check after each one. Fix ten records, look at them, then fix the next ten. This feels slow and it is the fastest reliable method, because a repair script that is subtly wrong stops after ten records instead of after four hundred.

Where the damage reached outside, recovery is communication rather than data work. A correction message sent quickly, naming what happened plainly, costs far less than the same message sent three days later after people have drawn their own conclusions. That is true for customers and doubly true for a client whose system you built.

What should you build in advance to make this possible?

Three things, and none of them are complicated. A record of what changed, so you can tell what to repair. A way to stop everything quickly, so the damage has an end. And a limit on how much can happen in a given period, so there is a window in which somebody could notice before it finishes.

The record is a log of every write, with the record identifier, the field, the old value, and the timestamp. Systems that keep revision history give you this free. Systems that do not need you to write it yourself, into a table nobody else touches. This is the single highest-value thing to add to any automation that modifies data.

The limit is a rate cap or a batch size. A workflow that can process a thousand records in a minute can also ruin a thousand records in a minute, and there is rarely a business reason it must move that fast. Slowing a workflow down deliberately buys you the window in which monitoring can catch something, which is the whole point of watching for silent failures.

Should you test the rollback itself?

Yes, and almost nobody does. A rollback procedure that has never been executed is a hypothesis. The first time you run it should not be during an incident, because that is when you discover that the export is missing a field or that the restore needs a permission you do not have.

The test does not need to be elaborate. Break something small on purpose in a safe environment, then recover it using only your written procedure. If you have to improvise, the procedure is incomplete and you have learned that cheaply.

Do this once when you build, and again if the systems involved change significantly. It is the same argument as testing with deliberately awkward data before going live, which I have covered in rehearsing an automation before real customer data. Rehearsal is what separates a plan from an intention.

What should change after an incident?

The design, not just the bug. Fixing the specific fault that caused this incident is necessary and insufficient, because the reason it did damage was almost always structural rather than incidental. The fault was only the trigger. The blast radius, the speed, and the absence of anything to stop it were the design.

The questions worth asking are why it ran unattended, why nothing stopped it earlier, and whether the irreversible step needed to be automatic. Those produce changes that prevent a whole category of incidents rather than one repeat, and they are easier to justify immediately afterwards than at any other time.

Write it down while it is fresh, in a few honest paragraphs. What happened, what the impact was, what you changed. Not for ceremony, but because the same workflow will be modified in eight months by somebody who needs to know why the batch limit is there. Undocumented safety measures get removed as unnecessary friction.

What should you do next?

Pick your riskiest automation and answer three questions right now, without opening anything. Where is the off switch. Could you reach it from your phone on a Saturday. What record exists of what it changed yesterday. Any one you cannot answer is a gap worth closing today.

Then look at the single most irreversible thing that workflow does and ask honestly whether it has to be automatic. Sending, deleting, charging, publishing. Putting one human confirmation in front of that step costs a small amount of convenience and removes most of the scenarios where you would need a rollback plan at all.

If you are running automations where you genuinely do not know what would happen if they went wrong, that is worth an afternoon before anything else gets built on top. Reach out if you want help thinking through the failure cases.