Why Does Your Automation Keep Creating Duplicate Records?

Why does your automation keep creating duplicate records?

Almost always because something retried and nothing was watching for a repeat. Networks time out, webhooks fire twice, a polling trigger sees the same row again after a change. The automation does exactly what you told it: create a record. It creates another one, because nobody told it the first one already exists.

This is the failure mode I get called about most often, and it is never dramatic. Nothing errors. No alert fires. You find out three weeks later when someone in sales says the same lead is in the CRM four times and the follow-up emails went out four times with it.

The fix has a name. It is idempotency, and it means an operation produces the same result whether you run it once or ten times. Most good platforms have a version of it built in. The trouble starts in the gaps between them, which is where most of us actually build.

What is an idempotency key, actually?

It is a unique string you attach to a request so the receiving system can recognise a retry. Stripe's API reference describes the pattern plainly: the API supports idempotency for safely retrying requests without accidentally performing the same operation twice, and when creating or updating an object you supply an idempotency key.

The mechanism is worth understanding because it is more subtle than it sounds. Stripe's documentation says its idempotency works by saving the status code and body of the first request made for any given key, regardless of whether that request succeeded or failed. Subsequent requests with the same key return that same saved result, including 500 errors.

Read that twice. A retry does not get a second chance at the operation. It gets a replay of what happened the first time. That is the whole point. If the first attempt failed, you learn it failed rather than accidentally creating a second object while trying again.

Stripe's guidance on the keys themselves is specific. It suggests version 4 UUIDs or another random string with enough entropy to avoid collisions, notes that keys can be up to 255 characters long, and advises against using sensitive data such as email addresses or personal identifiers as keys. It also says keys can be removed automatically once they are at least 24 hours old, and that reusing a key after the original is pruned generates a new request.

How does Zapier decide whether data is new?

Through deduplication on the trigger. Zapier's help documentation says that in trigger steps, Zapier uses a process called deduplication to ensure only new data triggers a Zap. It compares each piece of data's unique ID against IDs that Zap has seen before. A new ID runs the Zap. A familiar one does not.

The developer platform documentation is more specific about how that works for polling triggers. It says polling triggers deduplicate using unique id fields as primary keys, reverse-chronological sorting, and 30-second polling intervals, and that by default the field with the key id is used as the primary key. It states directly that for deduplication to work, the id field should always be supplied and unique among all items in the result.

There is a detail in there that surprises people. Zapier's docs describe what happens when a Zap is first switched on: Zapier makes an initial call to your API, then caches and stores each id field. When the Zap is turned off, that stored list is cleared. So turning a Zap off and on again is not a neutral act. It resets the memory of what has already been seen.

Two more constraints from the same page are worth writing on a sticky note. Your API must return results in reverse-chronological order, because Zapier's polling triggers do not automatically fetch additional pages. And all requests and processing code for a trigger must finish within 30 seconds. If your source is slow or sorted the wrong way, deduplication is not the thing that breaks first, but it will break eventually.

Why do instant triggers behave differently?

Because they are built on a different assumption. Zapier's help documentation states that instant triggers do not use deduplication, because apps only send new data. That is a reasonable assumption about a well-behaved sender. It is not a guarantee about every sender.

The consequence is straightforward. If a system pushes the same event twice, whether from a retry, a misconfigured integration, or someone clicking submit again, there is no deduplication step on that path to catch it. Whatever your action step does, it does twice.

I do not read that as a flaw. It is an honest division of responsibility. Polling means Zapier sees your whole list and can compare, so it dedupes. Pushing means the sender decides what to send, so the sender owns correctness. The mistake is assuming the same safety net exists on both paths when the documentation says it does not.

This is the single most common reason I find duplicate records when I audit someone's setup. They tested with a polling trigger, watched deduplication work, then switched to an instant trigger for speed and carried the assumption across. The behaviour changed and nothing announced it.

What happens when two automations watch the same trigger?

They both run. Zapier's help documentation says that if you have more than one Zap using the same trigger, such as two Zap workflows watching the same form, they will all trigger, because the deduplication system only checks within the same Zap.

That line explains a whole category of confusing duplicates. Deduplication is scoped per Zap, not per source. So the moment you split one workflow into two, perhaps one to notify Slack and one to write to the CRM, you have two independent memories of what has been seen. If both write records, you get two records, and both systems believe they are behaving correctly.

The practical version of this rule: decide early whether a source feeds one workflow that branches, or several workflows that each listen. The first is easier to reason about. The second is easier to build. I have shipped both, and I now default to the first whenever anything downstream writes data rather than just reading it.

How do you build a dedupe check when the tool will not do it for you?

You keep your own list of what you have already processed, and you check it before you write. That is the entire idea. The implementation is boring, which is a feature.

The shape I use looks like this. Every incoming event needs a stable identifier that the source controls, not one you generate on arrival. A form submission ID, an order ID, a message ID. If the source gives you nothing stable, you can build one by hashing the fields that define uniqueness, though that is a weaker guarantee and I treat it as a fallback rather than a plan.

Then you need somewhere to remember it. In the Ajust automation I run on Airtable with WhaleSync, which has delivered more than 25,000 cases, the memory is a table. A dedupe table with the source ID, a timestamp, and the outcome is enough. Before the write step, you search that table. If you find the ID, you stop. If you do not, you write the record and then write the ID.

The ordering matters more than people expect. Write the record first, then the marker, and a crash between the two leaves you duplicating on retry. Write the marker first and a crash leaves you silently dropping a record that was never created. Neither is free. You are choosing which failure you would rather explain, and for most marketing and sales workflows I would rather explain a duplicate than a missing lead. Choose deliberately, then write down which one you chose, because your future self will not remember. I have made the same argument about planning a rollback for an automation that writes to your CRM, and the reasoning is identical.

What does Stripe's design teach you about retries?

That the safest retry is one that cannot surprise you. Stripe's documentation says the idempotency layer compares incoming parameters against those of the original request and errors if they are not the same, which prevents accidental misuse of a key.

That is a good habit to copy even without a payments API. If you reuse an identifier, reuse it for the same operation with the same data. An identifier that means one thing on Monday and another on Tuesday is worse than no identifier, because it gives you confidence you have not earned.

Stripe also notes that results are saved only after execution of an endpoint begins, and that if incoming parameters fail validation or the request conflicts with another request executing concurrently, the result is not saved. In plain terms: a request that never really started is not remembered as having happened. That is the correct behaviour, and it is the behaviour your own dedupe table should imitate. Mark work as done when it is done, not when it is attempted.

When is a duplicate record actually the cheaper outcome?

More often than automation enthusiasts admit. A duplicate row in a reporting table costs you a confusing number. A duplicate row in a billing table costs you a customer. The correct amount of engineering depends entirely on what sits downstream, and treating every workflow as equally critical is how small teams end up maintaining infrastructure instead of marketing.

Zapier's own documentation makes the downstream dependency explicit on the action side. It says that if an app allows duplicate data, Zapier will create a duplicate record, and if an app does not allow duplicate data, Zapier will return an error message you can see in the Zap run details. So part of your answer is already decided by the destination, and it is worth checking what your destination does before building protection it already provides.

My rule after six years of building these things is to grade by blast radius. If a duplicate triggers an outbound email, a payment, or anything a customer sees, build the dedupe check. If it lands in a spreadsheet you clean weekly, do not. Spend the effort where a mistake reaches a person. The same triage logic applies when you are trying to stop an AI automation from sending bad data to your CRM, where the question is never whether checks are good but which writes deserve them.

What should you do next?

Open your most important automation and answer one question: if this exact event arrived twice, what would happen? If you cannot answer confidently, that is your project. Add a stable source identifier, add a check before the write, and decide on purpose whether you would rather duplicate or drop.

Then do the boring part. Write down which failure you chose and why, next to the automation itself. Most of the damage I have cleaned up was not caused by a bad decision. It was caused by a reasonable decision nobody recorded, inherited by someone who assumed the opposite. If you are weighing platforms while you are at it, I have compared Make, Zapier, and n8n for small businesses and the same question applies to all three.

If you want a second pair of eyes on an automation that has started producing records you cannot explain, reach out. Untangling this is most of what I do, and it is usually simpler than it looks from the inside.