Why Your Automation Keeps Creating Duplicate Records, and How to Stop It

Why does my automation keep creating duplicate records?

Because it creates rather than matches. Most integrations are built to take an input and make a record, and nothing in that flow asks whether the record already exists. Every retry, every resubmitted form, every re-run after a fix produces another one, and none of it looks like an error.

This is the most common data problem I see in automation work, and it is almost never a fault in the destination system. It is a design omission in the thing doing the writing, and it is cheap to fix before it happens and expensive to fix afterwards.

Here is what actually causes duplicates, how to choose the key that prevents them, and what to do about the pile you already have.

What actually causes a duplicate?

Four things, in roughly this order. A retry after a timeout where the first attempt actually succeeded. A person submitting a form twice. The same person arriving through two different entry points. And a re-run of a batch job over records it already processed.

The retry case is the one people underestimate. If your automation calls an API, does not get a response in time, and tries again, the first call may well have worked. From the automation's point of view nothing happened. From the destination's point of view two identical records arrived.

The multiple entry point case is the one that causes arguments, because both records are legitimate. Someone downloads a guide with a personal address and later books a demo with a work address. Neither submission was wrong, and you now have two records describing one human being.

What is a dedupe key, and how do you choose one?

A dedupe key is the field, or combination of fields, that decides whether two records are the same thing. Choosing it is the whole job. Everything else about duplicate prevention is mechanics, and the mechanics are easy once you have decided what sameness means.

The strongest keys are identifiers that your own systems control, such as a customer number or an external ID you generate and store in both places. They are strong precisely because nobody can retype them slightly differently, and because they do not change when a person changes jobs or a company rebrands.

The weakest keys are human-entered text. Names are the classic mistake. Two records with the same name may be two different people, and one person may appear under three spellings. Any key that a human types in a form will eventually contain a typo, and the typo creates the duplicate you were trying to prevent.

Why is email a worse key than people think?

Because it is stable enough to feel safe and unstable enough to fail. People change jobs and lose the address. Teams share a single inbox address. The same person uses a personal address once and a work address later. Each of those produces either a false match or a missed one.

Email is still usually the best available key for contacts, and I use it. The point is to use it knowing what it does not cover, and to combine it with something else where the stakes are high. Email plus company domain is meaningfully better than email alone for B2B, because it survives one of the two common failure modes.

The habit that helps most is normalising before comparing. Lowercase it, trim the whitespace, and decide deliberately whether you treat address variations with tags or dots as the same person. Doing that consistently at every write point removes a surprising share of duplicates without any other change.

Should the automation check first, or use an upsert?

Use an upsert where the destination offers one, because a single operation cannot race against itself the way a check followed by a create can. HubSpot documents an upsert operation for contacts alongside the ordinary create endpoints, and where that exists it is the better shape.

The reason check-then-create fails is timing. Between your lookup and your write, another run of the same automation can insert the record you just failed to find. Both runs then create, both believe they were correct, and you get exactly the duplicate you built the check to prevent.

Where no upsert exists, the fallback is to keep your own record of what you have already written. Store the dedupe key and the resulting record identifier on your side, check that first, and treat it as the authority. It is more work and it is the only reliable option when the destination will not do the matching for you.

What does HubSpot give you once duplicates already exist?

A duplicates manager. HubSpot's documentation describes it as a tool to identify, review, and resolve duplicate contact and company records, and says HubSpot automatically compares record property values daily to surface potential duplicates. You can merge or reject pairs individually or in bulk, and create custom rules to define your own matching criteria.

The access rules are worth knowing before you plan around it. HubSpot's documentation states that a Professional or Enterprise subscription is required to manage individual duplicates and display up to 10,000 duplicate pairs using the duplicates manager tool, and that a Data Hub Professional or Enterprise subscription is required to bulk manage duplicates and display more, at up to 30,000 pairs on Professional and up to 100,000 on Enterprise.

There are permission requirements too. HubSpot lists data quality tools access along with edit permissions set to all contacts or all companies, or super admin permissions. If you are planning a cleanup with someone else's account, check that before you schedule the work rather than during it.

When should a duplicate block the automation rather than merge?

When merging could combine two different people or two different companies. A merge is destructive in the sense that it is hard to reason about afterwards, so anything ambiguous should stop and wait for a human rather than resolve itself confidently.

My rule is that an exact match on a strong key merges automatically, and a fuzzy match on a weak key raises a flag. That means a matching external ID proceeds without comment, while two records with similar names at the same company get routed to someone who can look at them.

The cost of getting this wrong runs in one direction. A missed merge leaves you with two records, which is annoying. A wrong merge combines two customers into one, which is a data incident and occasionally a privacy problem. Bias the rules toward caution.

How do you stop it happening again?

Put the dedupe key in the automation, not in the cleanup tool. A daily deduplication pass is a safety net, and treating it as the primary mechanism means you have accepted that your systems produce bad data and you will tidy it later.

The second prevention is a count you watch. If the number of records created per day is consistently higher than the number of real events, something is duplicating, and you will see it in the counts long before anyone notices in the interface. That is the same comparison habit I described in the daily reconciliation check I build for two connected systems.

The third is to make every writing path use the same matching logic. Duplicates often come from having three integrations that each decide sameness differently, so two of them agree and the third quietly disagrees. One shared rule, applied everywhere, removes that entire category. And if you do run a bulk cleanup on the backlog, keep a record of what you merged, for the same reason I keep a rollback path for any other bulk change in how to roll back an automation change without making it worse.

Does this apply to event-driven automations too?

Especially to those. An event can be delivered more than once, and a receiver that creates a record on every delivery will duplicate for reasons that have nothing to do with your data quality. The fix is the same key, applied to the event rather than to the person.

For anything driven by a webhook, I derive a key from the event itself and record every key already processed, so a repeated delivery is recognised and discarded. That single check is what separates a receiver you can trust from one you have to watch, and I set out the rest of the pattern in setting up a Webflow webhook that fires when your CMS content changes.

The general principle is worth stating plainly. Any automation that can run twice will eventually run twice, so it should be designed to produce the same result when it does. Everything else here is an application of that one idea.

What should you do next?

Open your CRM and search for the two or three contacts you deal with most. If any of them appears more than once, you have a live duplication problem rather than a historical one, and the cause is still running.

Then look at the automation that writes those records and find the line that decides whether to create. If there is no such line, that is the fix, and adding it is usually an afternoon rather than a project.

If you are not sure which key is right for the kind of records you hold, tell me what you are matching and where the data comes from. I will tell you what I would use and what I would watch for. Let's chat.