How Do You Stop a Content Pipeline Publishing a Duplicate Post?

How do you stop an automated content pipeline from publishing the same post twice?

Check in two ways. First, check for an exact duplicate: the same slug or the same source record already in the CMS. Second, check for a near duplicate: a new topic that covers the same reader problem as an existing post under a different title. Exact checks are easy. Near-duplicate checks need a scoring rule and a human threshold.

Content pipelines are now common. A team uses an AI model to draft articles, a database like Airtable to manage the queue, and an automation to push finished items into a CMS such as Webflow. It is fast, and when it works, it frees writers to focus on ideas and editing.

The failure that shows up first is duplication. A retry creates the same item twice. A topic list generated in March overlaps with one generated in June. Two people queue the same idea with different wording. Without checks, the archive slowly fills with posts competing against each other.

Why are duplicate posts a problem for search and AI answers?

Duplicate and near-duplicate posts split your signals. Search engines have to choose which page to show, and they may choose the weaker one. AI assistants may cite whichever version they find first. Readers who land on two similar posts lose trust. One strong page on a topic beats three thin ones.

There is also a maintenance cost. When a fact changes, every duplicate needs updating. Teams rarely remember all of them, so the older copies drift out of date and contradict the newer one. That kind of contradiction is exactly what makes a site harder for AI systems to describe accurately.

So deduplication is not housekeeping. It is part of quality control, and it belongs inside the pipeline, before anything is created, not in a cleanup project months later.

How do you prevent exact duplicates?

Give every item a stable key and check for it before you create anything. The slug is the natural key in most CMSs. Before the create step, look up the slug in the CMS. If it exists, stop. Also store a unique ID from your source table on the CMS item, so retries can match on it.

Exact duplicates usually come from retries. An automation times out after the CMS has already created the item, retries, and creates a second copy. The fix is the same idempotency idea used everywhere in automation. I covered the general pattern in how idempotency keys stop duplicate records in automations.

Check how your CMS behaves when you send a slug that already exists. Some systems reject the request. Others accept it and quietly modify the slug, so you end up with a second item that has a slightly different URL. If yours does the second, your pipeline needs to catch it, because the create call will look like a success.

On Webflow, the Data API's list endpoint can filter collection items by slug, which makes the lookup a single call. I run that lookup immediately before every create, not just at the start of a batch, so items created by another process in the meantime are still caught. My piece on what the Webflow CMS API can and cannot do for a content pipeline covers the wider API picture.

What is a near duplicate, and why is it harder?

A near duplicate is a new post that answers the same reader question as an existing one, with different words. "How to set up lead routing in HubSpot" and "HubSpot lead routing setup guide" are different strings but the same article. No exact-match check will catch them. You need a similarity rule.

Near duplicates come from topic generation. AI models are good at producing plausible topic lists, and they will happily produce variations of topics you already covered. Without a check against the live archive, each new list adds a few posts that compete with older ones.

The hard part is that some overlap is fine. Two posts can share keywords and still serve different readers. A setup tutorial and an audit checklist on the same tool are different jobs. The check has to flag overlap for review, not reject everything that shares a word.

How do you score topic overlap in a pipeline?

Strip common words from each candidate slug or title, reduce the rest to simple stems, and count how many stems the candidate shares with every live post. Set two thresholds: a high one that blocks the topic automatically, and a middle band that sends it to a person to compare angles. Below that, let it through.

The stopword list matters. Words like your brand, the year, "guide," "how," and "what" appear in most slugs and add noise. Remove them first, or every post looks similar to every other. What remains should be the words that describe the actual topic.

A simple stem count is crude, and that is a feature. It is easy to explain, cheap to run against thousands of slugs, and good at catching the obvious repeats. For a large archive, you can add an embedding similarity check on titles and excerpts later. I would start with the simple version and see what it misses.

Log the top few overlaps for every approved topic. When someone later asks why two similar posts both exist, the log shows that the overlap was reviewed and the angles were judged different.

Where should the duplicate checks sit in the pipeline?

Run the near-duplicate check when topics are chosen, before any drafting, so you do not spend money writing articles you will reject. Run the exact check again immediately before the create step, so retries and parallel processes cannot slip a duplicate in. Both checks should read the live CMS, not a cached list.

The live part matters. A cached slug list from the start of the day will not include posts published an hour ago. If two runs work from the same cache, they can both approve the same topic. Pulling the current list at the start of each run, and checking the exact slug again at create time, closes that gap.

A staging step helps too. Drafts that land in a review table before they reach the CMS give you one more chance to spot a duplicate. I explained that pattern in why a staging table should sit before content reaches your CMS.

What should happen when a duplicate slips through anyway?

Decide which version to keep, usually the stronger or older one with more links. Merge anything useful from the other into it, redirect the weaker URL to the kept one, and remove the duplicate from the CMS. Then find out which check failed and fix it, so the same path cannot produce another.

Redirects are the step teams skip. If the duplicate has been live for even a few days, it may have been crawled or shared. A redirect keeps any value it gathered and sends readers to the right page. Simply deleting it leaves a broken link.

Treat each slip as a test failure, not an accident. Was the slug check skipped? Did a retry bypass it? Was the near-duplicate threshold too loose? The answer tells you exactly what to change.

What should you do next?

Add an exact slug lookup immediately before every create step, and store a source ID on each CMS item. Then add a near-duplicate score at topic selection, with a block threshold and a review band, run against the live archive. Log overlaps for approved topics, and clean up any existing duplicates with redirects.

A content pipeline that cannot tell when it is repeating itself will eventually compete with its own archive. Two simple checks, placed at the right points, are enough to stop that.

If you want help building duplicate checks into your content pipeline, or reviewing the automation behind your CMS, reach out. This is the kind of AI automation QA I build into the content systems I set up.