What Breaks First When an Automation Starts to Scale?

What breaks first when an automation starts to scale?

Rate limits, almost always. Your logic is usually fine at a hundred records and fine at a thousand, but the platforms you are calling publish hard ceilings on requests per minute, and a workflow that hums along on a small dataset will hit those ceilings the first time someone runs it on the whole table.

This is the least glamorous failure mode and the most common one. It is also the easiest to design around, because unlike most scaling problems the limits are published in advance. You can read the number before you write the automation, which is not true of much else in this work.

What follows is the order I actually see things fail in, with the published limits from the platforms I build on most.

What do the published rate limits actually say?

They are specific and they differ by plan. Webflow's Data API documentation gives 60 requests per minute on Starter and Basic site plans, 120 on CMS, eCommerce and Business plans, and custom limits on Enterprise. Airtable's documentation states a limit of 5 requests per second per base.

Both describe the same failure signature. Webflow says exceeding the limit returns an HTTP 429 Too Many Requests, accompanied by a Retry-After header telling you how long to wait, and notes that this reset time is typically 60 seconds. Airtable says you will receive a 429 status code and will need to wait 30 seconds before subsequent requests succeed.

Airtable adds a second ceiling that catches people running several automations from one account: a limit of 50 requests per second for all traffic using personal access tokens from a given user or service account. Two well behaved automations can be badly behaved together.

Why does this catch people so late?

Because development happens at a scale where the limit is invisible. Testing against twenty records at a comfortable pace never approaches 60 requests a minute. The first production run against 800 records does, and the failure arrives at the worst possible moment, which is the moment something is finally being useful.

The maths is worth doing once, deliberately. If you need one request per item and you have 800 items, that is at minimum roughly fourteen minutes on a 60 per minute ceiling, before you count the requests you make to check things. Most people discover this by watching a run fail rather than by dividing two numbers.

There is also a compounding effect that surprises people. An automation that reads before it writes makes two requests per item, not one, and one that verifies afterwards makes three. Your ceiling in items per minute is the request ceiling divided by requests per item, and that divisor is usually bigger than you assumed.

What breaks second?

The budget, not the code. Zapier's documentation is explicit that each successful tool call through an MCP server consumes two tasks at a fixed rate, and that failed calls do not consume tasks. It also spells out that adding five rows to a spreadsheet is five tool calls and ten tasks.

Zapier also states what happens at the ceiling: if you reach your plan's task limit, MCP tool calls stop working until your allowance resets at the start of the next billing period or you upgrade. That is a hard stop rather than a slowdown, and it arrives without any code having changed.

The uncomfortable part is that the budget failure looks exactly like a working automation that stopped caring. Nothing errors in your logic. The platform simply declines. I wrote about modelling this before you deploy in budgeting automation costs before they run away.

What breaks third?

Your assumptions about the data. At small volume you have effectively seen every record. At large volume you have seen a sample, and the records you never looked at contain the empty fields, the unexpected characters, the duplicate names and the row somebody created in 2019 with a different convention.

This is where automations start producing wrong results rather than no results, which is considerably worse. A rate limit failure is loud. A field that is null for three percent of records and silently becomes the string undefined in an output is not.

The defence is to run the automation in read-only mode across the entire dataset before you let it write anything. Count the records that would fail your assumptions. That single dry run finds more real problems than any amount of careful coding against the twenty rows you happened to test with.

What breaks fourth, and worst?

Your ability to notice. A small automation is watched. A scaled one runs unattended, and the gap between something going wrong and someone finding out grows with exactly the success that made it worth scaling.

The specific danger is the silent partial failure. Half the batch succeeded, the rate limiter rejected the rest, the run reported completion because nothing threw, and the state is now inconsistent in a way nobody will detect until a customer does. I have come to think unattended automations need a success metric more than they need error handling.

So define what a normal run looks like numerically, and alert on the absence of that number rather than on the presence of an exception. Processed zero records is a far more common symptom of breakage than crashed. That thinking belongs in the runbook, which I argued for in writing an automation runbook before you ship.

How do you design around the published limits?

Back off, batch, and cache, in that order of importance. Airtable's documentation says that on receiving a 429 you should back off and wait before retrying, and notes that its official JavaScript client has built-in back-off and retry logic. Using the vendor's client rather than raw requests solves this for free.

Batching is the second lever and usually the bigger one. If an API lets you create or update several records in one call, the number of items you can process per minute goes up by that multiple without touching the rate limit. This is why I check the batch size of every write endpoint before designing a loop.

Caching is the third and the most overlooked. Airtable explicitly recommends a caching proxy if you anticipate a higher read volume. Webflow makes an equivalent point from the other side: it says cached requests to its content delivery API effectively have no rate limits, while uncached requests to the origin server do count against your plan's limit.

Which limits are easy to miss entirely?

The endpoint-specific ones. Webflow's documentation notes that while general limits apply to most requests, some endpoints carry additional constraints, and gives site publish as an example, limited to one successful publish per minute. A workflow that publishes after every item will fail on its second item regardless of your overall budget.

Per-key accounting is the other easy miss. Webflow states that rate limits are applied on a per API key basis, so each key is tracked independently. That is useful, because splitting workloads across keys is a legitimate design lever, and it is also a trap when you assume a shared key is fine because each automation is small.

Airtable's account-wide ceiling works in the opposite direction and needs the opposite reflex. Per base you have room. Across every automation using one personal access token, you share a pool. Design for whichever accounting model the vendor actually uses rather than the one that would be convenient.

Should you scale the automation at all?

Not always, and this is the question I ask before the engineering ones. An automation that works well at small volume because a human glances at its output has that human as part of its design. Removing them is not scaling the same system, it is deploying a different one that happens to share the code.

The honest test is what happens when it is wrong. If a bad output at small scale means somebody notices and fixes it in a minute, and a bad output at large scale means four hundred wrong records and a client email, the risk profile changed even though the logic did not.

Sometimes the right answer is to keep the human and raise the volume more slowly, or to add an approval step for the subset that fails a confidence check. I went through the version of this question that comes earlier in when not to automate a workflow.

What should you do next?

Before your next run, look up the published rate limit for every API your automation touches and write the numbers at the top of the file. It takes ten minutes, the vendors publish them plainly, and it converts the most common scaling failure into an arithmetic problem you solve in advance.

Then do one read-only pass over the full dataset and count how many records would break your assumptions. Those two steps catch most of what goes wrong in the first four failures described above. If you want someone to look at an automation before you point it at real volume, reach out and tell me what it touches.