Who Is Accountable When an AI Agent Acts on Your Site?

Who is accountable when an AI agent acts on your site?

The person who sent it, according to the companies building these agents. Google, Cloudflare, OpenAI, and Perplexity all draw the same line in their own documentation: a fetch triggered by a human is treated as that human's action, and it is governed differently from an automatic crawl.

That sounds like a technicality until you realise it decides which of your controls applies. Your robots file governs crawlers. It does not govern a person using a tool, and the vendors say so explicitly rather than leaving it implied.

I want to work through what that actually means for a site owner, because the practical consequences are real and most of the advice circulating right now is still written as though every non-human request is a crawler.

How do the vendors themselves answer this?

By naming the category. Google's documentation on verifying its crawlers splits them into three types, and the third is the interesting one. It calls them user-triggered fetchers, describes them as tools and product functions where the end user triggers a fetch, and states that because the fetch was requested by a user, these fetchers ignore robots.txt rules.

Cloudflare defines the same thing from the other side. Its bot documentation says it classifies bots by behaviour rather than by a single AI label, and defines the agent behaviour as automated activity acting in real time on a person's behalf to get something done, naming chat fetch bots and browser-use agents as examples. It also notes that a single bot can have more than one behaviour.

The AI companies use the same split in their own crawler documentation. OpenAI names a separate agent for certain user actions in ChatGPT and Custom GPTs, distinct from the crawlers it uses for search and for training. Perplexity does the same, describing one agent as supporting user actions within Perplexity and controlling which sites those user requests can access.

Four companies, four documentation sets, one consistent distinction. An automatic crawl is the company's action. A fetch someone asked for is the person's action, carried out by the company's software. That is the answer they have all converged on, and it is worth knowing before you form your own view.

Why does acting on a person's behalf change your rules?

Because a robots file is a request to automated crawlers, not an access control system, and it was never intended to govern what a person does with a browser. When the software is standing in for a person, the vendors treat it as being on the person's side of that line.

Here is the concrete version. If somebody opens your page in a browser, reads it, and copies a paragraph, no directive of yours applied. If somebody asks an assistant to open your page and tell them what it says, the vendors' documented position is that the same logic holds. The intermediary changed. The requester did not.

You can disagree with that framing, and plenty of publishers do. My point is narrower and more useful: whatever you think should be true, the documented behaviour is what will actually happen on your server tomorrow, so your controls need to be designed against the documented behaviour rather than the preferred one.

The practical translation is simple. If you want to stop automatic crawling, robots directives are the documented tool and well-behaved crawlers are documented as obeying them. If you want to stop or shape requests made on a person's behalf, you need an access rule at the edge, because a directive will not do it.

What does this mean for your robots file?

That it is doing less than you think, and doing that part well. A robots file remains the right control for the thing it controls, and it is the wrong control for everything people currently complain about.

Google's own categories make the split legible. Common crawlers, the ones used for its products, are described as always respecting robots rules for automatic crawls. Special-case crawlers may or may not. User-triggered fetchers ignore them because a user asked. Three behaviours, one file, and only one of the three fully bound by it.

So the audit question is not whether your robots file is comprehensive. It is whether each rule in it is aimed at something that is documented to obey it. A directive aimed at a user-triggered fetcher is a comment, not a control, and the request will arrive regardless.

This is also why identity verification matters more than it used to. Before you decide which category a request belongs to, you have to know it is genuinely from who it claims, and I have written the method for that in proving a crawler is who it says it is. Category-based reasoning on unverified user agents is reasoning on self-reported data.

Who is accountable for an agent you sent somewhere?

You are, by the same logic, and this is the half most people skip because it is less comfortable. If the vendors treat a user-triggered fetch as the user's action, then when you point an agent at somebody else's site, that is your action carrying your name.

I run agents against my own systems and against public documentation, and I hold myself to the rule that if I would not do it manually at that rate, I do not do it automatically. That is not a legal position. It is a practical one, and it keeps me on the right side of every argument I have seen so far.

The specific behaviours worth self-policing are volume, repetition, and access to things that were not obviously public. An agent that fetches a documentation page once is a person reading documentation. An agent that fetches four hundred pages an hour is a crawler, whatever the request headers say, and the person who started it owns that characterisation.

There is a governance version of this inside your own business too. Webflow's MCP documentation states that every agent works within your existing permissions and roles, and that each site can provide its own agent instructions to guide how agents work on it. So when an agent acts on your own site, the accountable party is whoever granted it that role, which is a question with a name attached rather than a shrug.

What evidence would you actually have?

Less than you would want, unless you set it up first. This is the gap I find most often, and it is entirely fixable in advance and almost impossible to fix afterwards.

For actions inside your own systems, the evidence is an activity log, and it usually requires its own permission. Webflow exposes this as separately grantable scopes, one for site activity and one for workspace audit logs, which means you can grant an agent the ability to work without granting it the ability to read the record of its own work.

For requests arriving from outside, the evidence is request-level data with the source address preserved, which is why this usually has to live at the edge rather than at the host. Once you have the address, the vendors' published address lists let you sort a request into a category rather than guessing from a name.

Write down what you would need before an incident forces you to want it. The list is short: what happened, when, from where, under whose credential, and whether the request was verified. A site that can answer those five is a site that can have a calm conversation about an agent. A site that cannot is reduced to asserting things it cannot show.

Does this change how you design forms?

It should, and mostly in the direction of being clearer rather than more defensive. If software is increasingly filling in your forms on someone's behalf, then a form that only makes sense visually is a form that will be filled in wrongly.

The fixes are unglamorous and they help humans too. Label every field properly rather than relying on placeholder text. Say what a field expects in words rather than through formatting hints. Make error messages state what was wrong and what to do, because whatever is reading the page has to act on the text and cannot infer from a red outline.

What I would not do is treat every automated submission as an attack. Some of them are a real person who asked their assistant to book a call. Blocking that person because of how they arrived is a conversion decision disguised as a security decision, and I have seen it cost real leads. The broader shift is covered in AI agents filling website forms.

Where I would keep friction is anything that commits money or shares private data. Those steps deserve a human confirmation regardless of who or what is on the other end, and that is one of the few places where an extra step is worth the conversion cost. The evolution of this for browser-based agents is worth reading about in how agent-mode browsers change forms.

Where does the current answer run out?

At the point where the person and the software disagree about what was asked for. The vendors' framing assumes an agent does what its user intended. When it does not, the documentation I have read does not say who owns the outcome, and I am not going to invent an answer.

I should be clear about the limits of what I am telling you. I am not a lawyer, none of this is a legal position, and I have not verified any regulatory or contractual source on agent liability. What I have verified is how four companies describe their own software's behaviour, which is a different and narrower thing.

The second unresolved area is scale. The user-behalf framing was built for one person asking for one page. It stretches uncomfortably when one instruction produces hundreds of requests, and nothing I have read resolves where that line sits. Treat that as genuinely open rather than settled.

What should you do next?

Separate your controls by what they actually govern. Put crawler directives in your robots file and accept that they bind automatic crawls from well-behaved crawlers. Put everything else in an access rule at the edge, where it applies regardless of what a request calls itself.

Then make sure you could answer the five evidence questions if you had to: what, when, from where, under whose credential, and verified or not. Grant yourself the log-reading permission now, separately from whatever your automations use, because the day you want it is the worst day to be configuring it.

If you are trying to work out which of your rules are real controls and which are polite requests, reach out. It is usually a short review, and it tends to end with a shorter robots file and a clearer set of rules somewhere that can actually enforce them.