Diagram: a user asks, Google-Agent visits the site, and a firewall gate decides ALLOW or BLOCKED

Your Firewall Decides Whether Google's Agent Finishes the Job

Google's Nov 17 guide tells sites how to let its agent through their firewall. The agent ignores robots.txt, so the firewall is where the decision happens.

By Dalin de Graff

Your firewall is the gate Google's agent has to pass. Google's November 17 guide, Allow Google-Agent traffic, tells site owners how to let the agent through their CDN, firewall, or bot-detection service. The sentence that does the work says explicitly allowing this traffic "helps prevent legitimate requests from being blocked while preserving your existing security controls." Your bot protection can stop its agent. The fix is a narrow exception.

Google-Agent is the name for agents hosted on Google infrastructure that browse the web and act when a user asks, and it is not new: Google added it to its list of user-triggered fetchers on March 20, 2026. What changed on November 17 is the direction. Until now Google's pages described what the agent is. Now they describe what you have to change for it to work: the agent does more than read. Google's guide expects it to search a site, filter results, open detail pages, add items to a cart, and make bookings. A link preview needs one page. A booking needs the whole site to cooperate.

Robots.txt does not reach this visitor

Google classifies Google-Agent as a user-triggered fetcher, and its fetcher page states the rule plainly: a fetch requested by a user generally ignores robots.txt rules. A Disallow line asks a crawler to stay out, and this visitor does not follow it.

Gary Illyes made the same point on October 28: fetchers acting for a single user have no good reason to obey robots.txt, and faking Googlebot's user agent takes about 30 seconds. The guide Google published on November 17 does not mention robots.txt at all.

The network edge is where a site decides whether the agent gets in. Training is a separate decision with its own controls, covered in my November 1 post on the shared opt-out signal. The only controls that reach a live visitor sit in front of your site.

The four checks that identify the agent

Google's guide gives a short verification sequence: a request counts as Google-Agent when the User-Agent header contains the token and the source IP sits in Google's published file, user-triggered-agents.json. Some requests also carry a Web Bot Auth signature, an experimental cryptographic check under the identity agent.bot.goog. Verify it if present, then skip bot challenges.

The header alone proves nothing, because anyone can write Google-Agent into a header. That is why the IP file matters. The version Google served on November 17 lists 20 address blocks, 12 IPv4 and 8 IPv6 prefixes, refreshed daily. A copy pasted into a firewall rule last month is stale. The signature is the rarest check: Arkose Labs reported in October that only 2.3% of the legitimate AI agent sessions it classified across all products, on protected actions in Q3 2026, carried a valid Web Bot Auth signature, and Google says only a subset of its requests is signed at all. For now the IP file does the real work.

Four-step flow: User-Agent contains Google-Agent, IP in user-triggered-agents.json, signature verified if provided, skip bot challenges

Six ways the agent still gets stopped

Its troubleshooting section lists six ways a legitimate request still gets blocked after a site allowed it. Each is a different layer of the same site, and each fails silently: the person who asked for the booking never sees the block.

What still blocks itWhat breaksThe check
Allow rule ranked below broad blocksAn ASN block or AI agent category block overrides the allow ruleRank the allow rule above ASN blocks, managed bot rules, and category blocks
Interactive challenges left onA CAPTCHA or JavaScript challenge fires on search or form submissionSkip CAPTCHAs and JavaScript challenges for verified requests
Only the homepage allowedThe agent loads the homepage, then hits a wall on search, filtering, detail pages, add to cart, or bookingCover the full journey plus the XHR, GraphQL, and api.* calls behind it
Per-IP rate limitsMany users share one egress pool, so throttling one IP throttles every user behind itExempt or account for the verified Google-Agent ranges in rate limits
Stale IP rangesThe firewall holds a stale copy of the range fileSync IP rules against user-triggered-agents.json on a schedule
Proxies stripping headersA load balancer drops the original IP or signature headersPreserve the client IP, User-Agent, and signature headers through every proxy

The check your citation panel cannot run

I measure citations with a fixed panel: ten money questions, three times each, across four engines, logged in and out. That panel tells me what the answers say about a brand, and nothing about this. An agent that fails at a CAPTCHA leaves no citation, no Search Console impression, and no trace in an AI visibility dashboard. It shows up only in the server log.

The log check takes ten minutes and needs no new tool. Pull the last 30 days of access logs and filter for the Google-Agent token. For each hit, record the path, status code, and IP, then check it against the current user-triggered-agents.json file. A 403 on a search or checkout path is a blocked agent; a 200 on the homepage and nothing after is an agent that got in and then stalled, or only ever needed one page. Check the logs for its task before concluding it failed. Run it monthly, because the IP file refreshes daily and a rule written against last month's ranges quietly goes stale.

What it costs, and how big it is today

Blend reported on November 4 that its AI Shopping Audit 2026, tested September 8 to 15, found bot protection blocked its AI shopping agent at 29 of 200 UK and US retailers, 14.5%. One vendor's agent, not Google's, and an unbranded one no site could allowlist by name, so 14.5% is an upper bound on what a named, verifiable agent like Google-Agent would face. The blocking layer is the same: the same firewall rules, the same bot challenges, the same blocked checkout calls.

Bar chart: bot protection blocked the AI shopping agent at 29 of 200 retailers in the Blend AI Shopping Audit 2026, 171 not blocked

Google's agent is a small share of agent traffic today. HUMAN's October data, from the sites and apps it protects, put Meta's Muse at 59% of observed agent requests, and the product it called Google Agent (Gemini) at 0.8%. Google has named no product in the new guide and published no volume figures.

The cost is one verified exception, written once and synced on a schedule. Skipping it costs a booking or a cart that fails with no error anyone on your side sees. OpenAI published a merchant playbook on November 13 asking stores to recognise signed requests the same way, through Web Bot Auth at the CDN, and Google's own Universal Cart moved checkout into Search the month before.

Allow the verified agent through, with the four checks and the six troubleshooting items handled, or block it deliberately at the edge, accepting that bookings and carts from that agent will fail. Most sites do neither: no decision at all, while the firewall makes it for them. Search your logs this week. Has Google-Agent visited, and did it get past the front door?

Dalin de Graff

About the author

Dalin de Graff is a digital marketing expert based in Winnipeg, Canada, with 9 years of experience across SEO, GEO, and AI search. At First Rank, he built the agency's GEO practice and leads its work on AI search and LLM visibility. His results include +311% organic clicks, +300% calls, and +98% AI search traffic in 30 days. He writes about search, AI, and marketing at dalindegraff.com.