The thing most likely to stop a Google agent reading a restaurant's menu is the restaurant's own security stack. Google said as much on 17 November, publishing a technical guide on allowing Google-Agent through a content delivery network and logging it in its crawling changelog: "Published a technical guide on how to ensure that Google-Agent has access to your website." Barry Schwartz had it at Search Engine Roundtable by 7am.
Robots.txt is the wrong file for this
Google-Agent is not a crawler. It appeared in March among Google's user-triggered fetchers, where the documentation describes it as "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request". Search Engine Roundtable's write-up at the time named Project Mariner as an example.
The classification matters more than the name, because Google's documentation says of that whole class: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules."
So the file a developer would reach for first does nothing here. There is no Google-Agent line to add and no disallow to lift. The decision is taken somewhere a restaurant rarely looks: the CDN, the web application firewall, or whichever bot-detection service the host switched on by default.
Google asks to be recognised by signature
The guide sets out three ways to identify the traffic. The User-Agent header contains Google-Agent. The source address appears in a published list at user-triggered-agents.json. And some requests are now signed: Google is using RFC 9421 HTTP Message Signatures under the identity https://agent.bot.goog, sending Signature-Agent: g="https://agent.bot.goog" alongside Signature and Signature-Input headers, with public keys served from a well-known directory on that domain.
Google's instruction for requests that verify is to "skip bot challenges", and its stated reason for working this way is that allowlisting does the job while "preserv[ing] your existing security controls". This is not a request to turn the firewall off.
The troubleshooting list describes a route through a menu
The most revealing part of the page is the section on requests that still get blocked. Google asks that the allow rule evaluate before broad data-centre ASN blocks, managed bot rules and AI agent block policies. It asks that the rule skip CAPTCHAs and JavaScript challenges, "including in-page CAPTCHAs triggered on search or form submission". It asks that the allowance cover "search, filtering, detail pages, add-to-cart, booking, and underlying XHR, GraphQL, or api.* endpoints". And it warns that rate limits need adjusting, because many users share the same egress pool.
Read that as a sequence and it is a path through a restaurant site: arrive, search, filter, open a dish, book. Each step Google names is one a modern menu page implements as an asynchronous call to its own API, and each is a separate opportunity for a challenge to fire. A bot check on the dietary filter does not return an error anyone sees. It returns a page with nothing on it.
Google has not said what blocking costs
What the guide does not do is argue. It never claims a site loses anything by leaving Google-Agent blocked, and the changelog frames the audience narrowly: "To help publishers that want to enable Google-Agent for their website."
Apple was willing to be explicit about the equivalent question, writing of its own opt-out crawler name that "site rules for Applebot-Extended are not considered in ranking for Search". Google has published no comparable sentence about its agent, in either direction. That leaves the decision sitting with whoever configured the firewall, on no stated information about the consequences.
A page that answers on first fetch needs no allowlist
There is one configuration none of this reaches. If the dish names, prices, sections, allergens and dietary flags are in the HTML and the JSON-LD of the first response, the agent has what it came for before any of the interactions on Google's list begin. No filter to operate, and no challenge to survive.
That is not a security posture, and it will not rescue a site whose firewall drops the first request outright. What it does is narrow the problem to a single fetch of a single URL, which is the smallest surface a restaurant can leave for a bot rule to misread.
