Google updates its documentation by adding a new user agent called Google-CWS. This agent is used by the Chrome Web Store as part of the “user-triggered fetchers,” meaning content fetches triggered by user actions.
Key takeaways:
- Google added a new user agent called Google-CWS, specific to the Chrome Web Store.
- This user agent is triggered when a user interacts with a Chrome extension or theme and an external URL is requested.
- “User-triggered fetchers” do not always follow robots.txt directives because they depend on a human action rather than an automated crawl.
- You can identify this fetcher in server logs as "Mozilla/5.0 (compatible; Google-CWS)".
A new agent appears in the logs
With the arrival of Google-CWS, Chrome extension developers will now see a new agent appear in their logs. This fetcher is directly tied to actions performed within the Chrome Web StoreIt is triggered when a user views, installs, or interacts with an extension, and that extension references an external URL in its metadata.
Google specifies that these requests are not made by the regular crawler (Googlebot), but by a distinct, occasional mechanism initiated by the user. This behavior falls into the family of “user-triggered fetchers” already known for other cases, such as Google Site Verifier or certain features hosted on Google Cloud Platform (GCP). As a reminder, a few weeks ago, Google had added NotebookLM to its User-Triggered Fetchers.
Why this matters for SEOs
Identifying this new user agent in logs can avoid confusion with potential suspicious activity. Unlike Googlebot, Google-CWS does not aim to index pages, but the retrieval/viewing of content related to extensions.
This also helps publishers and SEO experts better understand certain external requests that bypass robots.txt filters, while confirming their legitimate origin via Google.
The article “Google adds Google-CWS (Chrome Web Store) to User-Triggered Fetchers” was published on the site Abondance.
