Definition
Robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may and may not request. It follows the Robots Exclusion Protocol, published as RFC 9309. It controls crawling, not indexing. Every Shopify store serves a default robots.txt, which you can change with the robots.txt.liquid theme template.
How robots.txt works
A well-behaved crawler fetches one file before any page. The rules must be in a file named /robots.txt, all lowercase, in the top-level path of the site[1]. The format is the Robots Exclusion Protocol, published by the IETF as RFC 9309 in September 2022[1]. A file applies only to the host, protocol and port it is served from[2], so www.example.com and shop.example.com each need their own.
The file is made of groups. Each group names a user agent, then lists Allow and Disallow rules for URL paths. A crawler obeys the group that matches its name, compared without regard to case, and falls back to the * group when no group matches[1]. When several rules match one URL, the most specific rule wins, which means the one with the longest path[1]. When an allow and a disallow rule conflict, Google uses the least restrictive one[2].
Robots.txt controls crawling, not indexing. Google says the file is used mainly to manage crawler traffic and is not a mechanism for keeping a web page out of Google[3]. A blocked URL can still appear in search results, only without a description[3]. It is not security either: RFC 9309 states that the rules "are not a form of access authorization"[1], and Google notes that not every search engine supports them[3].
Google reads the file with limits of its own. It enforces a maximum size of 500 KiB and ignores anything after it, and it generally caches the file for up to 24 hours[2]. It supports four fields: user-agent, allow, disallow and sitemap; crawl-delay is not supported[2]. On a 5xx error, Google stops crawling the site for the first 12 hours[2].
What Shopify's default robots.txt blocks
Every Shopify store serves a generated file at /robots.txt, and Shopify says the default works for most stores[4]. It opens with Allow: /, so crawlers can reach products, collections, pages, blogs and policies, then disallows the URLs that should stay out of crawlers' way[4]:
| Default rule | What it keeps crawlers out of |
|---|---|
Disallow: /admin | The store admin[4] |
Disallow: /cart/ | Cart links, which are unique to each customer[4] |
Disallow: /checkout, /checkouts/ | Checkout pages[4] |
Disallow: /account, /orders | Customer account and order pages[4] |
Disallow: /collections/*+* | Filtered collection pages, which duplicate existing content[4] |
Disallow: /collections/*sort_by* | Sorted collection pages, for the same reason[4] |
A default Shopify robots.txt also has a separate group for adsbot-google, a Sitemap line pointing to /sitemap.xml, and, in the files we checked in November 2026, comment lines at the top that point AI shopping agents to the store's /agents.md file.
How to customise robots.txt on Shopify
You cannot upload your own robots.txt file to Shopify. Instead you add a theme template, robots.txt.liquid, and it replaces Shopify's generated output[5]. In the admin, go to Online Store, Themes, open the theme's menu, choose Edit code, click Add a new template and select robots[4].
The template supports only six Liquid objects: robots, group, rule, user_agent, sitemap and request[5]. robots.default_groups holds Shopify's default groups, each with a user agent, rules and an optional sitemap[5]. Shopify strongly recommends building on these objects rather than plain-text rules, and asks you to save a copy of your current /robots.txt first, because the rendered template doesn't always match the generated file exactly[5]. With separate domains per market in Shopify Markets, request.host lets you write rules for one domain[5].
Shopify is blunt about the risk. Editing robots.txt.liquid is an unsupported customization, Shopify Support can't help with it, and incorrect use can result in loss of all traffic[4]. To return to the default, delete the template[4].
AI crawlers and robots.txt
Each AI crawler token does a different job, and the settings are independent: you can allow one and block another[6].
| Token | Operator | What a Disallow: / does |
|---|---|---|
GPTBot | OpenAI | Signals that your content should not be used to train its generative AI foundation models[6] |
OAI-SearchBot | OpenAI | Keeps your pages out of ChatGPT search answers; they may still appear as navigational links[6] |
ChatGPT-User | OpenAI | Little: it acts on a user's request, so robots.txt rules may not apply[6] |
Google-Extended | Limits use for Gemini training and grounding; no effect on inclusion or ranking in Google Search[7] |
Google-Extended is not a separate crawler. Google crawls with its usual user agents and reads the token only as a control[7]. On Shopify there is one more layer: if you sell through agentic storefronts such as ChatGPT or Microsoft Copilot, Shopify Catalog sends your product data to them independently of robots.txt, so blocking AI crawlers affects only open-web discoverability[4].
Example: blocking internal search URLs
Take a hypothetical store with 1,500 products whose crawl stats show crawlers requesting endless internal search URLs such as /search?q=red+clogs. The fix is one extra rule in the * group, which is also Shopify's own documented example[5]:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /*?q=*' }}
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
The loop prints every default group and rule unchanged, then adds Disallow: /*?q=* to the group for all crawlers. Before publishing, compare the rendered /robots.txt with the saved copy line by line[5]. After publishing, the robots.txt report in Search Console shows which file Google fetched and when, and lets you request a recrawl in urgent cases[8].
The default file also shows how precedence works. It contains both Disallow: /account and Allow: /account/login. For the URL /account/login, both rules match, but the allow rule's path has 14 characters and the disallow rule's has 8, so the allow rule is more specific and wins[1]. The login page stays crawlable, the rest of the account area does not. For the wider crawl plan, see our guide to crawling optimization on Shopify Plus.
Robots.txt vs noindex
| Robots.txt disallow | Noindex | |
|---|---|---|
| Controls | Whether a crawler requests the URL | Whether the page is shown in search results |
| Blocked URL in Google | Can still appear, without a description[3] | Dropped once Google recrawls the page |
| Use together? | Not on the same URL: if robots.txt blocks the page, Google never sees the noindex[9] | |
FAQ
Where is the robots.txt file on Shopify?
At yourdomain.com/robots.txt. Shopify generates it automatically, and you can't upload a replacement. To change it, add a robots.txt.liquid template to your theme; it replaces the generated output[5]. Delete the template to return to the default[4].
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing, and a blocked URL can still appear in results without a description[3]. To remove a page, use noindex and keep the page crawlable, because Google must crawl the page to see the rule[9].
Can I add a crawl-delay rule?
Shopify's template lets you add one for specific crawlers[4], but Google ignores it: Google supports only user-agent, allow, disallow and sitemap[2].
Should I block GPTBot on my store?
It depends on what you want. Blocking GPTBot signals that your content should not be used for training, while OAI-SearchBot controls whether pages appear in ChatGPT search answers[6]. Most stores that want AI visibility allow OAI-SearchBot even if they block GPTBot.
The bottom line
Robots.txt tells crawlers which URLs they may request, following RFC 9309. It controls crawling, not indexing, and it is not a security tool. Shopify's default already blocks admin, cart, checkout, account and duplicate collection URLs, so most stores never need to touch it. If you do, build robots.txt.liquid on the default groups and test the output; for a second pair of eyes, our Shopify SEO service starts with crawling and indexing.
Article sources
We use primary sources only: official documentation, product pages and laws. Each link was checked on the date shown.
- RFC Editor (IETF). RFC 9309: Robots Exclusion Protocol. Accessed 18th Nov 2026.
- Google Search Central. How Google interprets the robots.txt specification. Accessed 18th Nov 2026.
- Google Search Central. Introduction to robots.txt. Accessed 18th Nov 2026.
- Shopify Help Center. Editing robots.txt.liquid. Accessed 18th Nov 2026.
- Shopify. Customize robots.txt. Accessed 18th Nov 2026.
- OpenAI. Overview of OpenAI crawlers. Accessed 18th Nov 2026.
- Google Search Central. Google's common crawlers (Google-Extended). Accessed 18th Nov 2026.
- Google Search Console Help. robots.txt report. Accessed 18th Nov 2026.
- Google Search Central. Block Search indexing with noindex. Accessed 18th Nov 2026.
Need this on your store?
See the serviceThe checklist we run before touching a client's code
Get it by email, plus a few short notes over the next two weeks: what we see most often on Shopify stores and what actually fixes it.