Ultimate Guide to Shopify Plus Crawling Optimization (2026)

Crawling optimization on Shopify Plus means getting Googlebot to spend its visits on the products, collections and content you want indexed, instead of on filter combinations, sort orders, search results and duplicate URLs. Shopify already generates canonical tags, robots.txt and sitemap.xml, so most of the work is checking those defaults, customizing robots.txt.liquid only where you have a clear reason, keeping theme links clean and fixing what Google Search Console reports.

Shopify Plus doesn't crawl differently from other plans. The same robots.txt, canonical tags and sitemaps apply. What changes is scale: bigger catalogs, more filter values, more markets, sometimes expansion stores or a headless storefront. Every one of those multiplies the URLs a crawler can find. This guide covers robots.txt.liquid, faceted and filter URLs, sitemaps, internal search pages, duplicate collection paths and the Search Console reports that show where crawling goes wrong.

Does crawl budget matter for a Shopify store?

Google defines crawl budget as the set of URLs that Google can and wants to crawl on a site. It depends on two things: crawl capacity (how much load your server handles without slowing down or returning errors) and crawl demand (how many URLs Google knows about, how popular they are and how often they change). Your Shopify plan plays no part in it.

Google's crawl budget guide is written for sites with more than a million unique pages that change weekly, sites with more than 10,000 pages that change daily, and sites with many URLs stuck in "Discovered - currently not indexed". Few stores have a million products. But on Shopify the number of crawlable URLs can be far larger than the number of products, because every filter value, tag, sort order and collection path creates more URLs. Google counts those duplicates in its "perceived inventory" too.

These are the URL patterns that usually take crawl time on a Shopify store:

What does Shopify already handle for crawling?

Before changing anything, know what the platform does for you. We checked the live behavior on Shopify's Dawn demo store and on several stores that use the default settings in October 2026.

What changed in Shopify's default robots.txt in 2026?

Shopify rewrote its generated robots.txt. Stores without a custom template now get a file that opens with comments pointing AI agents to the store's /agents.md file, starts its main rule group with Allow: /, and no longer blocks /search or /policies/. The rules that stop crawl traps are still there. This is the relevant part of the generated file on the Dawn demo store in October 2026:

# Filters, sort, previews, language-picker crawl traps
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /collections/*filter*&*filter*
Disallow: /*/collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /*?*oseid=*
Disallow: /*?*preview_theme_id=*
Disallow: /*?*preview_script_id=*

Two of these rules matter most for collections. /collections/*+* blocks combinations of two or more tags. /collections/*filter*&*filter* blocks URLs with two or more filter parameters. A URL with a single filter stays crawlable and relies on its canonical tag.

If your theme has a robots.txt.liquid template, your file can differ. Shopify's Help Center says a template that outputs the default rule groups can still include rules the current default no longer has, such as Disallow: /search and Disallow: /policies/. To get the current default, delete the template. Open yourstore.com/robots.txt and compare it with the excerpt above before you decide anything else.

How do you customize robots.txt.liquid safely?

Shopify treats robots.txt edits as an unsupported customization: Shopify Support can't help with them, and the Help Center warns that a mistake can cost you all of your search traffic. Work through it in this order:

1. Save a copy of your current /robots.txt output.
2. In the Shopify admin, go to Online Store > Themes, open the code editor for your theme, add a new template and choose robots.
3. Keep Shopify's Liquid loop over robots.default_groups, so the default rules stay current as Shopify updates them. Shopify advises against replacing the template with plain text.
4. Add your rules inside the loop, for the right user agent.
5. Compare the rendered file with your saved copy, line by line.
6. In Search Console, open the robots.txt report and request a recrawl.

Here is the pattern from Shopify's documentation, with one extra rule for the * group. This rule blocks single-filter collection URLs as well, which suits a large catalog where Crawl Stats show Googlebot spending a lot of time on filters:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /*collections/*?*filter.' }}
  {%- endif -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

The leading /* also covers market subfolders such as /en-ca/collections/. Add each further rule the same way and check the rendered output, because a missing line break joins two rules into one invalid line.

Other changes the template supports:

• Remove a default rule by skipping it in the loop, for example with unless rule.directive == 'Disallow' and rule.value == '/policies/'.
• Add a group for one crawler as plain text after the loop, such as User-agent: GPTBot followed by Disallow: /.
• Add an extra sitemap with a Sitemap: line, for example for content served from another system.
• Add host-specific rules with request.host, but only if you use Shopify Markets with separate domains that need different crawling rules.

What not to do: don't block theme CSS or JavaScript, don't block /products/ or /collections/ as a whole, and don't try to noindex pages from robots.txt. Google doesn't support a noindex rule there, and a blocked page can't show Google the noindex tag on the page itself.

How should you handle filter and faceted URLs?

Shopify's storefront filtering writes filters into URL parameters. A product type filter becomes filter.p.product_type=shoes, a variant option filter becomes filter.v.option.color=red, and several filters are joined with &. Older themes also filter by tag in the path, as in /collections/shoes/red.

Google's faceted navigation guide gives two routes. If you don't need filtered URLs in search, disallow them in robots.txt. Google describes canonical tags and rel="nofollow" as generally less effective in the long term. If you do want filtered URLs indexed, keep filter order consistent, use & as the separator and return a 404 when a combination has no results.

If a filtered view matches real search demand, such as "black leather boots", build it as its own collection. A real collection gets a clean URL, its own title and description, a place in your navigation and an entry in the sitemap. A filter URL gets none of those. For pagination, our guide to canonical tags on Shopify pagination explains why page 2 should not point its canonical to page 1.

Check how your filter app renders the page, too. Shopify's performance guidance is to render essential content in Liquid and HTML, because content that only appears through client-side JavaScript delays LCP and harms SEO. If a third-party filter app replaces the product grid with JavaScript, make sure the unfiltered collection still has product links and crawlable pagination links in the HTML. Google ignores URL fragments, so filters that only change the # part of a URL don't create crawlable URLs at all.

How do you fix duplicate collection and product URLs?

Collection-aware product URLs

The within filter turns /products/chelsea-boot into /collections/boots/products/chelsea-boot. Shopify's own documentation warns that a standard product page and a product page in the context of a collection have the same content on separate URLs. The canonical tag points to /products/chelsea-boot, but every collection a product sits in still adds a URL that Google can crawl.

Fix it at the source. Search your theme's sections and snippets folders for within: collection and link product cards to product.url instead. Dawn's demo store already links product cards straight to /products/.

Variant URLs

Variant URLs with ?variant= carry a canonical to the main product URL. That is the right setup, and Search Console will list these URLs as "Alternate page with proper canonical tag". No action is needed.

Automatic listing pages

Shopify generates pages such as /collections/all and vendor or type listings like /collections/vendors?q=Brand, which link_to_vendor creates. If your theme links to them from every product page, Google will crawl them. Decide whether they are useful landing pages. If they aren't, remove the links or build a proper brand collection instead.

Secondary domains

In Settings > Domains, every extra domain should be a redirect domain, which sends visitors to the primary domain. An alias domain doesn't redirect and keeps its own name in the address bar, so the same store answers on two hostnames.

Markets and expansion stores

With Shopify Markets, hreflang tags, canonical URLs and market sitemaps are generated for you, but only for markets that have a distinct domain, subdomain or subfolder. Automatic hreflang covers the markets inside one store. If you run separate expansion stores per region, connecting them is your job. Our hreflang guide for Shopify Plus covers both setups.

How do you keep Shopify sitemaps clean?

Shopify generates /sitemap.xml as an index that links to child sitemaps for products, pages, collections, blogs and articles, plus the same set for each market subfolder. Product entries include the primary product image and a lastmod date. The file notes that it can't be edited manually and is kept up to date in real time. In October 2026 the index on the Dawn demo store also listed a sitemap_agentic_discovery.xml file that points to the store's agents.md.

You can't edit the sitemap directly, so you control it through the pages themselves:

• seo.hidden metafield. Create a metafield definition with the namespace and key seo.hidden, type Integer, and set it to 1 on a page, blog post or product. Shopify then hides it from sitemaps, search engines and storefront search.
• Unlisted product status. An Unlisted product gets noindex and nofollow tags, drops out of the sitemap and doesn't show in Shopify-powered collection pages, search results or recommendations. Older Admin API versions, before 2025-10, report these products as active.
• Extra sitemaps. Content that lives outside Shopify can be added with a Sitemap: line in robots.txt.liquid.

Submit sitemap.xml in Search Console under Indexing > Sitemaps. With international domains, submit the sitemap for each domain. Google recommends keeping sitemaps current and including lastmod, which Shopify already does for products.

Should Shopify's internal search pages be crawled?

Search result pages are thin and endless: any word typed into the box creates a new URL. Shopify's current approach is to let crawlers fetch them and tell search engines not to index them. On three default-configured stores we checked in October 2026, /search?q= pages returned X-Robots-Tag: noindex, nofollow, and the generated robots.txt no longer blocked /search.

That order matters. Google's documentation says a noindex rule only works if the page isn't blocked by robots.txt. A store with an old template that still disallows /search hides the noindex header from Google. If other sites link to search URLs, those URLs can then show up in Search Console as "Indexed, though blocked by robots.txt".

• Use the current default, or remove the /search rule from your template, so Google can see the noindex.
• If your theme needs a fallback, Shopify's Help Center shows a meta robots tag for the search template: {% if template contains 'search' %}<meta name="robots" content="noindex">{% endif %}.
• Block /search again only if Crawl Stats show a large share of Googlebot requests going to search URLs, and only after those URLs have dropped out of the index.

How do you fix crawl issues in Google Search Console?

The Page indexing report tells you why URLs aren't indexed. Read it with Shopify's URL patterns in mind. Several statuses are normal on a healthy store.

Then use the tools around the report:

1. URL Inspection. Test a live URL to see whether Google can fetch it, which canonical it picks and whether robots.txt blocks it.
2. robots.txt report. It shows which robots.txt files Google found for your top hosts, when it last fetched them and any parsing errors. After a change, request a recrawl from the same report.
3. Crawl Stats. Open Settings > Crawl stats to see total requests, response times and host status, and to break requests down by response code, file type, purpose (discovery or refresh) and Googlebot type. This is where you see whether filters or search pages eat your crawling.
4. Validate fix. After you fix every URL behind a status, click Validate Fix. Validation usually takes up to about two weeks.

Google also lists soft 404s and long redirect chains as crawl budget waste, so clear them even when the counts look small.

What about JavaScript, headless and mobile crawling?

• Mobile first. Google indexes the mobile version of your pages, crawled with its smartphone agent. If your mobile layout drops links or content that desktop has, such as a collapsed menu that isn't in the HTML, Google may never see them.
• Liquid over client-side rendering. Product information, navigation and collection grids should be in the HTML that Liquid renders, not added later by JavaScript.
• Hydrogen storefronts. A headless storefront serves its own robots.txt and sitemap routes, separate from the Online Store. Shopify's Hydrogen SEO docs note that Oxygen serves robots.txt only on the production deployment with a custom domain and blocks crawlers on preview deployments. Check both files after every major upgrade.

Does crawling affect AI search?

Yes, because AI assistants that browse the web need to reach your pages too. OpenAI uses OAI-SearchBot to surface sites in ChatGPT search and GPTBot to collect training data. A site that blocks OAI-SearchBot won't appear in ChatGPT search answers, apart from navigational links. You can allow or block these crawlers by user agent in robots.txt.liquid.

Shopify's Help Center adds an important point: if you sell through agentic storefronts such as ChatGPT or Microsoft Copilot, your product data reaches them through Shopify Catalog, independently of robots.txt. Blocking a crawler only affects open-web discovery. That is why we treat crawling as the base layer: SEO alone is no longer enough, but a store that crawlers can't read cleanly starts behind. Nobody can guarantee a place in an AI answer. If you want help with that layer, see our Shopify AI search optimization service.

Shopify Plus crawling checklist

1. Open /robots.txt and compare it with Shopify's current default. Delete or update an old robots.txt.liquid template.
2. Confirm every template outputs a canonical tag built from canonical_url.
3. Replace within: collection in product cards with product.url.
4. Decide how to handle single-filter URLs. Turn filtered views that have real search demand into collections.
5. Stop linking to tag paths and automatic listing pages you don't want indexed.
6. Make sure search pages can show Google their noindex header.
7. Hide pages from the sitemap with seo.hidden or the Unlisted status, not by deleting and recreating them.
8. Set every extra domain to redirect to the primary domain.
9. Check that automatic hreflang is on, and connect expansion stores yourself.
10. Submit the sitemap for every domain in Search Console.
11. Review Page indexing and Crawl Stats every month, and validate fixes.

Large catalogs, several markets and custom themes are where crawling problems multiply. If you would like us to audit yours, see our Shopify SEO service.

FAQ

Can you edit robots.txt on Shopify Plus?

Yes. On Shopify Plus and every other plan, you add a robots.txt.liquid template in the theme code editor. Keep Shopify's default rule loop and add your rules inside it. Shopify treats this as an unsupported customization, so Shopify Support can't help with it, and a mistake can block your whole store from search.

Does Shopify still block /search in robots.txt?

Not in the file Shopify generates in 2026. Search pages are sent with an X-Robots-Tag: noindex, nofollow header instead, so Google can fetch them and drop them from the index. Stores with an older robots.txt.liquid template may still output Disallow: /search, which hides that header from Google.

Should I block filter URLs or use noindex?

Pick one per URL, because Google can't see a noindex tag on a page that robots.txt blocks. Blocking in robots.txt saves crawling, and it is what Google recommends for facets you don't need in search. Shopify already blocks combinations of two or more filters. Single-filter URLs carry a canonical to the collection. If a filtered view deserves to rank, make it a real collection.

How do I remove a page from the Shopify sitemap?

Set the seo.hidden metafield to 1 on the page, blog post or product, which hides it from sitemaps, search engines and storefront search. For products, the Unlisted status does the same and also removes the product from collection pages, search results and recommendations.

Why does Search Console show my products as "Alternate page with proper canonical tag"?

Those are usually variant URLs, collection-aware product URLs or filtered collection URLs that point their canonical to the main page. It means the canonical tags work. Check that the main /products/ URL for each product is indexed.

Does Shopify Plus get a bigger crawl budget?

No. Google sets crawl budget per site, from how fast and reliably your server responds and how much demand there is for your URLs. Your Shopify plan doesn't change it. What you control is how many low-value URLs compete for that budget.