Shopify generates your sitemap.xml automatically. It includes your products, collections, blog posts, and pages. That’s the good news. The bad news: automatically included does not mean correctly optimized. Indexing bloat, faceted navigation errors, and redirect chains silently kill rankings on stores that have never looked at their sitemap past the initial submission.
This guide covers the full Shopify technical SEO stack — sitemap auditing, robots.txt configuration, duplicate content fixes, and crawl budget management for stores with 500+ products.
Key Takeaways
- Shopify auto-generates sitemaps, but many include non-indexable and duplicate URLs
- The /products/ and /collections/ URL structure creates inherent duplicate content risks
- Canonical tags are your primary tool for resolving Shopify’s duplicate content issues
- Large catalogs (500+ products) require active crawl budget management to rank efficiently
How Shopify’s Sitemap Works
What’s Automatically Included in Your Shopify Sitemap
Shopify’s sitemap lives at yourstore.com/sitemap.xml. It’s structured as a sitemap index — a parent file that points to child sitemaps for each content type:
sitemap_products_1.xml— product pagessitemap_collections_1.xml— collection pagessitemap_blogs_1.xml— blog postssitemap_pages_1.xml— standard pages
Each child sitemap includes the URLs for that content type. Shopify automatically updates sitemaps when you add, remove, or change content.
What Shouldn’t Be Indexed
Shopify’s default robots.txt already blocks crawling of several high-noise paths: /cart, /checkout, /account, /orders, /admin. These aren’t in the sitemap and are correctly disallowed in robots.txt.
However, some URL patterns that Shopify includes in sitemaps are problems. Tagged collection pages (/collections/shirts?sort_by=price) create hundreds of duplicate URLs with no unique content. Unavailable product variant pages indexed after products are discontinued create soft 404s that dilute crawl budget.
Shopify Sitemap URL Structure
Your sitemap is always at yourstore.com/sitemap.xml. Submit this exact URL to Google Search Console. Don’t submit individual child sitemaps — the parent file automatically points to them.
For stores with large catalogs, Shopify automatically paginates the child sitemaps (sitemap_products_1, sitemap_products_2, etc.) when the URL count exceeds Shopify’s internal limits.
Submitting Your Shopify Sitemap to Google
Step-by-Step: Google Search Console Submission
- Open Google Search Console for your property
- Select Sitemaps from the left sidebar
- Enter
sitemap.xmlin the input field (just the filename — your domain is pre-filled) - Click Submit
- Refresh the page after 30 seconds to see the initial status
Google typically processes a new sitemap submission within 24–48 hours for small stores. For large stores (1,000+ URLs), full processing can take 1–2 weeks.
Monitoring Indexing Status After Submission
After submission, monitor three things in Search Console:
Sitemaps report: Shows submitted URLs vs. indexed URLs. A large gap (submitted 800, indexed 200) signals an indexing problem — possibly canonical issues, low-quality content, or crawl budget exhaustion.
Coverage report: Shows page status categories. “Excluded” URLs need investigation. “Discovered — currently not indexed” means Google found the URL but hasn’t gotten to it. “Crawled — currently not indexed” means Google evaluated it and decided not to index it — a quality signal problem.
Page Indexing report: New in 2023, provides specific reasons why URLs aren’t indexed.
Common Submission Errors and Fixes
The most common issue: Shopify-generated sitemap URLs don’t match the actual live URLs due to 301 redirects or password protection left on a store. Verify your store is public (not password-protected) before submitting.
Sarah inherited an 8-year-old Shopify store as part of a business acquisition. The Google Search Console showed 340 URLs submitted in the sitemap but only 89 indexed. A sitemap audit revealed 140 soft 404s from discontinued products still in the sitemap, 80 tagged collection pages driving duplicate content signals, and 31 redirect chains (old URL → new URL → newer URL). After cleaning the sitemap, resolving the redirect chains, and setting canonical tags on tagged collection pages, indexed URLs reached 280 within 90 days and organic impressions improved 47%.
Shopify robots.txt — What You Can and Can’t Control
Default Shopify robots.txt Rules
Shopify’s default robots.txt disallows crawling of administrative, transactional, and account paths. View it at yourstore.com/robots.txt. It already handles most crawl hygiene for standard stores.
What it doesn’t handle: URL parameters from faceted navigation, sorting parameters, and tag-filtered collection pages. These require either robots.txt additions or canonical tag implementation.
How to Edit robots.txt in Shopify
Shopify added the ability to edit robots.txt as a Liquid template in 2021. In your theme code editor, find robots.txt.liquid under Templates. If it doesn’t exist, you can create it.
The Liquid template approach lets you add custom Disallow rules. Common additions for Shopify technical SEO:
Disallow: /collections/*?sort_by=
Disallow: /collections/*?filter.
Disallow: /search?What to Disallow and Why
Sort-by parameters: /collections/shirts?sort_by=price-ascending and ?sort_by=price-descending are the same content sorted differently. Disallow all sort_by parameter variations to prevent Googlebot from crawling hundreds of identical collection pages.
Filter parameters: If you use filtering apps, they generate URLs like /collections/shoes?filter.p.m.color=blue. Block the filter parameter pattern unless each filter combination has genuinely unique content worth indexing.
Search pages: /search?q=... URLs are almost never worth indexing. Block the search path entirely.
Technical SEO Issues Unique to Shopify
Duplicate Content from Collection + Product URL Structures
This is Shopify’s most significant structural SEO issue. Every product accessible through a collection has two valid URLs:
/products/leather-wallet(canonical product URL)/collections/wallets/products/leather-wallet(collection-contextualized URL)
Both URLs return the same page content. Without a canonical tag, Google sees them as duplicates and may index either, dilute authority across both, or choose the wrong one as the canonical.
Shopify themes should include a canonical tag on all product pages pointing to the /products/ URL. Check your theme’s product.liquid template (or the head section) for:
<link rel="canonical" href="{{ canonical_url }}" />Shopify populates canonical_url with the correct /products/ URL automatically — as long as your theme includes this tag.
Canonical Tag Implementation in Shopify Themes
Verify your theme includes canonical tags by inspecting a product page’s HTML source. Search for rel="canonical". If it’s missing, add the Liquid tag to your theme’s <head> section.
For collection pages with filtered/sorted variants, add canonical tags pointing back to the base collection URL:
{% if collection.current_sort_type %}
<link rel="canonical" href="{{ shop.url }}/collections/{{ collection.handle }}" />
{% else %}
<link rel="canonical" href="{{ canonical_url }}" />
{% endif %}For a full breakdown of Shopify schema and structured data, see our guide to Shopify schema markup implementation.
Pagination and Infinite Scroll SEO
Shopify collections paginate by default. /collections/shirts shows page 1; /collections/shirts?page=2 shows page 2.
Don’t add rel="prev"/"next" — Google deprecated support for these tags. Instead, ensure paginated pages are crawlable and let Google discover them naturally through internal links. Don’t block pagination in robots.txt.
Infinite scroll is an SEO risk unless implemented with History API URL updates so each scroll-loaded section has a unique URL that Googlebot can reach independently.
Faceted Navigation and URL Parameter Bloat
Filtering apps (Boost Commerce, Filterist, etc.) often create hundreds of URL parameter combinations. A collection with 50 products, 5 colors, 4 sizes, and 3 sorting options can generate thousands of combinatorial URLs — none with unique content.
The fix: either configure your filtering app to use JavaScript-only filtering (no URL changes) or add comprehensive Disallow rules in robots.txt for filter parameter patterns.
Need help with a technical SEO audit? Our Shopify technical SEO audit covers sitemap issues, duplicate content, and crawl budget optimization with a prioritized fix list.
Crawl Budget Management for Large Shopify Stores
Why Crawl Budget Matters at Scale
For stores under 200 products, crawl budget is rarely a concern. Google crawls small sites thoroughly. For stores with 500+ products, 10+ collections, and an active blog, crawl budget becomes a real constraint.
Googlebot has a finite crawl allocation per site. If it’s spending 40% of that allocation on paginated sort-by pages and discontinued product URLs, it’s spending 40% less on your actual valuable content. Rankings suffer silently.
Removing Low-Value Pages From Indexation
Audit your Search Console Coverage report for “Crawled — currently not indexed” URLs. These are pages Googlebot evaluated and rejected. They still consumed crawl budget. Common culprits:
- Discontinued product pages (keep the page with a 301 redirect to the category, or add
<meta name="robots" content="noindex">) - Internal search result pages
- Filtered collection pages without canonical tags
Internal Link Architecture That Guides Googlebot
Googlebot follows links. Pages with many internal links pointing to them get crawled more frequently. Pages with zero internal links (orphan pages) may never be discovered.
Audit your internal link structure with Screaming Frog. Every product page should be reachable from at least one collection page. Every blog post should be reachable from your blog index and at least one other post or page. No page worth indexing should be an orphan.
Shopify Technical SEO Audit Checklist
Run this audit quarterly for stores with 200+ products:
- Sitemap status: Open GSC > Sitemaps. Compare submitted vs. indexed. Gap > 30% needs investigation.
- Canonical tags: Inspect 5 product pages via source. Confirm
rel="canonical"points to/products/URL. - robots.txt: View
yourstore.com/robots.txt. Confirm sort-by and filter parameters are blocked. - Duplicate content: Search Google for
site:yourstore.com/collections/*/products/— these are collection-path duplicates. - Redirect chains: Use Screaming Frog to crawl your sitemap URLs. Any chain with 2+ hops needs flattening.
- Soft 404s: GSC Coverage report > “Not found (404)”. Investigate and redirect or remove from sitemap.
- Orphan pages: Screaming Frog crawl log — URLs in your sitemap with zero internal links pointing to them.
- Page speed on crawled pages: GSC’s Core Web Vitals report shows which URL groups are failing.
- Mobile usability: GSC > Mobile Usability. Fix any errors flagged here.
- Index coverage trend: Track indexed URL count over time. A declining trend without deliberate exclusions is a warning signal.
Marcus noticed his collection pages had stopped ranking after a theme update. An audit revealed the update had removed the canonical tag from product pages. For six weeks, Googlebot had been indexing /collections/shirts/products/[slug] URLs as the canonical versions instead of /products/[slug]. After restoring the canonical tag and resubmitting the sitemap, correct canonicalization was established within three weeks and collection page rankings recovered to previous levels.
Conclusion
Shopify’s sitemap automation handles the basics. It doesn’t handle the edge cases — and the edge cases are where most stores silently lose rankings. Duplicate content from collection-path URLs, faceted navigation parameter bloat, and redirect chains from poor migration or redesign history are the most common culprits.
The audit is not complex. The discipline of running it quarterly is what separates stores that maintain clean Shopify technical SEO from those that wonder why their rankings drift.
Start with Google Search Console’s Coverage report. Look at your submitted vs. indexed URL count. Investigate the gap. Most stores find the answer to their indexing problem within 20 minutes of looking.
For a complete technical SEO audit by a developer who understands Shopify’s structural quirks, our Shopify development services and Shopify technical audit packages cover the full stack — sitemap, canonicals, crawl budget, and redirect architecture.
Frequently Asked Questions
Does Shopify automatically create a sitemap?
Yes. Every Shopify store has a sitemap automatically generated at yourstore.com/sitemap.xml. It updates automatically when you add or remove products, collections, pages, and blog posts. You cannot customize the content of the auto-generated sitemap itself, but you can control what gets indexed via canonical tags and robots.txt.
How do I submit my Shopify sitemap to Google?
Go to Google Search Console > Sitemaps > enter sitemap.xml > Submit. Use just the filename — Search Console pre-fills your domain. You only need to submit once; Google will re-crawl it automatically on its regular schedule.
Can I customize the Shopify sitemap?
You can’t edit the sitemap content directly. You control what appears in the sitemap indirectly: products and collections must be set to “visible” to be included. To exclude specific pages, you can add noindex meta tags to those pages, which signals to Google not to index them (though they’ll still appear in the sitemap XML).
What causes duplicate content on Shopify?
The main cause is Shopify’s dual URL structure for products: /products/[slug] and /collections/[handle]/products/[slug]. Both URLs serve identical content. Canonical tags in your theme resolve this by designating /products/[slug] as the canonical version. Filtered and sorted collection pages are the secondary cause.
How do I fix crawl errors in Shopify?
Open Google Search Console > Page Indexing. Identify error types: 404s need redirects or removal, “Soft 404s” need real content or noindex tags, “Server errors” need hosting investigation. For systematic crawl errors across many URLs, the cause is usually a pattern (discontinued product category, removed blog section) that can be resolved with bulk redirects or noindex configuration.