Home Station → Journey → Dabbo 05 📄 → 5.2

📄 5.2 — Robots.txt & Sitemap Properly

Dabbo 05 · On-Page SEO ·

Robots.txt Google ne kahe chhe su crawl karvu ane su nahi. Sitemap.xml Google ne site na badha pages nu map aape chhe. Banne technical SEO nu base chhe - kharab robots.txt important pages ne crawl thi roki shake; sitemap index ni guarantee nathi aaptu, pan discovery fast kare.

5.2.A Robots.txt na niyamo

  1. Location: domain.com/robots.txt ma accessible hovu joiye.
  2. Crawl block karo: admin pages, cart, checkout, internal search result pages. Duplicate/filter URLs ane thank-you pages mate pehla canonical/noindex vaparo — robots.txt block fakt crawl waste motu hoy tyare (ane index mathi nikli gaya pachhi).
  3. Kyarey block na karo: CSS/JS files (Google render karvu joiye), main content pages.
  4. Sitemap line add karo: Sitemap: https://domain.com/sitemap.xml

5.2.B Sitemap na niyamo

  1. Generate karo: WordPress ma Yoast/RankMath auto-banave; custom site ma generator tool.
  2. Fakt 200-status, canonical URLs sitemap ma hova joiye - no 404s, no redirects, no noindex pages.
  3. Submit karo: Google Search Console → Sitemaps → submit. Bing Webmaster Tools ma pan.
  4. Update automatic hovu joiye new page add thay tyare.

5.2.C Common mistakes (aa bhool na karo)

  • "Disallow: /" staging ma hoy ane live site par j reh jay - site deindex!
  • Dead URLs sathe nu old sitemap
  • Sitemap submit karel pan GSC ma errors check na karela

5.2.D Checklist

  • robots.txt accessible ane sitemap line sathe
  • Admin/cart/search pages block karela, content pages open
  • Sitemap GSC + Bing ma submit karel chhe, koi errors nathi
  • New pages sitemap ma auto-add thay

5.2.E Ecommerce mate full rulebook (ChatGPT Plus thi verify karela, 19 Nov 2026)

4 vastu yaad rakho: robots.txt = crawl control, noindex = indexing control, canonical = duplicate ma thi preferred URL, sitemap = je pages index karavva chhe teni discovery list. Badha alag kaam kare chhe.

1. Robots.txt — su allow ane su block

URL typeRule
Homepage, product pages, collection/category pages, blog, important content pagesAllow + index
Cart (/cart), checkout (/checkout), account/loginDisallow
Internal search (/search?q=...)Disallow; already index thayel hoy to noindex
Thank-you / order confirmationNoindex + crawl restrict (order data protect karo)
Sort URLs (?sort_by=) ane filter URLs (?filter=)Generally disallow — PAN jo koi filter URL intentional SEO landing page hoy (dakhla "black running shoes" category) to allow + canonical + sitemap ma nakho
Tracking URLs (?utm_=)Canonical clean URL par; crawl block optional
CSS, JS, product imagesAllow — page render karva jaruri
Category paginationGenerally allow (crawl thavu joie)

2. Robots.txt example (custom site mate — Shopify ma blindly paste na karo)

User-agent: * | Disallow: /cart | Disallow: /checkout | Disallow: /account/ | Disallow: /search | Disallow: /thank-you | Disallow: /*?*sort_by= | Disallow: /*?*filter= | chelle: Sitemap: https://(site).com/sitemap.xml

3. Kaay pan block NA karo

  • Product, collection, category ke rank karvo hoy eva content pages
  • CSS/JS/images je page render mate jaruri
  • Je page par noindex chhe e page — block karsho to Google noindex joi j na shake
  • Akhi site par Disallow: / (production ma kyay nahi)

4. Sitemap rules

  • Include: canonical product, collection/category, homepage, important pages, indexable blogs, SEO-worthy filter landing pages
  • Exclude: cart, checkout, login/account, search URLs, thank-you, non-indexable filter/sort URLs, tracking parameter URLs, redirected URLs, 404/410, noindex pages, canonicalized duplicates
  • Darek sitemap URL: HTTP 200, indexable, HTTPS canonical host, internal link thi milay evu
  • Limit: 1 sitemap ma max 50,000 URLs ke 50MB — motu ho to split karo (sitemap-products-1.xml, sitemap-collections.xml, sitemap-pages.xml, sitemap-blogs.xml) ane sitemap index ma link karo
  • lastmod accurate rakho — fakt rebuild thayo etle update na karo; changefreq/priority Google ignore kare chhe
  • Product add/remove/change thay tyare sitemap update; weekly error check, monthly full audit

5. Shopify vs custom platform

  • Shopify robots.txt ane sitemap.xml auto-generate kare chhe — pehla review karo, pachhi j change karo
  • Shopify ma custom rules mate robots.txt.liquid edit karo — default rule groups replace na karo, add karo
  • /collections/category/products/product URL no canonical /products/product par j hoy — multiple collections ma product hoy to canonical audit karo
  • Sitemap Shopify auto-maintain kare (products, collections, pages, blogs)
  • Product hide karvo hoy to seo.hidden = 1 ke Unlisted status — pan storefront visibility pehla check karo
  • Custom platform ma: robots.txt approved template thi, sitemap database thi auto-generate, delete/redirect/draft/private URLs exclude, removed URL par real 404/410 ke 301, JS store ma crawlable product links confirm karo

6. Trainee checklist (darek client mate)

  1. /robots.txt kholo — production globally blocked nathi ne?
  2. Products, categories, blogs crawlable chhe?
  3. Cart, checkout, search, account, thank-you handling barabar chhe?
  4. Real filter/sort/pagination URLs test karya?
  5. CSS/JS/images block to nathi ne?
  6. /sitemap.xml ane child sitemaps load thay chhe?
  7. Sitemap URLs: 200, indexable, canonical, HTTPS?
  8. Size limit ane lastmod barabar?
  9. GSC ma sitemap submit karyu ane errors check karya?
  10. Changes pachhi GSC ma sample product/collection URLs inspect karya?

Final rule: sitewide robots.txt ke sitemap change karta PEHLA representative URLs test karo, ane deploy pachhi GSC review karo — especially mota ecommerce sites ma.