📄 5.2 — Robots.txt & Sitemap Properly
Dabbo 05 · On-Page SEO ·
Robots.txt Google ne kahe chhe su crawl karvu ane su nahi. Sitemap.xml Google ne site na badha pages nu map aape chhe. Banne technical SEO nu base chhe - kharab robots.txt important pages ne crawl thi roki shake; sitemap index ni guarantee nathi aaptu, pan discovery fast kare.
5.2.A Robots.txt na niyamo
- Location: domain.com/robots.txt ma accessible hovu joiye.
- Crawl block karo: admin pages, cart, checkout, internal search result pages. Duplicate/filter URLs ane thank-you pages mate pehla canonical/noindex vaparo — robots.txt block fakt crawl waste motu hoy tyare (ane index mathi nikli gaya pachhi).
- Kyarey block na karo: CSS/JS files (Google render karvu joiye), main content pages.
- Sitemap line add karo: Sitemap: https://domain.com/sitemap.xml
5.2.B Sitemap na niyamo
- Generate karo: WordPress ma Yoast/RankMath auto-banave; custom site ma generator tool.
- Fakt 200-status, canonical URLs sitemap ma hova joiye - no 404s, no redirects, no noindex pages.
- Submit karo: Google Search Console → Sitemaps → submit. Bing Webmaster Tools ma pan.
- Update automatic hovu joiye new page add thay tyare.
5.2.C Common mistakes (aa bhool na karo)
- "Disallow: /" staging ma hoy ane live site par j reh jay - site deindex!
- Dead URLs sathe nu old sitemap
- Sitemap submit karel pan GSC ma errors check na karela
5.2.D Checklist
- robots.txt accessible ane sitemap line sathe
- Admin/cart/search pages block karela, content pages open
- Sitemap GSC + Bing ma submit karel chhe, koi errors nathi
- New pages sitemap ma auto-add thay
5.2.E Ecommerce mate full rulebook (ChatGPT Plus thi verify karela, 19 Nov 2026)
4 vastu yaad rakho: robots.txt = crawl control, noindex = indexing control, canonical = duplicate ma thi preferred URL, sitemap = je pages index karavva chhe teni discovery list. Badha alag kaam kare chhe.
1. Robots.txt — su allow ane su block
| URL type | Rule |
|---|---|
| Homepage, product pages, collection/category pages, blog, important content pages | Allow + index |
| Cart (/cart), checkout (/checkout), account/login | Disallow |
| Internal search (/search?q=...) | Disallow; already index thayel hoy to noindex |
| Thank-you / order confirmation | Noindex + crawl restrict (order data protect karo) |
| Sort URLs (?sort_by=) ane filter URLs (?filter=) | Generally disallow — PAN jo koi filter URL intentional SEO landing page hoy (dakhla "black running shoes" category) to allow + canonical + sitemap ma nakho |
| Tracking URLs (?utm_=) | Canonical clean URL par; crawl block optional |
| CSS, JS, product images | Allow — page render karva jaruri |
| Category pagination | Generally allow (crawl thavu joie) |
2. Robots.txt example (custom site mate — Shopify ma blindly paste na karo)
User-agent: * | Disallow: /cart | Disallow: /checkout | Disallow: /account/ | Disallow: /search | Disallow: /thank-you | Disallow: /*?*sort_by= | Disallow: /*?*filter= | chelle: Sitemap: https://(site).com/sitemap.xml
3. Kaay pan block NA karo
- Product, collection, category ke rank karvo hoy eva content pages
- CSS/JS/images je page render mate jaruri
- Je page par noindex chhe e page — block karsho to Google noindex joi j na shake
- Akhi site par Disallow: / (production ma kyay nahi)
4. Sitemap rules
- Include: canonical product, collection/category, homepage, important pages, indexable blogs, SEO-worthy filter landing pages
- Exclude: cart, checkout, login/account, search URLs, thank-you, non-indexable filter/sort URLs, tracking parameter URLs, redirected URLs, 404/410, noindex pages, canonicalized duplicates
- Darek sitemap URL: HTTP 200, indexable, HTTPS canonical host, internal link thi milay evu
- Limit: 1 sitemap ma max 50,000 URLs ke 50MB — motu ho to split karo (sitemap-products-1.xml, sitemap-collections.xml, sitemap-pages.xml, sitemap-blogs.xml) ane sitemap index ma link karo
- lastmod accurate rakho — fakt rebuild thayo etle update na karo; changefreq/priority Google ignore kare chhe
- Product add/remove/change thay tyare sitemap update; weekly error check, monthly full audit
5. Shopify vs custom platform
- Shopify robots.txt ane sitemap.xml auto-generate kare chhe — pehla review karo, pachhi j change karo
- Shopify ma custom rules mate robots.txt.liquid edit karo — default rule groups replace na karo, add karo
- /collections/category/products/product URL no canonical /products/product par j hoy — multiple collections ma product hoy to canonical audit karo
- Sitemap Shopify auto-maintain kare (products, collections, pages, blogs)
- Product hide karvo hoy to seo.hidden = 1 ke Unlisted status — pan storefront visibility pehla check karo
- Custom platform ma: robots.txt approved template thi, sitemap database thi auto-generate, delete/redirect/draft/private URLs exclude, removed URL par real 404/410 ke 301, JS store ma crawlable product links confirm karo
6. Trainee checklist (darek client mate)
- /robots.txt kholo — production globally blocked nathi ne?
- Products, categories, blogs crawlable chhe?
- Cart, checkout, search, account, thank-you handling barabar chhe?
- Real filter/sort/pagination URLs test karya?
- CSS/JS/images block to nathi ne?
- /sitemap.xml ane child sitemaps load thay chhe?
- Sitemap URLs: 200, indexable, canonical, HTTPS?
- Size limit ane lastmod barabar?
- GSC ma sitemap submit karyu ane errors check karya?
- Changes pachhi GSC ma sample product/collection URLs inspect karya?
Final rule: sitewide robots.txt ke sitemap change karta PEHLA representative URLs test karo, ane deploy pachhi GSC review karo — especially mota ecommerce sites ma.