In this article · 10 sections
- What robots.txt does
- What robots.txt does NOT do
- What an XML sitemap does
- What belongs in your sitemap
- Large or dynamic sites
- How to submit your sitemap
- Common mistakes we fix
- What about AI crawlers?
- Frequently Asked Questions
- Does every website need a robots.txt file?
- How often should the sitemap be updated?
- Will submitting a sitemap make my pages rank?
- Should images have their own sitemap?
- Get a technical SEO check
Every website has two small files that quietly influence how search engines see it: robots.txt and the XML sitemap. Most business owners never look at them — until a single wrong line blocks the whole site from Google. This guide explains what each file does, what a good version looks like, and the mistakes we see most often when auditing websites.
What robots.txt does
Robots.txt lives at yourdomain.com/robots.txt. It gives instructions to crawlers (Googlebot, Bingbot and others) about which parts of your site they may crawl. A simple version:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://example.com/sitemap.xmlThis says: all crawlers may access everything except the admin and cart areas, and here is where the sitemap is.
What robots.txt does NOT do
- It doesn't remove pages from Google. A blocked page can still appear in results (without a description) if other sites link to it. To keep a page out of search, use a
noindexmeta tag and allow crawling so Google can see it. - It isn't security. Anyone can read your robots.txt. Never use it to "hide" private folders — protect them with a login instead.
- Not every bot obeys it. Reputable search engines do; spam bots often don't.
What an XML sitemap does
An XML sitemap is a list of the URLs you want search engines to know about, usually with the date each was last updated. It helps Google discover pages faster, especially new pages, deep pages and pages on newer sites with few links. A minimal entry looks like:
<url>
<loc>https://example.com/website-design-ludhiana</loc>
<lastmod>2026-10-01</lastmod>
</url>A sitemap is a suggestion, not a guarantee. Google still decides what to crawl and index based on quality and links.
What belongs in your sitemap
| Include | Leave out |
|---|---|
| Homepage, service pages, location pages with unique content | Admin, login, cart and checkout pages |
| Published blog posts | Search results, tag and filter URLs |
| Product and category pages | Pages with noindex |
| Only the final, canonical URL of each page | Redirecting URLs, 404 pages, duplicate versions |
A clean sitemap is a signal of a well-maintained site. A sitemap full of redirects and errors suggests the opposite.
Large or dynamic sites
A single sitemap file can hold up to 50,000 URLs. Larger sites use a sitemap index that points to several sitemaps — for example one for pages, one for blog posts and one for products. For PHP sites, the sitemap is best generated dynamically from the database so new pages appear automatically.
How to submit your sitemap
- Open Google Search Console and select your property.
- Go to Indexing → Sitemaps.
- Enter the sitemap URL (for example
sitemap.xml) and click Submit. - Check the status after a day or two — it should say "Success" with a count of discovered URLs.
Also list the sitemap in robots.txt so other search engines, including Bing, can find it. You can submit it in Bing Webmaster Tools too.
Common mistakes we fix
- "Disallow: /" left over from development — this blocks the entire site. Always check robots.txt after a launch.
- Blocking CSS and JavaScript folders — Google needs these to render pages properly.
- Sitemap listing http or www URLs when the site uses https and non-www.
- Old sitemap plugins left active after moving away from WordPress, producing broken URLs.
- Thousands of near-duplicate URLs in the sitemap, diluting attention from important pages.
- lastmod dates that change on every page load — use the real date the content changed.
What about AI crawlers?
Robots.txt is also where you decide whether AI crawlers such as GPTBot or ClaudeBot can access your site. We cover this in detail in our guide on AI crawlers and robots.txt.
Frequently Asked Questions
Does every website need a robots.txt file?
It isn't required — without one, crawlers assume they can access everything. But a simple file that points to your sitemap and blocks admin areas is good practice.
How often should the sitemap be updated?
Whenever pages are added, removed or meaningfully changed. Dynamic sitemaps do this automatically.
Will submitting a sitemap make my pages rank?
No. It helps discovery. Rankings depend on content quality, relevance, links and user experience.
Should images have their own sitemap?
For most small business sites it isn't necessary. It can help image-heavy sites such as photographers or e-commerce stores.
Get a technical SEO check
Joboo Web reviews robots.txt, sitemaps and indexing during every audit. See SEO services, our indexing problems guide, or contact us.