How to Use Custom Robots.txt with Your Blog? – Blogspot/Blogger SEO

 Robots.txt is a text file on the server in that you can write which directories, web pages, or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt is available for Blogger. In Blogger search option is related to Labels. If you are not using labels wisely per post, you should disallow the crawl of the search link. In Blogger, by default, the search link is disallowed to crawl. In this robots.txt, you can also write the location of your sitemap file. A sitemap is a file located on a server that contains all posts’ permalinks of your website or blog. Mostly sitemap is found in XML format, i.e., sitemap.xml.

Presently Blogger is working on sitemap.xml. Now Blogger is reading sitemap entries through the feed. By this method, the most recent 25 posts are submitted to search engines. If you want search engine, bots, only work on the most recent 25 posts, and then you should use robots.txt type 1 given below. If you set robots.txt like this, then Google Adsense bot is allowed to crawl the entire blog for best Adsense performance.

Table of Contents  hide 

Robots.txt Type 1

Robots.txt Type 2

Robots.txt Type 3

Manage Blogger custom robots.txt

Robots.txt Type 1

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/sitemap.xml

Note: You may alter Blogspot’s default robots.txt like above.’

Robots.txt Type 2

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/feeds/posts/default?orderby=updated

Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain. If you want search engine bots to crawl the most recent 500 posts, then you should need to use following robots.txt type 2. If already you have more than 500 posts on your blog, then you can add one more sitemap line highlighted in red. Robots.txt

Robots.txt Type 3

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /b
Allow: /

Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=501&max-results=500

Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain.

Mathematical expression for Blogger robots.txt sitemap entries:

Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*0)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*1)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*3)+1&max-results=m
.
.
.
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m

Where m=500 and n=1, 2, 3, 4,…, n. If you have organized post labels in a good format and good experience of search engine optimization (SEO), then you can remove the following line –

Disallow: /search

Most important: if you don’t want to submit any blogger post or page to search engines then you can add them like that: For Post add a line like that –

Disallow: /yyyy/mm/post-name.html

For Page add a line like that –

Disallow: /p/page-name.html

Manage Blogger custom robots.txt

For this, please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes To get a better understanding of this you can take reference of the image given below:

I hope you’ll get the benefit of this post and get better search engine presence and ranking.

Ref: https://www.techprevue.com