Blogspot/Blogger: Custom Robots.txt – How to? SEO

It is possible to customise robots.txt, a server-side text file that search engine bots can read. It tells search engine bots which directories, web pages, or links to index or not index in search results. Search engines can be restricted from crawling specific directories and pages or links on your site or blog. Custom robots.txt for Blogspot is now available. Labels are the subject of the search function in Blogger. Crawling of search results pages should be disabled if labels are not being used properly. The search URL in Blogger is by default blocked from crawling. The path of your sitemap can also be specified in the robots.txt file. A sitemap is a list of all the permalinks on your website or blog in one place. In most cases, sitemaps can be found in an XML file with the extension sitemap.xml.
The sitemap.xml has been completed by Blogger at this time. The sitemap feed is now being used by Blogger to read sitemap entries. Submission of the last 25 posts to search engines can be made using this strategy. It’s recommended that you use the robots.txt type 1 listed below if you want search engine bots to just operate on the last 25 posts. To get the most out of Google Adsense, you’ll want to configure your robots.txt file in this manner.

Robots.txt Type 1

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: https://www.pktopweb.blogspot.com/sitemap.xml

Note: The default robots.txt file on Blogspot can be modified as shown above.’

Robots.txt Type 2

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: https://www.pktopweb.blogspot.com/feeds/posts/default?orderby=updated

The https://www.pktopweb.blogspot.com should be changed to your own blog or custom domain. The following robots.txt type 2 should be used if you want search engine bots to crawl the last 500 posts on your site. The red sitemap line can be added if your blog already has 500 or more seats. Robots.txt

See also  How do I Test my Windows 10 surround Sound Speakers?

Robots.txt Type 3

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /b
Allow: /
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=501&max-results=500

The https://www.pktopweb.blogspot.com should be changed to your own blog or custom domain.

Blogger robots.txt sitemap entries can be expressed mathematically as the following:

Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=(m*0)+1&max-results=m
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=(m*1)+1&max-results=m
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=(m*3)+1&max-results=m
.
.
.
Sitemap: https://www.pktopweb.blogspot.com/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m

Where m=500 and n=1, 2, 3, 4,…, n. You can eliminate the following line if you have arranged post labels in a suitable format and have good expertise in search engine optimization (SEO).

Disallow: /search

If you don’t want any of your blog posts or pages to be indexed by search engines, you can simply add them like this: Add a line like this to post:

Disallow: /yyyy/mm/post-name.html

For Page, add a line like that.

Disallow: /p/page-name.html

Manage Blogger custom robots.txt:

For this, please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes

I wish you the best of luck with this post and a higher search engine rating resulting from it.

Related Articles

Leave a Reply

Your email address will not be published.

Back to top button