What is a robot.txt file and what is its importance? 

The robot.txt file is a file that Webmaster Tools creates and its primary purpose is to organize and give instructions to search engine robots on how to archive some pages or prevent them from being archived at all and how to follow existing links in terms of following them and archiving the content in them or not, which helps in improving your SEO. On your blog,

in short,

the robots.txt file directs search engine robots to the pages that should be archived and prevents them from archiving some other pages that would be better not to be archived and appear in the search engine.

Examples of pages that it is better not to archive

* Search pages (pages in the main menu)
*
About the blog page* Privacy policy page
* Terms of use page
* And any page that you do not want to be archived and appear in the search engine   

How to obtain a robots.txt file

We talked about in Previous topics on

the importance of registering in Webmaster Tools, and here comes one of the things we will need.1- Go to Webmaster Tools and move to the old version
2- Go to the crawl and then test the robots.txt file
3- Copy the existing robot file And keep it in a notepad so we can use it
4- In the last line of the robots.txt file, you will find the word sitemap. Erase this line and add your sitemap that you previously created for your blog, the

basic form of the robots.txt file.

How to add a robots.txt file to a Blogger blog 

1- Go to your Blogger blog
2- Go to Settings, then Search Preferences
3- When the word “Custom robot.txt file” click on “Edit”
4- Paste the robots.txt file in the Notepad file and then click on Save Changes

On your blog, you will find next to the word “Edit” the word “inactive.” This is because you have not added the robots.txt file yet, but I have already added it. Therefore, you will find the word “activated” when

adding the robots.txt file. Follow the instructions carefully because if any errors occur, archiving may be completely prevented.

For more information about the robots.txt file
for the site,*
The article about the robots.txt file in Webmaster Tools*The robots.txt file in the Moz directory

Head signs, how to adjust them, and their importance 

Warning:
When working on adjusting header tags or the robots.txt file, follow the steps carefully because if any errors occur, the entire archiving of your blog or site may be prevented.

How to set custom robots header tags for Blogger blogs

Setting header tags and configuring them correctly is one of

the important SEO factorsthat you must pay attention to. These tags prepare your site for search engines.The importance of header tags?

It is natural for you to ask yourself why you adjust the header tags and what is the purpose of that. Therefore, you must know that the importance of adjusting these tags is to prevent the duplication of some pages by preventing the archiving of pages such as the archive and search pages that are already on your blog.

Google, when it finds most duplicate pages more than once, may It considers these pages as spam and may prevent the archiving of both pages.

It contributes to solving the problem of duplicate meta codes in webmaster tools.

You may also like: 30 ways to attract thousands of visitors to your blog 

How to set header tags for robots correctly? 

2- Go to Settings, then Search Preferences, and at the header tabs designated for robots, click Edit 

3- After that, select the tags as in this image, then click on Save Changes 

How to correctly set header tags for robots on a Blogger blog

Thus, you have set the header tags for your Blogger blog correctly 

Did you like the topic? 

If you like the topic, share it with your friends so that everyone can benefit 

* If you are a blogger, don’t forget to subscribe to the blog and follow us on Facebook to receive all new updates