How to View Robots.txt file of any Website?

How to View Robots.txt file of any Website?

Hey, if do not know how to view Robots.txt file of any website, then just type robots.txt in front of the main domain of any site.

Let us explain you with an example:

Consider, Domain = https://www.xyz.com/

then you have to add “robots.txt” after domain like this: https://www.xyz.com/robots.txt

How to View Robots.txt file of InfoCompile?

If you wish you can test this on our website’s domain. Right now our website’s domain is something like this:

https://www.infocompile.com/how-to-view-robots-txt-file-of-any-website/

So, now just remove this:

how-to-view-robots-txt-file-of-any-website/

After doing this add this:

robots.txt

At last, your URL will look like this:

https://www.infocompile.com/robots.txt

When you will visit, this URL you will see our robots.txt file like this:

User-Agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
 
Sitemap: https://www.infocompile.com/post-sitemap.xml
Sitemap: https://www.infocompile.com/page-sitemap.xml

What to do if nothing comes up?

In our case, it is sure that the robots.txt file will be visible to you. But in your case, if nothing comes up, this means you do not have any robots.txt file yet. Don’t worry, we are here to help you, you can quickly learn about how to create a robots.txt file. <— Click here

So by the end of this article, you have learnt how to view a website’s robots.txt file. Stay Connected for more informative articles.?

More on InfoCompile:

Google Adsense High CPC Ads Tips | 100% Working Trick

Checklists for SEO, Keyword research, Content Writing, Link building

What is Keyword in SEO? Why are keywords important for SEO?

How to cleverly use Focus Keyword in first paragraph

How to check Website backlinks? Backlink Checker

Leave a Comment

Your email address will not be published. Required fields are marked *