.Google's John Mueller answered an inquiry concerning why Google.com marks webpages that are refused from crawling through robots.txt and also why the it is actually secure to overlook the associated Browse Console documents regarding those crawls.Robot Visitor Traffic To Inquiry Specification URLs.The individual talking to the concern recorded that robots were actually creating links to non-existent query parameter URLs (? q= xyz) to webpages along with noindex meta tags that are also obstructed in robots.txt. What caused the question is actually that Google is creeping the links to those webpages, receiving obstructed through robots.txt (without noticing a noindex robots meta tag) then obtaining shown up in Google Search Console as "Indexed, though shut out through robots.txt.".The person inquired the observing question:." But below is actually the big question: why would certainly Google index web pages when they can not even see the information? What is actually the advantage during that?".Google's John Mueller verified that if they can't creep the page they can not observe the noindex meta tag. He likewise produces a fascinating reference of the website: search operator, advising to dismiss the results due to the fact that the "common" users will not find those outcomes.He created:." Yes, you are actually appropriate: if we can't creep the page, our experts can not observe the noindex. That stated, if our team can't crawl the webpages, at that point there is actually certainly not a whole lot for us to mark. Thus while you could view some of those web pages with a targeted internet site:- query, the normal customer will not see all of them, so I wouldn't bother it. Noindex is actually likewise great (without robots.txt disallow), it merely means the URLs will wind up being actually crept (as well as wind up in the Search Console record for crawled/not catalogued-- neither of these statuses result in problems to the rest of the web site). The fundamental part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the limits in operation the Site: search accelerated hunt driver for diagnostic factors. Some of those main reasons is actually because it's not linked to the routine hunt index, it's a different trait altogether.Google.com's John Mueller commented on the site hunt operator in 2021:." The quick response is actually that a site: concern is actually certainly not implied to be complete, neither utilized for diagnostics purposes.A site query is a details type of search that limits the results to a particular website. It's primarily merely the word web site, a digestive tract, and then the site's domain.This question restricts the end results to a certain internet site. It's certainly not meant to be an extensive selection of all the webpages coming from that website.".2. Noindex tag without making use of a robots.txt is great for these sort of scenarios where a robot is actually linking to non-existent web pages that are actually receiving discovered through Googlebot.3. Links along with the noindex tag are going to create a "crawled/not indexed" item in Search Console and also those won't possess an unfavorable result on the rest of the web site.Review the inquiry as well as address on LinkedIn:.Why will Google index web pages when they can't even view the content?Included Image by Shutterstock/Krakenimages. com.