.Google's John Mueller answered a question regarding why Google.com indexes web pages that are forbidden from creeping through robots.txt and also why the it is actually risk-free to ignore the associated Explore Console reports about those creeps.Crawler Visitor Traffic To Inquiry Specification URLs.The person asking the question chronicled that crawlers were actually producing web links to non-existent query criterion URLs (? q= xyz) to pages with noindex meta tags that are actually also shut out in robots.txt. What triggered the inquiry is actually that Google.com is crawling the web links to those webpages, receiving obstructed through robots.txt (without watching a noindex robots meta tag) after that getting shown up in Google.com Look Console as "Indexed, though blocked by robots.txt.".The person talked to the observing inquiry:." But here's the significant question: why would certainly Google.com mark webpages when they can't even observe the content? What's the conveniences during that?".Google's John Mueller affirmed that if they can not creep the webpage they can not see the noindex meta tag. He also creates an exciting reference of the website: search driver, advising to disregard the outcomes since the "typical" consumers will not observe those outcomes.He wrote:." Yes, you're proper: if our team can not crawl the page, our team can't observe the noindex. That said, if our experts can't creep the webpages, at that point there is actually certainly not a whole lot for us to mark. Thus while you might see several of those webpages along with a targeted web site:- question, the normal consumer will not see them, so I wouldn't fuss over it. Noindex is actually also fine (without robots.txt disallow), it only suggests the URLs are going to wind up being crept (and also end up in the Look Console report for crawled/not indexed-- neither of these standings create issues to the rest of the site). The essential part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the limitations being used the Site: search accelerated search operator for diagnostic explanations. Among those causes is actually due to the fact that it is actually not hooked up to the routine hunt index, it is actually a separate point completely.Google's John Mueller discussed the internet site hunt driver in 2021:." The quick solution is that a website: question is not suggested to become full, neither utilized for diagnostics purposes.A web site inquiry is actually a particular kind of hunt that limits the results to a specific web site. It is actually generally merely the word internet site, a colon, and afterwards the website's domain name.This query limits the end results to a particular website. It's certainly not suggested to become an extensive selection of all the web pages from that site.".2. Noindex tag without utilizing a robots.txt is great for these sort of circumstances where a bot is actually connecting to non-existent web pages that are acquiring discovered through Googlebot.3. Links along with the noindex tag will definitely generate a "crawled/not catalogued" entry in Explore Console and that those will not possess a damaging impact on the remainder of the internet site.Review the inquiry as well as answer on LinkedIn:.Why would Google mark webpages when they can not also see the content?Included Photo by Shutterstock/Krakenimages. com.