Disclosure: This post contains affiliate links. I may receive compensation when you click on links to products in this post. For an explanation of my Advertising Policy, visit this page. Thanks for reading!
What does it mean when Google says, “Crawled, but not indexed“?
According to Google, “Crawled but not indexed” means: “Crawled – currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.”
So, Google already knows that the site, page, post or URL exists, but has not indexed it in their Search Index.“
These could be due for a number of reasons, the 2 most common are:
- Technical reasons. For example, Google may have tried to crawl the URL but the site’s servers were overloaded or offline at the time of the crawl.
- URL has poor quality content.
These also mean that although Google visited the page on your website and found out some information about your page, such as its title, what keywords you used to link to this page etc. but for some reason, Google decided not to index the page and make it accessible to web users.
In this post, I am going to go over the possible reasons why Google crawled your site, but not indexed it and what you can do to get indexed!
Here are some more reasons why Google crawled your site, but not indexed it.
-The content on your page was considered ‘low quality’. Google may have decided that the information on the page is either irrelevant or of low value for website users, so they did not add it to their index.
-Your website might have one or more restrictive robots.txt files, such as ‘no Google’ which prevents Google from indexing your website.
-Your site might have a lot of dynamic URLs that change frequently or don’t follow a common pattern (for example, instead of having ‘/category/article-name’, it might be ‘/article-name’). This confuses Google and makes it difficult for them to crawl and index your website.
-The content might have been submitted to Google by a webmaster, but their spider was unable to find it or get to it at the time of submission.
If the content is not indexed, you need to make sure that all links from other websites point back to your website.
This will give Google a better chance of finding your content and indexing it for your site.
-If you have moved the page using a 301 redirect, or by modifying the actual destination URL (for example, if you changed http://www.example.com/article-name to http://www.exmaple.com/my-new-article-name), then Google will not be able to index the page and it will be invisible to users and search engines.
Has Google crawled your site, but not indexed it? Fix it!
You can get your content indexed by following these steps:
–Make sure that Google is crawling your entire website (that they are not ignoring any part of it) by checking Google webmaster tools.
-Submit a post or page URL into the “Inspect URL” box in your Google Search Console and click “Request Indexing”.
-Make sure that you are adding enough new content to your website on a regular basis – Google is more likely to index pages that are updated frequently).
-Enable the Google sitemap plugin for WordPress or manually create a Google sitemap. You can learn how to do this on the Google sitemaps page.
-Use the Yoast WordPress SEO plugin (and other similar plugins) to consolidate all of your SEO efforts into one, user-friendly dashboard. These plugins also have built-in site maps Google can crawl easily.
-Instead of using an SEO plugin, you can use the Site Kit by Google WordPress plugin. This plugin not only has a Google site map built in, but has all kinds of Google tools like Google Analytics and Google Search Console, all of which you can access via one convenient interface right in your WordPress Dashboard.
-Make sure that the website is not blocked from Google by a restrictive robots.txt file.
FAQs about Google indexing:
How often does Google index a site?
Google bots crawl millions of websites every day, but they aren’t all indexed immediately. It also depends on how often Google finds something new on the site. Google will index more frequently if you are adding content to your website or making changes with sitemaps.
What is a Googlebot?
The Googlebot is Google’s search spider, which crawls websites in order to find content that can be indexed by Google. Googlebot can also see what Google sees (you can check Google’s indexing of your site using your Google Search Console)
What is Google’s crawl rate?
Google only indexes the links that it finds on a website. Google can’t crawl all pages at once, but crawls each page in order to find new content (such as newly posted blog articles) and content that it has not yet indexed.
Google can’t index everything all at once either, but Google is constantly crawling the web, so your sites will usually be included in Google’s index within a few weeks (if not sooner).
Google usually indexes websites within 48 hours of crawling them, as long as the Googlebot is able to crawl the pages.
Remember, Google isn’t crawling your site all the time, Google is only periodically crawling a website depending on how often Google crawls a certain website and this can change at any time.
In addition, Google might not be crawling your website because Google’s web crawler “Googlebot” has been prevented from accessing your site.
This could be because Google doesn’t think that there is useful content on your site, or Google has no crawl budget.
The Googlebot can index a website instantly if the Googlebot crawler detects new or updated content on your website. As a result, Google will only update search results when Google crawls a page.
How often does Google recrawl a website?
Google will recrawl a website if Googlebot can’t find & index all the pages on a website.
Google will recrawl a website after Google has removed a page from Google’s index due to spamming or technical issues.
Google may recrawl your website more frequently as Google detects new/changed content on your site.
The Googlebot will recrawl a site when Google refreshes its search results to display the latest content.
The Googlebot will also recrawl a site if Google detects unusual fluctuations in the way that pages are being displayed or links are being followed.
The Googlebot will recrawl a website after Google shuts down its previous web crawler “Googlebot” as well, which can be due to updating Google’s web crawling technology.
Google crawled your site, but not indexed it, summary.
To sum up, Google will not index a website if Google believes that there is no useful content on the site, Google has crawled the site and found inaccessible pages or Google has blocked your website from being crawled.
If Google hasn’t indexed your new content within 48 hours of publishing, Google might have blocked some of your pages or images from being indexed.
You can submit a Google reconsideration request to Google using Google Search Consoles “Request Indexing” feature.
You can also improve Google’s crawl rate by making your website faster and ensuring that Google will not block your website from being crawled again in the future by keeping a robots.txt file updated with the Googlebot.
If your posts, pages or site isn’t indexed, no-one will be able to find you, which will result in little to no traffic, Yikes!
So make sure that Google has not only crawled all of your pages, but has indexed them as well.
