How to Ask Google to ‘Recrawl’ Your WordPress Website

Posted on October 1, 2018 by in Tips & Tricks | 18 comments

How to Ask Google to ‘Recrawl’ Your WordPress Website

Usually, Google is pretty good when it comes to listing your site’s content. However, sometimes the search engine may not be up-to-date with all the changes to your website, even if you’re following good Search Engine Optimization (SEO) practices. In those situations, you may need to ‘force’ (or ask nicely for) a Google recrawl so it can take another look at your site.

In this article, we’ll explain what a Google recrawl is and when you might need to ask for one. Then we’ll go over two ways you can ask Google to recrawl your website.

Let’s get to work!

What ‘Crawling’ Is (And How It Affects Your Site)

For search engines to display your website in their results, they first need to know it exists and what content it contains. To do that, search engines use bots that ‘crawl’ the web, jumping from one URL to another using the links on your pages.

According to Google, this process can sometimes take days or weeks. In our experience, though, the bots search engines use are pretty darn good when it comes to capturing all the relevant pages on your website.

It’s important to understand that you can’t ‘force’ the crawling process to go any faster. You have to be patient since there a lot of new sites going up all the time, and even Google can’t index them instantly.

However, it is possible to ask Google to recrawl your website or make it easier for the bot to get a full picture of what its pages are. We’ll dig into how that works next.

When You Might Need to Request a Google Recrawl

There are a few reasons why your site might not be properly indexed by Google. For instance, in rare cases, web crawlers can miss key content on your website. It might also be you made changes to a page or published new content and Google hasn’t had a chance to index it yet. In these cases, you can ask Google to take a second look at your site, which is referred to as ‘recrawling’.

Keep in mind – Google will eventually recrawl your website on its own if you leave it to its own devices. Even so, it’s impossible to predict the frequency of the crawling process. However, you can use Google Webmaster Tools to check out your site’s crawl stats.

The Google Search Console homepage.

We’ll talk more about how to use Google Webmaster Tools in the next section. For now, suffice it to say if you take the time to set up a sitemap and a robots.txt file, you shouldn’t run into situations where Google doesn’t index your site properly or misses key pages.

How to Request a Google Recrawl for Your WordPress Website (2 Methods)

In this section, we’re going to explore two ways you can request a recrawl to make sure all your site’s pages are fully indexed by Google. Let’s start with the sitemap approach.

1. Create and Submit a Sitemap to Google Search Console

As you may know, a sitemap is essentially a list of all the pages that make up your website. Sitemaps come in the .xml format and although you could put one together yourself, it’s far more efficient to generate it using a tool. For example, Yoast SEO automatically sets up a sitemap for you and keeps it updated:

The Yoast SEO plugin.

To access your sitemap, install and activate Yoast SEO. Once that’s done, navigate to the SEO > General page and jump to the Features tab. You’ll notice an option that reads XML sitemaps, which should be set to On:

Yoast SEO's sitemap settings.

Go ahead and click on the question mark icon next to the setting to make some links appear. One of these contains more information about sitemaps and the one to the left enables you to access yours:

Accessing your sitemap.

Copy your sitemap’s URL now and jump over to the Google Search Console. If you haven’t done so already, go ahead and connect your website to the service so you can get access to its features.

When you’re ready, navigate to the Sitemaps tab within Google Search Console. At the top of the screen, you’ll find an option to enter the URL of your sitemap:

Entering the URL of your sitemap.

After you add the URL, the sitemap should appear on the list right below. You’ll also see a Success message under the Status column:

The status of your sitemap's upload.

Once Google has access to your sitemap, the crawling process should go much more smoothly and chances are it won’t miss any pages.

2. Use the Google Search Console URL Inspector Tool

If you look around your Google Search Console dashboard, you’ll notice there’s an option that reads URL Inspection to the left of your screen. This feature enables you to check the status of any page on your site as it relates to Google. For example, here’s what it shows when you inspect an URL that’s not indexed:

Checking out an URL that's not indexed.

The easiest way to find if there are any pages that Google isn’t indexing is to use the URL inspector to check out your entire website. When you run into a result like the one you see above, you can go ahead and click on the REQUEST INDEXING option:

Request that Google indexes your website.

As we mentioned earlier, this doesn’t force Google to crawl or index your website immediately. However, making a request tells them there’s a URL they need to look at and usually, their web crawlers will get around to it pretty fast.

In practice, it’s a good idea to regularly use Google Search Console to check if there are any indexing errors for your website. After all, organic traffic is key to any endeavor and you don’t want to miss out on potential visits due to indexing errors.

Conclusion

For most of us, search engine traffic is a critical factor to our site’s success. That means if Google can’t find one or more of your pages, you could potentially lose out on valuable traffic. The easy way to fix this is to ask Google to recrawl your website.

In this article, we’ve shown you how to request a Google recrawl in two ways:

  1. Create and submit a sitemap to Google Search Console.
  2. Use the Google Search Console URL inspector tool.

Do you have any questions about how to get Google to recrawl your website? Let’s talk about them in the comments section below!

Article thumbnail image by Anatolir / shutterstock.com

18 Comments

  1. Hello John thanks for the great article does google automatically crawl you site when you post a new article. Or do you have to resubmit a re crawl on the site

    • John Hughes

      You’re welcome, Gerard. Google crawls all sites eventually even if you never request it, but it won’t do it automatically just because you post something new. 🙂

  2. Yoast has simplifies the work for SEO. With some little knowledge, you can set up basic SEO for your websites. Thumbs up for the perks.

  3. Search console is truly helpful if you are serious about serp and you nicely describe recrawl section. Appreciate it.

  4. There is another Search Console option that should be in here: fetch as Google.

    After you have uploaded a site map, it can still take weeks for a site to be indexed. To speed that up:
    1. In search console, go to crawl » fetch as Google
    2. Leave the text box blank so it only shows the root
    3. Keep it on desktop and click the [Fetch] button
    4. Click the [Request indexing] button you get after a few seconds
    5. Choose ‘Crawl this URL and its direct links’ and click [Go]
    6. Then repeat 3~5 for mobile.

    • John Hughes

      Thank you very much for sharing this, John-Pierre! 🙂

  5. What if I do not want google bots (and other bots) constantly crawling my site?
    I see in my page views in analytics they crawl my website every week. I do not need that.
    I just want them crawling like once a month.
    Is there something I can include in my robots or .htaccess file to tell them not to crawl the website more than once a month?

    I only update my website once a month. It is a small business and there is no need to have all bots crawling it everyday.
    Thanks

    • Hi Margarida, I think you have a wider issue than search engines constantly crawling your site if you are seeing these things in your analytics reports.
      Assuming your analytics is JavaScript based (such as Google Analytics) and not server analytics (such as Piwik/Matomo) bots should not be seen in your analytics unless it is spam visits that affects your reporting in general.

      I’d suggest you have a review of your analytics platform and setup some filters to clear out some of this spam.

  6. Thank you, I do SEO for my clients and now, with this information, you enlightened me on how to explain easy to them.

    • John Hughes

      I’m really happy to hear you found it useful! 🙂

  7. Hello John,
    Great information on Crawling of google bots in site URL…

  8. Nice post.. Very helpful.. I thought Yoast will do everything.. He would submit my sitemap but i noticed oneday that the date of submitting sitemap was 10 days earlier than i posted my last post. Now i came to know i can do it on my own. Thanks for this valuable post.

Leave a Reply

Comments are reviewed and must adhere to our comments policy.

500,591 Customers Are Already Building Amazing Websites With Divi. Join The Most Empowered WordPress Community On The Web

We offer a 30 Day Money Back Guarantee, so joining is Risk-Free!

Sign Up Today

Pin It on Pinterest