How to Ask Google to ‘Recrawl’ Your WordPress Website

Last Updated on September 12, 2023 by 19 Comments

How to Ask Google to ‘Recrawl’ Your WordPress Website
Blog / Tips & Tricks / How to Ask Google to ‘Recrawl’ Your WordPress Website

Usually, Google is pretty good when it comes to listing your siteโ€™s content. However, sometimes the search engine may not be up-to-date with all the changes to your website, even if youโ€™re following good Search Engine Optimization (SEO) practices. In those situations, you may need to โ€˜forceโ€™ (or ask nicely for) a Google recrawl so it can take another look at your site.

In this article, weโ€™ll explain what a Google recrawl is and when you might need to ask for one. Then weโ€™ll go over two ways you can ask Google to recrawl your website.

Letโ€™s get to work!

What โ€˜Crawlingโ€™ Is (And How It Affects Your Site)

For search engines to display your website in their results, they first need to know it exists and what content it contains. To do that, search engines use bots that โ€˜crawlโ€™ the web, jumping from one URL to another using the links on your pages.

According to Google, this process can sometimes take days or weeks. In our experience, though, the bots search engines use are pretty darn good when it comes to capturing all the relevant pages on your website.

Itโ€™s important to understand that you canโ€™t โ€˜forceโ€™ the crawling process to go any faster. You have to be patient since there aย lotย of new sites going up all the time, and even Google canโ€™t index them instantly.

However, itย isย possible to ask Google to recrawl your website or make it easier for the bot to get a full picture of what its pages are. Weโ€™ll dig into how that works next.

When You Might Need to Request a Google Recrawl

There are a few reasons why your site might not be properly indexed by Google. For instance, in rare cases, web crawlers can miss key content on your website. It might also be you made changes to a page or published new content and Google hasnโ€™t had a chance to index it yet. In these cases, you can ask Google to take a second look at your site, which is referred to as โ€˜recrawlingโ€™.

Keep in mind โ€“ Google will eventually recrawl your website on its own if you leave it to its own devices. Even so, itโ€™s impossible to predict the frequency of the crawling process. However, you can use Google Webmaster Tools to check out your siteโ€™s crawl stats.

The Google Search Console homepage.

Weโ€™ll talk more about how to use Google Webmaster Tools in the next section. For now, suffice it to say if you take the time to set up a sitemap and a robots.txt file, you shouldnโ€™t run into situations where Google doesnโ€™t index your site properly or misses key pages.

How to Request a Google Recrawl for Your WordPress Website (2 Methods)

In this section, weโ€™re going to explore two ways you can request a recrawl to make sure all your siteโ€™s pages are fully indexed by Google. Letโ€™s start with the sitemap approach.

1. Create and Submit a Sitemap to Google Search Console

As you may know, a sitemap is essentially a list of all the pages that make up your website. Sitemaps come in theย .xmlย format and although you could put one together yourself, itโ€™s far more efficient to generate it using a tool. With WordPress, itโ€™s easy to find a dedicated sitemap plugin to help you. All of the top WordPress SEO plugins will help you create a sitemap. For example, Yoast SEO automatically sets up a sitemap for you and keeps it updated:

The Yoast SEO plugin.

To access your sitemap, install and activate Yoast SEO. Once thatโ€™s done, navigate to theย SEO > Generalย page and jump to theย Featuresย tab. Youโ€™ll notice an option that readsย XML sitemaps, which should be set toย On:

Yoast SEO's sitemap settings.

Go ahead and click on the question mark icon next to the setting to make some links appear. One of these contains more information about sitemaps and the one to the left enables you to access yours:

Accessing your sitemap.

Copy your sitemapโ€™s URL now and jump over to the Google Search Console. If you havenโ€™t done so already, go ahead and connect your website to the service so you can get access to its features.

When youโ€™re ready, navigate to theย Sitemapsย tab within Google Search Console. At the top of the screen, youโ€™ll find an option to enter the URL of your sitemap:

Entering the URL of your sitemap.

After you add the URL, the sitemap should appear on the list right below. Youโ€™ll also see aย Successย message under theย Statusย column:

The status of your sitemap's upload.

Once Google has access to your sitemap, the crawling process should go much more smoothly and chances are it wonโ€™t miss any pages.

2. Use the Google Search Console URL Inspector Tool

If you look around your Google Search Console dashboard, youโ€™ll notice thereโ€™s an option that readsย URL Inspectionย to the left of your screen. This feature enables you to check the status of any page on your site as it relates to Google. For example, hereโ€™s what it shows when you inspect an URL thatโ€™s not indexed:

Checking out an URL that's not indexed.

The easiest way to find if there are any pages that Google isnโ€™t indexing is to use the URL inspector to check out your entire website. When you run into a result like the one you see above, you can go ahead and click on theย REQUEST INDEXINGย option:

Request that Google indexes your website.

As we mentioned earlier, this doesnโ€™t force Google to crawl or index your website immediately. However, making a request tells them thereโ€™s a URL they need to look at and usually, their web crawlers will get around to it pretty fast.

In practice, itโ€™s a good idea to regularly use Google Search Console to check if there are any indexing errors for your website. After all, organic traffic is key to any endeavor and you donโ€™t want to miss out on potential visits due to indexing errors.

Conclusion

For most of us, search engine traffic is a critical factor to our siteโ€™s success. That means if Google canโ€™t find one or more of your pages, you could potentially lose out on valuable traffic. The easy way to fix this is to ask Google to recrawl your website.

In this article, weโ€™ve shown you how to request a Google recrawl inย two ways:

  1. Create and submit a sitemap to Google Search Console.
  2. Use the Google Search Console URL inspector tool.

Do you have any questions about how to get Google to recrawl your website? Letโ€™s talk about them in the comments section below!

Article thumbnail image by Anatolir / shutterstock.com

Divi

Want To Build Better WordPress Websites? Start Here! ๐Ÿ‘‡

Take the first step towards a better website.

Get Started
Divi
Premade Layouts

Check Out These Related Posts

Splice Video Editor: An Overview and Review

Splice Video Editor: An Overview and Review

Updated on March 10, 2023 in Tips & Tricks

Video is a valuable form of content for social media. Unfortunately, creating quality videos is usually a long process that involves moving mobile footage to a desktop app for editing. However, mobile editing is on the rise. Apps such as Splice Video Editor make it possible to efficiently create...

View Full Post
How to Use Font Awesome On Your WordPress Website

How to Use Font Awesome On Your WordPress Website

Updated on September 16, 2022 in Tips & Tricks

When given the choice between using a vector icon or a static image, it’s a good idea to go with the vector. They’re small and fast to load, and they can scale to any size without a loss of resolution. Font Awesome is a superb library of vector icons that you can use on your websites,...

View Full Post

19 Comments

  1. Thank you for useful tips, John!

  2. Nice post.. Very helpful.. I thought Yoast will do everything.. He would submit my sitemap but i noticed oneday that the date of submitting sitemap was 10 days earlier than i posted my last post. Now i came to know i can do it on my own. Thanks for this valuable post.

    • You’re welcome, Punit. ๐Ÿ™‚

  3. Hello John,
    Great information on Crawling of google bots in site URL…

    • Thank you, Kalyani ๐Ÿ™‚

  4. Thank you, I do SEO for my clients and now, with this information, you enlightened me on how to explain easy to them.

    • I’m really happy to hear you found it useful! ๐Ÿ™‚

  5. What if I do not want google bots (and other bots) constantly crawling my site?
    I see in my page views in analytics they crawl my website every week. I do not need that.
    I just want them crawling like once a month.
    Is there something I can include in my robots or .htaccess file to tell them not to crawl the website more than once a month?

    I only update my website once a month. It is a small business and there is no need to have all bots crawling it everyday.
    Thanks

    • Hi Margarida, I think you have a wider issue than search engines constantly crawling your site if you are seeing these things in your analytics reports.
      Assuming your analytics is JavaScript based (such as Google Analytics) and not server analytics (such as Piwik/Matomo) bots should not be seen in your analytics unless it is spam visits that affects your reporting in general.

      I’d suggest you have a review of your analytics platform and setup some filters to clear out some of this spam.

  6. There is another Search Console option that should be in here: fetch as Google.

    After you have uploaded a site map, it can still take weeks for a site to be indexed. To speed that up:
    1. In search console, go to crawl ยป fetch as Google
    2. Leave the text box blank so it only shows the root
    3. Keep it on desktop and click the [Fetch] button
    4. Click the [Request indexing] button you get after a few seconds
    5. Choose ‘Crawl this URL and its direct links’ and click [Go]
    6. Then repeat 3~5 for mobile.

    • Thank you very much for sharing this, John-Pierre! ๐Ÿ™‚

  7. great site!!! thx from austria, agnes

  8. Search console is truly helpful if you are serious about serp and you nicely describe recrawl section. Appreciate it.

  9. Yoast has simplifies the work for SEO. With some little knowledge, you can set up basic SEO for your websites. Thumbs up for the perks.

  10. VERY NICE! VERY HELPFUL! THANK YOU!

    • You’re welcome! ๐Ÿ™‚

  11. Hello John thanks for the great article does google automatically crawl you site when you post a new article. Or do you have to resubmit a re crawl on the site

    • You’re welcome, Gerard. Google crawls all sites eventually even if you never request it, but it won’t do it automatically just because you post something new. ๐Ÿ™‚

Leave A Reply

Comments are reviewed and must adhere to our comments policy.

Get Started With Divi