A question that often pops up for sites that are chasing “hot topics” is: how to get Google to crawl our site faster? This could be either because you have a big site and want to see changes faster or because you’re optimizing a news site. There are a few things you can do to improve your crawl speed. We’ll cover them in this post.
There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The other reasons are technical: Google has too much to crawl on your site, your site is too slow, or it’s encountering too many errors.
Your site doesn’t have enough authority
When your site doesn’t have enough quality inbound links, Google will not crawl your site very quickly. You can imagine it doesn’t necessarily want to spend much time on a site that it doesn’t consider important. When your site is brand new, this will be the case.
So, when you need more crawl action on your site, start doing some link building. We’ve written about that topic before, I’d suggest reading these three posts. They cover what you need to know to get started, so I won’t go into it any further here:
Link building from a holistic SEO perspective
5 link building DON’Ts you didn’t know about
6 steps to a successful link building strategy
If the above three posts are not enough for you, there’s more in our link building archive.
Technical reasons for slow crawling
The technical reasons for Google to crawl your site slowly can be divided into three groups: your site is too slow, you have too many errors, or you have too many URLs.
Your server is slow
The main reason why we see Google crawling sites slowly is when the site itself is actually slow. If your website is slow to respond to requests, or loads lots of bulky resources, Google might intentionally lower their crawl frequency and depth (or limit your crawl budget) to try to prevent things from breaking.
If that’s the case, you probably have larger issues with site speed, and should read up on how to measure your site speed, and how to speed things up!
Too many errors on your site
If you have a lot of errors on your site for Google, Google will start crawling slowly too. To speed up the crawl process, fix those errors. Simply 301 redirect those erroring pages to proper URLs on your site. If you don’t know where to find those errors: log into Google Search Console. If you have access to your site’s access logs, you can also look at those, preferably with a tool like Screaming Frog’s Log file analyzer. To prevent your site from being crawled slowly, it’s important that you regularly look at your site’s errors and fix them. We have a more extensive article on fixing 404 errors to help with that.
When your site is down too often, Google will also slow down incredibly hard. If this happens because of your hosting, switch hosting. Make sure to check your site’s uptime with a tool like Pingdom.
Too many URLs
If you simply have too many URLs on your site, Google might crawl a lot but it will never be enough. This can happen because of faceted search navigation for instance, or another system on your site that simply generates too many URLs. To figure out whether this is the case for you, it’s always wise to regularly crawl your own site. You can either do that manually with Screaming Frog’s SEO spider, or with a tool like Ryte.
Tips to increase your crawl speed
A couple of simple tips to increase your site’s crawl speed:
Using the methods above, find and fix all the errors.
Make sure your site is fast.
Add an XML sitemap to your site and submit it to the search engines.
If all of that fails to improve your crawl speed, start link building!
Specifically for news sites, if you want to be indexed faster, you should also look at our News SEO plugin. When you’re in (or want to be in) Google News, it can help make sure you meet all the technical requirements.