Site Speed: Reducing Page Load Times and Increasing Pages Crawled 23

I’ve always been a big fan of quick page load times on sites from overall User Experience standpoint. But, ever since Google announced that they were using site speed in web search ranking I have become a big time stickler about this. After watching this for the last couple months, I’ve realized that my peskiness about the whole thing warranted just cause.

Proving that Site Speed is directly correlated Pages Crawled

I tend to keep an eye on Google Webmaster tools to see any blips in crawl rate, pages crawled, etc. I don’t want any big issues to show up that catch me by surprise. When Google made the announcement I made sure to keep an eye on site speed. Here are a couple examples of the DIRECT CORRELATION

Website 1: Page Load times shoot up, Pages Crawled drops:
First look at graph three and notice the massive upswing in the page load times or as google titles it “time spent downloading a page.” Then, from there, look at the top two graphics and notice the massive drop off in the number of pages crawled per day and the amount of content downloaded per day.

Page Load time increase

Website 2: Page Load times go down, Pages Crawled dramatically increases:
On the flip side of the above chart, looking at another site, check out graph three again first. Notice that the time spent downloading a page (thus page load time) has decreased dramatically. Now, as you look at graphs one and two, notice the massive uptick in the number of pages crawled and amount of content downloaded daily.

Pages Crawled Increases

I can tell you that this was a big positive for website 2, to notice a HUGE positive in the number of pages crawled daily.

What this means for SEO and User Experience

Understanding how to diagnose and address these issues is extremely important to any SEO and Webmaster now at days. I was a huge advocate of this at PayPal and lent a helping hand in diagnosing this and addressing it in hopes of increasing the user experience. Again though, we now know what a big part this obviously plays, as I’ve shown the direct correlation above with the number of pages crawled increasing or decreasing based on the page load times.

I was very happy to see this as it does increase the overall user and searcher experience by coming to sites that are loading pages faster. But, it is extremely important for an SEO to note and understand the importance of site speed, especially with sites that have extremely large indexes.

Diagnosing Site Speed in Google Webmaster Tools

Start by logging into webmaster tools and talking a look at the Diagnostics -> Crawl Stats section and make note of any increases or decreases in the third graph about page download times. If there are any big rises or drops, check out the above two graphs to see if it had an effect on your pages crawled. If you are seeing relatively steady movement, move on to the Labs -> Site Performance section.

On the Site Performance page you’ll notice a few things. To start with the graph that represents the avg. number of seconds it takes to load your pages. Something to take note of that they’ve highlighted the optimal time to download a page as 1.5 seconds or faster. Depending on your site, they’ll list about 10 URLs and their respective page load time. Finally, they provide some very useful info (which I’ll get to in a minute) on how to address these concerns.

There are a couple tools that you can use to diagnose site speed and page load times. My favorite is yslow and Google recently released their Page Speed tool, both of these plug directly into my favorite Firefox Plugin: Firebug.

Addressing Site Speed and Page Load time Concerns

Google Webmaster Tools and their Page Speed tool, along with YSlow provide you ample amounts of information to start addressing the site speed concerns one step at a time. Some of the biggest mistakes that people make run into is the size of images and image optimization, number of HTTP request and number of external JavaScript and CSS files. Here is a list of issues and how to address them:

Making fewer HTTP Requests
The number of HTTP requests you make can put a massive strain on the page load times. As mentioned in the screenshot below, try to combine external files to a very minimal amount. I’ve seen sites that have as many as 20 external JavaScript files and some of those files with only 1-4 lines of code. That is the type of thing that can be combined to reduce the number of http requests. The same goes with CSS files and optimizing the number of CSS calls into one main CSS document.

http requests

JavaScript Minification and Compression
Yahoo!’s Developer Network blog has a post dedicated to how to minify javascript. The main reason for doing this is to shrink the overall size of the JavaScript file by getting rid of comments, new lines, etc. that aren’t necessary for the JavaScript to process. There are a couple tools out there to do this, JSMin and YUI Compressor.

Using Gzip to compress components
You can drastically reduce the HTTP Requests and Responses using Gzip compression, and, it is the most popular and effective form right now. In Apache you can use mod_gzip and in IIS you can do this through IIS Manager and adding a Web Service extension using these IIS compression tips.

CSS Sprites
CSS Sprites are extremely useful for reducing the number of image requests (thus lowering http requests). Having your background images in one large image and using CSS for the background image and positions you can use a single image for many purposes. A List Apart has a great resource that goes into the details of the HTML Markup and CSS that is necessary to make this dream a reality.

Much much more!
There are a ton more ways of improving site performance and reducing page load time to not only help your page crawl rate and pages crawled on your site, but also, increase the user experience. Some of the things I didn’t even get to touch on like adding expires headers, reducing the number of DNS lookups, using a CDN (Content Delivery Network), etc. are all very important to reducing page load times.

Have you noticed similar things with page load times and pages crawled? How are you dealing with Site Speed concerns for both SEO and User Experience?

23 thoughts on “Site Speed: Reducing Page Load Times and Increasing Pages Crawled

  1. Reply Shane Jul 29, 2010 6:17 pm

    Site speed is probably a big factor, but it's not a direct correlation. Check out the stats on these two different sites:

  2. Pingback: Tweets that mention Site Speed: Reducing Page Load Times and Increasing Pages Crawled --

  3. Reply Cyrrus Jul 29, 2010 11:34 pm

    Indexing is just part of the game. Once you see increased SEO traffic, then you can celebrate ;)

  4. Reply Gary Chiro Jul 30, 2010 5:55 am

    Tony do you think 100/100 page speed is possible? I see that even can't mange it they get 92/100

    • Reply tonyadam Jul 30, 2010 6:08 am

      Gary, I don't think 100/100 is realistic, my goal is to get pages to under 1.5 second page load times….when i'm there, I'm happy :)

    • Reply Matthijs Aug 2, 2010 1:30 am

      Why wouldnt it? The question is, is 100 out of 100 important or is the actually loadtime important? Well from what I can say its the loadtime ;-)

      The score is just an indication and a guidance. It shows what you did and can do to improve the pagespeed on-site. Beside you can have a 100/100 score but if your server (due what so ever reason) is slow your pagespeed will still be slow.

  5. Reply Gareth Jul 31, 2010 9:03 pm

    Those graphs a pretty interesting, I'd like to see this info over a large data set.

    • Reply tonyadam Aug 2, 2010 4:32 am

      Gareth, some of these are over VERY large data sets…unless you mean different sites…i've seen identical results with 5 different sites, email me and let me know how i can help! :)

  6. Pingback: July ‘10: Best Search/Marketing Posts

  7. Reply Pieter Aug 1, 2010 7:40 am

    We've done great efforts in bringing down page load speeds and recently moved up from no 2 to no1 on the most important keyword in our market. Our website is way faster than our nearest competitor and former no 1, so it seems like Google really rewards good performance!

    • Reply tonyadam Aug 2, 2010 4:34 am

      Congrats Pieter on taking the #1 spot! I'm seeing the same thing in rankings as I am indexing/crawling…and it looks like you were the recipient of the same! :)

  8. Reply Sean - SEO Aug 1, 2010 11:53 pm

    Google is telling exactly what it has been following. We all know how good is Google in providing results with fast and reliable information. Also, Google many times stated that its objective is to provide the information with high relevancy and speed. Though it is a bit hard to dynamic websites, I think it is now the time to re-design/ alter at least their main product pages.

    • Reply tonyadam Aug 2, 2010 4:35 am

      I couldn't agree more Sean, I've been a big fan of quick page loads for years because of the user experience aspect…having it effect traffic will only make people look at it much closer, which makes me very happy!

  9. Pingback: July ‘10: Best Search/Marketing Posts | Google Seo Guide

  10. Pingback: Internet Marketing highlights for July 2010 | The Blog of

  11. Reply Pierce Aug 3, 2010 3:30 pm

    I don't know if this is a true correlation. From experience all I have seen is the more pages google downloads the lower the time spent downloading.

    I see the upper limit of those graphs have been blanked out, but if its similar to mine with 31,000 pages downloaded per day with an average of 343ms per page load.. in all honest thats not going to make much difference if its 600ms or 343ms but you still get those lovely up and down graphs.

    What I am trying to say is the more pages google downloads the better the average it seems to get.

    If you want to make a truly significant difference enable gzip on your pages. That way google wont mind downloading many more pages for the same amount of bandwidth ;)

  12. Reply rolfbroer Aug 3, 2010 7:56 pm

    Nice test! Always interesting to see the movements of the Googlebot.

  13. Reply evdenevenakliyat Dec 8, 2010 4:25 pm

    interesting analysis, visitor-friendly sites, do we understand this point of..Thankssssss

  14. Reply Vlad Jan 20, 2011 7:57 am

    Everyone has been saying that increasing the site's speed would increase the number of pages crawled, but I haven't seen much proof of that until I saw this article. Thanks for the demonstration Tony

Leave a Reply