PageSpeed, Pingdom, WebPageTest, GTmetrix – Which one is the best for measuring web performance?

If you’ve used any of these tools to measure web and WPO performance, you may wonder why the results are sometimes different.

I hope this article serves to highlight the key differences in these performance analysis tools, and that is that PageSpeed Insights, Pingdom Tools, WebPageTest, and GTmetrix all offer similar features, but there are a few things I think you need to know about their differences.

Test locations

Where the test is performed from affects the performance results.

Different distances in test locations will cause factors such as latency and quality of network connection to influence page performance.

In fact, this is the main reason why content delivery networks (CDNs) are a crucial aspect in serving a fast web.

In addition, geo-specific content can be activated in several regions due to third-party resources or advertisements.

Here are the test regions that each web tool offers:

Summary

The test site closest to your target audience provides the most accurate representation of your page load…

• GTmetrix, Pingdom Tools and WebPagetest offer multiple locations for analysis in order to best represent the performance of your site, both in terms of actual load time and best performance practices.
• WebPagetest is able to offer as many test locations because they allow anyone to host a test location for them.
• Google PageSpeed Insights does not give you the option to choose where to test from; because it does not measure how fast your page loads, only if your site follows a set of rules.
• Naturally, the test location closest to your target audience provides the most accurate representation of your page’s load, as experienced by your visitors.

Choose the tool with the test location closest to your audience.

Ratings and recommendations

Recommendations will differ between tools.

Each of these tools evaluates the websites according to its own set of recommendations.

Most of them come from Google’s original open source PageSpeed library, and have probably been customized or modified.

GTmetrix uses a set of modified PageSpeed and YSlow recommendations to evaluate sites based on what they consider to be important metrics.

This is what each tool uses to determine its score:

Summary

While they are all likely to be based on the original open source Google PageSpeed library, there are some key differences.

After Google made general changes to the scoring algorithm and recommendations years ago, they have not updated the open source library since then.

The online tool PageSpeed Insights as a whole seems to contain a new set of rules (which is not open source).

Don’t compare these tools alike and ask yourself why your scores differ from one tool to another; all of them use different sets of recommendations.

That’s why you’ll see that the rules of each service are different.

Downtime Test

When each tool decides that an analysis is complete it will affect its final report.

This is an incredibly important distinction between these tools.

In general terms, a page analysis can be determined to be complete at two different points, each with its own characteristics:

Loading time

When page processing is complete and all page resources (images, CSS, etc.) have finished downloading.

The browser will activate window.onload when this happens.

Problems with the use of this event:

Some elements of the page load may not arrive before this event is triggered – such as JavaScript based image carousels – causing inconsistent page load times and inaccurate screenshots.
You may also report faster-than-real page load times.

Full load time

The point after the Onload event is triggered and there has been no network activity for 2 seconds. This ensures more consistency with testing.

Potential problems with using this event: This event is triggered only when a page stops loading content completely, including advertisements and underneath folding elements.

Your site could have loaded quickly above the fold and be usable, however, since the analysis is now waiting for the entire site to stop loading data, the reported page load time could be longer.

This is when each tool decides to stop its analysis:

GTmetrix and WebPagetest allow you to switch between full load time and loading time.

Pingdom only has the default loading time. This is why you may see the results of Pingdom loading faster than GTmetrix and/or WebPagetest.

PageSpeed Insights uses the first pint of content and the content of the loaded DOM (FCP and DCL) to determine the loading time of your page, but this is based on the Chrome user experience report, which collects FCP and DCL data from real Chrome users around the world who visit your page.

Depending on the FCP and DCL median, your page will be marked as fast, slow, or average.

Because this service requires a sufficient amount of FCP and DCL data to generate an average score, your page must have good traffic, which means that pages with low traffic have no time value.

Summary

Due to network variations or the way the page was designed to load (e.g. asynchronous loading), resources loaded after the browser triggered window.onload may not reach the reports.

The result is a report that indicates that the page loaded faster than it actually did.

This means that while the tool received a “page finished loading” trigger at the time of loading, the actual user is still experiencing resources being downloaded.

While interesting, Chrome’s user experience reports in PageSpeed Insights are limited at the moment.

Expect to see empty values if your page doesn’t have a large number of Chrome visitors.

Still, the use of first content painting as a time metric is questionable.

Real browsers VS. Headless/emulator browsers – How do you actually load your page?

The accuracy of the page performance evaluation may be affected by the method in which the page was loaded.

Real browsers and headless/emultiple browsers are used in the field of web performance testing tools, but they each load websites in different ways.

Real browsers

The tool opens a real instance of a browser (Firefox, Chrome, Internet Explorer, etc.), enters its URL and captures the resulting load data.

This is the closest you’ll get to a real human visiting your URL, as it uses the real desktop browser applications to perform the URL analysis.

Headless/Emulated Browsers

The tool enters its URL into a browser script to load your website, capturing the resulting loading data.

There is no user interface, which makes it lighter and faster in its operation.

The most popular headless browsers are PhantomJS and SlimerJS.

Here is the type of browser that each tool uses to analyze its URL:

GTmetrix, WebPagetest and Pingdom use real browsers and devices in their performance tests.

This means that the results captured are representative of what a real user would see on their end using the same browser/device combination.

Other services may instead use headless/emulator browsers, which could slightly alter the loading time or behaviour of a web page.

Summary

• Real browsers provide a better indication of your website’s performance.
• Most headless/emultiple browsers are based on engines such as Webkit or Gecko.
• Because these engines are complex, they are often outdated, and therefore cannot run full or updated functionality that their real-world browser counterparts can.
• Things like HTTP/2 or Flash may not work or be supported, which means your page load may not be fully reflected.

Other Features

Along with the key differences listed above, there are other features that distinguish each service from the others.

Here are some key features that each service has:

Here is a breakdown of the main features and why they are important:

Connection Throttle

Naturally, the speed at which the user connects to your website determines the upload speed.

Both GTmetrix and WebPagetest offer the possibility to speed up connections to simulate the different types of Internet connections your users may have.

Pingdom Tools does not have any acceleration function.

HTTP/2 support

HTTP/2 is an improved version of HTTP/1.1 that attempts to correct some of the major drawbacks of the original protocol.

Generally speaking, if implemented and used correctly, a web page would load faster over HTTP/2 than over normal HTTP/1.1.

Some of these improvements are:

• The ability to allow multiple requests over the same connection (unlike HTTP/1.1, which almost always requires a per-request connection)
• A more “computer-friendly” message protocol
• Additional techniques to reduce server load

To experience this performance improvement, both the client and the server must have HTTP/2 support, otherwise normal HTTP/1.1 will be used by default.

HTTP/2 support is continually changing, with W3Techs reporting that about 15% of the top 10 million websites worldwide support HTTP/2 as of July 2017.

Both GTmetrix and WebPagetest support HTTP/2. Pingdom Tools and PageSpeed Insights do not.

Screen resolution

The size of the browser in which the page loads also affects its performance, as different sizes can offer different resources.

The adaptive (responsive) design has made this especially evident.

GTmetrix offers the possibility to change the resolution of your screen for tests, as well as to simulate devices (more than 40 different variants of phones and tablets).

The Pingdom/WebPageTest/PageSpeed Insights tests are set at 1024×878 and 1024×696 respectively.

Hardware Provisioning

The performance of the device hardware itself is also a huge factor when it comes to quickly loading websites.

The functionality of JavaScript or sophisticated CSS animations can vary in softness or responsiveness depending on how powerful the hardware is.

WebPagetest servers can be run by anyone who wants to provide a test site.

There are minimum system requirements, however it can be assumed that not all servers have the same hardware, and therefore not the same performance.

Conclusions

GTmetrix, Pingdom Tools, WebPagetest and PageSpeed Insights provide web performance analysis using their own testing methodologies and configurations.

It is important to understand how each tool works before relying on them for any type of data or results.

Consistency and the ability to choose the metrics you are interested in for website performance analysis are key when it comes to testing and benchmarking.

Summary of the main conclusions

• The test location closest to your target audience provides the most accurate representation of your page load.
• Don’t compare these tools in the same way and ask yourself why your results differ from one tool to another.
• The load time can distort the true load time of your page.
• Real browsers provide a better indication of your website’s performance.
• It is important to understand how each tool works before relying on them for any type of data or results.

So, what tool should I use?

It depends.

I think the use cases for each tool can be divided into the following categories:

• Quick checks: GTmetrix, Pingdom Tools, and PageSpeed Insights
• Consistency/Historical Tracking: GTmetrix and WebPagetest
• In-depth analysis: GTmetrix and WebPagetest
• SEO Testing: PageSpeed Insights, GTmetrix, and Pingdom Tools
• Mobile devices: WebPagetest, GTmetrix and PageSpeed Insights
• Location dependency: WebPagetest, GTmetrix and Pingdom Tools
• Advanced options: WebPagetest and GTmetrix

But, above all, always use the same one, learn from its metrics and recommendations to improve the performance parameters of your website, and don’t even think about comparing them, you’ll go crazy, and if you don’t, here you have captures of the same website, this blog, in the different tools…

Which one do you use?

As a general rule we use GTMetrix because it seems to me to be the most complete and has a good balance between consistency in measurements and detail of recommendations, which in the end is what matters, that it offers you analysis tools that help you improve performance and optimize your web pages.

Read this post in Spanish: PageSpeed, Pingdom, WebPageTest, GTmetrix ¿cuál es mejor para medir el rendimiento web?