Why speed tests give different results

TL;DR

There are several reasons why speed tests give different results. These are:

1. The Internet Protocol is an unreliable protocol. As a result, it can never be guaranteed that the test conditions are exactly the same
2. The speed test used may or may not provide a subsequent calculation of the results obtained
3. The speed test is "optimized" for a specific speed. Practice shows that the most accurate speed test varies depending on the speed to be measured
4. The amount of data used for testing varies per speed test

The internet protocol is an unreliable protocol

When you test your internet speed with different speed tests, you will get different results even if you have the same test conditions.

Unfortunately, in practice you cannot speak of "Same test conditions". A speed test measures the speed of an inherently unreliable network.

The Internet protocol (IP), on which all Internet traffic is based, is designed as an unreliable protocol. Data corruption, packet loss, and duplication might occur.

The Internet protocol is designed to choose "the best" available path for each packet, based on the destination IP address.

Note that "the best" is not tightly defined, because Internet traffic uses dynamic routing.

It is therefore normal that if you run multiple speed tests you will get different results even it you have the same test conditions.

Subsequent calculation

The standard explanation for this is that each speed test is programmed in a unique way. However that is not the main reason.

Of course, the operation of any speed test is basically the same: divide the number of downloaded megabits by the number of seconds and you have the download speed in megabits per second (Mbps).

The main reason for the large differences are the assumptions about the reliability of the intrinsically unreliable internet protocol made by the development teams of the speed tests.

It turns out that speed tests can be divided into realistic, mainstream, and conservative categories. In addition to these main categories, two more categories (ambiguous and miscellaneous) were added later.

In short:

1. A realistic speed test gives an indication of the actual (maximal) download speed provided by your ISP. There is probably no subsequent calculation. Where the actual download speed is approx 12% higher than the advertised speed;
2. An ambiguous speed test gives results between realistic and mainstream speed tests;
3. A mainstream speed test gives gives an indication of the advertised download speed. Ookla's Speedtest -the gold standard-, adviced by many ISP's shows the advertised download speed. The following subsequent calculation is performed for this purpose: Displayed speed = 90% of the measured speed;
4. A conservative speed test gives an indication lower than the advertised download speed;
5. Miscellaneous speed tests doesn't match criteria mentioned in which speed tests to test.

The way of programming does not seem to be the main reason for the observed differences. The objectives of the companies that do speed tests seem to explain the estimate made of the reliability of the internet protocol and thus the large differences that exist in the measurement results.

This is most evident by looking at the possible business objectives per main category.

Speed tests by category

The speed tests belong to a certain category as follows:

Realistic speed tests

Creators of realistic speed tests probably try to measure the actual speed test as good as possible.

Customer retention

Some of these realistic speed tests are sold to or used by ISP's. Examples are:

1. Comparitech
2. nPerf
3. Astound
4. SpeedOf.me
5. Xfinity Speed Test

As ISP you want to keep your customers. One way to do that is by displaying the actual internet speed as it is 10-12% higher than the advertised speed.

Mainstream speed tests

The Ookla Speedtest is the gold standard. The majority of ISP's advice their customers to use an Ookla Speedtest or an Ookla based speed test. Hence it is logical that a lot of speed test vendors create a speed test similar to the Ookla Speedtest.

Note

The business objectives range from "Inform the public" to "Selling speed tests".

1. Inform the public (Bredbandskollen, Measurement Lab)
2. Collecting and selling speed test data (Broadband Speed Checker)
3. Showcase (Cloudflare, Google Fiber, SamKnows)
4. Selling speed tests (Ookla, Fireprobe)
5. Attract visitors interested in a new ISP broadband contract (DSLReports)

Conservative speed tests

Some conservative speed tests are owned by ISP resellers. A low speed test can be a trigger for a customer to switch to another ISP. Examples of ISP resellers are:

1. Bandwidth Place
2. Which Broadband Checker

Fast

Fast is a selfservice tool for Nexflix users to detect if they have internet problems. When you as user conclude that your internet speed is low, you will not bother the Netflix helpdesk when you have problems watching Netflix.

The speed to be measured is an important factor

Based on hunderds of tests we have learned that the most accurate speed test varies depending on the speed to be measured.

The amount of data used

As mentioned earlier, each speed test is programmed in a unique way. In particular, the amount of data used for testing differs per speed test.

Strangely enough (aside from Internet Speed at a Glance) there seems to be an inverse correlation between the accuracy of the speed test and the amount of data used. Google Fiber, Measurement Lab and TestMy.net are very accurate speed tests but use relatively little data for a test.

Which speed test to use?

In general the advice is to use the interactive decision tree to answer the question which speed test to use.