Why speed tests give different results

When you test your internet speed with different speed tests, you will get different results even if you have the same test conditions.

Unfortunately, in practice you cannot speak of "Same test conditions". A speed test measures the speed of an inherently unreliable network.

The Internet protocol (IP), on which all Internet traffic is based, is designed as an unreliable protocol. Data corruption, packet loss, and duplication might occur.

TCP makes the internet reliable

The reliability of data traffic over the internet is due to the Transmission Control Protocol (TCP). That is why the term TCP/IP is often used.

The Internet protocol is designed to choose "the best" available path for each packet, based on the destination IP address.

Note that "the best" is not tightly defined, because Internet traffic uses dynamic routing.

It is therefore normal that if you run multiple speed tests you will get different results even it you have the same test conditions.

Differences in speed tests

The standard explanation for this is that each speed test is programmed in a unique way. However that is not the main reason.

Of course, the operation of any speed test is basically the same: divide the number of downloaded megabits by the number of seconds and you have the download speed in megabits per second (Mbps).

The main reason for the large differences are the assumptions about the reliability of the intrinsically unreliable internet protocol.

It seems that ISPs assume that their customers will experience a 10% reduction in Internet speed as offered by the ISP due to things like data corruption, packet loss and duplication.

It turns out that speed tests can be divided into realistic, mainstream and conservative categories.

In short:

  1. A realistic speed test gives an indication of the actual (maximal) download speed provided by your ISP (This actual download speed is approx 12% higher than the advertised speed);
  2. A mainstream speed test gives gives an indication of the advertised download speed. Ookla's Speedtest -the gold standard-, adviced by many ISP's shows the advertised download speed;
  3. A conservative speed test gives an indication lower than the advertised download speed.

The way of programming does not seem to be the main reason for the observed differences. The objectives of the companies that do speed tests seem to explain the estimate made of the reliability of the internet protocol and thus the large differences that exist in the measurement results.

This is most evident by looking at the possible business objectives per category.

Speed tests by category

The speed tests belong to a certain category as follows: A venn diagram of the speed test categories

Realistic speed tests

Creators of realistic speed tests probably try to measure the actual speed test as good as possible.

Customer retention

Some of these realistic speed tests are sold to or used by ISP's. Examples are:

  1. CompareSpeedTest
  2. nPerf
  3. RCN
  4. SpeedOf.me
  5. Xfinity Speed Test

As ISP you want to keep your customers. One way to do that is by displaying the actual internet speed as it is 10-12% higher than the advertised speed.

Mainstream speed tests

The Ookla Speedtest is the gold standard. The majority of ISP's advice their customers to use an Ookla Speedtest or an Ookla based speed test. Hence it is logical that a lot of speed test vendors create a speed test similar to the Ookla Speedtest.

The business objectives range from "Inform the public" to "Selling speed tests".

  1. Inform the public (Bredbandskollen, Measurement Lab)
  2. Collecting and selling speed test data (Broadband Speed Checker)
  3. Showcase (Cloudflare, Google Fiber, SamKnows)
  4. Selling speed tests (Ookla, Fireprobe)
  5. Attract visitors interested in a new ISP broadband contract (DSLReports)

Conservative speed tests

Some conservative speed tests are owned by ISP resellers. A low speed test can be a trigger for a customer to switch to another ISP. Examples of ISP resellers are:

  1. Bandwidth Place
  2. Which Broadband Checker

Fast

Fast is a selfservice tool for Nexflix users to detect if they have internet problems. When you as user conclude that your internet speed is low, you will not bother the Netflix helpdesk when you have problems watching Netflix.

Which speed test to use?

Which speed test to use depends on what you want. The best internet speed test or the most accurate speed test.

The actual download speed is best measured with SpeedOf.me (use the maximum download speed).

The speed test of Cloudflare is closest to the advertised download speed.

Overall the best speed test is Google Fiber.