Why our tables say “Not measured”

Every case study on this site ends with a table. Some rows hold a number. Some say “Not measured”. We would like the second kind to be read as information, which it is.

The rule behind the table is short. A number is printed only with the date it was taken and the instrument that took it. If either is missing, the site does not build.

Two kinds of number

Speed is measured in two ways, and they answer different questions.

A lab measurement is one machine loading the page under fixed conditions. It says what the page can do. A field measurement is what the browsers of real visitors recorded. It says what happened. Google’s PageSpeed Insights describes the difference like this:

The lab data is based on a simulated load of a page on a single device and fixed set of network conditions. As a result, the values may differ.

Core Web Vitals, the three measurements Google’s ranking systems use, are judged in the field. The target is “the 75th percentile of page loads, segmented across mobile and desktop devices.” A perfect score in a lab tool is therefore not a Core Web Vitals result. When we print a lab number, we call it one.

What can be measured on the first day

A lab number needs no visitors. How many requests a page makes, how many bytes it sends, what Lighthouse scores it, how many accessibility faults an automated check finds: all of that can be taken before a site is live.

Two cautions come with it, both from the people who make the tools.

The first is that one run is not a measurement. Lighthouse’s own documentation:

Lighthouse performance scores will change due to inherent variability in web and network technologies, even if there hasn’t been a code change.

It adds: “The median Lighthouse score of 5 runs is twice as stable as 1 run.” It also warns against collecting several reports at the same time on one machine. So we run it five times, one after another, and print the middle result.

The second is that a lab run cannot see everything. Interaction to Next Paint measures how quickly a page answers a tap or a key. A run that only loads the page has nobody tapping. Google:

Tools like Lighthouse that load pages in a simulated environment without a user cannot measure INP, as there is no user input.

Such a run reports Total Blocking Time in its place, which the same page calls “a proxy for INP”. A proxy is printed as a proxy.

What needs visitors

Field data comes from the Chrome User Experience Report. A site is included when it is, in Chrome’s terms, “publicly discoverable” and “sufficiently popular”. How popular is not stated: “An exact number is not disclosed.” The figures cover “the previous 28-day collection period”.

A site that went live last week has neither the visitors nor the 28 days. PageSpeed Insights says what happens then:

Sometimes the origin may also have insufficient data, in which case PSI will be unable to show any real-user experience data.

That is where this site stands today. The honest entry is “Not measured”, with the reason beside it.

What only the owner can see

How a site does in search is recorded in Search Console: which searches showed it, how often it was clicked, its average position. That record belongs to the owner of the site. Google’s help states the rule from the owner’s side: “A property must have at least one verified owner, or no users will have access to the property.”

We can read it for a site only when its owner adds us. For any other site, including one we built, we cannot. Tools that offer a figure anyway are estimating, and Google is plain about what they have to go on: “Third-party tools don’t have access to our internal ranking data.”

So a case study of ours will say “Not measured” under search visibility until the owner has shown us the record, and will say whose record it is when they have.

A zero that meant nothing

An instrument can also be wrong in your favour. It happened to us while writing another article.

We were measuring how far a page moves when its fonts arrive late, with and without a fix. One run printed 0.0000 for the page without the fix: a perfect result for the version we knew was worse.

Nothing had gone right. The machine was busy, the page was drawn late, and by then the fonts had already arrived. There was no swap to measure. The script reported zero because it had nothing to report.

The script now checks that the text was on screen before the first font arrived. A load that fails that check is repeated, and if it keeps failing, the answer is “no reading” and not a number.

The rule, in full

• A value carries its date, its instrument and its conditions.
• A lab number is called a lab number.
• What could not be measured is written down as not measured, with the reason.
• A reading that could not have failed is not trusted.

A table with gaps in it looks weaker than a full one. We think it is the stronger of the two, because the other rows can then be believed.