by Daniel Sim
In this series we are looking at three methods of measuring your website performance: 1. Sampling your visitors 2. Analysis and recommendation tools 3. Time measurement. All are distinctly different ways of measuring the performance of your website, with advantages and disadvantages to each.
Sampling Your Visitors
Measures Things Outside Of Your Control
How long a page takes to load for a user is dependent on a lot of different factors.
It starts with the device the user is using: is it a slow phone or an ultra-fast gaming machine. Do they have a bunch of browser addons slowing down the loading. What about other software that's running.
Then there's the network they are on. Is it a shaky mobile network, home connection with everyone in the family streaming, or a big corporate connection.
Where they are in the world comes into play. Are they in a remote part of a distant country or right down the street from where your shop is served from. A network is made up of lots of different computers relaying messages from one to the next. Some of the hops from computer to computer can be slowed due to congestion.
Even when the user's request gets to the big server room where your shop is served from it can be temporarily slower or faster than usual if other shops are experiencing more or less traffic than normal.
As we can see, a lot of these factors are completely outside of our control. There's nothing we can do to speed it up if the user has a slow device, network and is in a remote part of the world.
This is something to be very aware of when using visitor sampling. However some tools do let you remove factors outside of your control from the measurements, focussing in on just the first meaningful paint measurement we recommend.
Can Be Erratic
Just like the road network, the internet is not a steady speed all of the time. A user with a fast experience today may have a slow loading time tomorrow. Similarly, if you measure 10 users today they may all be slow due to a factor like they come from a distant country. 10 users tomorrow may be much closer by and therefore load faster.
For websites hosted with a hosting company that just provides servers (vs. a hosted software company like Shopify or Bigcommerce) it may be that erratic performance can be because of other websites on the same server or problems with how your site is coded or server configured.
However, on Shopify you are much less likely to see big erratic peaks and troughs of speed. Shopify monitor their infrastructure constantly and consistently achieve a dependable network and server speed.
Insufficient Sample Sizes Give Misleading Results
Imagine you opened a restaurant. On the first day you welcomed 50 diners and you asked three of them what they thought of the food. Two of them said they didn't like your food. The other diner really enjoyed it. Does that mean 2/3 of your 50 diners didn't like your food? With just three opinions it is no way possible to say.
The same is true of sampling your shop visitors. Some tools select a certain percentage of your visitors to measure. If this sample is too small we cannot have confidence that their experience reflects the experience of most of your visitors. There's a neat sample size calculator here to give you an idea of the numbers involved.
The best option here is to (if available) change the sample size to 100% after checking any side effects in the tool.
Lag In Measuring Before And After Changes
If we make a change today to try to speed up our shop we cannot measure the effect on our visitors immediately by sampling them.
This is because of something related to sample sizes: statistical significance. If we sampled 1000 visitors' speed up until today and compared it to the 10 visitors we've had so far after the change, this is not statistically significant. That is, we cannot say that these 10 visitors experience is an indication that we've improved vs. the 1000 visitors in the days before and it is not just random chance or an uncontrollable factor.
Because of this we need to wait until we have gathered enough samples after the change to compare to the samples before the change. That may mean waiting a number of days, weeks or we may never reach a statistically significant result.
When It Should Not Be Used
Due to the reasons we've covered above, visitor sampling is not useful for measuring the changes before and after speed optimization work. Something more unaffected by uncontrollable factors, steady and immediate is required.
It should also not be used for shops with a small number of visitors since there just isn't enough data.
When It Is Useful
Visitor sampling can uncover interesting patterns that could be worth further investigation. Do your users in Japan experience a slower shop than North America? Why is it on the 14th of the month your shop slows down? Why do Firefox users take almost twice as long to load vs. Chrome?
Visitor speed sampling can be used as an interesting lead to hone in on factors which may or may not be under your control but which give your users a different speed experience while visiting the same shop.
It can also be used to track speed changes over longer periods of time like month to month. But here bear in mind the limitations detailed above.
Visitor speed sampling can be an interesting tool to see the speeds that users are experiencing your shop. It shouldn't be used as the primary method of deciding if your shop needs its speed optimized or not, but it can help you to get a holistic picture of all of the factors in and out of your control affecting your shop speed.