How Do Real User Monitoring (RUM) Tests Work?

RUM tests

Introduction to RUM Tests

RUM tests are a crucial monitoring tool for performance-oriented content delivery companies. While they are not exclusive to CDNs, they are important in measuring real-world performance to/from a variety of destinations: whether it is a website’s load time, or latency to an edge node, RUM tests (or Real User Monitoring) measure application performance for both analytics and status monitoring (e.g. seeing degraded performance with users in a specific region due to, say, a fibre cut).

The Difference Between Real User Monitoring (RUM) and Synthetic Tests

RUM tests are fundamentally different from synthetic tests. The latter generally requires a set of servers, owned and run by a single entity. In other words, synthetic testing can measure performance almost instantaneously, but they typically don’t represent a real-world experience (which RUM tests show).

Unfortunately, even with all the listed benefits of RUM testing, the information is received as users view a page, rather than instantaneously. Also, there are many factors that cannot be measured that can affect a RUM test: the mode of which a user has chosen to connect to the Internet (Wi-Fi, a LTE hotspot, etc), device (an old Android phone will load a page slower than a modern iPhone), and more. Essentially, RUM tests should be taken with a “grain of salt.”

More Examples of Real User Monitoring (RUM) Testing

• Latency to other servers (using embedded JS), through ping
• Application performance (whether or not, say, an update has improved or made performance to a tool worse)
• Issues with a page’s programming (measuring if a user has to wait, ex. 10 seconds before they see any semblance of a polished page)
• How a user experiences and uses an application (this is typically done to improve UI/UX; positioning of certain buttons or fixing hard-to-read links)

“A Grain of Salt”

As hinted previously, there is a simple reason why RUM tests shouldn’t be an exclusive form of testing: the tests can be notoriously unreliable due to a variety of factors on a user’s device:

An example of a RUM test can be something simple; here is one written in JS (with jQuery):

$(document).ready(function() {
  var startTime = (new Date()).getTime();
    $(".dummy").load("pageToMonitor/", function(resp, status, xhr){
    if(status == "success") {
  	var loadTime = (new Date()).getTime() - startTime;
	// Send loadTime to server and capture page speed
    } else if (status == "error") {
  	console.log("An error occurred: " + xhr.statusText);
    }
  });
});

(The code above waits for a page to fully load, then sends a test request to pageToMonitor/ and forwards the result back)

Having said that, you can see pretty quickly why this testing falls apart in certain situations:

• User has a poor connection (skewing page load times, on average, for other users with good connections)
• Slow device

Furthermore, the testing data is generated completely on a user’s device! Simply put: while RUM tests are simple conceptually, no data can be fully trusted when analysing results.

Conclusion

RUM testing is an incredibly valuable tool for monitoring users’ experiences. In the case of a CDN, such as bunny.net, RUM monitoring (even when provided by 3rd party services) offers valuable insight to real world changes in network performance compared to synthetic testing servers located in a well-connected datacentre.