[General] Baseline vs SLA Parameters

Intro

One of the main features of the timings API is that it compares measured performance against a historical baseline. The API parameters that are most often discussed/confused/misunderstood are related to this feature. This page will explain what these parameters are and how you can use them. Overview of the parameters:

sla:
  {metric}: {int}
baseline:
  days: {int}
  perc: {int}
  padding: {float}
flags:
  assertBaseline: {boolean}
  passOnFailedAssert: {boolean}

How the API processes performance data

Every time you run a test for a particular URL, the timings client will collect performance data from the browser and send it to the API. The API will then:

• Determine the measured value (pageLoadTime or visualCompleteTime)
• Calculate the baseline (for example: the 75th percentile of the last 7 days)
• Determine the threshold by taking the lower of these two values:
  • baseline * padding or
  • the SLA value of the metric
• Set the assert field
  • false if the measured value is higher than the threshold
  • true if the measured value is within the threshold
• Send back the response message that includes the assert field

Example

Below is an example of the baseline at work!

This chart shows the actual performance (green), the baseline (purple), and the threshold (blue) over time. The measured values are between ~300ms and ~1000ms which results in a baseline of ~600ms and a threshold of ~900ms (includes padding). You can see five test runs where the measured value exceeded the threshold.