Beginner’s Guide to Load Testing with k6 — Part 2

6 minute read

Performance testing is an umbrella term for a group of tests that encompasses many types of tests, as discussed in the first part of this series. Each test type tries to answer a set of questions to make the performance testing process more goal-oriented. This means that just running tests is not enough, you have to have a set of goals to reach.

Since we’re testing an API or a website, the following goals may be relevant, in which you can choose either one or more:

  • Concurrency: systems which would set this goal, usually have a concept of end-user and need to see how the system behaves while many concurrent users try to access the system. They basically want to test how many of requests fail/pass under high loads of users. This both includes many concurrent users and each requesting multiple resources at the same time.
  • Throughput: systems with no concept of end-users, would set this goal to see how the system behaves overall, while there is a ton of requests/responses coming in/out of the system.
  • Server response time: this goal signifies the time it takes from the initial request from the client to the server up until a response is sent back from the server.
  • Regression testing: sometimes the goal is not to put “heavy load” on the system, but is more about “normal load” and functional and regression testing to see how a change would affect our system’s performance and if it still adheres to our defined SLAs. The general idea is to measure how a system or system of systems behave(s) under heavy load, in terms of speed, scalability, stability and resiliency. Each of which can be measured by these goals.
  1. Speed can be measured by time it takes for request to be handled by the server and how much time it takes for this request/response to happen.
  2. Scalability can be measured by how well the system scales if the load is increased and by measuring if it sustains over a period of time under this load.
  3. Stability can be measured by how well the system sustains the load and to see if it stands against a high number of errors and events and still stays responsive and stable.
  4. Resiliency can be measured by how the system recovers from crashes and down-times and responds to requests, after putting too much or too frequent load on it and eventually crashing the system.

Rerunning Tests to Verify the Results

You can rerun the tests to see if they hold almost the same results during different tests and compare the tests to see if they deviate.

If they are almost the same, you can analyze the tests and derive your results, otherwise you should pinpoint where it deviates and try to find a way to prevent it from happening, like a bottleneck.

k6 and the Metrics

k6 supports a set of built-in and custom metrics that can be used to measure various things and to either achieve the above mentioned goals or prove them wrong. The metrics that can be used to define custom metrics are: Counter, Gauge, Rate and Trend.

k6 built-in metrics
k6 built-in metrics

As you’ve probably seen above, these following tables describes reported built-in metrics, present on all tests:

metric type description
vus Gauge Current number of active virtual users
vus_max Gauge Max possible number of virtual users (VU resources are preallocated to ensure performance will not be affected when scaling up the load level)
iterations Counter The aggregate number of times the VUs in the test have executed the JS script (the default function). Or in case the test is not using a JS script but accessing a single URL the number of times the VUs have requested that URL
data_received Counter The amount of received data
data_sent Counter The amount of data sent
checks Rate Number of failed checks.

Credits: k6 built-in metrics

metric type description datatype
http_reqs Counter How many HTTP requests has k6 generated in total integer
http_req_blocked Trend Time spent blocked (waiting for a free TCP connection slot) before initiating request float
http_req_looking_up Trend Time spent looking up remote host name in DNS float
http_req_connecting Trend Time spent establishing TCP connection to remote host float
http_req_tls_handshaking Trend Time spent handshaking TLS session with remote host float
http_req_sending Trend Time spent sending data to remote host float
http_req_waiting Trend Time spent waiting for response from remote host (a.k.a. time to first byte or TTFB) float
http_req_receiving Trend Time spent receiving response data from remote host float
http_req_duration Trend Total time for the request. It’s equal to http_req_sending + http_req_waiting + http_req_receiving (i.e. how long did the remote server take to process the request and respond (without the initial DNS lookup/connection times) float

Credits: k6 HTTP-specific built-in metrics

Custom (non-built-in) Metrics

1. Counter

This is a simple cumulative counter that can be used to measure any cumulative value like number of errors during the test.

import { Counter } from "k6/metrics";
import http from "k6/http";

var myErrorCounter = new Counter("my_error_counter");

export default function() {
  let res = http.get("https://test.loadimpact.com/404");
  if(res.status === 404) {
    myErrorCounter.add(1)
  }
};

k6 Counter metric

As you can see in the above example, it counts the number of 404 errors that are returned by the test. The result is evident in the screenshot below:

Results of k6 Counter metric
Results of k6 Counter metric

Since it is a beginner’s guide, I try to stick with simple examples, but you can extend and customize them to your specific case.

2. Gauge

This metric lets you keep the last thing that is added to it. It’s a simple over-writable metric that holds its last added value.

This metric can be used to retain the last value of any test item, be it response time, delay or any other user-defined value.

If you run the following code, you’ll see that it catches the latest error code, which is 404.

import { Gauge } from "k6/metrics";
import http from "k6/http";

var myGauge = new Gauge("my_gauge");

export default function() {
  let res = http.get("https://test.loadimpact.com/404");
  myGauge.add(res.status);
};

k6 Gauge metric

The result of the test is presented in the screenshot below.

Results of k6 Gauge metric
Results of k6 Gauge metric

3. Rate

This built-in metric keeps the rate between non-zero and zero/false values. For example if you add two false and one true value, the percentage becomes 33%.

It can be used to keep track of the rate of successful request/responses and compare them with errors.

In the following piece of code, you can see that I added res.error_code as a measure to see how many errors I’ll catch.

import { Rate } from "k6/metrics";
import http from "k6/http";

var myRate = new Rate("my_rate");

export default function() {
  let res = http.get("https://test.loadimpact.com/404");
  myRate.add(res.error_code);
};

k6 Rate metric

Below is the result of the test, which is 100% errors.

Results of k6 Rate metric
Results of k6 Rate metric

4. Trend

This metric allows you to statistically calculate your custom value. It will give you minimum, maximum, average and percentiles, as is evident in the above screenshots for http_req* requests.

import { Trend } from "k6/metrics";
import http from "k6/http";

var myTrend = new Trend("my_trend");

export default function() {
  let res = http.get("https://test.loadimpact.com/");
  myTrend.add(res.timings.sending + res.timings.receiving);
};

k6 Trend metric

The above example of trend metric shows how to calculate the sending and receiving time without taking into account the waiting time. The result is shown below:

Results of k6 Trend metric
Results of k6 Trend metric

In this part, I’ve tried to describe the goals of performance testing and how one can use metrics to achieve those goals. In the next sections, I’ll try to go more in-depth and present you more details on how to define custom metrics and how to use them.

Now that you have a good grasp of performance goals and k6 metrics, you can move to the next article, in which I try to show you how to write and run a k6 script.

Updated:

Comments