The Regression Detector, owned by Single Machine Performance, is a tool that detects if there are more-than-random performance changes to a target program -- here, the Agent -- across a variety of experiments and goals. This directory contains the experiments for Agent. A similar one exists in Vector. Please do add your own experiments, instructions below. If you have any questions do contact #single-machine-performance; we'll be glad to help.

Experiments prefixed with quality_gate_ represent the strongest claims made about the Agent and its performance. These are discussed in more detail on this page

In order for SMP's tooling to properly read a experiment directory please adhere to the following structure. Starting at the root:

• config.yaml -- Required Configuration that applies to all experiments.
• cases/ -- Required The directory that contains each experiment. Each sub-directory is a separate experiment and the name of the directory is the name of the experiment, for instance tcp_syslog_to_blackhole. We call these sub-directories 'cases'.

The structure of each case is as follows:

• lading/lading.yaml -- Required The lading configuration inside its own directory.

• datadog-agent/ -- Required This is the configuration directory of your program. Will be mounted read-only in the container build from Dockerfile above at /etc/datadog-agent.

• experiment.yaml -- Required Set any experiment-specific configuration. The "optimization goal" determines what metric the Regression Detector will analyze at the conclusion of the experiment.

  Eg:

  optimization_goal: ingress_throughput

  Supported values of optimization_goal are ingress_throughput and egress_throughput.

In order to run a regression experiment locally, you need two CLI utilities available:

• smp -- build from source repo
• lading -- See the notes in the below documentation about architecture, lading needs to be compatible with the architecture of the image being run.

See full docs here

An example command may look like this:

smp local-run --experiment-dir ~/dev/datadog-agent/test/regression/ --case uds_dogstatsd_to_api --target-image datadog/agent-dev:nightly-main-fe13dead-py3