DEV Community: Oscar Reyes

Test Observability for AWS Lambda with Grafana Tempo and OpenTelemetry Layers

Oscar Reyes — Wed, 26 Jun 2024 20:18:38 +0000

I got great feedback from my Pulitzer award-winning blog post, "Testing AWS Lambda & Serverless with OpenTelemetry". The community wanted a guide on using the official OpenTelemetry Lambda layers instead of a custom TypeScript wrapper. 😄

I decided to write this follow-up but to spice it up a little 🥵. Today I’m using Grafana Cloud, which has become one of my favorite tools! We use it extensively at Tracetest for our internal tracing, metrics, profiling, and overall observability.

See the full code for the example app you’ll build in the GitHub repo, here.

OpenTelemetry Lambda Layers

With a decade of development experience, one thing I’ve learned is that no-code solutions help save time and delegate maintenance and implementation to a third party. It becomes even better when it's free 🤑 and from the OpenTelemetry community!

There are two different layers we will use today:

The Node.js auto-instrumentation for AWS Lambda enables tracing for your functions without writing a single line of code, as described in the official OpenTelemetry docs, here and on GitHub, here.
The OpenTelemetry collector AWS Lambda layer enables the setup to be 100% serverless without any need to maintain infrastructure yourself. You still need to pay for it though 👀.

Grafana Cloud

Grafana Cloud has become a staple tool to store everything related to observability under one umbrella. It allows integration with different tools like Prometheus for metrics or Loki for logs.

In this case, I’ll use Tempo, a well-known tracing backend where you store the OpenTelemetry spans generated by the Lambda functions.

Trace-based testing everywhere and for everyone!

Trace-based testing involves running validations against the telemetry data generated by the distributed system’s instrumented services.

Tracetest, as an observability-enabled testing tool for Cloud Native architectures, leverages these distributed traces as part of testing, providing better visibility and testability to run trace-based tests.

The Service under Test

Who said Pokemon? We truly love them at Tracetest, so today we have a new way of playing with the PokeAPI!

Using the Serverless Framework, I’ll guide you through implementing a Lambda function that sends a request to the PokeAPI to grab Pokemon data by id, to then store it in a DynamoDB table.

Nothing fancy, but this will be enough to demonstrate how powerful instrumenting your Serverless functions and adding trace-based testing on top can be! 💥

Requirements

Tracetest Account

Sign up to app.tracetest.io or follow the get started docs.
Create an environment.
Select Application is publicly accessible to get access to the environment's Tracetest Cloud Agent endpoint.
Select Tempo as the tracing backend.
Fill in the details of your Grafana Cloud Tempo instance by using the HTTP integration. Check out the tracing backend resource definition, here.
Test the connection and save it to finish the process.

AWS

Have access to an AWS Account.
Install and configure the AWS CLI.
Use a role that is allowed to provision the required resources.

What are the steps to run it myself?

If you want to jump straight ahead to run this example yourself ⭐️.

First, clone the Tracetest repo.

git clone https://github.com/kubeshop/tracetest.git
cd examples/quick-start-serverless-layers

Then, follow the instructions to run the deployment and the trace-based tests:

Copy the .env.template file to .env.
Fill the TRACETEST_API_TOKEN value with the one generated for your Tracetest environment.
Set the Tracetest tracing backend to Tempo. Fill in the details of your Grafana Cloud Tempo instance by using the HTTP integration including headers looking like authorization: Basic <base 64 encoded>. It should be encoded base64 with the format of username:token. Follow this guide to learn how. And, check out this tracing backend resource definition. You can apply it with the Tracetest CLI like this tracetest apply datastore -f ./tracetest-tracing-backend.yaml.
Fill the authorization header in the collector.yaml file from your Grafana Tempo Setup. It should be encoded base64 with the format of username:token. Follow this guide to learn how.
Run npm i.
Run the Serverless Framework deployment with npm run deploy. Use the API Gateway endpoint from the output in your test below.
Run the trace-based tests with npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com.

Now, let’s dive into the nitty-gritty details. 🤓

The Observability Setup

Instrumenting a Lambda function is easier than ever, depending on your AWS region, add the ARN of the OpenTelemetry Collector and the Node.js tracer.

# serverless.yaml
functions:
  api:
      # Handler and events definition
    handler: src/handler.importPokemon
    events:
      - httpApi:
          path: /import
          method: post

          # ARN of the layers
    layers:
      - arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-nodejs-0_6_0:1
      - arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-collector-amd64-0_6_0:1

Next, add a couple of environment variables to configure the start of the handler functions and the configuration for the OpenTelemetry collector.

# serverless.yaml
environment:
    OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml
    AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler

The opentelemetry-nodejs layer will spin off the Node.js tracer, configure the supported auto-instrumentation libraries, and set up the context propagators.

While the opentelemetry-collector layer is going to spin off a version of the collector executed in the same context as the AWS lambda layers, configured by the collector.yaml file.

# collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  otlp:
    endpoint: tempo-us-central1.grafana.net:443
    headers:
      authorization: Basic <your basic64 encoded token>

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Easy peezy lemon squeezy 🍋 right? well, this is everything you need to do to start your observability journey!

For every trace, there should be a test!

After having the observability setup, now is time to go to the next level by leveraging it by running some trace-based tests. This is our test case:

Execute an HTTP request against the import Pokemon service.
This is a two-step process that includes a request to the PokeAPI to grab the Pokemon data.
Then, it executes the required database operations to store the Pokemon data in DynamoDB.

What are the key parts we want to validate?

Validate that the external service from the worker is called with the proper POKEMON_ID and returns 200.
Validate that the duration of the DB operations is less than 100ms.
Validate that the response from the initial API Gateway request is 200.

Running the Trace-Based Tests

To run the tests, we are using the @tracetest/client NPM package. It allows teams to enhance existing validation pipelines written in JavaScript or TypeScript by including trace-based tests in their toolset.

The code can be found in the tracetest.ts file.

import Tracetest from '@tracetest/client';
import { TestResource } from '@tracetest/client/dist/modules/openapi-client';
import { config } from 'dotenv';

config();

const { TRACETEST_API_TOKEN = '' } = process.env;

const [raw = ''] = process.argv.slice(2);

let url = '';

try {
  url = new URL(raw).origin;
} catch (error) {
  console.error(
    'The API Gateway URL is required as an argument. i.e: `npm test https://75yj353nn7.execute-api.us-east-1.amazonaws.com`'
  );
  process.exit(1);
}

const definition: TestResource = {
  type: 'Test',
  spec: {
    id: 'ZV1G3v2IR',
    name: 'Serverless: Import Pokemon',
    trigger: {
      type: 'http',
      httpRequest: {
        method: 'POST',
        url: '${var:ENDPOINT}/import',
        body: '{"id": "${var:POKEMON_ID}"}\n',
        headers: [
          {
            key: 'Content-Type',
            value: 'application/json',
          },
        ],
      },
    },
    specs: [
      {
        selector: 'span[tracetest.span.type="database"]',
        name: 'All Database Spans: Processing time is less than 100ms',
        assertions: ['attr:tracetest.span.duration < 100ms'],
      },
      {
        selector: 'span[tracetest.span.type="http"]',
        name: 'All HTTP Spans: Status  code is 200',
        assertions: ['attr:http.status_code = 200'],
      },
      {
        selector:
          'span[name="tracetest-serverless-dev-api"] span[tracetest.span.type="http" name="GET" http.method="GET"]',
        name: 'The request matches the pokemon Id',
        assertions: ['attr:http.url  =  "https://pokeapi.co/api/v2/pokemon/${var:POKEMON_ID}"'],
      },
    ],
  },
};

const main = async () => {
  const tracetest = await Tracetest(TRACETEST_API_TOKEN);

  const test = await tracetest.newTest(definition);
  await tracetest.runTest(test, {
    variables: [
      {
        key: 'ENDPOINT',
        value: url.trim(),
      },
      {
        key: 'POKEMON_ID',
        value: `${Math.floor(Math.random() * 100) + 1}`,
      },
    ],
  });
  console.log(await tracetest.getSummary());
};

main();

Get True Test Observability

Make sure to apply the Tempo tracing backend in Tracetest. Create your Basic auth token, and use this resource file for reference. View the tracetest-tracing-backend.yaml resource file on GitHub, here.

type: DataStore
spec:
  id: tempo-cloud
  name: Tempo
  type: tempo
  tempo:
    type: http
    http:
      url: https://tempo-us-central1.grafana.net/tempo
      headers:
        authorization: Basic <base 64 encoded>
      tls: {}

Apply the resource with the Tracetest CLI.

tracetest config -t TRACETEST_API_TOKEN
tracetest apply datastore -f ./tracetest-tracing-backend.yaml

Or, add it manually in the Tracetest Web UI.

With everything set up and the trace-based tests executed against the PokeAPI, we can now view the complete results.

Run the test with the command below.

npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com

Follow the links provided in the npm test command output to find the full results, which include the generated trace and the test specs validation results.

[Output]

> tracetest-serverless@1.0.0 test
> ENDPOINT="$(sls info --verbose | grep HttpApiUrl | sed s/HttpApiUrl\:\ //g)" ts-node tracetest.ts https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com/import

Run Group: #618f9cda-a87e-4e35-a9f4-10cfbc6f570f (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_a613d93805243f83/run/618f9cda-a87e-4e35-a9f4-10cfbc6f570f)
Failed: 0
Succeed: 1
Pending: 0

Runs:

✔ Serverless: Import Pokemon (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_a613d93805243f83/test/ZV1G3v2IR/run/22) - trace id: d111b18ca75fb6dbf170b66d963363f9

Find the trace in Grafana Cloud Tempo

The full list of spans generated by the AWS Lambda function can be found in your Tempo instance, these are the same ones that are displayed in the Tracetest App after fetching them from Tempo.

👉 Join the demo organization where you can start playing around with the Serverless example with no setup!! 👈

From the Tracetest test run view, you can view the list of spans generated by the Lambda function, their attributes, and the test spec results, which validate the key points.

Key Takeaways

Simplified Observability with OpenTelemetry Lambda Layers

In this post I’ve highlighted how using OpenTelemetry Lambda layers allows for automatic tracing without additional code, making it easier than ever to set up observability for your Serverless applications.

Powerful Integration with Grafana Cloud

Grafana Cloud has become an essential tool in our observability toolkit. By leveraging Grafana Tempo for tracing, we can store and analyze OpenTelemetry spans effectively, showcasing the seamless integration and its benefits.

Enhanced Trace-Based Testing with Tracetest

Tracetest is a game-changer for trace-based testing. By validating telemetry data from our instrumented services, it provides unparalleled visibility and testability, empowering us to ensure our distributed systems perform as expected.

Would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out today by signing up for free!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Testing AWS Lambda Functions (Serverless Framework) with OpenTelemetry and Tracetest

Oscar Reyes — Tue, 12 Mar 2024 16:12:18 +0000

A year ago, I published a blog post about how to use Tracetest with Lambda and AWS. That post took me on an adventure as I tried to figure out the best way to create a simple, repeatable, easy-to-understand approach to setting up a complete FaaS (Function as a Service) distributed system. I thought I’d figured it all out. However, reading it again, I believe that wasn’t the case 😅.

Since then, the ecosystem has changed. Using the Serverless Framework makes deployment simpler. We released the managed Tracetest App making any serverless-based systems simpler to instrument and test. You can now test public-facing apps with no infra overhead!

Buckle up and get ready for the second round; this time improved, faster, bigger and more efficient! With explosions… Ok, just kidding! 💥💣

Why should I care about this?

You know the drill. Cloud Native systems can become a pain to debug. Information moves around to different places, pipelines, services, workers, message brokers, you name it.

The story often repeats itself. Our small team must provision infrastructure, write code, and figure out bugs often working late into the hours of a Friday night to solve production issues with only logs to accompany us in those dark moments.

After that, we promise ourselves that we’ll come back and fix all of it… this time for real.

Well, that day has come, because today I’m going to show you how you and your team can easily instrument a Node.js Serverless App using the revamped Pokeshop Demo Serverless implementation.

And all this while I teach you how to take it to the next level by using trace-based testing with Tracetest tools and libraries! 🕺🏽

I have never heard about Tracetest and Trace-based testing. What is that?

Excellent question my friend! 🤝🏽

Tracetest is an observability-enabled testing tool for Cloud Native architectures, leverages these distributed traces as part of testing, providing you with better visibility and testability enabling you to run trace-based tests.

Trace-based testing is the technique of running validations against the telemetry data generated by the distributed system’s instrumented services.

What are we building today?

Today, we are going to be provisioning the Serverless version of the Pokeshop Demo API which is a fully distributed and instrumented Node.js application running on AWS Lambda. Here’s the list of resources that will be used outside of the regular Serverless setup.

AWS RDS (Postgres).
AWS SQS.
AWS ElastiCache.

The networking will be handled by the serverless-vpc plugin, which is a simple way to spin off the required resources to manage ingress and egress rules, as well as protecting our precious services behind a private network!

Requirements

Tracetest Account:

Sign up to app.tracetest.io or follow the get started docs.
Create an environment.
Select Application is publicly accessible to get access to the environment's Tracetest Cloud Agent endpoint.
Select OpenTelemetry as the tracing backend.
Create an environment token.

AWS:

Have access to an AWS Account.
Install and configure the AWS CLI.
Use a role that is allowed to provision the required resources.

What are the steps to run it myself?

For the self-made developers out there, here’s what you need to run to do it yourself 🦾.

First, clone the Pokeshop repo.

git clone https://github.com/kubeshop/pokeshop.git
cd pokeshop/serverless

Then, follow the instructions to run the deployment and the trace-based tests:

Copy the .env.template file to .env.
Fill the TRACETEST_AGENT_ENDPOINT value from your environment’s tracing backend information. It should be formatted like this https://agent-<redacted>-<redacted>.tracetest.io:443.
Fill the TRACETEST_API_TOKEN value with the one generated for your Tracetest environment. It’ll look like this tttoken_***************.
Run npm i.
Run the Serverless Framework deployment with npm run deploy. Use the API Gateway endpoint from the output in your test below.
Run the trace-based tests with npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com.

Now, let’s dive-in into the nitty-gritty details. 🤓

Instrumenting the AWS Lambda Functions

First, each Lambda function is preloading the OpenTelemetry configuration by executing the setup file before the actual handler execution.

environment:
    NODE_OPTIONS: --require ./src/setup

This is going to execute the createTracer function from the src/telemetry/tracing.ts file that configures the trace provider with the exporter options.

let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
  const provider = new NodeTracerProvider();

  const spanProcessor = new BatchSpanProcessor(
    new OTLPTraceExporter({
      url: COLLECTOR_ENDPOINT,
    })
  );

  provider.addSpanProcessor(spanProcessor);
  provider.register();

  registerInstrumentations({
    instrumentations: [
      new AwsLambdaInstrumentation({
        disableAwsContextPropagation: true,
      }),
    ],
  });

  const tracer = provider.getTracer(SERVICE_NAME);

  globalTracer = tracer;

  return globalTracer;
}

async function getTracer(): Promise<opentelemetry.Tracer> {
  if (globalTracer) {
    return globalTracer;
  }

  return createTracer();
}

The telemetry data generated by the AWS Lambda function is going to be sent to the COLLECTOR_ENDPOINT, which, in this case, is set to the Tracetest Cloud Agent, with extra no setup, no collectors, no side carts. The Tracetest platform is ready to ingest your traces.

That’s it, that’s all you need to instrument your AWS Lambda functions. You don’t believe me?! Take a look at the official OpenTelemetry Serverless docs.

Test Case: Importing a Pokemon

This is what we are going to be using as test case:

Execute an HTTP request against the import Pokemon service.
This is a two-step process that includes an initial handler that puts a message into SQS.
Then, a worker picks up the message to trigger an external service (PokeAPI) request to grab the raw Pokemon data.
Finally the worker executes the required database operations to store the Pokemon data to both RDS Postgres and ElastiCache.

What are the key parts we want to validate?

Validate that the external service from the worker is called with the proper POKEMON_ID and returns 200.
Validate that the duration of the DB operations is less than 100ms.
Validate that the response from the initial API Gateway request is 200.

Running the Trace-Based Tests

Because, who doesn’t like JavaScript, right? …Right? 👀

The code can be found in the tracetest.ts file.

import Tracetest from '@tracetest/client';
import { TestResource } from '@tracetest/client/dist/modules/openapi-client';
import { config } from 'dotenv';

config();

const { TRACETEST_API_TOKEN = '' } = process.env;
const [url = ''] = process.argv.slice(2);

// The Tracetest test JSON definition
const definition: TestResource = {
  type: 'Test',
  spec: {
    id: 'ZV1G3v2IR',
    name: 'Serverless: Import Pokemon',
    trigger: {
      type: 'http',
      httpRequest: {
        method: 'POST',
        url: '${var:ENDPOINT}/import',
        body: '{"id": ${var:POKEMON_ID}}\n',
        headers: [
          {
            key: 'Content-Type',
            value: 'application/json',
          },
        ],
      },
    },
    specs: [
      // Validate the external service from the worker is called with the proper POKEMON_ID and returns 200
      {
        selector: 'span[tracetest.span.type="http" name="GET" http.method="GET"]',
        name: 'External API service should return 200',
        assertions: ['attr:http.status_code   =   200', 'attr:http.route  =  "/api/v2/pokemon/${var:POKEMON_ID}"'],
      },
      // Validate the duration of the DB operations is less than 100ms.
      {
        selector: 'span[tracetest.span.type="database"]',
        name: 'All Database Spans: Processing time is less than 100ms',
        assertions: ['attr:tracetest.span.duration < 100ms'],
      },
      // Validate the response from the initial API Gateway request is 200
      {
        selector: 'span[tracetest.span.type="general" name="Tracetest trigger"]',
        name: 'Initial request should return 200',
        assertions: ['attr:tracetest.response.status = 200'],
      },
    ],
  },
};

const main = async () => {
  if (!url)
    throw new Error(
      'The API Gateway URL is required as an argument. i.e: `npm test https://75yj353nn7.execute-api.us-east-1.amazonaws.com`'
    );

  // configure
  const tracetest = await Tracetest(TRACETEST_API_TOKEN);

  // create
  const test = await tracetest.newTest(definition);

  // run! 
  await tracetest.runTest(test, {
    variables: [
      {
        key: 'ENDPOINT',
        value: `${url.trim()}/pokemon`,
      },
      {
        key: 'POKEMON_ID',
        value: `${Math.floor(Math.random() * 100) + 1}`,
      },
    ],
  });

  // and wait and log results (optional)
  console.log(await tracetest.getSummary());
};

main();

Visualizing the Results

With everything set up and the trace-based tests executed against the Pokeshop demo, we can now view the complete results. Follow the links provided in the npm test command output to find the full results, which include the generated trace and the test specs validation results.

npm test https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com

[Output]
> api@1.0.0 test
> ts-node tracetest.ts https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com

Successful: 1
Failed: 0

[✔️ Serverless: Import Pokemon] #5 - https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_a7c6870903f808ce/test/ZV1G3v2IR/run/5

👉 Join the demo organization where you can start playing around with the Serverless example with no setup!! 👈

From the Tracetest test run view, we can view the list of spans generated by the Lambda function, their attributes, and the test spec results, which validate the key points.

Takeaways

You have seen how simple it can be to instrument an AWS Lambda Function but, not only that, you now know how to run trace-based tests against an asynchronous Serverless process.

You are now ready to face the world and give it a try by yourself. Remember that testing and observability is a process, but as everything it can always be improved, so don’t be afraid to start with something small!

Have questions? you can find me lurking around the Tracetest Slack channel - join, ask, and we will answer!

Announcing the Tracetest Integration with Azure App Insights

Oscar Reyes — Thu, 06 Jul 2023 18:26:30 +0000

Today, we are happy to announce that Tracetest now works with Azure App Insights. This integration enables you and your team to fully leverage your system's observability from Azure Monitor. By using Tracetest in conjunction with Azure, you can improve your instrumentation setup, enhance quality using the new Tracetest Analyzer, test system behavior by creating assertions based on traces, and integrate it into your automated pipelines.

Check out this hands-on example of how Tracetest works with Azure App Insights, or check out the recipe in the docs!

https://lh5.googleusercontent.com/yl3yGiCgWG7sUb9Gz8cetqkrTzoqfsFiUHoYcWhbU-Rz3QNmeqNrffqMop1cIkyQXe85m1sJ37KphR69oV74Kb_sIEnEC5VPXdPHiEn53ZXEoN8h0z7KLv9-90Z3EDfLqXCYPvLYyaFtOcIA_fo4N6E

What is Azure App Insights?

Azure App Insights is an extension of Azure Monitor and provides application performance monitoring (APM) features. APM tools are useful to monitor applications from development, through test and into production in the following ways:

Proactively understand how an application is performing.
Reactively review application execution data to determine the cause of an incident.

What is Tracetest?

Tracetest is an open-source project, part of the CNCF landscape. It allows you to quickly build integration and end-to-end tests, powered by your distributed traces.

Tracetest uses your existing distributed traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source or send traces to Tracetest directly!

Tracetest makes it possible to:

Define tests and assertions against every single microservice that a trace goes through.
Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.
Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.
Save and run the tests manually or via CI build jobs with the Tracetest CLI.

Tracetest Now Works with Azure App Insights!

Tracetest now works with Azure App Insights, enabling you to use Azure Monitor as a data store to pull traces generated by your distributed system.

In your existing observability setup, you can have the OpenTelemetry instrumentation configured in your code with OpenTelemetry Collector and send telemetry data to both Azure App Insights and Tracetest.

Or, you can decide to use the native approach by instrumenting your app directly with any of the available Azure Monitor distros to send data directly to the cloud provider and connect Tracetest to fetch data directly from Azure Monitor API.

Why is it important?

Enhancing your test suites with Azure App Insights and Trace-based Testing is crucial. When running integration tests, it can be difficult to pinpoint exactly where an HTTP transaction went wrong within a network of microservices. By enabling tracing, Tracetest can run tests with assertions against existing trace data for every service involved in the transaction. These tests can be used as part of your CI/CD process to ensure system functionality and catch regressions.

https://lh5.googleusercontent.com/yl3yGiCgWG7sUb9Gz8cetqkrTzoqfsFiUHoYcWhbU-Rz3QNmeqNrffqMop1cIkyQXe85m1sJ37KphR69oV74Kb_sIEnEC5VPXdPHiEn53ZXEoN8h0z7KLv9-90Z3EDfLqXCYPvLYyaFtOcIA_fo4N6E

Combining the test creation capability of Tracetest with Azure App Insights allows you to take advantage of your existing instrumentation setup to explore new frontiers. You will now have the ability to evaluate complex system behaviors beyond just the initial request response, by analyzing the full list of steps taken to complete an asynchronous transaction.

Try Tracetest with Azure App Insights

For this blog post, we are going to focus on the native connection from Tracetest to Azure App Insights which pulls telemetry data through the Azure Monitor API. Showcasing how you can instrument a simple Node.js containerized application, validate the traces from Tracetest and add a test to the generated telemetry data.

The first task is to gather the necessary information from your Azure Cloud. This information will be used to connect the Node.js app to your App Insights instance and for Tracetest to pull data from the Azure Monitor API.

The things you need are:

After gathering the required information, the next step is to download the quick start example from the Tracetest repo that can be found here.

This repo contains the code to run the Node.js app, the docker-compose setup to run the services, and the configuration for the Tracetest server.

/src # the Node.js application code
/tests # Tracetest test YAML definitions 
/tracetest # Tracetest server setup
.env # environment variables
docker-compose.yaml # main docker compose configuration
Dockerfile # Node.js app dockerization

Then, you can start overriding the template values for the configuration with the information you gathered in the first step.

In the .env file add the CONNECTION_STRING information

CONNECTION_STRING="<your-connection-string>"

This will enable the Node.js instrumentation code to send traces to the Azure Cloud.

In the tracetest/tracetest-provision.yaml file add the resourceArmId and the accessToken

---
type: Config
spec:
  analyticsEnabled: true
---
type: PollingProfile
spec:
  name: Custom Profile
  strategy: periodic
  default: true
  periodic:
    timeout: 5m
    retryDelay: 5s
---
type: DataStore
spec:
  name: azureappinsights
  type: azureappinsights
  azureappinsights:
    connectionType: direct
    resourceArmId: <your-arm-id>
    accessToken: <your-access-token>
    useAzureActiveDirectoryAuth: false

This will enable Tracetest to connect with the Azure Monitor API and fetch traces from the Azure Cloud. With the configuration ready, you can now execute the example by running:

docker compose -f ./docker-compose.yaml -f ./tracetest/docker-compose.yaml up -d

[Output]
[+] Running 3/3
 ⠿ Container tracetest-azure-app-insights-tracetest-1  Started                                                                                                    2.3s
 ⠿ Container tracetest-azure-app-insights-app-1        Started                                                                                                    0.5s
 ⠿ Container tracetest-azure-app-insights-postgres-1   Healthy

This will start the Tracetest server on port 11633.

To validate that the configuration is correct and Tracetest is ready to start running tests, you can open the UI by opening [http://localhost:11663](http://localhost:11663) in a browser.

Then, head to the settings page where you’ll find the Data Store configuration and you should see the Azure App Insights option selected with the configuration we added earlier.

At the bottom of the settings page, you will find the Test Connection button, click on it to validate that Tracetest is able to connect and fetch traces from Azure App Insights.

Now that we have everything set up, let’s execute a test!

To begin with, start by installing the Tracetest CLI:

brew install kubeshop/tracetest/tracetest

Note: Check out the download page for more info.

Then, from the example root folder, let’s use the test definition from the tests folder.

# tests/test.yaml

type: Test
spec:
  id: 4F1jCHJVR
  name: App Insights
  description: App Insights
  trigger:
    type: http
    httpRequest:
      url: http://app:8080/http-request
      method: GET
      headers:
        - key: Content-Type
          value: application/json
  specs:
    - selector: span[tracetest.span.type="http"]
      assertions:
        - attr:tracetest.selected_spans.count   =   2
        - attr:tracetest.span.duration   <   1s
    - selector: span[tracetest.span.type="http" name="GET /"]
      assertions:
        - attr:http.target =   "/"

By executing the following command:

tracetest test run -d tests/test.yaml -s http://localhost:11633

✔ App Insights (http://localhost:11633/test/<test-id>/run/1/test)

After executing the test, you can follow the result link back to the UI, which after a couple of minutes, should display the generated trace and test results.

This is just one of three recipes we have created covering different setups and ways you can combine the power of Azure App Insights and Tracetest.

To learn more, head to the following links:

What's next?

Would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out today by downloading it today! Want to learn more about Azure App Insights? Read more here.

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Tracetest Analyzer: Identify patterns and issues with code instrumentation

Oscar Reyes — Thu, 06 Jul 2023 18:21:59 +0000

Are you ready!? I’m thrilled to announce a new feature that has been in the works for the past few weeks.

The Tracetest Analyzer!

The Tracetest Analyzer is a plugin-based framework used to analyze OpenTelemetry traces to help teams improve instrumentation data, find potential problems and provide tips to fix problems with code instrumentation. 🔥

We are always looking for ways to help users have the best experience when testing their distributed systems.

Currently, executing a test with Tracetest is composed of three main modes:

Trigger
Trace
Test

Today we have some exciting news to share about improvements to the Trace mode. With the release of v0.11.10 you will now have access to the Tracetest Analyzer. Let’s dive into what this means.

Let’s talk about the problem

Instrumenting a distributed system is an overwhelming task.

When deciding to introduce tracing using OpenTelemetry, it is important to follow rules and standards to ensure the quality of the telemetry data. The data should be readable and useful for developers to debug and find potential issues, as well as help SREs sleep better at night.

All of the standards and rules necessary for instrumenting the system can be found spread across various documentation pages and libraries on hosting platforms like Github. This can make the job of instrumenting code much more difficult.

I took a moment to collect them all here for you:

Spreading this knowledge across different resources makes enforcing standards an even bigger hurdle.

It gets exponentially more complex depending on the architecture and team organization. A distributed system can be composed of different sets of programming languages and tools. As well as for some microservice-driven systems, where teams can also be divided into entirely separated cells with little communication between each other.

Most of the time, SREs and QAs end up getting caught in the middle… 😢

Ideation 💡

From one of the early brainstorming sessions we had as a team, Matheus came up with the idea of being able to analyze traces to find possible problems. This would catch problems by evaluating the metadata in trace spans or finding patterns in the entire distributed trace.

The idea grew based on several community requests. Community members created three different issues.

One by Mark de Jong, a cloud-native architect at ING, and two opened by Adam Gardner a CNCF ambassador working at Dynatrace.

We got the idea to include a scoring system similar to Google’s Lighthouse. It allows users to find potential problems that exist within the representation of a flow within the system. But, also enable them to have a set goal to improve the overall quality of the telemetry data. Plus, it fixes potential problems with the system.

The culmination of our effort is called the Tracetest Analyzer!

Solution: The Tracetest Analyzer 🐙

The first-ever tool to analyze traces! The Tracetest Analyzer can analyze traces, identify patterns, and fix issues with code instrumentation.

As you already saw from the beginning of the article, we are thrilled to share that we have released the initial version of the Tracetest Analyzer. Currently in beta.

This is the culmination of all the planning, conversations and community requests that we have had across several months.

During the first-ever Kubeshop Hackathon held from May 15th to May 18th, 2023, the team decided to work on the initial version of the Analyzer, code-named “The Dream”.

What is the Tracetest Analyzer?

The Tracetest Analyzer is an evaluation framework composed of plugins. It analyzes OpenTelemetry traces to help teams improve instrumentation data, find potential problems and provide tips to fix problems with code instrumentation.

Each plugin contains a set of rules which can be applied to each of the spans or to the entire trace.

It is included as one of the steps by the Tracetest test runner, which executes the analyzer every time you run a test and allows you to visually see the global score, plugin score and passed/failed results of each of the rules alongside tips on how to fix them.

Key Concepts

***********Plugin.*********** The encapsulation of the metadata (name, description), the list of rules and the logic on how to evaluate them. Evaluating a plugin will yield a score depending on the weight of each passed and failed rule and each of the individual results.

Rule. A unitary validation against any section of the trace. It can be:

a span
a set of spans
the entire trace

Evaluating the result will output a passed or failed status.

How does the Tracetest Analyzer work?

After getting the distributed trace from the data store, Tracetest will run the analyzer step to evaluate each of the registered plugins, gather the results and calculate the global score. If the global score is below the minimum you’ve configured, the test run will fail!

Registered Plugins

For the beta version, we have defined three plugins.

*************OpenTelemetry Semantic Conventions.************* Enforces teams to follow the semantic conventions defined by the Otel community, it includes rules to validate span names and required attributes.

*************Security.************* Allows teams to identify possible security problems by evaluating the trace. It currently has rules to prefer HTTPS over HTTP and to avoid API Keys Leaks.

*****************************Common Problems.***************************** Helps users catch some of the common problems, the enabled rule evaluates the usage of DNS over IPs endpoints.

The beta list of plugins and rules is not configurable at the moment but it represents a subset of things we will be doing in the near future. We are asking for community feedback on what you’ll want to see added! Add your thoughts to this Issue or Discord!

How to enable the Tracetest Analyzer?

If you are using Tracetest version v0.11.10 or above, the Analyzer will be enabled by default. We are not enforcing a minimum score. You can enforce it by heading to the analyzer settings page in the UI and selecting the minimum score. It’s the percentage average of all the plugin scores combined. A test will fail if its global score is below the configured minimum score.

You can also disable the Analyzer entirely on the same page if you would like to do so.

Tracetest Analyzer Video Guide

I’ve also recorded a 6-minute video guide on how to quickly get started with the Analyzer.

https://www.loom.com/share/af06e7e0b97947888ced120a48d31961

Tracetest Analyzer Webinar

We’re dropping a webinar with Adriana Villela from On-Call Me Maybe Podcast tomorrow! Save the date and tune in.

https://www.youtube.com/watch?v=AZiEST7EUhU

What’s Next?

First and foremost, we would love your feedback on the first version of the Tracetest Analyzer!

We understand that there is a lot of room for improvement. We want to make sure the features we’re building are aligned with what the community needs!

Last, but not least, do you want to learn more about Tracetest and what it brings to the table? Check the docs and try it out by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Trace-based Testing AWS Lambda with Tracetest, ECS Fargate, and Terraform

Oscar Reyes — Thu, 06 Jul 2023 18:09:48 +0000

Since the initial release of AWS Lambda back in 2014, developers have seen the great potential of having a FaaS (Function as a Service) distributed system. It avoided the pain of maintaining EC2 instances, load balancers, target groups and more. Soon enough, we saw the birth of frameworks like Serverless or SAM that make managing these serverless functions easier.

We started building REST APIs, creating workers and cron jobs using AWS CloudWatch and listening to different AWS services events like S3. However, we did not realize that at some point it would get complex to understand how all of these services interact with each other. It would get even harder to ensure full visibility of every step that is taken to complete several tasks that move through the distributed system.

Today, standard protocols like OpenTelemetry exist to help teams implement observability as part of the development process and provide a clear understanding of what their serverless application is doing.

After reading this blog post, you will learn how to instrument a basic Node.js Lambda function using OpenTelemetry to provide better insights into what the code is doing, store the tracing data in Jaeger where you can visualize and query the information and run trace-based testing with Tracetest to include automated validations based on the generated telemetry data.

You can also find a recipe for this use case in the Tracetest official docs, here.

Let’s begin!

Basic Concepts

First, let’s explain a few basic concepts to give you context and clarity about what we are going to do once we begin the setup.

What is AWS Lambda?

AWS Lambda is a computing service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling and logging. With Lambda, you can run code for virtually any type of application or back-end service. All you need to do is supply your code in one of the languages that Lambda supports.

Currently, there are several frameworks to manage Lambda functions by enabling a streamlined way for you to create APIs, workers, cron jobs, etc. So your team can focus on the product requirements and business logic. Some of them are:

What is AWS ECS Fargate?

AWS Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers. AWS Fargate is compatible with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

We are going to be using ECS Fargate as a streamlined way of deploying Tracetest to the AWS cloud.

What is Terraform?

HashiCorp Terraform is an infrastructure as code (IaC) tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse and share. You can then use a consistent workflow to provision and manage all of your infrastructure throughout its lifecycle. Terraform can manage low-level components like computing, storage and networking resources, as well as high-level components like DNS entries and SaaS features.

Terraform will be the base framework we’ll use to manage all AWS services, as well as package and deploy the Lambda function

What is Tracetest?

Tracetest is an open-source project, part of the CNCF landscape. It allows you to quickly build integration and end-to-end tests, powered by your OpenTelemetry traces.

Tracetest uses your existing OpenTelemetry traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Tracetest makes it possible to:

Define tests and assertions against every single microservice that a transaction goes through.
Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.
Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly and without errors.
Save and run the tests manually or via CI build jobs with the Tracetest CLI.

Setting up the AWS Infrastructure

We are going to use Terraform to provision all of the required AWS infrastructure. It includes:

*******************Networking.******************* VPCs, Security Groups, Internal, and Public Load Balancers.
The REST API. Lambda, API Gateway, S3 bucket.
***************Telemetry Side Cart.*************** ECS Cluster, Fargate Task definitions, Jaeger, Tracetest.

Cloud Infrastructure Diagram

What do I need to run the Use Case?

Here are the requirements to run the use case:

Initial Terraform Definitions

To encourage a better understanding of the Terraform code definitions, we’ll be using external and local variables defined in the [variables.tf](http://variables.tf) file, which also, includes the initial setup for the required providers such as AWS.

// variables.tf

// defining the dependencies and packages
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.55.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "4.0.4"
    }
  }
}

provider "aws" {
  region = local.region
}

data "aws_caller_identity" "current" {}
data "aws_availability_zones" "available" {}

// external variables 
variable "aws_region" {
  type    = string
  default = "us-west-2"
}

variable "tracetest_version" {
  type    = string
  default = "latest"
}

variable "environment" {
  type    = string
  default = "dev"
}

// prefixes, naming convention and static variables
locals {
  name            = "tracetest"
  region          = var.aws_region
  tracetest_image = "kubeshop/tracetest:${var.tracetest_version}"
  environment     = var.environment

  db_name     = "postgres"
  db_username = "postgres"

  vpc_cidr = "192.168.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)


  // used to provision the Tracetest server
  provisioning = <<EOF
---
type: PollingProfile
spec:
  name: default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: jaeger
  type: jaeger
  jaeger:
    endpoint: ${aws_lb.internal_tracetest_alb.dns_name}:16685
    tls:
      insecure_skip_verify: true
  EOF

  tags = {
    Name    = local.name
    Example = local.name
  }
}

In the following Terraform definition files, you’ll find references to the local variables defined in this file.

Network Definitions

First, we need to set up the base definitions that will allow communication between the different services we will be generating. Part of this section is to provide public access to certain services like the Serverless API, Jaeger UI and Tracetest as well as ensure VPC protection for others like the RDS Postgres instance.

// network.tf
// creating the main VPC resources, including private and public subnets
module "network" {
  source                                      = "cn-terraform/networking/aws"
  name_prefix                                 = local.name
  vpc_cidr_block                              = local.vpc_cidr
  availability_zones                          = ["${local.region}a", "${local.region}b", "${local.region}c", "${local.region}d"]
  public_subnets_cidrs_per_availability_zone  = ["192.168.0.0/19", "192.168.32.0/19", "192.168.64.0/19", "192.168.96.0/19"]
  private_subnets_cidrs_per_availability_zone = ["192.168.128.0/19", "192.168.160.0/19", "192.168.192.0/19", "192.168.224.0/19"]
}

// configuration pieces for the public ALB
module "tracetest_alb_security_group" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = local.name
  description = "Load balancer security group"
  vpc_id      = module.network.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 11633
      to_port     = 11633
      protocol    = "tcp"
      description = "HTTP access for Tracetest"
      cidr_blocks = "0.0.0.0/0"
      }, {
      from_port   = 16686
      to_port     = 16686
      protocol    = "tcp"
      description = "HTTP access for Jaeger UI"
      cidr_blocks = "0.0.0.0/0"
      }, {
      from_port   = 16685
      to_port     = 16685
      protocol    = "tcp"
      description = "HTTP access for Jaeger API"
      cidr_blocks = "0.0.0.0/0"
  }]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "-1"
      description = "HTTP access to anywhere"
      cidr_blocks = "0.0.0.0/0"
  }]
}

resource "aws_lb" "tracetest-alb" {
  name               = "tracetest-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [module.tracetest_alb_security_group.security_group_id]
  subnets            = module.network.public_subnets_ids

  enable_deletion_protection = false
  tags                       = local.tags
}

// creation of the internal ALB
module "internal_tracetest_alb_security_group" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = local.name
  description = "Internal Load balancer security group"
  vpc_id      = module.network.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 16685
      to_port     = 16685
      protocol    = "tcp"
      description = "HTTP access for Jaeger API"
      cidr_blocks = local.vpc_cidr
      }, {
      from_port   = 4318
      to_port     = 4318
      protocol    = "tcp"
      description = "HTTP access for Jaeger Collector"
      cidr_blocks = local.vpc_cidr
  }]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "-1"
      description = "HTTP access to Anywhere"
      cidr_blocks = "0.0.0.0/0"
  }]
}

resource "aws_lb" "internal_tracetest_alb" {
  name               = "tracetest-internal-alb"
  internal           = true
  load_balancer_type = "application"
  security_groups    = [module.internal_tracetest_alb_security_group.security_group_id]
  subnets            = module.network.private_subnets_ids

  enable_deletion_protection = false
  tags                       = local.tags
}

// security groups for services
module "lambda_security_group" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = "${local.name}_lambda_security_group"
  description = "Lambda security group"
  vpc_id      = module.network.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "-1"
      description = "HTTP access from anywhere"
      cidr_blocks = "0.0.0.0/0"
  }]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "-1"
      description = "HTTP access to anywhere"
      cidr_blocks = "0.0.0.0/0"
  }]
}

module "tracetest_ecs_service_security_group" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = "tracetest_ecs_service_security_group"
  description = "ECS Service security group"
  vpc_id      = module.network.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "tcp"
      description = "HTTP access from VPC"
      cidr_blocks = local.vpc_cidr
  }]

  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "-1"
      description = "HTTP access to anywhere"
      cidr_blocks = "0.0.0.0/0"
  }]
}

REST API Definitions

The REST API is composed of two major sections. The first section contains the Node.js handler code for the Lambda functions. The second covers the creation of AWS resources.

Let’s start with the Node.js handler code and the instrumentation code to enable distributed tracing.

You can find the code described in this section in the Tracetest GitHub repo under the examples folder. There you can find the src and api.tf files.

The handler code for this use case is included in the src folder and is composed of the package.json file, the main handler function called hello.js and the tracing.js file which includes the OpenTelemetry instrumentation.

The Node.js Handler Code for the Lambda Function

In this case, we’ll be building a pretty basic Javascript function that returns a “Hello, World!” message on each request.

// src/hello.js
const handler = async (event) => {
  console.log("Event: ", event);
  let responseMessage = "Hello, World!";

  return {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      message: responseMessage,
    }),
  };
};

module.exports.handler = handler;

Instrumenting the Handler Function with OpenTelemetry

Next, let’s add a second Javascript file that will include all of the setup and configuration to enable automated instrumentation for distributed tracing with OpenTelemetry.

// src/tracing.js
const { Resource } = require("@opentelemetry/resources");
const api = require("@opentelemetry/api");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "tracetest",
  }),
});
const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter());

provider.addSpanProcessor(spanProcessor);
provider.register();

registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-aws-lambda": {
        disableAwsContextPropagation: true,
        eventContextExtractor: (event) => {
          // Tracetest sends the traceparent header as Traceparent
          // passive aggressive author comment 🥴 The propagators doesn't care about changing the casing before matching
          event.headers.traceparent = event.headers.Traceparent;
          const eventContext = api.propagation.extract(api.context.active(), event.headers);

          return eventContext;
        },
      },
    }),
  ],
});

Installing the Dependencies

Now that we have the base JavaScript code ready, its time to include the required modules to run the app, this is the package.json we are going to be using:

# src/package.json
{
  "name": "tracetest-aws",
  "version": "1.0.0",
  "description": "",
  "main": "hello.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@opentelemetry/auto-instrumentations-node": "^0.36.3",
    "@opentelemetry/instrumentation": "^0.35.1",
    "@opentelemetry/resources": "^1.9.1",
    "@opentelemetry/sdk-node": "^0.35.1",
    "@opentelemetry/sdk-trace-node": "^1.9.1",
    "@opentelemetry/semantic-conventions": "^1.9.1"
  }
}

To match the Lambda Node.js versions, switch any 12.x version to install the packages

Then, the last step is to run the npm install command to have all of the packages installed.

Provision a Serverless API with Terraform

To finalize the API section, we will define the required services using Terraform.

// api.tf
resource "aws_lambda_function" "hello_world" {
  function_name = "HelloWorld"

  s3_bucket = aws_s3_bucket.lambda_bucket.id
  s3_key    = aws_s3_object.lambda_hello_world.key

  runtime = "nodejs12.x"
  handler = "hello.handler"
  timeout = 30

  // This allows the tracing.js file to be loaded before the handler hello.js function
/ // to ensure the tracing logic is the first piece of code to be executed
  environment {
    variables = {
      NODE_OPTIONS                = "--require tracing.js"
      OTEL_EXPORTER_OTLP_ENDPOINT = "http://${aws_lb.internal_tracetest_alb.dns_name}:4318"
    }
  }

  source_code_hash = data.archive_file.lambda_hello_world.output_base64sha256

  vpc_config {
    subnet_ids         = module.network.public_subnets_ids
    security_group_ids = [module.lambda_security_group.security_group_id]
  }

  role = aws_iam_role.lambda_exec.arn
}

resource "aws_cloudwatch_log_group" "hello_world" {
  name = "/aws/lambda/${aws_lambda_function.hello_world.function_name}"

  retention_in_days = 30
}

resource "aws_iam_role" "lambda_exec" {
  name = "${local.name}_serverless_lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Sid    = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role       = aws_iam_role.lambda_exec.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_iam_role_policy_attachment" "lambda_network_policy_attachment" {
  role       = aws_iam_role.lambda_exec.name
  policy_arn = aws_iam_policy.lambda_network_policy.arn
}

resource "aws_iam_policy" "lambda_network_policy" {
  name   = "${local.name}_network_policy"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNetworkInterfaces",
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:DescribeInstances",
        "ec2:AttachNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

resource "aws_apigatewayv2_api" "lambda" {
  name          = "${local.name}_lambda_gw"
  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_stage" "lambda" {
  api_id = aws_apigatewayv2_api.lambda.id

  name        = local.environment
  auto_deploy = true

  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.api_gw.arn

    format = jsonencode({
      requestId               = "$context.requestId"
      sourceIp                = "$context.identity.sourceIp"
      requestTime             = "$context.requestTime"
      protocol                = "$context.protocol"
      httpMethod              = "$context.httpMethod"
      resourcePath            = "$context.resourcePath"
      routeKey                = "$context.routeKey"
      status                  = "$context.status"
      responseLength          = "$context.responseLength"
      integrationErrorMessage = "$context.integrationErrorMessage"
      }
    )
  }
}

resource "aws_apigatewayv2_integration" "hello_world" {
  api_id = aws_apigatewayv2_api.lambda.id

  integration_uri    = aws_lambda_function.hello_world.invoke_arn
  integration_type   = "AWS_PROXY"
  integration_method = "POST"
}

resource "aws_apigatewayv2_route" "hello_world" {
  api_id = aws_apigatewayv2_api.lambda.id

  route_key = "GET /hello"
  target    = "integrations/${aws_apigatewayv2_integration.hello_world.id}"
}

resource "aws_cloudwatch_log_group" "api_gw" {
  name = "/aws/api_gw/${aws_apigatewayv2_api.lambda.name}"

  retention_in_days = 30
}

resource "aws_lambda_permission" "api_gw" {
  statement_id  = "AllowExecutionFromAPIGateway"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.hello_world.function_name
  principal     = "apigateway.amazonaws.com"

  source_arn = "${aws_apigatewayv2_api.lambda.execution_arn}/*/*"
}

resource "random_id" "server" {
  byte_length = 8
}

resource "aws_s3_bucket" "lambda_bucket" {
  bucket = "${local.name}-${random_id.server.hex}"
}

resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.lambda_bucket.id
  acl    = "private"
}

data "archive_file" "lambda_hello_world" {
  type = "zip"

  source_dir  = "${path.module}/src"
  output_path = "${path.module}/src.zip"
}

resource "aws_s3_object" "lambda_hello_world" {
  bucket = aws_s3_bucket.lambda_bucket.id

  key    = "src.zip"
  source = data.archive_file.lambda_hello_world.output_path

  etag = filemd5(data.archive_file.lambda_hello_world.output_path)
}

This file includes packaging the JavaScript source code, creating the AWS S3 bucket, uploading the package to S3, creating the Lambda functions, and the AWS API Gateway requirements to expose the functionality to the public using REST.

Telemetry Side Cart Definitions

In this section, we’ll focus on creating the infrastructure to handle telemetry data on AWS. We’ll be following a serverless containerized approach using ECS Fargate to run tasks.

The ECS Cluster

The first thing we need to do is create the cluster to include all the tasks and services we need for the side cart.

# cluster.tf
// creating the ecs cluster
resource "aws_ecs_cluster" "tracetest-cluster" {
  name = "tracetest"
  tags = local.tags
}

// generic task execution role for both jaeger and tracetest
resource "aws_iam_role" "tracetest_task_execution_role" {
  name = "tracetest_task_execution_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = ["ecs-tasks.amazonaws.com", "ecs.amazonaws.com"]
        }
      },
    ]
  })
  managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"]

  tags = local.tags
}

resource "aws_iam_role_policy" "tracetest_task_execution_role_policy" {
  name = "tracetest_task_execution_role_policy"
  role = aws_iam_role.tracetest_task_execution_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "logs:PutLogEvents",
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:DescribeLogStreams",
          "logs:DescribeLogGroups",
        ]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

Creating the Jaeger Service and Task

Once the cluster is defined, we can focus on adding all services and tasks we need for our telemetry data. In this case, we’ll use the [jaeger-all-in-one Docker image](https://www.jaegertracing.io/docs/1.43/getting-started/#all-in-one) as it provides everything we need for a basic demo.

// jaeger.tf

// creating the task definition
resource "aws_ecs_task_definition" "jaeger" {
  family                   = "${local.name}_jaeger"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024
  memory                   = 2048
  execution_role_arn       = aws_iam_role.tracetest_task_execution_role.arn
  container_definitions = jsonencode([
    {
      "name" : "jaeger",
      # using the all in one docker image
      "image" : "jaegertracing/all-in-one:1.42",
      "cpu" : 1024,
      "memory" : 2048,
      "essential" : true,
      // enabling the OTLP listeners
      "environment" : [{
        "name" : "COLLECTOR_OTLP_ENABLED",
        "value" : "true"
      }],
      // exposing the required ports
      "portMappings" : [
        {
          "hostPort" : 14269,
          "protocol" : "tcp",
          "containerPort" : 14269
        },
        {
          "hostPort" : 14268,
          "protocol" : "tcp",
          "containerPort" : 14268
        },
        {
          "hostPort" : 6832,
          "protocol" : "udp",
          "containerPort" : 6832
        },
        {
          "hostPort" : 6831,
          "protocol" : "udp",
          "containerPort" : 6831
        },
        {
          "hostPort" : 5775,
          "protocol" : "udp",
          "containerPort" : 5775
        },
        {
          "hostPort" : 14250,
          "protocol" : "tcp",
          "containerPort" : 14250
        },
        {
          "hostPort" : 16685,
          "protocol" : "tcp",
          "containerPort" : 16685
        },
        {
          "hostPort" : 5778,
          "protocol" : "tcp",
          "containerPort" : 5778
        },
        {
          "hostPort" : 16686,
          "protocol" : "tcp",
          "containerPort" : 16686
        },
        {
          "hostPort" : 9411,
          "protocol" : "tcp",
          "containerPort" : 9411
        },
        {
          "hostPort" : 4318,
          "protocol" : "tcp",
          "containerPort" : 4318
        },
        {
          "hostPort" : 4317,
          "protocol" : "tcp",
          "containerPort" : 4317
        }
      ],
      "logConfiguration" : {
        "logDriver" : "awslogs",
        "options" : {
          "awslogs-create-group" : "true",
          "awslogs-group" : "/ecs/jaeger",
          "awslogs-region" : "us-west-2",
          "awslogs-stream-prefix" : "ecs"
        }
      },
    }
  ])
}

// creating the jaeger service
resource "aws_ecs_service" "jaeger_service" {
  name            = "jaeger-service"
  cluster         = aws_ecs_cluster.tracetest-cluster.id
  task_definition = aws_ecs_task_definition.jaeger.arn
  desired_count   = 1
  launch_type     = "FARGATE"

// defining three entry points to the Jaeger task using albs
  load_balancer {
    target_group_arn = aws_lb_target_group.tracetest-jaeger-tg.arn
    container_name   = "jaeger"
    container_port   = 16686
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.tracetest-jaeger-api-tg.arn
    container_name   = "jaeger"
    container_port   = 16685
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.tracetest-jaeger-collector-tg.arn
    container_name   = "jaeger"
    container_port   = 4318
  }

  network_configuration {
    subnets          = module.network.private_subnets_ids
    security_groups  = [module.tracetest_ecs_service_security_group.security_group_id]
    assign_public_ip = false
  }
}

resource "aws_lb_target_group" "tracetest-jaeger-api-tg" {
  name             = "tracetest-jaeger-api-tg"
  port             = 16685
  protocol         = "HTTP"
  protocol_version = "GRPC"
  vpc_id           = module.network.vpc_id
  target_type      = "ip"
}

resource "aws_lb_target_group" "tracetest-jaeger-collector-tg" {
  name        = "tracetest-jaeger-collector-tg"
  port        = 4318
  protocol    = "HTTP"
  vpc_id      = module.network.vpc_id
  target_type = "ip"

  health_check {
    path              = "/"
    port              = "4318"
    protocol          = "HTTP"
    healthy_threshold = 2
    matcher           = "200-499"
  }
}

resource "aws_lb_target_group" "tracetest-jaeger-tg" {
  name        = "tracetest-jaeger-tg"
  port        = 16686
  protocol    = "HTTP"
  vpc_id      = module.network.vpc_id
  target_type = "ip"
}

// alb listeners for the public and internal ALBS, depending on port
resource "aws_lb_listener" "tracetest-jaeger-alb-listener" {
  load_balancer_arn = aws_lb.tracetest-alb.arn
  port              = "16686"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tracetest-jaeger-tg.arn
  }
}

resource "aws_lb_listener" "tracetest-jaeger-collector-alb-listener" {
  load_balancer_arn = aws_lb.internal_tracetest_alb.arn
  port              = "4318"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tracetest-jaeger-collector-tg.arn
  }
}

resource "aws_lb_listener" "tracetest-jaeger-api-alb-listener" {
  load_balancer_arn = aws_lb.internal_tracetest_alb.arn
  port              = "16685"
  protocol          = "HTTPS"
  certificate_arn   = aws_acm_certificate.cert.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tracetest-jaeger-api-tg.arn
  }
}

resource "tls_private_key" "tracetest_private_key" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "tracetest_self_signed_cert" {
  private_key_pem = tls_private_key.tracetest_private_key.private_key_pem

  subject {
    common_name  = "tracetest.com"
    organization = "Tracetest"
  }

  validity_period_hours = 720

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "cert" {
  private_key      = tls_private_key.tracetest_private_key.private_key_pem
  certificate_body = tls_self_signed_cert.tracetest_self_signed_cert.cert_pem
}

To enable Jaeger’s gRPC OTLP endpoint and expose it through an Application Load Balancer (ALB), you have to provide a valid certificate, as ALBs require HTTPS as protocol and a valid cert to work with GRPC.

Deploying Tracetest to ECS Fargate

One of Tracetest’s biggest goals is to provide numerous easy ways to deploy a Tracetest instance anywhere. The goal is to fit into any user’s current setup with minimal edits, or none at all. The case of ECS Fargate is no different. It’s a user request that was turned into a use case!

When deploying Tracetest in a container-based infrastructure you can configure the Docker image using environment variables. This makes the process easier, because you do not need to provide any configuration files.

// tracetest.tf

// creating the Tracetest task definition
resource "aws_ecs_task_definition" "tracetest" {
  family                   = "tracetest"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024
  memory                   = 2048
  execution_role_arn       = aws_iam_role.tracetest_task_execution_role.arn
  container_definitions = jsonencode([
    {
      "name" : "${local.name}",
      "image" : "${local.tracetest_image}",
      "cpu" : 1024,
      "memory" : 2048,
      "essential" : true,
      "portMappings" : [
        {
          "containerPort" : 11633,
          "hostPort" : 11633,
          "protocol" : "tcp"
        }
      ],
      // configuring and provisioning Tracetest using env variables
      "environment" : [
        {
          "name" : "TRACETEST_POSTGRES_HOST",
          "value" : "${module.db.db_instance_address}"
        },
        {
          "name" : "TRACETEST_POSTGRES_PORT",
          "value" : "${tostring(module.db.db_instance_port)}"
        },
        {
          "name" : "TRACETEST_POSTGRES_DBNAME",
          "value" : "${module.db.db_instance_name}"
        },
        {
          "name" : "TRACETEST_POSTGRES_USER",
          "value" : "${module.db.db_instance_username}"
        },
        {
          "name" : "TRACETEST_POSTGRES_PASSWORD",
          "value" : "${module.db.db_instance_password}"
        },
        {
          "name" : "TRACETEST_PROVISIONING",
          "value" : base64encode(local.provisioning),
        }
      ],
      "logConfiguration" : {
        "logDriver" : "awslogs",
        "options" : {
          "awslogs-create-group" : "true",
          "awslogs-group" : "/ecs/tracetest",
          "awslogs-region" : "us-west-2",
          "awslogs-stream-prefix" : "ecs"
        }
      },
    }
  ])
}

// tracetest service
resource "aws_ecs_service" "tracetest-service" {
  name            = "${local.name}-service"
  cluster         = aws_ecs_cluster.tracetest-cluster.id
  task_definition = aws_ecs_task_definition.tracetest.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  // exposing the UI and API listeners using ALB
  load_balancer {
    target_group_arn = aws_lb_target_group.tracetest-tg.arn
    container_name   = "tracetest"
    container_port   = 11633
  }

  network_configuration {
    subnets          = module.network.private_subnets_ids
    security_groups  = [module.tracetest_ecs_service_security_group.security_group_id]
    assign_public_ip = false
  }
}

// Postgres RDS instance used as main store
module "db" {
  source = "terraform-aws-modules/rds/aws"

  identifier = local.name

  engine               = "postgres"
  engine_version       = "14"
  family               = "postgres14"
  major_engine_version = "14"
  instance_class       = "db.t4g.micro"

  allocated_storage     = 20
  max_allocated_storage = 100

  db_name  = local.db_name
  username = local.db_username
  port     = 5432

  create_db_subnet_group = true
  subnet_ids             = module.network.private_subnets_ids

  vpc_security_group_ids = [module.db_security_group.security_group_id]
  deletion_protection    = false

  tags = local.tags
}

module "db_security_group" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "~> 4.0"

  name        = local.name
  description = "PostgreSQL security group"
  vpc_id      = module.network.vpc_id

  ingress_with_cidr_blocks = [
    {
      from_port   = 5432
      to_port     = 5432
      protocol    = "tcp"
      description = "PostgreSQL access from within VPC"
      cidr_blocks = local.vpc_cidr
    },
  ]

  tags = local.tags
}

resource "aws_lb_target_group" "tracetest-tg" {
  name        = "tracetest-tg"
  port        = 11633
  protocol    = "HTTP"
  vpc_id      = module.network.vpc_id
  target_type = "ip"
}

// allowing public access to Tracetest
resource "aws_lb_listener" "tracetest-alb-listener" {
  load_balancer_arn = aws_lb.tracetest-alb.arn
  port              = "11633"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tracetest-tg.arn
  }
}

Take a moment to look at this part of the file:

{
  "name" : "TRACETEST_PROVISIONING",
  "value" : base64encode(local.provisioning),
}

The provisioning configuration for Tracetest can be found in the variables.tf file under the locals configuration which uses the internal load balancer DNS endpoint to build the data store entry.

provisioning = <<EOF
---
type: PollingProfile
spec:
  name: default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: jaeger
  type: jaeger
  jaeger:
    endpoint: ${aws_lb.internal_tracetest_alb.dns_name}:16685
    tls:
      insecure_skip_verify: true
  EOF

  tags = {
    Name    = local.name
    Example = local.name
  }
}

  EOF

This file is base64 encoded and passed into the TRACETEST_PROVISIONING environment variable. To learn more about provisioning a Tracetest service, check out the official docs.

Defining Terraform Outputs

The final setup step is to generate outputs to provide links that we can use to reach the generated services.

// outputs.tf
output "api_endpoint" {
  value       = "${aws_apigatewayv2_stage.lambda.invoke_url}/hello"
  description = "The API endpoint"
}

output "tracetest_url" {
  value       = "http://${aws_lb.tracetest-alb.dns_name}:11633"
  description = "Tracetest public URL"
}

output "jaeger_ui_url" {
  value       = "http://${aws_lb.tracetest-alb.dns_name}:16686"
  description = "Jaeger public URL"
}

output "internal_jaeger_api_url" {
  value       = "${aws_lb.internal_tracetest_alb.dns_name}:16685"
  description = "Jaeger internal API URL"
}

Provisioning the AWS Infra

Finally! We have written down every required definition piece we need to generate the AWS resources, now it's time to take it to the next level by actually creating them using the Terraform CLI.

The first step is to initialize the Terraform project and install dependencies.

terraform init

[Output]
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/tls from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.55.0
- Using previously-installed hashicorp/tls v4.0.4
- Using previously-installed hashicorp/random v3.4.3
- Using previously-installed hashicorp/archive v2.3.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The second step is to run the apply command.

terraform apply

This is going to prompt you with the question to continue or not, as well as show a list of changes it will execute.

Plan: 100 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + tracetest_url = (known after apply)
  + jaeger_ui_url = (known after apply)
  + internal_jaeger_api_url = (known after apply)
  + api_endpoint = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

By explicitly typing yes and hitting enter, the provisioning process will start.

This is where you can go grab lunch, walk your dog or do any pending chores… or just stare at the Terminal until it’s done 😅.

You will see something similar to this when the provisioning is done.

You’ll see four URLs as outputs, the same ones we defined in the Terraform files.

We can go one by one to validate that everything is working as expected. First, let’s try following the jaeger_ui_url. You should be able to see the angry Gopher and the Jaeger search page.

After that, we can continue with the tracetest_url, where you should see the Tracetest dashboard page with an empty test list.

From the /settings page we can validate that Tracetest has access to the Jaeger GRPC endpoint by clicking the Test Connection button.

Lastly, we can use the following CURL command to validate the api_endpoint.

curl <api_endpoint>

[Output]
{"message": "Hello, World!"}

Good job 🎉! With this set of steps, we have validated that the AWS Infrastructure is ready. In the following section, we’ll move on to validating the tracing data.

Trace-based Testing with Tracetest

You made it! You’ve crossed the desert of creating the Terraform definitions and conquered the fires of waiting an eternity to provision AWS services. Now it’s time to have some fun!

Let’s add some validations based on the Node.js Lambda telemetry. To do so, we’ll create a test based on the following checks:

There should be one span of type FaaS (Function as a Service) called HelloWorld.
The HelloWorld span should take less than 1 second.

# tests/test.yaml

type: Test
spec:
  id: 4iiL0j1Vg
  name: Hello World
  trigger:
    type: http
    httpRequest:
      url: <your_api_endpoint>
      method: GET
      headers:
        - key: Content-Type
          value: application/json
  specs:
    - selector: span[tracetest.span.type="faas" name="HelloWorld"]
      assertions:
        - attr:tracetest.selected_spans.count = 1
        - attr:tracetest.span.duration < 1s

Save this file in your tests folder as test.yaml.

Remember to update the <your_api_endpoint section of the test with the api_endpoint Terraform output.

Running the Trace-based Test

First, configure the Tracetest CLI to point to the proper public ALB and Tracetest port, using the following:

# Replace the <tracetest_url> placeholder for the Terraform output
tracetest configure --endpoint <tracetest_url> --analytics

Then, create and run the test with this command:

tracetest test run -d tests/test.yaml

[Output]
✔ Hello World (<tracetest_url>/test/4iiL0j1Vg/run/1/test)

Finally, follow the output response URL from the run command to find the test run alongside the test specs we just created.

Tracetest allows you to create assertions against any span from the trace, using the selector query language to pick exactly the spans you want to validate to then add checks based on attributes from the matched spans.

Not only that, but you can create, run and validate your trace-based tests entirely from the CLI, enabling your team to have them be part of your CI/CD pipelines.

Conclusion

At Tracetest we share a mix of different technical backgrounds, we have team members like the awesome Matheus with distributed Mobile Game services experience, or Jorge with experience in huge enterprise front-end applications.

Every single one of us has been in a position where we are moments away from throwing our computer in the air to test if airplane mode actually transforms it into an actual plane! It always boils down to wishing we could have better ways of understanding what’s happening in the code.

With that in mind, I wanted to display the power of trace-based testing and how it can help teams deliver quality software and alleviate those nightmare scenarios we have all been through.

In summary, you’ve learned how to:

Provision an Observability side cart with Tracetest and Jaeger on AWS Fargate using Terraform.
Write, provision and instrument an AWS Lambda function with Node.js using OpenTelemetry and Terraform.
Use trace-based testing with Tracetest to validate the generated telemetry data.

Last, but not least - if would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Trace-based testing cloud-native apps with AWS X-Ray and Tracetest

Oscar Reyes — Mon, 10 Apr 2023 17:18:50 +0000

When working with cloud-native applications, having a clear understanding of what every part of the system is doing can become quite complex. Teams might struggle to correctly have accurate integration tests that could validate entire flows because they usually only focus on the user-facing parts of the system like APIs, WebSockets, and FE apps, and their immediate response. Therefore, testing back-end processes like messaging queues, ETL (Extract, Transform, and Load) tasks and any type of asynchronous process can become a pain point.

That’s where the power of distributed tracing alongside AWS X-Ray comes to play. Enabling teams to add automatic and/or custom checkpoints, called spans, that generate step-by-step details for a complete end-to-end process, a distributed trace, from anywhere in your distributed cloud infrastructure. Even if you’re using services on AWS like Lambda, Fargate, EKS, EC2, or others.

Then, ultimately combining this with the power of Tracetest allows teams to run automatic trace-based tests by defining test specs and assertions against the data from distributed traces.

Check out this hands-on basic X-Ray recipe with Tracetest and Docker!

After you finish reading, you’ll learn how to combine AWS X-Ray’s tracing capabilities with the power of Tracetest’s trace-based testing!

What is AWS X-Ray?

AWS X-Ray is a distributed tracing system included in the AWS cloud platform, that enables developers to monitor, analyze, and debug distributed applications running on AWS infrastructure. It provides information on how an application is performing and allows developers to identify and resolve performance issues quickly. X-Ray traces requests as they travel through an application, providing a comprehensive view of its performance. It also integrates with other AWS services, such as Amazon EC2 and AWS Lambda, providing developers with a complete view of their applications.

What is Tracetest?

Tracetest is an open-source project, part of the CNCF landscape. It allows you to quickly build integration and end-to-end tests, powered by your OpenTelemetry traces.

Tracetest uses your existing OpenTelemetry traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Tracetest makes it possible to:

Define tests and assertions against every single microservice that a transaction goes through.
Work with your existing distributed tracing solution, allowing you to build tests based against your already instrumented system.
Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.
Save and run the tests manually or via CI build jobs with the Tracetest CLI.

Why is the Tracetest Integration with X-Ray Important?

AWS X-Ray is commonly used to provide observability into distributed systems that have multiple moving pieces, some using queues, AWS Lambda functions, calling external services, and/or storing information in databases.

Therefore, combining it with Tracetest allows you to get more out of the instrumentation work done to introduce observability into your system. It enables trace-based testing, which can be used to run assertions against any part of the generated trace, allowing you to have automated ways of validating your distributed system’s behavior using real-life inputs as well as improving the overall test coverage.

You can also utilize these tests as part of your CI/CD process to ensure system functionality and to catch regressions.

Try Tracetest with AWS X-Ray

One of the core values of the Tracetest team and the project, in general, is to always aim for flexibility and allow teams to use Tracetest as part of their ecosystem with minimal disruption. In this blog post, we are not only showing you one way to combine X-Ray and Tracetest but rather three different use cases so you can choose what’s best for your team.

Node.js Instrumented API with X-Ray SDK, X-Ray Daemon, and Tracetest. A basic Express server that sends an HTTP GET request to amazon.com and uses the AWS X-Ray SDK auto instrumentation, the X-Ray Daemon as a middleware to send trace data to AWS, and Tracetest for trace-based testing.
Node.js Instrumented API with AWS X-Ray SDK, ADOT, and Tracetest. A basic Express server that sends an HTTP GET request to amazon.com and uses the AWS X-Ray SDK auto instrumentation, the AWS Distro for OpenTelemetry as a middleware to send tracing data to AWS, and Tracetest for trace-based testing.
The Pokeshop API with Otel Instrumentation, ADOT, X-Ray, and Tracetest. Using Tracetest's own distributed Node.js Pokemon demo API, instrumented using official OpenTelemetry packages, the AWS Distro for OpenTelemetry as middleware to send tracing data to AWS, and Tracetest for trace-based testing.

To run trace-based testing with AWS X-Ray, you need to install the following tools:

Node.js Instrumented API with X-Ray SDK, X-Ray Daemon, and Tracetest

In this use case, we have a dockerized Node.js Express application that listens for a GET request on the /http-request route, triggers a request to amazon.com, and returns a response based on the request.

This demo will showcase the following:

Instrumenting the Node.js app using the AWS X-Ray SDK
Sending trace data to AWS Cloud using the AWS X-Ray daemon
Triggering, visualizing, and applying trace-based testing with Tracetest.

To better follow this use case you can download and run the example from the Tracetest GitHub repo by following these steps:

git clone https://github.com/kubeshop/tracetest.git
cd examples/tracetest-amazon-x-ray
docker compose up

Instrumenting the Node.js Express App

The first task at hand is to create the instrumented Node.js App using the AWS X-Ray SDK. This can be achieved by:

Including the aws-x-ray-sdk module in the project.
Configuring the SDK to wrap and generate automatic telemetry data for packages (https, aws-sdk).
Updating the daemon endpoint to send tracing data.
Creating the express routes and logic.

// src/index.js

const AWSXRay = require("aws-xray-sdk");
const XRayExpress = AWSXRay.express;
const express = require("express");

// sends tracing data to the local x-ray daemon instance
AWSXRay.setDaemonAddress("xray-daemon:2000");

// Capture all AWS clients we create
const AWS = AWSXRay.captureAWS(require("aws-sdk"));
AWS.config.update({
  region: process.env.AWS_REGION || "us-west-2",
});

// Capture all outgoing https requests
AWSXRay.captureHTTPsGlobal(require("https"));
const https = require("https");

const app = express();
const port = 3000;

app.use(XRayExpress.openSegment("Tracetest"));

app.get("/", (req, res) => {
  const seg = AWSXRay.getSegment();
  const sub = seg.addNewSubsegment("customSubsegment");
  setTimeout(() => {
    sub.close();
    res.sendFile(`${process.cwd()}/index.html`);
  }, 500);
});

app.get("/http-request/", (req, res) => {
  const endpoint = "https://amazon.com/";
  https.get(endpoint, (response) => {
    response.on("data", () => {});

    response.on("error", (err) => {
      res.send(`Encountered error while making HTTPS request: ${err}`);
    });

    response.on("end", () => {
      res.send(`Successfully reached ${endpoint}.`);
    });
  });
});

app.use(XRayExpress.closeSegment());
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

The Docker Compose Configuration

The Docker configuration will be spinning up all of the required services to run the Node.js Express App, the AWS X-Ray Daemon, and the Tracetest server.

# docker-comopose.yaml

version: "3"
services:
  app:
    build: .
    environment:
      AWS_REGION: ${AWS_REGION}
    ports:
      - "3000:3000"

  tracetest:
    image: kubeshop/tracetest:${TAG:-latest}
    platform: linux/amd64
    volumes:
      - type: bind
        source: ./tracetest-config.yaml
        target: /app/tracetest.yaml
      - type: bind
        source: ./tracetest.provision.yaml
        target: /app/provisioning.yaml
    ports:
      - 11633:11633
    command: --provisioning-file /app/provisioning.yaml
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      postgres:
        condition: service_healthy
      xray-daemon:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:11633"]
      interval: 1s
      timeout: 3s
      retries: 60

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  xray-daemon:
    image: amazon/aws-xray-daemon:latest
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN}
      AWS_REGION: ${AWS_REGION}
    ports:
      - 2000:2000

Alongside the Docker compose configuration, a .env file is used to provide the AWS credentials to the AWS X-Ray daemon.

AWS_ACCESS_KEY_ID="<your-accessKeyId>"
AWS_SECRET_ACCESS_KEY="<your-secretAccessKey>"
AWS_SESSION_TOKEN="<your-session-token>"
AWS_REGION="us-west-2"

Configuring Tracetest

The only requirement to run the Tracetest server is to have a working Postgres instance and provide the connection details either over a config file and/or environment variables. This looks like the following:

Note: The name of the file should match the volume configuration part of the docker config under the tracetest configuration.

# tracetest-config.yaml

---
postgres:
  host: postgres
  user: postgres
  password: postgres
  port: 5432
  dbname: postgres
  params: sslmode=disable

In this case, we’ll be provisioning the Tracetest server with the AWS X-Ray configuration so we can ensure it's fully configured to use it as a Data Store from startup. For that, the provisioning file should be the following:

Note: The name of the file should match the volume configuration part of the docker config under the tracetest configuration.

# tracetest-provision.yaml

---
type: PollingProfile
spec:
  name: default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: awsxray
  type: awsxray
  awsxray:
    accessKeyId: <your-accessKeyId>
    secretAccessKey: <your-secretAccessKey>
    sessionToken: <your-session-token>
    region: "us-west-2"

Getting the AWS Credentials

If you are familiar with AWS, getting a pair of credentials should be straightforward, but we recommend using temporary credentials to validate this use case. To do so you can run aws sts get-session-token from your terminal to get them.

aws sts get-session-token

[Output]
{
    "Credentials": {
        "AccessKeyId": "<your-accessKeyId>",
        "SecretAccessKey": "<your-secretAccessKey>",
        "SessionToken": "<your-session-token>",
        "Expiration": "2023-03-29T21:35:14+00:00"
    }
}

Running the Use Case

If you are following this and have the example from the repo open you can run docker compose up -d.

Then you can open the Tracetest UI at http://localhost:11633.

And to validate that the provisioning works as expected and Tracetest can communicate with AWS X-Ray, you can go to the /settings page and click the Test Connection button.

Trace-based Testing with Tracetest

Now that we have the infra ready, we can move on to the fun part!

We’ll be using the following test definition for this example:

# tests/test.yaml

type: Test
spec:
  id: 4F1jCHJVR
  name: XRay
  description: XRay
  trigger:
    type: http
    httpRequest:
      url: http://app:3000/http-request
      method: GET
      headers:
        - key: Content-Type
          value: application/json
  specs:
    - selector: span[tracetest.span.type="http"]
      assertions:
        - attr:tracetest.selected_spans.count   =   3
        - attr:tracetest.span.duration   <   1s
    - selector: span[tracetest.span.type="http" name="amazon.com" http.method="GET"]
      assertions:
        - attr:http.url =  "https://amazon.com/"

Note: If you want to know more about how the test definitions work, refer to the Tracetest main docs.

Let’s start by configuring the Tracetest CLI and adding the test into the system:

tracetest configure --endpoint http://localhost:11633 --analytics

Next, create the test:

tracetest test run -d tests/test.yaml
✔ XRay (http://localhost:11633/test/4F1jCHJVR/run/1/test)

By following the link we can find the initial trigger information, describing the HTTP request and the response data.

Moving to the Trace tab, using the top navigation, we can visually validate the tracing information the Node.js Express App generated.

We can find the following:

The Tracetest Trigger generated a span.
The incoming HTTP request.
Two HTTP request spans to amazon.com.

This is reflected in AWS X-Ray where we can see the generated trace.

As good as it is to be able to visualize the full trace information to understand what an application is doing, having an automated way of validating it makes it more efficient.

Thus, we can move to Trace mode by using the top navigation to continue with the final step to create test specs and assertions based on the generated spans.

The test definition we executed includes the following validations:

All HTTP spans have a duration of less than 1 second.
The count of all HTTP spans should be 3.
HTTP outgoing requests should go to http://amazon.com.

As you can see from this use case, Tracetest enables you to use the generated telemetry data from your application, to have deeper and more specific tests targeting every part of the system.

Node.js Instrumented API with AWS X-Ray SDK, ADOT, and Tracetest

Next, we’ll have the same Node.js Express app example, that listens for a GET request in the /http-request route, trigger a request to amazon.com, and returns a response based on the request. But we’ll be doing some changes by introducing the AWS Distro for OpenTelemetry as a middleware for both X-Ray and Tracetest.

Here’s what the demo will showcase:

Instrumenting the Node.js app using the AWS X-Ray SDK.
Sending tracing data to both the AWS Cloud and Tracetest using the ADOT.
Triggering, visualizing, and applying trace-based testing with Tracetest.

To better follow this use case you can download the example from the repo by following the next steps:

git clone https://github.com/kubeshop/tracetest.git
cd examples/tracetest-amazon-x-ray-adot
docker compose up -d

Instrumenting the Node.js Express App

Similar to the previous use case, the first task is to create an instrumented version of the basic Node.js App using the AWS X-Ray SDK, but, in this case, the data should be sent to the ADOT collector instead of the X-Ray daemon instance.

// src/index.js
const AWSXRay = require("aws-xray-sdk");
const XRayExpress = AWSXRay.express;
const express = require("express");
// tracing data sent to the adot collector
AWSXRay.setDaemonAddress("adot-collector:2000");

// Capture all AWS clients we create
const AWS = AWSXRay.captureAWS(require("aws-sdk"));
AWS.config.update({
  region: process.env.AWS_REGION || "us-west-2",
});

// Capture all outgoing https requests
AWSXRay.captureHTTPsGlobal(require("https"));
const https = require("https");

const app = express();
const port = 3000;

app.use(XRayExpress.openSegment("Tracetest"));

app.get("/", (req, res) => {
  const seg = AWSXRay.getSegment();
  const sub = seg.addNewSubsegment("customSubsegment");
  setTimeout(() => {
    sub.close();
    res.sendFile(`${process.cwd()}/index.html`);
  }, 500);
});

app.get("/http-request/", (req, res) => {
  const endpoint = "https://amazon.com/";
  https.get(endpoint, (response) => {
    response.on("data", () => {});

    response.on("error", (err) => {
      res.send(`Encountered error while making HTTPS request: ${err}`);
    });

    response.on("end", () => {
      res.send(`Successfully reached ${endpoint}.`);
    });
  });
});

app.use(XRayExpress.closeSegment());
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

The Docker-Compose Configuration

The Docker configuration will be spinning up all of the required services to run the app, the ADOT collector, and the Tracetest server.

# docker-compose.yaml
version: "3"
services:
  app:
    build: .
    environment:
      AWS_REGION: ${AWS_REGION}
    ports:
      - "3000:3000"

  tracetest:
    image: kubeshop/tracetest:${TAG:-latest}
    platform: linux/amd64
    volumes:
      - type: bind
        source: ./tracetest-config.yaml
        target: /app/tracetest.yaml
      - type: bind
        source: ./tracetest.provision.yaml
        target: /app/provisioning.yaml
    ports:
      - 11633:11633
    command: --provisioning-file /app/provisioning.yaml
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      postgres:
        condition: service_healthy
      adot-collector:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:11633"]
      interval: 1s
      timeout: 3s
      retries: 60

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  adot-collector:
    image: amazon/aws-otel-collector:latest
    command:
      - "--config"
      - "/otel-local-config.yaml"
    volumes:
      - ./collector.config.yaml:/otel-local-config.yaml
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN}
      AWS_REGION: ${AWS_REGION}
    ports:
      - 4317:4317
      - 2000:2000

Alongside the Docker Compose configuration, a .env file is used to provide the AWS credentials to the ADOT collector.

AWS_ACCESS_KEY_ID="<your-accessKeyId>"
AWS_SECRET_ACCESS_KEY="<your-secretAccessKey>"
AWS_SESSION_TOKEN="<your-session-token>"
AWS_REGION="us-west-2"

Configuring Tracetest

# tracetest-config.yaml

---
postgres:
  host: postgres
  user: postgres
  password: postgres
  port: 5432
  dbname: postgres
  params: sslmode=disable

In this case, we’ll be provisioning the Tracetest server with the basic OTLP configuration so we can ensure it's fully configured to use it as a Data Store from startup. For that, the provisioning file should be the following:

# tracetest-provision.yaml
---
type: DataStore
spec:
  name: otlp
  type: otlp

Configuring the ADOT Collector

As the Node.js Express App generates X-Ray format telemetry data, the collector needs to be configured to receive the specific format similar to the Daemon we used for the first use case.

Then, it needs to send telemetry data to both Tracetest and the AWS Cloud, thus the configuration needs to include exporters for both destinations.

Finally, specifying the pipelines for both AWS X-Ray and Tracetest to complete the setup.

# collector.config.yaml

receivers:
  awsxray:
    transport: udp

processors:
  batch:

exporters:
  awsxray:
    region: ${AWS_REGION}
  otlp/tt:
    endpoint: tracetest:21321
    tls:
      insecure: true

service:
  pipelines:
    traces/tt:
      receivers: [awsxray]
      processors: [batch]
      exporters: [otlp/tt]
    traces/xr:
      receivers: [awsxray]
      exporters: [awsxray]

Getting the AWS Credentials

aws sts get-session-token

[Output]
{
    "Credentials": {
        "AccessKeyId": "<your-accessKeyId>",
        "SecretAccessKey": "<your-secretAccessKey>",
        "SessionToken": "<your-session-token>",
        "Expiration": "2023-03-29T21:35:14+00:00"
    }
}

Running the Use Case

If you are following this and have the example from the repo open, you can run docker compose up -d.

Then, open the Tracetest UI at http://localhost:11633

To validate that the provisioning works as expected, go to the /settings page and validate that the OpenTelemetry Data Store is selected.

Trace-based Testing with Tracetest

After finalizing the setup, we are back to the fun part!

We’ll be using a similar test as the previous use case to trigger the API endpoint, fetch the trace from the ADOT, and validate the spans.

# tests/test.yaml

type: Test
spec:
  id: 4F1jCHJVR
  name: XRay
  description: XRay
  trigger:
    type: http
    httpRequest:
      url: http://app:3000/http-request
      method: GET
      headers:
        - key: Content-Type
          value: application/json
  specs:
    - selector: span[tracetest.span.type="http"]
      assertions:
        - attr:tracetest.selected_spans.count   =   3
        - attr:tracetest.span.duration   <   1s
    - selector: span[tracetest.span.type="http" name="amazon.com" http.method="GET"]
      assertions:
        - attr:http.url =  "https://amazon.com/"

Note: If you want to know more about how the test definitions work, refer to the Tracetest main docs.

First, let’s configure the Tracetest CLI:

tracetest configure --endpoint http://localhost:11633 --analytics

Then, we create and run the test:

tracetest test run -d tests/test.yaml
✔ XRay (http://localhost:11633/test/4F1jCHJVR/run/1/test)

By following the link we can find the initial trigger information, describing the HTTP request and the response data.

Moving to the Trace tab, using the top navigation, we can visually validate the tracing information the Node.js Express App generated.

We can find the following:

The Tracetest Trigger generated a span.
The incoming HTTP request.
Two HTTP request spans to amazon.com.

This is reflected in AWS X-Ray where we can see the generated trace.

As good as it is to be able to visualize the full trace information to understand what an application is doing, having an automated way of validating it becomes more efficient.

Thus, we can move to trace mode by using the top navigation to continue with the final step to create test specs and assertions based on the generated spans.

The test definition we executed includes the following validations:

All HTTP spans have a duration of less than 1 second.
The count of all HTTP spans should be 3.
HTTP outgoing requests should go to http://amazon.com.

As you can see in this use case, enabling tracing for your distributed system can be done in multiple ways, allowing you to choose what’s the best match for your team and current setup. Tracetest is easy to integrate with the path that suits you best, without losing the ability to use the generated telemetry data from your application and have deeper and more specific tests targeting every part of the system.

The Pokeshop API with Otel Instrumentation, ADOT, X-Ray, and Tracetest

Now that we have learned the basics to instrument an application with AWS X-Ray and run some trace-based testing using Tracetest, it's time to remove the training wheels and go for something closer to a real-life scenario.

For this use case, we are going to be using Tracetest’s own Pokeshop API which is a fully distributed Node.js application. It showcases some of the hurdles and complexities a distributed system can have. Including:

Asynchronous processes
Cache layers
HTTP/Grpc APIs
External API requests
Database storage

Note: The use case will be focused on testing the import Pokemon async process

In this example, we are sending trace data from the Pokeshop API to the AWS Distribution for Open Telemetry (ADOT), which receives the OpenTelemetry protocol format and then exports it to AWS Cloud as the final destination. Then, Tracetest fetches the full trace directly from AWS X-Ray.

To better follow this use case you can download the example from the repo by following these steps:

git clone https://github.com/kubeshop/tracetest.git
cd examples/tracetest-amazon-x-ray-pokeshop
docker compose up -d

The Pokeshop API

The Pokeshop API is a fully instrumented and distributed application that uses queues, cache layers, databases, and processing workers to simulate a real-life application.

Note: To find out more about the Pokeshop API, you can take a look at the official docs.

The Docker Compose Configuration

The Docker configuration will be spinning off all of the required services to run the Pokeshop API services, the ADOT collector, and the Tracetest server.

version: "3"
services:
  tracetest:
    image: kubeshop/tracetest:${TAG:-latest}
    platform: linux/amd64
    volumes:
      - type: bind
        source: ./tracetest.config.yaml
        target: /app/tracetest.yaml
      - type: bind
        source: ./tracetest.provision.yaml
        target: /app/provisioning.yaml
    ports:
      - 11633:11633
    command: --provisioning-file /app/provisioning.yaml
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      postgres:
        condition: service_healthy
      adot-collector:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:11633"]
      interval: 1s
      timeout: 3s
      retries: 60

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  adot-collector:
    image: amazon/aws-otel-collector:latest
    command:
      - "--config"
      - "/otel-local-config.yaml"
    volumes:
      - ./collector.config.yaml:/otel-local-config.yaml
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN}
      AWS_REGION: ${AWS_REGION}
    ports:
      - 4317:4317
      - 2000:2000

  cache:
    image: redis:6
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 1s
      timeout: 3s
      retries: 60

  queue:
    image: rabbitmq:3.8-management
    restart: unless-stopped
    healthcheck:
      test: rabbitmq-diagnostics -q check_running
      interval: 1s
      timeout: 5s
      retries: 60

  demo-api:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: api
    ports:
      - "8081:8081"
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:8081"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

  demo-worker:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: worker
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

  demo-rpc:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: rpc
    ports:
      - 8082:8082
    healthcheck:
      test: ["CMD", "lsof", "-i", "8082"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

Alongside the Docker Compose configuration, a .env file is used to provide the AWS credentials to the ADOT collector.

AWS_ACCESS_KEY_ID="<your-accessKeyId>"
AWS_SECRET_ACCESS_KEY="<your-secretAccessKey>"
AWS_SESSION_TOKEN="<your-session-token>"
AWS_REGION="us-west-2"

Configuring Tracetest

# tracetest-config.yaml

---
postgres:
  host: postgres
  user: postgres
  password: postgres
  port: 5432
  dbname: postgres
  params: sslmode=disable

In this case, we’ll be provisioning the Tracetest server with the AWS X-Ray configuration so we can ensure it's fully configured to use it as a Data Store from startup. For that the provisioning file should be the following:

# tracetest-provision.yaml

---
type: PollingProfile
spec:
  name: default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: awsxray
  type: awsxray
  awsxray:
    accessKeyId: <your-accessKeyId>
    secretAccessKey: <your-secretAccessKey>
    sessionToken: <your-session-token>
    region: "us-west-2"

Configuring the ADOT Collector

As the Pokeshop API generates OpenTelemetry tracing data, the collector needs to be configured to receive that specific format.

Then, export the telemetry data to the AWS Cloud using the AWS X-Ray format.

Finally, specify the pipelines for AWS X-Ray.

# collector.config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  awsxray:
    region: ${AWS_REGION}

service:
  pipelines:
    traces/xr:
      receivers: [otlp]
      exporters: [awsxray]

Getting the AWS Credentials

If you are familiar with AWS, getting a pair of credentials should be straightforward, but we recommend using temporary credentials to validate this use case. You can run aws sts get-session-token from your terminal to get them.

aws sts get-session-token

[Output]
{
    "Credentials": {
        "AccessKeyId": "<your-accessKeyId>",
        "SecretAccessKey": "<your-secretAccessKey>",
        "SessionToken": "<your-session-token>",
        "Expiration": "2023-03-29T21:35:14+00:00"
    }
}

Running the Use Case

If you are following this and have the example from the repo open, you can run docker compose up -d.

Then, open the Tracetest UI at http://localhost:11633

To validate that the provisioning works as expected and Tracetest can communicate with AWS X-Ray you can go to the /settings page and click the Test Connection button.

Trace-based Testing with Tracetest

Okay, once more into the breach!

In this last use case, we are going to focus on testing the import endpoint, which executes an asynchronous process using a message queue and a task worker. It triggers an external API request to grab the Pokemon information by ID/name.

Validating these types of scenarios with integration or end-to-end tests can be quite difficult, as the immediate response clients get is a 201 Accepted response with no context of what is happening behind the scenes.

In this case, by using trace-based testing we can go further than just validating the initial request/response by also testing the worker process, external requests, and database queries.

Here’s the test definition we are going to be using:

# tests/test.yaml

type: Test
spec:
  id: -ao9stJVg
  name: Pokeshop - Import
  description: Import a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-api:8081/pokemon/import
      method: POST
      headers:
        - key: Content-Type
          value: application/json
      body: '{"id":6}'
  specs:
    - name: Import Pokemon Span Exists
      selector: span[tracetest.span.type="general" name="import pokemon"]
      assertions:
        - attr:tracetest.selected_spans.count = 1
    - name: Uses Correct PokemonId
      selector: span[tracetest.span.type="http" name="HTTP GET pokeapi.pokemon" http.method="GET"]
      assertions:
        - attr:http.url = "https://pokeapi.co/api/v2/pokemon/6"
    - name: Matching db result with the Pokemon Name
      selector: span[tracetest.span.type="general" name="postgres@postgres"]:first
      assertions:
        - attr:aws.xray.metadata.default   contains    "charizard"

The first step as before is to configure the Tracetest CLI:

tracetest configure --endpoint http://localhost:11633 --analytics

Then, we can create and run the test:

tracetest test run -d tests/test.yaml
✔ Pokeshop - Import (http://localhost:11633/test/-ao9stJVg/run/1/test)

Once we have received the confirmation, the test has been successfully created and we can follow the link to find the trigger information.

A POST request is sent to the /pokemon/import endpoint with a JSON body including the Pokemon Id.

And the 201 response from the API echoes the ID that will be imported.

After that, we can move to the Trace tab to visualize the list of spans Tracetest got from the transaction.

There you can see the two different processes divided by the message queue spans, which marks the end of the initial request logic and the start of the worker processing the job.

Now we are ready to move to the Trace section to start looking at the test specs. These are focused on validating that the process was completed as expected beyond the initial request and response. Here’s the list of checks:

Validating if the Import Span exists, picking up the message from the queue.
Using the correct Pokemon ID from the initial request to grab the Pokemon info from the external service.
Checking that the database saves the Pokemon we expect by matching it to the name.

With this last use case, we have demonstrated the advantages that trace-based testing provides for testing distributed systems. These are scenarios where the important part to test is not the initial request and response as we want to look deeper into the processes.

More about trace-based testing cloud-native apps

Well done! You have reached the end of the tutorial! 🎉

We at Tracetest hope that the information provided here is useful for you and your team. Most of us in the team have had situations where working with cloud-native systems became quite complex to debug. We all know about those late-night calls because some random microservice stopped working out of nowhere or the cache layer stopped working and now the database is down because of the number of requests.

With that in mind, a quick recap of this post is:

Tracetest now supports AWS X-Ray!
You set up three apps with three different ways of using X-Ray and Tracetest together, part of our core values is that Tracetest works with what you have.
You saw the power of Trace-Based Testing, by allowing you to test any part of the request, even if it is asynchronous.
Guide you through the three core sections of a Tracetest test run: Trigger, Trace, and Test.

Last, but not least - if would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Observability and Trace-based Testing in AWS with Serverless Framework, OpenSearch & Tracetest

Oscar Reyes — Mon, 19 Dec 2022 18:11:30 +0000

The OpenTelemetry ecosystem has been growing rapidly in the last year, as it solves one of the major pain points when it comes to observability and tracing for distributed systems.

A frequent question for developers wanting to start with OpenTelemetry is how to install the required toolset in their platforms to start instrumenting their applications.

The current technology trend is leaning towards Kubernetes and Serverless applications. In this tutorial, we want to showcase a way to instrument a distributed system that uses AWS Lambda and Serverless.

We’ll start with the basics and show how to set up the required infrastructure that makes it easy to get started, but also to scale. From there we’ll move on to networking, how to instrument a Node.js lambda function, and last, but not least, how you can introduce trace-based testing with the help of Tracetest.

This article will focus on answering three main issues:

How to provide the required AWS infrastructure for OpenTelemetry with OpenSearch.
How to instrument a Lambda using OpenTelemetry.
How to perform end-to-end tests on your instrumented Lambda's endpoints with Tracetest.

What are we building?

We came up with the idea of creating a basic Node.js Pokemon REST API called Pokeshop to help us validate the different scenarios a real-life application would have. The available endpoints are:

Get a list of Pokemon.
Create a Pokemon.
Delete a Pokemon.
Update a Pokemon.
Import by ID - An async process that uses the Pokeshop API, SQS, and a Lambda Worker.
Get Featured - Uses Redis.

After reading this tutorial, you will learn how we added OpenTelemetry instrumentation to the Pokeshop API and deployed it to AWS using the Serverless Framework.

Note: You can always deploy the demo yourself by following the instructions we have provided in our official Pokeshop Repo.

What are we using to build the distributed system?

Development Framework. As was mentioned, we’ll be using the Serverless Framework as the main tool to provision, deploy and update our application code and infrastructure.

Infrastructure. We’ll use AWS services as resources for this tutorial, providing an alternative for teams that are not using Kubernetes or some of its variants.

What does our OpenTelemetry Infrastructure contain?

There are several moving parts we need to configure to set up OpenTelemetry instrumentation.

VPC

To simplify the networking configuration, we’ll be using the Serverless VPC Plugin which provides an easy way to create the networking infrastructure for public and private services. It also generates security groups, bastion instances, and NAT gateways that we will be leveraging to interconnect the different AWS resources.

Amazon OpenSearch Domain

AWS provides a simple way to generate OpenSearch Domain instances. We’ll be using it as a trace data store and it will be a secure endpoint behind the VPC.

ECS Side Cart

A simple option to deploy and run containers in AWS is ECS. It lets you manage your cloud infrastructure by first connecting to instances, then by defining services and task definitions to execute tasks.

We’ll use ECS for the side cart services.

We are going to be using the ADOT collector which provides an AWS containerized version of the collector ready to be used. The ADOT collector will function as the main gateway to gather tracing information from the Pokeshop API Lambdas.

OpenTelemetry Tracing Data Flow

The final destination should be an OpenSearch domain that will be the store for all of our trace data.

When it comes to using OpenSearch as a store for OpenTelemetry data, there’s a particular caveat that needs to be addressed before it can be added to the stack. OpenSearch doesn’t automatically support the OpenTelemetry protocol so it requires a proxy service called Data Prepper to be used as a processing proxy to translate the data coming from the ADOT collector into a format that OpenSearch can use.

Tracetest Trace-based Testing Data Flow

The Tracetest ECS Service will fetch trace data from OpenSearch when you run a test against a lambda function endpoint. Check the task-definition.yaml to see how to configure Tracetest, ADOT, and the Data Prepper.

For more clarity, check out the entire config folder in the GitHub repo and the tracetest-config.yaml to see that Tracetest will fetch trace data from the OpenSearch instance.

To better visualize the data flow you can refer to the following diagram:

Pokeshop API AWS Resources

Lastly, we need to provide the required infrastructure used by the main application. The Pokeshop API uses the following external services to emulate a more complex distributed system:

RDS (Postgres). Main storage for the Pokeshop API.
Elasticache (Redis). In memory storage for featured Pokemons.
SQS. The async messaging queue for Pokemon imports.
S3. For Pokemon image upload.

After that, everything should be ready for us to start adding the code implementation for the services and the instrumentation.

Using Serverless to Deploy the Infrastructure

Serverless makes it possible to use custom Cloudformation Templates to provision the AWS resources we need. The goal is to have a single command to deploy all of the required infrastructure so you can start playing with the demo by yourself with minimal friction.

First, let’s have a look at the serverless.yaml to see the entire layout here. We’ll walk through the different sections and what they are used for.

You can find all of the definitions for the AWS resources within the /serverless/infra folder in the Pokeshop repo.

Here’s what the files looks like:

And, how we load them into the serverless.yaml as resources:

Note: If you are following the Pokeshop repo README you’ll find a step-by-step tutorial on how to deploy this demo yourself.

OpenTelemetry Instrumentation for AWS Lambda

Once we have defined the required resources and have everything ready, we can start working on the Pokeshop services and how to instrument them.

For simplicity, we are going to go over the creation of the get all pokemon service, which is going to return the total count plus the first 20 results from the database.

With Prisma we will generate the client based on the schema definition file. Then, use it to communicate with the database.

We’ll do this by executing the following command:

npx prisma generate

Once that’s done, we can proceed by adding the handler function where we can use the ORM client to fetch both the number of items and the list.

import { PromiseHandler } from '@lambda-middleware/utils';
import { prisma } from '../utils/db';
const get: PromiseHandler = async ({ queryStringParameters }, context) => {
  context.callbackWaitsForEmptyEventLoop = false;
  const { skip = '0', take = '20' } = queryStringParameters || {};
  const query = { skip: +skip, take: +take };
  const [items, totalCount] = await Promise.all([prisma.pokemon.findMany(query), prisma.pokemon.count()]);
  return {
    totalCount,
    items,
  };
};
export default get;

Having that ready, we can focus on integrating the instrumentation for our lambda functions. Start by creating a JavaScript named tracer.js that will be preloaded before the handler functions for each of the services.

We will be adding the base implementation for the Node provider, the creation of the GRPC exporter, the propagation configuration, and the auto instrumentation for Node.js and Prisma.

const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { PrismaInstrumentation } = require('@prisma/instrumentation');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { B3Propagator, B3InjectEncoding } = require('@opentelemetry/propagator-b3');
const api = require('@opentelemetry/api');
api.diag.setLogger(new api.DiagConsoleLogger(), api.DiagLogLevel.ALL);
api.propagation.setGlobalPropagator(new B3Propagator({ injectEncoding: B3InjectEncoding.MULTI_HEADER }));

const collectorOptions = {
  url: process.env.COLLECTOR_ENDPOINT,
};

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'pokeshop-api',
  }),
});
const exporter = new OTLPTraceExporter(collectorOptions);
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-aws-lambda': {
        disableAwsContextPropagation: true,
      },
    }),
    new PrismaInstrumentation(),
  ],
});
provider.register();

['SIGINT', 'SIGTERM'].forEach(signal => {
  process.on(signal, () => provider.shutdown().catch(console.error));
});

This file will be included as part of the bundle by configuring the package options in the serverless.yaml files.

To avoid including unwanted files and make sure the bundle is below the maximum size, we can decide which modules to include.

Your serverless.yaml should look like this:

package:
  patterns:
    - src/**
    - "!.vscode/**"
    - "!infra/**"
    - "!node_modules/typescript/**"
    - "!node_modules/aws-sdk/**"
    - "!node_modules/serverless-offline/**"
    - "!node_modules/.prisma/client/libquery_engine-*"
    - "node_modules/.prisma/client/libquery_engine-rhel-*"
    - "!node_modules/prisma/libquery_engine-*"
    - "!node_modules/@prisma/engines/**"

Next, we need to configure environment variables for our lambda functions to be able to connect to our different AWS resources. To avoid hardcoding any configuration we can use the dynamic refs that are available as part of the Cloudformation Template System.

environment:
  NODE_OPTIONS: --require ./src/utils/tracer.js
  COLLECTOR_ENDPOINT:
    Fn::Join:
      - ""
      - - "http://"
        - { Ref: ECSEIP }
        - ":4317"
  DATABASE_URL:
    Fn::Join:
      - ""
      - - "postgres://"
        - ${self:custom.databaseUsername}
        - ":"
        - ${self:custom.databasePassword}
        - "@"
        - ${self:custom.databaseEndpoint}
        - "/"
        - ${self:custom.databaseName}
        - "?schema=public"
  SQS_QUEUE_URL: { Ref: AssetsQueue }
  S3_ARN: { Ref: ImageBucket }
  POKE_API_BASE_URL: "https://pokeapi.co/api/v2"
  REDIS_URL:
    Fn::GetAtt: [PokeCache, "RedisEndpoint.Address"]

The first entry for NODE_OPTIONS specifies that our new tracer.js file will be required before any handler function. That way we can be sure the OpenTelemetry instrumentation configuration is always loaded first and available from the handler functions in case we want to add manual instrumentation.

Note: You can find the full serverless.yaml file in the Pokeshop repo.

Once we have the handler function, the tracer logic, and the serverless.yaml file ready. We can run the initial serverless deploy command that will start packaging the Node.js bundle and trigger the initial creation of all AWS resources.

sls deploy

Note: If you want to learn more about how to instrument a Serverless Lambda Function you can head to the OpenTelemetry official documentation.

OpenSearch Configuration

In this section, we’ll configure OpenSearch as our trace data store.

ADOT Collector configuration

The Docker version of the collector provides a way to inject the configuration from an environment variable, which matches a regular configuration for the OpenTelemetry collector.

We’ll be using SSM to store the variable and to be injected directly into the container during execution.

The collector should be pointing to the same EC2 instance we’ll be using for the data prepper. You can see this in the task-definition.yaml.

Data Prepper Configuration

At the time I’m writing this post, the Docker image for Data Prepper doesn’t support a way to inject the config through environment variables, so, in this case, we’ll need to use the Docker volume where we’ll be dropping the config files using SSH.

The Data Prepper config should point to the OpenSearch VPC endpoint which will be secured behind a private subnet.

AWS RDS Database Setup

Besides the resource creation, there is one more thing that needs to be done to prepare the database schema for the API queries. To achieve this, we will be using an SSH tunnel that binds the RDS Postgres port to your local machine so we can run the Prisma migrations against it.

Once we have the infrastructure ready to use, we can start executing some validation checks. In order to do this we have to gather some information first. The list of things we need and how to get them is the following:

Serverless API endpoints. You will find the endpoints as outputs from the serverless deploy command.
ECS Instance associated Public Elastic IP. You can find it as part of the outputs from the AWS Cloudformation stack.
OpenSearch VPC endpoint. This can also be found as an output from the stack.

Next, let’s create an SSH tunnel from your local machine to the OpenSearch secure endpoint. We are going to be using the created SSH key to achieve this, the command should look like this:

ssh -i "pokeshop-api.pem" -N -L 9200:<opensearch-vpc-endpoint>:443 ec2-user@<elastic-ip>

We can now access the localhost port to reach the OpenSearch instance. It should be accessible at localhost:9200/_dashboards. There you’ll find yourself looking at a login form that looks like this:

Add the username and password from the serverless.yaml file and you will have access to the dashboard.

Open the Trace Analytics section and you’ll find all of the data produced by the Serverless API endpoints.

Test the OpenTelemetry Instrumentation in your Infrastructure

Triggering one of the AWS Lambda functions will generate trace data.

The whole process consists of sending trace data to the OpenTelemetry collector. From there, it continues to the Data Prepper, and finally to OpenSearch where the trace data is stored.

Let’s run a test against the POST endpoint to create a new pokemon with CURL.

curl --location --request GET 'https://d0lith9uac.execute-api.us-east-1.amazonaws.com/pokemon'

Note: AWS Lambda cold start can be a little bit slow but it should speed up after the initial 3x requests.

After doing all of this, you can come back to the OpenSearch dashboard and find the tracing information.

Congratulations! You have successfully provided the required AWS infrastructure and instrumented your first AWS Lambda function using Serverless.

You can now tell your peers how great and important tracing is!

But, it doesn’t end here. In the next section, we’ll go into detail about trace-based testing and how it is a crucial component in your infrastructure.

Trace-based Testing

Testing large distributed systems and multiple microservices is hard. You need to understand how the whole system works, including how all microservices interconnect.

With tracing you get a map of all services which makes your whole system easier to understand. But, let’s take it a step further. With trace-based testing, you can enrich your end-to-end and integration tests and leverage trace data to improve assertion capabilities.

Now, you can set test specs and assertions against every single trace data point in a transaction, not just the response. Pretty cool, right?

That’s where Tracetest comes into play.

What is Tracetest?

Tracetest is the next-generation, open-source project for running end-to-end, system, and integration tests for distributed systems, powered by OpenTelemetry traces. It uses the observability you enabled by implementing a distributed tracing solution to empower a totally new, modern way of testing.

With Tracetest, you can create tests and transactions based on the common request patterns or pain points of your application. You can also create assertions based on the different attributes that are part of the spans generated by your application instrumentation.

Do you have an internal GRPc that you want to start validating? We have that covered.
Do you want to enable trace-based testing to the booking funnel of your application? That is possible with Tracetest.
Do you want to run trace-based validations as part of your CI/CD process? Done!

How to Use Tracetest

Tracetest will work alongside whatever instrumentation setup you have. We’ve used AWS and Serverless, but you can achieve the same results by using any type of infrastructure.

Tracetest supports triggers for:

HTTP
GRPc
CURL Commands
Postman Collections/Environments

There are two ways of using Tracetest.

First, you can use the visual interface. The GUI allows users to create different assertions to validate trace data. It empowers users with a more visual representation for traces, displaying different graphical views and shortcuts for easy test spec creation.

Second, you can use the Tracetest CLI. With the CLI you can add trace-based tests to your CI/CD pipelines. Now, you can run integration tests with test specs that assert against trace data from every single part of the transaction. You’re not just limited to unit tests and boring response-only integration tests anymore. Welcome to the future!

How to Create Trace-based Tests with Tracetest

For this, you need to get the ECS instance associated with the Public Elastic IP. You can find it as part of the outputs from the AWS CloudFormation stack. You can now access Tracetest through the ECS instance’s associated Elastic IP. Tracetest uses port number 11633.

In our infrastructure, it would be this link: ecs-domain.us-west-1.aws.com:11633.

Check your infrastructure configuration and open up your Tracetest instance in a browser.

Create a new test and run a quick validation check. Copy-paste the same endpoint we used above with CURL.

The first thing you’ll see after creating a test will be the initial service request and response, where you can validate if the endpoint you are using is accessible and if the setup is correct.

If everything is working as expected, the test run will change to a completed state. Now you can navigate to the test tab and start adding assertions based on the generated trace spans. Check out the Tracetest docs for more information about adding assertions.

Note: If you are using the automated script provided as part of the Pokeshop Serverless README Tracetest will be automatically deployed and configured.

Conclusion

OpenTelemetry is a great tool to better understand the internal processes of a microservices application. It outlines different techniques that are used to provide better insight into what is happening within the system, the different asynchronous processes, and how the data flows.

And as every team might be looking to start using it, coming from different backgrounds and using different stacks, we at Tracetest want to provide a way for teams that are using AWS and the Serverless Framework to start that process.

At Tracetest, we are committed to the community and we are always looking for ways to help and contribute to improving OpenTelemetry adoption by providing tutorials and assisting teams. If you have any questions or you want to reach out you can always find us in our official Discord channel. Feel free to give us a star on GitHub if you like Tracetest!

Thank you for reading this tutorial and happy coding!

Creating a Custom Language Code Editor Using React

Oscar Reyes — Fri, 02 Dec 2022 20:31:06 +0000

In this post, we will showcase how we at Tracetest built a custom front-end non-simple code editor based on React for the advanced query language created by the team to target spans across an OTEL Trace.

Introduction to Our Advanced Selector Query Language

Tracetest allows users to create assertions targeting specific spans from a trace simply and reliably.
A Trace can be composed by N number of spans that, at the same time, can be nested, repeated, or split into multiple queue producer/consumer sections.

Having a simple way to target the spans that matter for your assertions is instrumental to have good and reliable tests that will help you sleep at night.

To achieve this, Tracetest uses a custom span selector language to target spans. It inherits part of its syntax highlighting from CSS by using attribute matches, pseudo selectors, and child-parent operators.

This query language enables users to create the selectors the Tracetest system uses to apply the different assertions. Each section of the selector is composed of the following tokens:

Span Wrapper. To start describing a query, a span wrapper is the first thing that needs to be added.

Example: span[].

Attribute Matchers. Inside the span wrapper, a list of expressions can be added to narrow down the specific span that the user wants to match.

Example: name="get request" tracetest.span.type = "database".

Pseudo Selectors. Sometimes you’ll need to be more specific around what span to choose from a collection and a pseudo selector similar to CSS can help with that.

Example: span[]:first.

Span Operators. Currently, the query language we’ve built supports the child-parent (ancestor) operator and Or operator. You can write and combine both to match multiple spans or select one that is a child of a specific one.

Example: span[name="get request"] span[tracetest.span.type = "database"], span[http.status_code contains "200"]

Advanced Examples:

// selects all HTTP spans which status code is 200
span[tracetest.span.type = "http" http.status_code = "200"],

// selects the first database span which database name is pokeshop
span[tracetest.span.type = "database" db.name = "pokeshop"]:first,

// selects the second child of a span with name "POST /pokemon/import"
// which name is "validate request"
span[name = "POST /pokemon/import"] span[name = "validate request"]:nth_child(2)

For more information about advanced selectors and the query language, visit the Tracetest docs.

We Needed a Code Editor in Our UI

There are two main ways to use Tracetest.

The first is by using the CLI that can be fed by YAML text files allowing CI/CD processes and automation.

The second is by using the UI to create tests, execute requests and create assertions/checks. Tests created directly in YAML can be loaded into the UI and vice versa.

When using the YAML text files, creating advanced queries is pretty straightforward as you can define the different selectors as text to trigger the process.

The complexity comes when trying to implement the same level of functionality the language provides in the UI as there can be many branches, nested selectors, operators, etc.

The initial UI implementation only supported a single span wrapper with its attribute expressions and a separated input for pseudo selectors.

Another important issue is that if a user creates a selector that uses any of the language features that are not supported by the UI, then, when viewing it in the UI, the selector won't match the original text version. Trying to edit it will break the selector in most of the cases.

Creating the Non-simple Code Editor in React

At Tracetest, we knew we needed to let users build complex queries using some advanced query method in the near future. And with that in mind, we started thinking about the key features that we wanted to implement with the advanced editor.
They are:

Syntax Highlighting
Autocomplete
Lint and Error Prompt

In order to achieve this, we started researching potential solutions and found Code Mirror.

Code Mirror is an all-in-one code editor for the web that can be extended to support multiple standard languages and themes like Javascript, Go, Ruby, etc. It also provides a simple way to build your own custom parsers and themes.

And that’s just what we needed in order to accomplish the key features - to come up with our own parser. For that, we used Lezer, which is a tool to create custom grammar rules by specifying the different tokens and expressions.

As you learned at the beginning of this post, we defined the query language tokens and, based on that, we started to mix them to create our expressions.

Things like:

BaseExpression { Identifier Operator ComparatorValue }
// or
SpanOrMatch { expression Comma !list expression }

Once we had the grammar ready, we could finalize the language setup using Typescript.

import {LRLanguage, LanguageSupport} from '@codemirror/language';
import {styleTags, tags as t} from '@lezer/highlight';
import {parser} from './grammar';

export const tracetestLang = LRLanguage.define({
 parser: parser.configure({
   props: [
     styleTags({
       Identifier: t.keyword,
       String: t.string,
       Operator: t.operatorKeyword,
       Number: t.number,
       Span: t.tagName,
       ClosingBracket: t.tagName,
       Comma: t.operatorKeyword,
       PseudoSelector: t.operatorKeyword,
     }),
   ],
 }),
});

export const tracetest = () => {
 return new LanguageSupport(tracetestLang);
};

Next, we had to come up with a way to support Autocomplete and Linting.

The Code Mirror framework includes two separate packages to handle Autocomplete and Linting, where you can add the different rules that, combined with the parser, can enable these features.

We created two custom hooks to handle this functionality that can be found here.

When complete, the editor React component looked like this:

import {useMemo} from 'react';
import {noop} from 'lodash';
import CodeMirror from '@uiw/react-codemirror';
import {autocompletion} from '@codemirror/autocomplete';
import {tracetest} from 'utils/grammar';
import {linter} from '@codemirror/lint';
import useAutoComplete from './hooks/useAutoComplete';
import useLint from './hooks/useLint';
import useEditorTheme from './hooks/useEditorTheme';
import * as S from './AdvancedEditor.styled';

interface IProps {
 testId: string;
 runId: string;
 value?: string;
 onChange?(value: string): void;
}

const AdvancedEditor = ({testId, runId, onChange = noop, value = ''}: IProps) => {
 const completionFn = useAutoComplete({testId, runId});
 const lintFn = useLint({testId, runId});
 const editorTheme = useEditorTheme();

 const extensionList = useMemo(
   () => [autocompletion({override: [completionFn]}), linter(lintFn), tracetest()],
   [completionFn, lintFn]
 );

 return (
   <S.AdvancedEditor>
     <CodeMirror
       data-cy="advanced-selector"
       value={value}
       maxHeight="120px"
       extensions={extensionList}
       onChange={onChange}
       spellCheck={false}
       autoFocus
       theme={editorTheme}
     />
   </S.AdvancedEditor>
 );
};

export default AdvancedEditor;

Final Code Editor Look, Feel and Functionality

From the UI perspective, this was the final result:

https://www.youtube.com/watch?v=tmpZeA3zJXE

The video showcases the final version of the advance editor. It includes a demo of the main features and how the React application switches the state of the Diagram when updating the code within the editor.

It also shows the invalid query state error message which validates if the input is valid before triggering the request to the backend.

At Tracetest, we always try to provide you the best user experience. No matter the platform (UI/CLI) we want to ensure you have the tools to interact with the system in an easy way.

Having said that, if you have any comments or suggestions feel free to join our Discord Community - we are always looking for feedback that can help improve Tracetest in any way.

Using OpenTelemetry tracing (you should be!) and want to give Tracetest a try? Download Tracetest from Github (MIT Licensed) and experience the latest in trace-based testing!

Frontend Overhaul of OTel Demo: Go to Next.js

Oscar Reyes — Fri, 02 Dec 2022 19:20:02 +0000

Frontend Overhaul of the OpenTelemetry Demo (Go to Next.js)

One of the OpenTelemetry Project's many Special Interest Groups (SIG) is the OpenTelemetry Community Demo SIG which gives support to a set of instrumented backend microservices and a web frontend app that are primarily used to showcase how to instrument a distributed system using OpenTelemetry.

The application's main focus is to demonstrate the implementation process to instrument an application no matter what programming language, platform, or operating system your team is using, as well as providing different approaching techniques (automatic and manual instrumentation, metrics, baggage). All of this while following the standards and conventions defined by the official OpenTelemetry Documentation. More about the specific requirements can be found here.

At Tracetest, we have always focused on becoming part of and embracing the OpenTelemetry community. One of our goals this summer was to get more involved with a core OpenTelemetry project where we could provide a meaningful contribution. The OTel demo became the best match for achieving that goal as it would not only help the community, but we at Tracetest needed a good example to test and showcase what can be done with our tool.

During the version 0.7 project cycle, we created two specific tickets to get us closer to the community and start looking for things to pick up:

The first thing we did was to get in contact with Carter Socha, the organizer of the OTel Demo SIG. Carter was really welcoming and helped us identify where our contributions could be the most impactful. We started looking at the issue created by Austin Parker referencing a complete front-end overhaul that would involve moving the application away from Go (SSR) to an architecture that included a browser-side client (CSR), as well as improving the overall style, theme, and user experience. A fun aspect of the work was the request to move the store from a ‘normal’ store to an astronomy store to match the OpenTelemetry project’s overall branding.

Once we got confirmation and the green light from the rest of the OTel Demo SIG team the Tracetest team started working on the different changes included as part of the application frontend architecture overhaul.

OpenTelemetry Demo Application Description and Tech Stack

The demo app is an astronomy webstore that has the basic functionality to purchase online products such as a shopping cart, currency selector, product listing, and payment & checkout. It also includes features to display promotional items (ads) and related products depending on the context of the user.

The demo stack includes a set of multiple microservices built in different languages, one for each of the following programming languages:

Go
PHP
C++
.NET
Ruby
Python
Node.js
Rust
Elixir

Every microservice has a specific goal and can communicate with others by using a global GRPC definition. Persistent information is saved into a Postgres database and there are outbound services that connect with third-party services to trigger events (such as confirmation emails).
All of the microservices, including the front end, are connected to the same Open Telemetry collector instance, which uses Jaeger as one of the data stores for the traces and spans.

The front end was constituted by a Golang SSR application which sent the complete HTML to the browser client to be displayed. Each request and form call was redirecting the user back to the server so the new piece of information was shown.

Web App Styling Improvements, Theme Updates, and User Experience Redesign

Before the Tracetest team got involved in the development process, the frontend application wasn’t matching the theme that OpenTelemetry had been using in terms of colors, products, and overall user experience. In addition, the demo lacked a real frontend (browser side) application as the current implementation was a Go SSR application.

The first task at hand was to bring the demo to the modern age by updating the design, color schemes, and user experience. Our UX designer Olly walked into the fray to help us achieve this by creating a modernized version of the application. It included an improved way to display the products landing page, an updated product details page, a mini cart, and a fully compatible mobile version of the application.

Now we had an application design that would match the rest of the OpenTelemetry themes and colors and look more like the OpenTelemetry.io website.

Frontend Application Architecture Overhaul

The Tracetest team worked on an initial proposal that included the following bullet points:

Framework and tooling (Scaffolding, I/O, styling, UI library)
Code Architecture and structure (Directories, coding patterns)
Instrumentation
Deployment & Distribution
Testing (E2E, unit test) This proposal was presented to the OpenTelemetry Demo SIG during one of the weekly Monday meetings and we were given the green light to move ahead. As part of the changes, we decided to use Next.js to not only work as the primary frontend application but also to work as an aggregation layer between the frontend and the GRPC backend services.

As you can see in the Diagram, the application has two major connectivity points, one coming from the browser side (REST) to connect to the Next.js aggregation layer and the other from the aggregation layer to the backend services (GRPC).

OpenTelemetry Instrumentation

The next big thing we worked on was having a way to instrument both sides of the Next.js application. To accommodate this we had to connect the same application twice to the same open telemetry collector that was being used by all of the other microservices.
For the backend side, a simple solution was designed that involved using the official GRPC exporter in combination with the Nodejs SDK. You can find the full implementation here.
The basic instrumentation includes auto instrumentation for most of the commonly used libraries and tools for Node.js. But as part of having a better example for users, a manual instrumentation piece was added in the form of a route middleware that would catch the incoming HTTP request and create a span based on it, including the context propagation. The implementation can be found here.
For the frontend side, it became a little bit tricky, as the initial Next.js render comes from the server side so, in this case, we had to make sure to load the tracer from the browser side when the Javascript code is executed.

After adding validations to check for the browser side, we then can proceed to load the custom front-end tracing module which includes the creation of the web tracer provider and the automatic web instrumentations.
The automatic web instrumentations capture the most common user interactions such as click events, fetch requests, and page loads.
Another takeaway is that in order to allow the browser side to interact with the OTel collector there is a configuration change that needs to be done to enable incoming CORS requests from the web app. Having the collector receivers configuration looking similar to this:

Once the setup is complete, by loading the application from docker and interacting with the different features, we can start looking at the full traces that begin from the frontend user events all the way to the backend GRPC services.

Bringing the OTel Demo to Tracetest

The latest version of Tracetest includes the complete setup to have an instance of the OTel demo running side by side with Tracetest.
You can start the demo by simply cloning the Tracetest repository and, from the root folder, execute the following command:
docker compose -f ./examples/tracetest-otel-demo/docker-compose.yaml up
This will trigger the dockerized version of the app plus all of the services required for the OTel demo microservices.
After going through the test creation process on Tracetest and running the initial transaction by using one of the provided examples, we can see the full trace coming back.

Contributing to OpenTelemetry was Rewarding!

As a team focused on building an open source tool in the Observability space, the opportunity to help the overall OpenTelemetry community was important to us. In addition, having a complex microservice-based application that uses multiple different languages and technologies is directly useful for our team. It helps us when developing Tracetest and when showing the capabilities of Tracetest to the world.
P.S. Have any questions about Tracetest? You can add issues on Github or join us on our Discord channel to discuss the future of Trace-Based Testing with Tracetest.