This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to the k8S troubleshooting guide.

this tutorial will also utilize the OpenTelemetry Operator with:

• the OpenTelemetry Demo
• the hipster-shop
• istio gitg
• All the observability data generated by the environment would be sent to Dynatrace.

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

ZONE=europe-west3-a
NAME=isitobservable-troubleshooting
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

1. Download Istioctl

curl -L https://istio.io/downloadIstio | sh -

This command download the latest version of istio compatible with our operating system. 2. Add istioctl to you PATH

cd istio-1.22.0

this directory contains samples with addons . We will refer to it later.

export PATH=$PWD/bin:$PATH

git clone https://github.com/isitobservable/K8STroubleShoting
cd K8STroubleShoting

The application will deploy the entire environment:

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

in the Dynatrace folder , you will find the json file containing the dashboard for the API server. To deploy it click , on dashboard

Similar to the previous step, upload dynatrace/kubelet & Kubeproxy.json

you will find a workflow sending messages to the team with details on why their workload are failing. This Worflow would require to reconfigure the slack integraiton and to install the Slack application from the Hub.

To import the workflow you will need to click on worflow .

Click on Upload to import the following file: ddynatrace/wftpl_troubeshooting_k8s_workload.yaml