Add external logs provider support

LyndseyBu · Aug 6, 2020 · 3dbe0c9 · 3dbe0c9
1 parent 7e75e2e
commit 3dbe0c9
Show file tree

Hide file tree

Showing 20 changed files with 508 additions and 46 deletions.
diff --git a/cmd/dashboard/main.go b/cmd/dashboard/main.go
@@ -54,6 +54,7 @@ var (
 	logLevel           = flag.String("log-level", "info", "Minimum log level output by the logger")
 	logFormat          = flag.String("log-format", "json", "Format for log output (json or console)")
 	streamLogs         = flag.Bool("stream-logs", false, "Enable log streaming instead of polling")
+	externalLogs       = flag.String("external-logs", "", "External logs provider url")
 )
 
 func getCSRFAuthKey() []byte {
@@ -143,6 +144,7 @@ func main() {
 		IsOpenShift:        *isOpenshift,
 		LogoutURL:          *logoutUrl,
 		StreamLogs:         *streamLogs,
+		ExternalLogsURL:    *externalLogs,
 	}
 
 	resource := endpoints.Resource{

diff --git a/docs/dev/installer.md b/docs/dev/installer.md
@@ -63,6 +63,7 @@ Accepted options:
         [--ingress-url <url>]                   Will create an additional ingress with the specified url
         [--ingress-secret <secret>]             Will add ssl support to the ingress
         [--stream-logs]                         Will enable log streaming instead of polling
+        [--external-logs]                       External url to fetch logs from when not available in the cluster
         [--output <file>]                       Will output built manifests in the file instead of in the console
 ```
 

diff --git a/docs/walkthrough/README.md b/docs/walkthrough/README.md
@@ -5,6 +5,7 @@ We have the following walk-throughs available:
 
 - [Setup everything from scratch using a kind cluster](./walkthrough-kind.md)
 - [Securing access to the Dashboard behind an oauth2 proxy](./walkthrough-oauth2-proxy.md)
+- [Enabling logs persistence](./walkthrough-logs.md)
 
 ---
 

diff --git a/docs/walkthrough/walkthrough-logs-collected.png b/docs/walkthrough/walkthrough-logs-collected.png
diff --git a/docs/walkthrough/walkthrough-logs-logs.png b/docs/walkthrough/walkthrough-logs-logs.png
diff --git a/docs/walkthrough/walkthrough-logs-minio.png b/docs/walkthrough/walkthrough-logs-minio.png
diff --git a/docs/walkthrough/walkthrough-logs-nologs.png b/docs/walkthrough/walkthrough-logs-nologs.png
diff --git a/docs/walkthrough/walkthrough-logs.md b/docs/walkthrough/walkthrough-logs.md
@@ -0,0 +1,353 @@
+# Tekton Dashboard walkthrough - Logs persistence
+
+This guide walks you through installing a working Tekton Dashboard locally from scratch, collecting logs into an object storage solution like AWS S3, exposing those logs through a service and setting up the Tekton Dashboard to fallback on this service when the logs have been garbage collected by the cluster. It covers the following topics:
+
+* [Before you begin](#before-you-begin)
+* [Overview](#overview)
+* [Installing a working Tekton Dashboard locally from scratch](#installing-a-working-tekton-dashboard-locally-from-scratch)
+* [Installing minio as an object storage solution](#installing-minio-as-an-object-storage-solution)
+* [Collecting TaskRuns pod logs](#collecting-taskruns-pod-logs)
+* [Creating a service to serve collected logs](#creating-a-service-to-serve-collected-logs)
+* [Setting up the Dashboard logs fallback](#setting-up-the-dashboard-logs-fallback)
+* [Cleaning up](#cleaning-up)
+
+## Before you begin
+
+Before you begin, make sure the following tools are installed:
+
+1. [`kind`](https://kind.sigs.k8s.io/): For creating a local cluster running on top of docker.
+1. [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/): For interacting with your kubernetes cluster.
+1. [`helm`](https://helm.sh/docs/intro/install/): For installing helm charts in your kubernetes cluster.
+
+## Overview
+
+In this walkthrough you will deploy [minio](https://min.io), an object storage solution that can be accessed like AWS S3 or other cloud providers APIs.
+
+You will use this object storage to store pod logs collected with [banzaicloud logging operator](https://github.com/banzaicloud/logging-operator) from your `TaskRun`s.
+
+Then, you will create a service to serve those logs and will plug the Tekton Dashboard to fallback to this service when the logs you're trying to view have been garbage collected and are not available anymore from the kubernetes API.
+
+## Installing a working Tekton Dashboard locally from scratch
+
+If you didn't follow the [Tekton Dashboard walkthrough with Kind](./walkthrough-kind.md) yet, you should start there to get a local cluster with a working Tekton Dashboard installed.
+
+The following steps will focus on collecting, storing and serving pod logs to finally plug the logs service on the Tekton Dashboard.
+
+## Installing minio as an object storage solution
+
+First, you will need to deploy a storage solution to store the collected logs. For that, you're going to deploy [minio](https://min.io), it exposes the same API as AWS S3.
+
+Note that `minio` exposes other APIs similar to other cloud storage providers too.
+
+To deploy `minio` in your cluster, run the following command to install the `minio` helm chart:
+
+```bash
+cat <<EOF | helm upgrade --install --wait --create-namespace --namespace tools minio stable/minio --values -
+nameOverride: minio
+fullnameOverride: minio
+
+persistence:
+  enabled: false
+
+accessKey: "ACCESSKEY"
+secretKey: "SECRETKEY"
+
+ingress:
+  enabled: true
+  hosts:
+    - minio.127.0.0.1.nip.io
+
+resources:
+  requests:
+    memory: 100M
+EOF
+```
+
+The deployed instance will use hard coded access and secret keys `ACCESSKEY` / `SECRETKEY` and the service will be exposed externally at `http://minio.127.0.0.1.nip.io`.
+
+For this walkthrough `minio` will use in-memory storage but you can enable persistent storage by changing the config.
+
+The `minio` dashboard is available at the URL `http://minio.127.0.0.1.nip.io`.
+
+![Minio browser](./walkthrough-logs-minio.png)
+
+## Collecting TaskRuns pod logs
+
+Now you have a running storage, you can start collecting logs from the pods baking your `TaskRun`s and store those logs in `minio`.
+
+To collect logs, you will install [banzaicloud logging operator](https://github.com/banzaicloud/logging-operator). The `logging operator` makes it easy to deploy a fluentd/fluentbit combo for streaming logs to a destination of your choice.
+
+First, deploy the logging operator by running the following commands:
+
+```bash
+helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
+
+helm repo update
+
+helm upgrade --install --wait --create-namespace --namespace tools logging-operator banzaicloud-stable/logging-operator --set createCustomResource=false
+```
+
+The `logging operator` should now be deployed in your cluster.
+
+To start collecting logs you will need to create the logs pipeline using the available CRDs:
+
+- `Logging` will deploy the necessary fluentd/fluentbit workloads:
+
+```bash
+cat <<EOF | kubectl -n tools apply -f -
+apiVersion: logging.banzaicloud.io/v1beta1
+kind: Logging
+metadata:
+  name: logging
+spec:
+  fluentd: {}
+  fluentbit: {}
+  controlNamespace: tools
+EOF
+```
+
+This is a very simple deployment, please note that the position database and buffer volumes are ephemeral, this will stream logs again if pods restart.
+
+- `ClusterOutput` defines the output of the logs pipeline. In our case AWS S3 (through `minio`):
+
+```bash
+cat <<EOF | kubectl -n tools apply -f -
+apiVersion: logging.banzaicloud.io/v1beta1
+kind: ClusterOutput
+metadata:
+  name: s3
+spec:
+  s3:
+    aws_key_id:
+      value: ACCESSKEY
+    aws_sec_key:
+      value: SECRETKEY
+    s3_endpoint: http://minio.tools.svc.cluster.local:9000
+    s3_bucket: tekton-logs
+    s3_region: tekton
+    force_path_style: 'true'
+    store_as: text
+    path: \${\$.kubernetes.namespace_name}/\${\$.kubernetes.pod_name}/\${\$.kubernetes.container_name}/
+    s3_object_key_format: '%{path}%{time_slice}_%{index}.log'
+    buffer:
+      tags: time,\$.kubernetes.namespace_name,\$.kubernetes.pod_name,\$.kubernetes.container_name
+      timekey: 1m
+      timekey_wait: 1m
+      timekey_use_utc: true
+    format:
+      type: single_value
+      message_key: message
+EOF
+```
+
+The `ClusterOutput` above will stream logs to our `minio` storage in a `tekton-logs` bucket. It will buffer logs and will store one file per minute in the `<namespace_name>/<pod_name>/<container_name>/` path. All metadata will be omitted and only the log message will be stored, it will be the raw pod logs.
+
+- `ClusterFlow` defines how the collected logs are dispatched to the outputs:
+
+```bash
+cat <<EOF | kubectl -n tools apply -f -
+apiVersion: logging.banzaicloud.io/v1beta1
+kind: ClusterFlow
+metadata:
+  name: flow
+spec:
+  outputRefs:
+    - s3
+  selectors:
+    app.kubernetes.io/managed-by: tekton-pipelines
+EOF
+```
+
+The `ClusterFlow` above takes all logs from pods that have the `app.kubernetes.io/managed-by: tekton-pipelines` label (those are the pods baking `TaskRun`s) and dispatches them to the `ClusterOutput` created in the previous step.
+
+Running the `PipelineRun` below should produce logs and you should see corresponding objects being added in `minio` as logs get collected and stored by the logs pipeline.
+
+```bash
+cat <<EOF | kubectl -n tekton-pipelines create -f -
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+metadata:
+  generateName: sample-
+spec:
+  pipelineSpec:
+    tasks:
+      - name: gen-log
+        taskSpec:
+          steps:
+            - name: gen-log
+              image: ubuntu
+              script: |
+                #!/usr/bin/env bash
+                for i in {1..10}
+                do
+                  echo "Log line \$i"
+                  sleep 1s
+                done
+            - name: gen-log-2
+              image: ubuntu
+              script: |
+                #!/usr/bin/env bash
+                for i in {1..100}
+                do
+                  echo "Log line \$i"
+                  sleep 1s
+                done
+EOF
+```
+
+![Log files in minio](./walkthrough-logs-collected.png)
+
+## Creating a service to serve collected logs
+
+Now pod logs are collected and stored in your object storage, you will create a service to serve those logs.
+
+Given a `namespace`, `pod` and `container`, the service will list files from s3, then stream the content of those files to serve logs to the caller.
+
+Run the command below to create the kubernetes `Deployment` to serve your logs:
+
+```bash
+cat <<EOF | kubectl apply -n tools -f -
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: logs-server
+  labels: 
+    app: logs-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: logs-server
+  template:
+    metadata:
+      labels:
+        app: logs-server
+    spec:
+      containers:
+      - name: node
+        image: node:latest
+        ports:
+        - containerPort: 3000
+        command:
+        - bash
+        args:
+        - -c
+        - |
+          cat <<EOF > server.js
+          const express = require('express');
+          const aws = require('aws-sdk');
+
+          aws.config.update({
+            endpoint: 'minio.tools.svc.cluster.local:9000',
+            accessKeyId: 'ACCESSKEY',
+            secretAccessKey: 'SECRETKEY',
+            region: 'tekton',
+            s3ForcePathStyle: true,
+            sslEnabled: false
+          });
+
+          const s3 = new aws.S3();
+          const app = express();
+
+          const bucket = 'tekton-logs'
+
+          function streamLogs(namespace, pod, container, response) {
+            s3.listObjects({ Bucket: bucket, Delimiter: '', Prefix: namespace+'/'+pod+'/'+container+'/' })
+              .promise()
+              .then(files =>
+                files.Contents.sort((a, b) => a.Key.localeCompare(b.Key)).reduce((acc, file) =>
+                  acc.then(() =>
+                    new Promise(fulfill =>
+                      s3.getObject({Bucket: bucket, Key: file.Key})
+                        .createReadStream()
+                        .on("finish", fulfill)
+                        .pipe(response, { end: false })
+                    )
+                  ),
+                  Promise.resolve()
+                )
+              )
+              .then(() => response.end());
+          }
+          app.get('/logs/:namespace/:pod/:container', (req, res) => streamLogs(req.params.namespace, req.params.pod, req.params.container, res));
+          app.listen(3000, '0.0.0.0');
+          EOF
+
+          npm install aws-sdk express
+
+          node ./server.js
+EOF
+```
+
+This deployment will start a container running `nodejs`. It will install `express` web server and `aws-sdk` to interact with S3 (configured to hit your `minio` endpoint).
+
+Then it will run a web server exposing the `'/logs/:namespace/:pod/:container'` route to serve logs fetched from s3.
+
+To make this available you will need to deploy a `Service` and an `Ingress` rule to expose the `Deployment`:
+
+```bash
+cat <<EOF | kubectl apply -n tools -f -
+kind: Service
+apiVersion: v1
+metadata:
+  name: logs-server
+  labels:
+    app: logs-server
+spec:
+  ports:
+  - port: 3000
+    targetPort: 3000
+  selector:
+    app: logs-server
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: tekton-logs
+spec:
+  rules:
+  - host: logs.127.0.0.1.nip.io
+    http:
+      paths:
+      - backend:
+          serviceName: logs-server
+          servicePort: 3000
+EOF
+```
+
+The logs server should now be available ar `http://logs.127.0.0.1.nip.io`.
+
+## Setting up the Dashboard logs fallback
+
+The last step in this walkthrough is to setup the Tekton Dashboard to use the logs server you created above. The logs server will act as a fallback when the logs are not available anymore because the underlying pods baking `TaskRun`s are gone away.
+
+First, delete the pods for your `TaskRun`s so that the Dashboard backend can't find the pod logs:
+
+```bash
+kubectl delete pod -l=app.kubernetes.io/managed-by=tekton-pipelines -n tekton-pipelines
+```
+
+The Dashboard should now display the `Unable to fetch logs` message when browsing tasks.
+
+![Unable to fetch logs](./walkthrough-logs-nologs.png)
+
+Second, patch the Dashboard deployment to add the `--external-logs=http://logs-server.tools.svc.cluster.local:3000/logs` option:
+
+```bash
+kubectl patch deployment tekton-dashboard -n tekton-pipelines --type='json' \
+  --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--external-logs=http://logs-server.tools.svc.cluster.local:3000/logs"}]'
+```
+
+The logs should now be displayed again, fetched from the logs server configured in the previous steps.
+
+![Logs are available again](./walkthrough-logs-logs.png)
+
+## Cleaning up
+
+To clean up the local kind cluster, follow the [cleaning up instructions](./walkthrough-kind.md#cleaning-up) in Tekton Dashboard walkthrough with Kind.
+
+---
+
+Except as otherwise noted, the content of this page is licensed under the [Creative Commons Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/).
+
+Code samples are licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
diff --git a/overlays/patches/installer/deployment-patch.yaml b/overlays/patches/installer/deployment-patch.yaml
@@ -53,3 +53,7 @@
   path: /spec/template/spec/containers/0/args/-
   value:
     --stream-logs=--stream-logs
+- op: add
+  path: /spec/template/spec/containers/0/args/-
+  value:
+    --external-logs=--external-logs
-Original file line number
+Diff line change
@@ Expand Up / @@ -5,6 +5,7 @@ We have the following walk-throughs available: @@
     - [Setup everything from scratch using a kind cluster](./walkthrough-kind.md)
     - [Securing access to the Dashboard behind an oauth2 proxy](./walkthrough-oauth2-proxy.md)
+    - [Enabling logs persistence](./walkthrough-logs.md)
     ---
@@ Expand Down @@