go-sensor requires Go version 1.9 or greater.

The Instana Go sensor consists of three parts:

• Metrics sensor
• OpenTracing tracer
• AutoProfile™ continuous profiler

• Installation
  • Running in serverless environment
  • Using Instana to gather process metrics only
• Common Operations
  • Setting the sensor log output
  • Trace Context Propagation
  • Secrets Filtering
  • HTTP servers and clients
    • Instrumenting HTTP request handling
    • Instrumenting HTTP request execution
    • Capturing custom HTTP headers
  • Database Calls
    • Instrumenting sql.Open()
    • Instrumenting sql.OpenDB()
  • GRPC servers and clients
  • Kafka producers and consumers
• OpenTracing
• W3C Trace Context
• Events API
• AutoProfile™
  • Activation from within the application code
  • Activation without code changes
• Examples

To add Instana Go sensor to your service run:

$ go get github.com/instana/go-sensor

To activate background metrics collection, add following line at the beginning of your service initialization (typically this would be the beginning of your main() function):

func main() {
	instana.InitSensor(instana.DefaultOptions())

	// ...
}

The instana.InitSensor() function takes an *instana.Options struct with the following optional fields:

• Service - global service name that will be used to identify the program in the Instana backend
• AgentHost, AgentPort - default to localhost:42699, set the coordinates of the Instana proxy agent
• LogLevel - one of Error, Warn, Info or Debug
• EnableAutoProfile - enables automatic continuous process profiling when true
• MaxBufferedSpans - the maximum number of spans to buffer
• ForceTransmissionStartingAt - the number of spans to collect before flushing the buffer to the agent
• MaxBufferedProfiles - the maximum number of profiles to buffer
• IncludeProfilerFrames - whether to include profiler calls into the profile or not
• Tracer - tracer-specific configuration used by all tracers

Once initialized, the sensor performs a host agent lookup using following list of addresses (in order of priority):

1. The value of INSTANA_AGENT_HOST env variable
2. localhost
3. Default gateway

Once a host agent found listening on port 42699 (or the port specified in INSTANA_AGENT_PORT env variable) the sensor begins collecting in-app metrics and sending them to the host agent.

To use Instana Go sensor for monitoring a service running in a serverless environment, such as AWS Fargate or Google Cloud Run, make sure that you have INSTANA_ENDPOINT_URL and INSTANA_AGENT_KEY env variables set in your task definition. Note that the INSTANA_AGENT_HOST and INSTANA_AGENT_PORT env variables will be ignored in this case. Please refer to the respective section of Instana documentation for detailed explanation on how to do this:

• Configuring AWS Fargate task definitions
• Configuring AWS Lambda functions
• Configuring Google Cloud Run services

Services running in serverless environments don't use host agent to send metrics and trace data to Instana backend, therefore the usual way of configuring the in-app sensor via configuration.yaml file is not applicable. Instead there is a set of environment variables that can optionally be configured in service task definition:

Please refer to Insana documentation for more detailed description of these variables and their value format.

To use sensor without tracing ability, import the instana package and add the following line at the beginning of your main() function:

instana.InitSensor(opt)

The Instana Go sensor offers a set of quick features to support tracing of the most common operations like handling HTTP requests and executing HTTP requests.

To create an instance of the Instana sensor just request a new instance using the instana.NewSensor factory method and providing the name of the application. It is recommended to use a single instance only. The sensor implementation is fully thread-safe and can be shared by multiple threads.

var sensor = instana.NewSensor("my-service")

A full example can be found under the examples folder in example/http-database-greeter/.

The Go sensor uses a leveled logger to log internal errors and diagnostic information. The default logger.Logger uses log.Logger configured with log.Lstdflags as a backend and writes messages to os.Stderr. By default this logger only prints out the ERROR level messages unless the environment variable INSTANA_DEBUG is set.

To change the min log level in runtime it is recommended to configure and inject an instance of instana.LeveledLogger instead of using the deprecated instana.SetLogLevel() method:

l := logger.New(log.New(os.Stderr, "", os.Lstdflags))
instana.SetLogger(l)

// ...

l.SetLevel(logger.WarnLevel)

The logger.LeveledLogger interface is implemented by such popular logging libraries as github.com/sirupsen/logrus and go.uber.org/zap, so they can be used as a replacement.

Note: the value of INSTANA_DEBUG environment variable does not affect custom loggers. You'd need to explicitly check whether it's set and enable the debug logging while onfiguring your logger:

import (
	instana "github.com/instana/go-sensor"
	"github.com/sirupsen/logrus"
)

func main() {
	// initialize Instana sensor
	instana.InitSensor(&instana.Options{Service: SERVICE})

	// initialize and configure the logger
	logger := logrus.New()
	logger.Level = logrus.InfoLevel

	// check if INSTANA_DEBUG is set and set the log level to DEBUG if needed
	if _, ok := os.LookupEnv("INSTANA_DEBUG"); ok {
		logger.Level = logrus.DebugLevel
	}

	// use logrus to log the Instana Go sensor messages
	instana.SetLogger(logger)

	// ...
}

The Go sensor AutoProfile™ by default uses the same logger as the sensor itself, however it can be configured to use its own, for example to write messages with different tags/prefix or use a different logging level. The following snippet demonstrates how to configure the custom logger for autoprofiler:

autoprofile.SetLogger(autoprofileLogger)

Instana Go sensor provides an API to propagate the trace context throughout the call chain:

func MyFunc(ctx context.Context) {
	var spanOpts []ot.StartSpanOption

	// retrieve parent span from context and reference it in the new one
	if parent, ok := instana.SpanFromContext(ctx); ok {
	    spanOpts = append(spanOpts, ot.ChildOf(parent.Context()))
	}

	// start a new span
	span := tracer.StartSpan("my-func", spanOpts...)
	defer span.Finish()

	// and use it as a new parent inside the context
	SubCall(instana.ContextWithSpan(ctx, span))
}

Certain instrumentations provided by the Go sensor package, e.g. the HTTP servers and clients wrappers, collect data that may contain sensitive information, such as passwords, keys and secrets. To avoid leaking these values the Go sensor replaces them with <redacted> before sending to the agent. The list of parameter name matchers is defined in com.instana.secrets section of the Host Agent Configuration file and will be sent to the in-app tracer during the announcement phase (requires agent Go trace plugin com.instana.sensor-golang-trace v1.3.0 and above).

The default setting for the secrets matcher is contains-ignore-case with following list of terms: key, password, secret. This would redact the value of a parameter which name contains any of these strings ignoring the case.

The Go sensor module provides instrumentation for clients and servers that use net/http package. Once activated (see below) this instrumentation automatically collects information about incoming and outgoing requests and sends it to the Instana agent. See the instana.HTTPSpanTags documentation to learn which call details are collected.

With support to wrap a http.HandlerFunc, Instana quickly adds the possibility to trace requests and collect child spans, executed in the context of the request span.

Minimal changes are required for Instana to be able to capture the necessary information. By simply wrapping the currently existing http.HandlerFunc Instana collects and injects necessary information automatically.

That said, a simple handler function like the following will simple be wrapped and registered like normal.

The following example code demonstrates how to instrument an HTTP handler using instana.TracingHandlerFunc():

sensor := instana.NewSensor("my-http-server")

http.HandleFunc("/", instana.TracingHandlerFunc(sensor, "/", func(w http.ResponseWriter, req *http.Request) {
	// Extract the parent span and use its tracer to initialize any child spans to trace the calls
	// inside the handler, e.g. database queries, 3rd-party API requests, etc.
	if parent, ok := instana.SpanFromContext(req.Context()); ok {
		sp := parent.Tracer().StartSpan("index")
		defer sp.Finish()
	}

	// ...
}))

In case your handler is implemented as an http.Handler, pass its ServeHTTP method instead:

h := http.FileServer(http.Dir("./"))
http.HandleFunc("/files", instana.TracingHandlerFunc(sensor, "index", h.ServeHTTP))

Requesting data or information from other, often external systems, is commonly implemented through HTTP requests. To make sure traces contain all spans, especially over all the different systems, certain span information have to be injected into the HTTP request headers before sending it out. Instana's Go sensor provides support to automate this process as much as possible.

To have Instana inject information into the request headers, create the http.Client, wrap its Transport with instana.RoundTripper() and use it as in the following example.

req, err := http.NewRequest("GET", url, nil)
client := &http.Client{
	Transport: instana.RoundTripper(sensor, nil),
}

ctx := instana.ContextWithSpan(context.Background(), parentSpan)
resp, err := client.Do(req.WithContext(ctx))

The provided parentSpan is the incoming request from the request handler (see above) and provides the necessary tracing and span information to create a child span and inject it into the request.

The HTTP instrumentation wrappers are capable of collecting HTTP headers and sending them along with the incoming/outgoing request spans. The list of case-insensitive header names can be provided both within (instana.Options).Tracer.CollectableHTTPHeaders field of the options object passed to instana.InitSensor() and in the Host Agent Configuration file. The latter setting takes precedence and requires agent Go trace plugin com.instana.sensor-golang-trace v1.3.0 and above:

instana.InitSensor(&instana.Options{
	// ...
	Tracer: instana.TracerOptions{
		// ...
		CollectableHTTPHeaders: []string{"x-request-id", "x-loadtest-id"},
	},
})

This configuration is an equivalent of following settings in the Host Agent Configuration file:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'

By default the HTTP instrumentation does not collect any headers.

The Go sensor provides instana.InstrumentSQLDriver() and instana.WrapSQLConnector() (since Go v1.10+) to instrument SQL database calls made with database/sql. The tracer will then automatically capture the Query and Exec calls, gather information about the query, such as statement, execution time, etc. and forward them to be displayed as a part of the trace.

To instrument a database driver, register it using instana.InstrumentSQLDriver() first and replace the call to sql.Open() with instana.SQLOpen(). Here is an example on how to do this for github.com/lib/pq PostgreSQL driver:

// Create a new instana.Sensor instance
sensor := instana.NewSensor("my-daatabase-app")

// Instrument the driver
instana.InstrumentSQLDriver(sensor, "postgres", &pq.Driver{})

// Create an instance of *sql.DB to use for database queries
db, err := instana.SQLOpen("postgres", "postgres://...")

You can find the complete example in the Examples section of package documentation on pkg.go.dev.

The instrumented driver is registered with the name <original_name>_with_instana, e.g. in the example above the name would be postgres_with_instana.

Starting from Go v1.10 database/sql provides a new way to initialize *sql.DB that does not require the use of global driver registry. If the database driver library provides a type that satisfies the database/sql/driver.Connector interface, it can be used to create a database connection.

To instrument a driver.Connector instance, wrap it using instana.WrapSQLConnector(). Here is an example on how this can be done for github.com/go-sql-driver/mysql/ MySQL driver:

// Create a new instana.Sensor instance
sensor := instana.NewSensor("my-daatabase-app")

// Initialize a new connector
connector, err := mysql.NewConnector(cfg)
// ...

// Wrap the connector before passing it to sql.OpenDB()
db, err := sql.OpenDB(instana.WrapSQLConnector(sensor, "mysql://...", connector))

You can find the complete example in the Examples section of package documentation on pkg.go.dev.

github.com/instana/go-sensor/instrumentation/instagrpc provides both unary and stream interceptors to instrument GRPC servers and clients that use google.golang.org/grpc.

github.com/instana/go-sensor/instrumentation/instasarama provides both unary and stream interceptors to instrument Kafka producers and consumers built on top of github.com/Shopify/sarama.

Instana tracer provides an interface compatible with github.com/opentracing/opentracing-go and thus can be used as a global tracer. However, this approach is not recommended. The Go OpenTracing library only offers a low-level interface to create and maintain spans, which means that most of the co

In case you want to integrate Instana into an app that is already instrumented with OpenTracing, register an instance of Instana tracer as a global tracer at the beginning of your main() function:

import (
	instana "github.com/instana/go-sensor"
	opentracing "github.com/opentracing/opentracing-go"
)

func main() {
	opentracing.InitGlobalTracer(instana.NewTracerWithOptions(instana.DefaultOptions())
	// ...
}

This will automatically initialize the sensor and thus also activate the metrics stream. The tracer takes the same options that the sensor takes for initialization, described above.

The tracer is able to protocol and piggyback OpenTracing baggage, tags and logs. Only text mapping is implemented yet, binary is not supported. Also, the tracer tries to map the OpenTracing spans to the Instana model based on OpenTracing recommended tags. See the Instana OpenTracing integration example for details on how recommended tags are used.

The Instana tracer will remap OpenTracing HTTP headers into Instana headers, so parallel use with some other OpenTracing model is not possible. The Instana tracer is based on the OpenTracing Go basictracer with necessary modifications to map to the Instana tracing model. Also, sampling isn't implemented yet and will be focus of future work.

The Go sensor library fully supports the W3C Trace Context standard:

• An instrumented http.Client sends the traceparent and tracestate headers, updating them with the exit span ID and flags.
• Any http.Handler instrumented with instana.TracingHandlerFunc() picks up the trace context passed in the traceparent header, potentially restoring the trace from tracestate even if the upstream service is not instrumented with Instana.

The sensor, be it instantiated explicitly or implicitly through the tracer, provides a simple wrapper API to send events to Instana as described in its documentation.

To learn more, see the Events API document in this repository.

AutoProfile™ generates and reports process profiles to Instana. Unlike development-time and on-demand profilers, where a user must manually initiate profiling, AutoProfile™ automatically schedules and continuously performs profiling appropriate for critical production environments.

To enable continuous profiling for your service provide EnableAutoProfile: true while initializing the sensor:

func main() {
	instana.InitSensor(&instana.Options{
		EnableAutoProfile: true,
		// ...other options
	})

	// ...
}

To temporarily turn AutoProfile™ on and off from your code, call autoprofile.Enable() and autoprofile.Disable().

To enable AutoProfile™ for an app without code changes, set INSTANA_AUTO_PROFILE=true env variable. Note that this value takes precedence and overrides any attempt to disable profiling from inside the application code.

Following examples are included in the example folder:

• Greeter - an instrumented HTTP server that queries a database
• Doubler - an instrumented Kafka processor, that counsumes and produces messages
• event/ - Demonstrates usage of the Events API
• autoprofile/ - Demonstrates usage of the AutoProfile™
• OpenTracing - an example of usage of Instana tracer in an app instrumented with OpentTracing

For more examples please consult the godoc.