[processor/filter] Standardize configuration on OTTL #18642
Description
Is your feature request related to a problem? Please describe.
With #16369, OTTL was added to the filterprocessor. This allows us to standardize the way the filterprocessor configures conditions for when data should be dropped.
For spans, the existing non-ottl configuration options are:
- matching by service name, span name, attributes, scopes, resource attributes, and span kind.
- Both strict and regex is supported.
- When checking attributes, converting attribute values from bool, double, and int to string is supported
For metric, the existing non-ottl configuration options are:
- matching by metric name, resource attributes
- Both strict and regex is supported.
- matching by datapoint attributes via the expr expression engine.
- When checking attributes, converting attribute values from bool, double, and int to string is supported.
For logs, the existing non-ottl configuration options are:
- matching by resource attributes, attributes, severity text, body, and severity number.
- Both strict and regex is supported.
- When checking attributes, converting attribute values from bool, double, and int to string is supported.
- When matching severity number, a minimum log severity can be used and any values equal to or greater than the severity match. You can also specify if an undefined severity number can be match.
With the completion of #16413, OTTL can now handle each of these use cases, and more, in a uniform library.
In the past we only had what was in internal/filter
and as different components needed different condition features those packages grew. As users wanted access to different fields in the telemetry payload the signal-specific filters would change to reflect those needs. The filterprocessor grew with it, providing important value but with increased maintenance burden and configuration complexity. Reading through the filterprocessor's readme highlights the difficulty in maintaining and understanding the different features and fields available.
By unifying on OTTL we'll have a solution that has access to all fields on all signals. Users no longer need to worry about whether or not a field for their signal is available to use and maintainers no longer need to worry about adding more fields to filter on in the future (OTLP changes excluded). Due to OTTL's functions, adding more features to enable complex conditions is simpler as the functions encapsulate the logic and can be added without modifying the underlying libraries or configuration. On top of its field access and functions, OTTL's grammar also provides more robust conditions, allowing users to use inequalities, nil
, and arithmetic.
Steps to completion
- [internal/filter] Add bridge from filterspan to filterottl #21054
- [internal/filter] filtermetric to filterottl bridge #23141
- [internal/filter] Add bridge from filterlog to filterottl #22789
- Start logging the equivalent OTTL statements for filterspan
- Start logging the equivalent OTTL statements for filtermetric
- Start logging the equivalent OTTL statements for filterlog
- Switch filterspan bridge feature gate to beta
- Switch filtermetric bridge feature gate to beta
- Switch filterlog bridge feature gate to beta
- Switch filterspan bridge feature gate to Stable
- Switch filtermetric bridge feature gate to Stable
- Switch filterlog bridge feature gate to Stable
- Remove all non-filterottl filters