Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Fluentbit's unified networking interface #1217

Merged
merged 4 commits into from
Jun 25, 2024
Merged

Adding Fluentbit's unified networking interface #1217

merged 4 commits into from
Jun 25, 2024

Conversation

localleon
Copy link
Contributor

@localleon localleon commented Jun 21, 2024

What this PR does / why we need it:

This PR adds support for the fluentbit unified networking interface for many output plugins. See the offical docs for more information about the feature https://docs.fluentbit.io/manual/administration/networking

You can use the advanced networking options in most Clusteroutput-Plugins like this:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: tcp
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: "*"
  tcp:
    host: 127.0.0.1
    port: 9090
    format: json_lines
    networking:
      connectTimeout: 20
      connectTimeoutLogError: true
      DNSMode: "UDP"
      DNSPreferIPv4: false
      DNSResolver: "ASYNC"
      keepalive: "on"
      keepaliveIdleTimeout: 40
      keepaliveMaxRecycle: 3000
      maxWorkerConnections: 10

The operator will then generate the fluentbit-configuration according to the documentation. When running kubectl get secrets -n fluent fluent-bit-config -o json | jq -r '.data["fluent-bit.conf"]' | base64 -d we can see that our configuration now includes everything that we defined in the CRD!

[Input]
    Name    dummy
    Tag    my_dummy
[Output]
    Name    stdout
    Match    *
[Output]
    Name    tcp
    Match    *
    Host    127.0.0.1
    Port    9090
    Format    json_lines
    net.connect_timeout    20
    net.connect_timeout_log_error    true
    net.dns.mode    UDP
    net.dns.prefer_ipv4    false
    net.dns.prefer_ipv4    ASYNC
    net.keepalive    on
    net.keepalive_idle_timeout    40
    net.keepalive_max_recycle    3000
    net.max_worker_connections    10

Which issue(s) this PR fixes:

Fixes #247

Does this PR introduced a user-facing change?

Yes, most CRDs are updated to include the net-options and have the corresponding documentation updated

Implement the unified networking interface (https://docs.fluentbit.io/manual/administration/networking) for many supported output plugins (e.g splunk, tcp, syslog). Most CRDs of Output plugins now support setting common network settings e.g net.keepalive

Additional documentation, usage docs, etc.:

This PR is a 1:1 implementation of the provided documentation at https://docs.fluentbit.io/manual/administration/networking

Signed-off-by: localleon <me@lrau.xyz>
…t support the networking plugin (e.g where we currently already have the TLS interface)

Signed-off-by: localleon <me@lrau.xyz>
@benjaminhuo
Copy link
Member

That's great, thanks!

@benjaminhuo
Copy link
Member

benjaminhuo commented Jun 25, 2024

That's great, Thank you! @localleon
cc @wanjunlei

@benjaminhuo benjaminhuo merged commit db04760 into fluent:master Jun 25, 2024
9 of 10 checks passed
@localleon localleon deleted the fluentbit-networking branch June 25, 2024 06:40
@localleon
Copy link
Contributor Author

@benjaminhuo when do you guys usally do releases? I would like to get this feature into an official release soon because we need it for our company rollout of the operator ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to configure net.keepalive ?
2 participants