Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: Add filter to lift keys from a map or nest keys in a map #531

Merged
merged 2 commits into from
May 17, 2018
Merged

feature: Add filter to lift keys from a map or nest keys in a map #531

merged 2 commits into from
May 17, 2018

Conversation

michiel
Copy link
Contributor

@michiel michiel commented Mar 8, 2018 •
edited
Loading

This adds a new filter, called 'nest', that allows keys to be,

  • Nested under another key using a wildcard
  • Lifted up from a nest

Examples,

Nest 1

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

Output,

[0] mem.local: [1524795567.010244325, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1101372, "Mem.free"=>2949536}}]

{
  "Swap.total"=>1046524,
  "Swap.used"=>0,
  "Swap.free"=>1046524,
  "LAYER1"=>{
   "Mem.total"=>4050908,
   "Mem.used"=>1101372,
   "Mem.free"=>2949536
  }
}

Nest 1 - Lift 1

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER1

Output,

[0] mem.local: [1524795757.002111309, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Mem.total"=>4050908, "Mem.used"=>1112044, "Mem.free"=>2938864}]

{
  "Swap.total"=>1046524, 
  "Swap.used"=>0,
  "Swap.free"=>1046524, 
  "Mem.total"=>4050908, 
  "Mem.used"=>1112044, 
  "Mem.free"=>2938864
}

Nest 1/2/3

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

Output,

 [0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]

 {
   "Swap.total"=>1046524,
   "Swap.used"=>0,
   "Swap.free"=>1046524,
   "LAYER3"=>{
     "LAYER2"=>{
       "LAYER1"=>{
         "Mem.total"=>4050908,
         "Mem.used"=>1112036,
         "Mem.free"=>2938872
       }
     }
   }
 }

Nest 1/2/3 - Lift 3/2

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER3

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER2

Output,

[0] mem.local: [1524795834.019236872, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112152, "Mem.free"=>2938756}}]

{
  "Swap.total"=>1046524,
  "Swap.used"=>0,
  "Swap.free"=>1046524, 
  "LAYER1"=>{
    "Mem.total"=>4050908, 
    "Mem.used"=>1112152, 
    "Mem.free"=>2938756
  }
}

edsiper
edsiper reviewed Apr 2, 2018
View reviewed changes
plugins/filter_nest/nest.c Outdated
@@ -0,0 +1,284 @@
/* -*- Mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */

/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add missing header here:

/*  Fluent Bit
 *  ==========
 *  Copyright (C) 2015-2017 Treasure Data Inc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 77ac171

@michiel michiel force-pushed the feature/add_nest_filter branch from 030fadd to 94780d5 Compare April 9, 2018 23:56
@michiel
Copy link
Contributor Author

michiel commented Apr 9, 2018

Documentation PR added at fluent/fluent-bit-docs#62

@michiel michiel force-pushed the feature/add_nest_filter branch from 94780d5 to eedb6f1 Compare April 26, 2018 00:54
@michiel michiel changed the title feature: Add filter to nest specific keys under another key feature: Add filter to lift keys from a map or nest keys in a map May 2, 2018
@michiel michiel force-pushed the feature/add_nest_filter branch from 00cd563 to 58e6860 Compare May 2, 2018 22:54
@ffscl ffscl mentioned this pull request May 3, 2018
Closed
@michiel
Copy link
Contributor Author

michiel commented May 3, 2018 •
edited
Loading

Documentation PR updated to reflect addition of 'lift' operation

See https://github.com/michiel/fluent-bit-docs/blob/feature/add_filter_nest/filter/nest.md

filter: Add new nest filter
89183b8 
This adds a filter, called 'nest'. It allows you to,

 - Nest a set of key-values pairs under a new key
 - Lift a set of key-value pairs up to the top level

Signed-off-by: Michiel Kalkman <mkalkman@shelde.com>
@michiel michiel force-pushed the feature/add_nest_filter branch from f14545d to 89183b8 Compare May 9, 2018 12:03
@edsiper edsiper merged commit e97a293 into fluent:master May 17, 2018
@edsiper
Copy link
Member

edsiper commented May 17, 2018

thanks!

edsiper pushed a commit that referenced this pull request May 21, 2018
@michiel @edsiper
filter_nest: new filter to handle nest k/v (#531)
9f1d79f 
This adds a filter, called 'nest'. It allows you to,

 - Nest a set of key-values pairs under a new key
 - Lift a set of key-value pairs up to the top level

Signed-off-by: Michiel Kalkman <mkalkman@shelde.com>
Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
rawahars pushed a commit to rawahars/fluent-bit that referenced this pull request Oct 24, 2022
@nokute78 @edsiper
configuration: add log_level off (fluent#531)
694c896 
Signed-off-by: Takahiro Yamashita <nokute78@gmail.com>

Co-authored-by: Eduardo Silva <eduardo@treasure-data.com>
@gitworkflows gitworkflows mentioned this pull request Jan 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper edsiper left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@michiel @edsiper