Skip to content

Commit

Permalink
fix doc for new format in file plugin
Browse files Browse the repository at this point in the history
  • Loading branch information
@DmitryRomanov
DmitryRomanov committed Jun 11, 2024
1 parent 10edfee commit eea327b
Show file tree
Hide file tree
Showing 5 changed files with 71 additions and 106 deletions.
15 changes: 0 additions & 15 deletions plugin/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,6 @@ But update events don't work with symlinks, so watcher also periodically manuall
> ⚠ Use add_file_name plugin if you want to add filename to events.
**Reading docker container log files:**
```yaml
pipelines:
example_docker_pipeline:
input:
type: file
paths:
include:
- '/var/lib/docker/containers/**\/*-json.log' # remove \
exclude:
- '/var/lib/docker/containers/19aa5027343f4*\/*-json.log' # remove \
offsets_file: /data/offsets.yaml
persistence_mode: async
```
[More details...](plugin/input/file/README.md)
## http
Reads events from HTTP requests with the body delimited by a new line.
Expand Down
15 changes: 0 additions & 15 deletions plugin/input/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,21 +29,6 @@ But update events don't work with symlinks, so watcher also periodically manuall
> ⚠ Use add_file_name plugin if you want to add filename to events.
**Reading docker container log files:**
```yaml
pipelines:
example_docker_pipeline:
input:
type: file
paths:
include:
- '/var/lib/docker/containers/**\/*-json.log' # remove \
exclude:
- '/var/lib/docker/containers/19aa5027343f4*\/*-json.log' # remove \
offsets_file: /data/offsets.yaml
persistence_mode: async
```
[More details...](plugin/input/file/README.md)
## http
Reads events from HTTP requests with the body delimited by a new line.
Expand Down
15 changes: 15 additions & 0 deletions plugin/input/file/README.idoc.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# File plugin
@introduction

**Reading docker container log files:**
```yaml
pipelines:
example_docker_pipeline:
input:
type: file
paths:
include:
- '/var/lib/docker/containers/**/*-json.log'
exclude:
- '/var/lib/docker/containers/19aa5027343f4*/*-json.log'
offsets_file: /data/offsets.yaml
persistence_mode: async
```
### Config params
@config-params|description
64 changes: 31 additions & 33 deletions plugin/input/file/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,33 +27,20 @@ pipelines:
type: file
paths:
include:
- '/var/lib/docker/containers/**\/*-json.log' # remove \
- '/var/lib/docker/containers/**/*-json.log'
exclude:
- '/var/lib/docker/containers/19aa5027343f4*\/*-json.log' # remove \
- '/var/lib/docker/containers/19aa5027343f4*/*-json.log'
offsets_file: /data/offsets.yaml
persistence_mode: async
```
### Config params
**`watching_dir`** *`string`*

List of included pathes
*`string`*

List of excluded pathes
*`string`*

The source directory to watch for files to process. All subdirectories also will be watched. E.g. if files have
`/var/my-logs/$YEAR/$MONTH/$DAY/$HOST/$FACILITY-$PROGRAM.log` structure, `watching_dir` should be `/var/my-logs`.
Also the `filename_pattern`/`dir_pattern` is useful to filter needless files/subdirectories. In the case of using two or more
different directories, it's recommended to setup separate pipelines for each.

<br>

**`paths`** *`Paths`*

Paths.
> Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
Set paths in glob format

* `include` *`[]string`*
* `exclude` *`[]string`*

<br>

Expand All @@ -64,20 +51,6 @@ The filename to store offsets of processed files. Offsets are loaded only on ini

<br>

**`filename_pattern`** *`string`* *`default=*`*

Files that don't meet this pattern will be ignored.
> Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.

<br>

**`dir_pattern`** *`string`* *`default=*`*

Dirs that don't meet this pattern will be ignored.
> Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.

<br>

**`persistence_mode`** *`string`* *`default=async`* *`options=async|sync`*

It defines how to save the offsets file:
Expand Down Expand Up @@ -152,5 +125,30 @@ It turns on watching for file modifications. Turning it on cause more CPU work,

<br>

**`watching_dir`** *`string`*

**Deprecated format**

The source directory to watch for files to process. All subdirectories also will be watched. E.g. if files have
`/var/my-logs/$YEAR/$MONTH/$DAY/$HOST/$FACILITY-$PROGRAM.log` structure, `watching_dir` should be `/var/my-logs`.
Also the `filename_pattern`/`dir_pattern` is useful to filter needless files/subdirectories. In the case of using two or more
different directories, it's recommended to setup separate pipelines for each.

<br>

**`filename_pattern`** *`string`* *`default=*`*

Files that don't meet this pattern will be ignored.
> Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.

<br>

**`dir_pattern`** *`string`* *`default=*`*

Dirs that don't meet this pattern will be ignored.
> Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.

<br>


<br>*Generated using [__insane-doc__](https://github.com/vitkovskii/insane-doc)*
68 changes: 25 additions & 43 deletions plugin/input/file/file.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,21 +33,6 @@ But update events don't work with symlinks, so watcher also periodically manuall
> By default the plugin is notified only on file creations. Note that following for changes is more CPU intensive.
> ⚠ Use add_file_name plugin if you want to add filename to events.
**Reading docker container log files:**
```yaml
pipelines:
example_docker_pipeline:
input:
type: file
paths:
include:
- '/var/lib/docker/containers/**\/*-json.log' # remove \
exclude:
- '/var/lib/docker/containers/19aa5027343f4*\/*-json.log' # remove \
offsets_file: /data/offsets.yaml
persistence_mode: async
```
}*/

type Plugin struct {
Expand Down Expand Up @@ -84,14 +69,7 @@ const (
)

type Paths struct {
// > @3@4@5@6
// >
// > List of included pathes
Include []string `json:"include"`

// > @3@4@5@6
// >
// > List of excluded pathes
Exclude []string `json:"exclude"`
}

Expand All @@ -101,16 +79,10 @@ type Config struct {

// > @3@4@5@6
// >
// > The source directory to watch for files to process. All subdirectories also will be watched. E.g. if files have
// > `/var/my-logs/$YEAR/$MONTH/$DAY/$HOST/$FACILITY-$PROGRAM.log` structure, `watching_dir` should be `/var/my-logs`.
// > Also the `filename_pattern`/`dir_pattern` is useful to filter needless files/subdirectories. In the case of using two or more
// > different directories, it's recommended to setup separate pipelines for each.
WatchingDir string `json:"watching_dir"` // *

// > @3@4@5@6
// > Set paths in glob format
// >
// > Paths.
// > > Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
// > * `include` *`[]string`*
// > * `exclude` *`[]string`*
Paths Paths `json:"paths"` // *

// > @3@4@5@6
Expand All @@ -120,18 +92,6 @@ type Config struct {
OffsetsFile string `json:"offsets_file" required:"true"` // *
OffsetsFileTmp string

// > @3@4@5@6
// >
// > Files that don't meet this pattern will be ignored.
// > > Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
FilenamePattern string `json:"filename_pattern" default:"*"` // *

// > @3@4@5@6
// >
// > Dirs that don't meet this pattern will be ignored.
// > > Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
DirPattern string `json:"dir_pattern" default:"*"` // *

// > @3@4@5@6
// >
// > It defines how to save the offsets file:
Expand Down Expand Up @@ -194,6 +154,28 @@ type Config struct {
// >
// > It turns on watching for file modifications. Turning it on cause more CPU work, but it is more probable to catch file truncation
ShouldWatchChanges bool `json:"should_watch_file_changes" default:"false"` // *

// > @3@4@5@6
// >
// > **Deprecated format**
// >
// > The source directory to watch for files to process. All subdirectories also will be watched. E.g. if files have
// > `/var/my-logs/$YEAR/$MONTH/$DAY/$HOST/$FACILITY-$PROGRAM.log` structure, `watching_dir` should be `/var/my-logs`.
// > Also the `filename_pattern`/`dir_pattern` is useful to filter needless files/subdirectories. In the case of using two or more
// > different directories, it's recommended to setup separate pipelines for each.
WatchingDir string `json:"watching_dir"` // *

// > @3@4@5@6
// >
// > Files that don't meet this pattern will be ignored.
// > > Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
FilenamePattern string `json:"filename_pattern" default:"*"` // *

// > @3@4@5@6
// >
// > Dirs that don't meet this pattern will be ignored.
// > > Check out [func Glob docs](https://golang.org/pkg/path/filepath/#Glob) for details.
DirPattern string `json:"dir_pattern" default:"*"` // *
}

var offsetFiles = make(map[string]string)
Expand Down

0 comments on commit eea327b

Please sign in to comment.