Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pkg/stanza] Issue with star pattern with mixed files and folders #16995

Closed
atoulme opened this issue Dec 13, 2022 · 3 comments · Fixed by #17362
Closed

[pkg/stanza] Issue with star pattern with mixed files and folders #16995

atoulme opened this issue Dec 13, 2022 · 3 comments · Fixed by #17362
Labels
bug Something isn't working needs triage New item requiring triage

Comments

@atoulme
Copy link
Contributor

atoulme commented Dec 13, 2022

Component(s)

pkg/stanza

What happened?

Description

In a set up where we have a folder containing folders and files, the filelog receiver will fail to start scanning for files if told to scan for any content in the folder (/var/log/*).

Steps to Reproduce

Attached example as zip file.

Set up a docker-compose with the contents:

version: "3"
services:
  splunk-otel-collector:
    image: otel/opentelemetry-collector-contrib:0.67.0
    volumes:
    - ./config.yaml:/etc/otelcol-contrib/config.yaml
    - ./foo:/var/foo

Set up config.yaml file:

receivers:
    filelog:
      include: [ /var/foo/* ]

exporters:
  logging:

processors:
  batch:

extensions:
    health_check:
      endpoint: 0.0.0.0:13133
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679

service:
    extensions: [pprof, zpages, health_check]
    pipelines:
      logs:
        receivers: [filelog]
        processors: [batch]
        exporters: [logging]

Create a foo folder, containing a log file and a bar folder:

mkdir -p foo/bar
touch foo/my.log
touch foo/bar/other.log

Run the collector.

Expected Result

The collector starts.

Actual Result

The collector continuously outputs an error:

splunk-otel-collector_1  | 2022-12-13T01:28:20.189Z	error	fileconsumer/file.go:178	Failed creating fingerprint	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/foo/bar: is a directory"}
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeReaders
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:178
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:125
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).poll
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:120
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller.func1
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:101
splunk-otel-collector_1  | 2022-12-13T01:28:20.189Z	error	fileconsumer/reader.go:78	Failed during scan	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/foo/bar", "error": "scanner error: read /var/foo/bar: is a directory"}
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Reader).ReadToEnd
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/reader.go:78
splunk-otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
splunk-otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.67.0/fileconsumer/file.go:137
splunk-otel-collector_1  | 2022-12-13T01:28:21.388Z	error	fileconsumer/file.go:178	Failed creating fingerprint	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/foo/bar: is a directory"}

Collector version

0.67.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

See above

Log output

No response

Additional context

No response
@atoulme atoulme added bug Something isn't working needs triage New item requiring triage labels Dec 13, 2022
@atoulme
Copy link
Contributor Author

atoulme commented Dec 13, 2022

Pinging @djaglowski

@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@djaglowski
Copy link
Member

djaglowski commented Dec 13, 2022 •
edited
Loading

If I'm understanding correctly, the doublestar library does not provide a way to match only files in such a case. We may have to post process the results to filter out directories. However, I've opened bmatcuk/doublestar#79 in hopes that this can be supported natively in the library.

@djaglowski djaglowski mentioned this issue Jan 3, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage New item requiring triage
Projects
None yet
Milestone
No milestone
2 participants
@atoulme @djaglowski