-
Notifications
You must be signed in to change notification settings - Fork 909
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flag excess drops #561
Merged
Merged
Flag excess drops #561
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kaizhe
reviewed
Mar 15, 2019
Kaizhe
reviewed
Mar 15, 2019
mattpag
approved these changes
Mar 19, 2019
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just a couple of comments
mstemm
force-pushed
the
flag-excess-drops
branch
from
March 26, 2019 19:13
226b0e3
to
132034b
Compare
New argument --stats_interval=<msec> controls the interval at which statistics are written to the stats file. The default is 5000 ms (5 sec) which matches the prior hardcoded interval. The stats interval is triggered via signals, so an interval below ~250ms will probably interfere with falco's behavior.
A new method falco_outputs::handle_msg allows emitting generic messages that have a "rule", message, and output fields, but aren't exactly tied to any event and aren't passed through an event formatter. This allows falco to emit "events" based on internal checks like kernel buffer overflow detection.
Log messages from falco_logger::log may or may not have trailing newlines. Handle both by always adding a newline to stderr logs and always removing any newline from syslog logs.
New variant of get_sequence that allows fetching a list of items from a key + subkey, for example: key: subkey: - list - items - here Both use a shared method get_sequence_from_node().
Start actively monitoring the kernel buffer for syscall event drops, which are visible in scap_stats.n_drops, and add the ability to take actions when events are dropped. The -v (verbose) and -s (stats filename) arguments also print out information on dropped events, but they were only printed/logged without any actions. In falco config you can specify one or more of the following actions to take when falco notes system call drops: - ignore (do nothing) - log a critical message - emit an "internal" falco alert. It looks like any other alert with a time, "rule", message, and output fields but is not related to any rule in falco_rules.yaml/other rules files. - exit falco (the idea being that the restart would be monitored elsewhere). A new module syscall_event_drop_mgr is called for every event and collects scap stats every second. If in the prior second there were drops, perform_actions() handles the actions. To prevent potential flooding in high drop rate environments, actions are goverened by a token bucket with a rate of 1 actions per 30 seconds, with a max burst of 10 seconds. We might tune this later based on experience in busy environments. This might be considered a fix for #545. It doesn't specifically flag falco rules alerts when there are drops, but does make it easier to notice when there are drops.
Add unit tests for syscall event drop detection. First, add an optional config option that artifically increments the drop count every second. (This is only used for testing). Then add test cases for each of the following: - No dropped events: should not see any log messages or alerts. - ignore action: should note the drops but not log messages or alert. - log action: should only see log messages for the dropped events. - alert action: should only see alerts for the dropped events. - exit action: should see log message noting the dropped event and exit with rc=1 A new trace file ping_sendto.scap has 10 seconds worth of events to allow the periodic tracking of drops to kick in.
mstemm
force-pushed
the
flag-excess-drops
branch
from
March 26, 2019 23:39
e8a0456
to
dd45d44
Compare
Also added a wiki page documenting the feature: https://github.com/falcosecurity/falco/wiki/Actions-For-Dropped-System-Call-Events and notes to the falco config wiki page for the new config options. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For more information on this PR, read our blog post on CVE-2019-8339.
Changes to monitor for dropped system calls/kernel buffer exhaustion and take configurable actions when this occurs.
Certainly related to and might be a fix for #545.