Kernel Monitor: Add look back support and kernel panic handling #22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Random-Liu
force-pushed
the
add-look-back
branch
from
f398ef0 to
43bd62d
Compare
|
I've updated and verified the PR. Since this PR added look back support, if there was a kernel panic in last run, node problem detector will report a node event after reboot similar with: During look back, node problem detector will not keep historical node conditions (since the node has rebooted), but will report historical node events. |
| @@ -1,5 +1,7 @@ | |||
| { | |||
| "logPath": "/log/kern.log", | |||
| "lookback": "1h", | |||
There was a problem hiding this comment.
looking back 1h might be too long to find panic information? We only care about the most recent, right? Changing it to 5m, or 10m maybe?
1) Add lookback support in kernel monitor. After started, Kernel monitor will check some old logs to detect problems which happened before last node reboot. 2) Add `lookback` and `startPattern` in kernel monitor configuration. * `lookback` specifies how long time kernel monitor should look back. * `startPattern` specifies which log indicates the node is started. kernel monitor will clear all current node conditions once it finds a node start log. This makes sure that old problems won't change the node condition. 3) Add support for kernel panic monitoring, the null pointer and divide 0 kernel panic will be surfaced as event. Usually kernel monitor will report these events during looking back phase.
Random-Liu
force-pushed
the
add-look-back
branch
from
43bd62d to
532f933
Compare
|
@dchen1107 Addressed comments. |
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This helps kubernetes/kubernetes#27885.
This PR:
Add lookback support in kernel monitor. After started, Kernel monitor will check some old logs to detect old problems which happened before last node reboot.
Add
lookbackandstartPatternin kernel monitor configuration.lookbackspecifies how long time kernel monitor should look back.startPatternspecifies which log indicates the node is started. kernel monitor will clear all current node conditions once it finds a node start log. This makes sure that old problems won't change the node condition.I've cut a branch v0.1 before this PR, and bump up the image version to v0.2 in this PR.
@dchen1107
/cc @kubernetes/sig-node