-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix logrotate config on GCI #29139
Fix logrotate config on GCI #29139
Conversation
daily | ||
dateext | ||
dateformat -%Y%m%d-%s |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add a comment about the significance (i.e collisions, and that exceeding maxsize within a second will not rotate). A meta comment about how we expect logorate to happen will also help, something like: "logrotate daily or if the size exceeds maxsize, into a gzipped timestamped backup. Backups are discarded after ."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we going backwards on maxsize
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah. From my testing, this is the only combination (maxsize with dateformat) that gives us expected behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. So this will stamp out >100M files every 1h on busy clusters?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. I tried explain a bit in the comment. PTAL.
LGTM just comment nit |
@zmerlynn fyi, Aditya ran some tests and found the old config lacking. I tested this version, looks ok. |
reapplying lgtm |
we need to add the dateformat option so that the logrotate can create unique logfiles for each rotation. Without this, we logrotation is skipped with message like (generated in verbose mode of logrotate): rotating log /var/log/rotate-test.log, log->rotateCount is 5 dateext suffix '-20160718' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' destination /var/log/rotate-test2.log-20160718.gz already exists, skipping rotation Tested as follows: # config in '/etc/logrotate.d/rotate-test': /var/log/rotate-test.log { rotate 5 copytruncate missingok notifempty compress maxsize 100M daily dateext dateformat -%Y%m%d-%s create 0644 root root } # create 150Mb of /var/log/rotate-test.log $ dd if=/dev/zero of=/var/log/rotate-test.log bs=1048576 count=150 conv=notrunc oflag=append # run logrotate $ /usr/sbin/logrotate -v /etc/logrotate.conf ... rotating pattern: /var/log/rotate-test.log after 1 days (5 rotations) empty log files are not rotated, log files >= 104857600 are rotated earlier, old logs are removed considering log /var/log/rotate-test.log log needs rotating rotating log /var/log/rotate-test.log, log->rotateCount is 5 Converted ' -%Y%m%d-%s' -> '-%Y%m%d-%s' dateext suffix '-20160718-1468875268' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' copying /var/log/rotate-test.log to /var/log/rotate-test.log-20160718-1468875268 truncating /var/log/rotate-test.log compressing log with: /bin/gzip Repeating 'dd' and 'logrotate' commands now generate logfiles correctly.
@@ -77,12 +77,21 @@ function setup-logrotate() { | |||
compress | |||
maxsize 10M |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to bring this to your attention @zmerlynn @bprashanth . The 10M size for docker-container log seems arbitrary too. But I guess we don't encourage users to log to stdout/stderr from their containers. So this may not be an issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought stdout/stderr was the default way fluentd picked up the container logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind. I recalled some discussion on google-containers group recommending to redirect logs to files.
So, is 10Mb here fine?
Yes, since fluentd should be slurping the logs up, in theory. |
@kubernetes/goog-testing I've see this error quite a bit off late:
Does that mean rebase (github usually detcts rebase though right), or flake? |
That's #27462, it's a flake in the Jenkins plugin. |
GCE e2e build/test passed for commit 09b2c27. |
prio 0 because 1.3 |
@k8s-bot test this [submit-queue is verifying that this PR is safe to merge] |
GCE e2e build/test passed for commit 09b2c27. |
Automatic merge from submit-queue |
Commit found in the "release-1.3" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked. |
…pick-of-#29139-upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of kubernetes#29139 Cherry pick of kubernetes#29139 on release-1.3.
we need to add the dateformat option so that the logrotate
can create unique logfiles for each rotation. Without this,
logrotation is skipped with message like (generated in
verbose mode of logrotate):
rotating log /var/log/rotate-test.log, log->rotateCount is 5
dateext suffix '-20160718'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
destination /var/log/rotate-test2.log-20160718.gz already exists, skipping rotation
Tested as follows:
config in '/etc/logrotate.d/rotate-test':
/var/log/rotate-test.log {
rotate 5
copytruncate
missingok
notifempty
compress
maxsize 100M
daily
dateext
dateformat -%Y%m%d-%s
create 0644 root root
}
create 150Mb of /var/log/rotate-test.log
$ dd if=/dev/zero of=/var/log/rotate-test.log bs=1048576 count=150 conv=notrunc oflag=append
run logrotate
$ /usr/sbin/logrotate -v /etc/logrotate.conf
...
rotating pattern: /var/log/rotate-test.log after 1 days (5 rotations)
empty log files are not rotated, log files >= 104857600 are rotated earlier, old logs are removed
considering log /var/log/rotate-test.log
log needs rotating
rotating log /var/log/rotate-test.log, log->rotateCount is 5
Converted ' -%Y%m%d-%s' -> '-%Y%m%d-%s'
dateext suffix '-20160718-1468875268'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
copying /var/log/rotate-test.log to /var/log/rotate-test.log-20160718-1468875268
truncating /var/log/rotate-test.log
compressing log with: /bin/gzip
Repeating 'dd' and 'logrotate' commands now generate logfiles correctly.
#27754
@bprashanth can you please review?