Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix logrotate config on GCI #29139

Merged
merged 1 commit into from
Jul 19, 2016
Merged

Fix logrotate config on GCI #29139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions cluster/gce/gci/configure-helper.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,21 @@ function setup-logrotate() {
compress
maxsize 10M
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to bring this to your attention @zmerlynn @bprashanth . The 10M size for docker-container log seems arbitrary too. But I guess we don't encourage users to log to stdout/stderr from their containers. So this may not be an issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought stdout/stderr was the default way fluentd picked up the container logs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nevermind. I recalled some discussion on google-containers group recommending to redirect logs to files.
So, is 10Mb here fine?

daily
dateext
dateformat -%Y%m%d-%s
create 0644 root root
}
EOF

# Configuration for k8s services that redirect logs to /var/log/<service>.log
# files.
# files. Whenever logrotate is ran, this config will:
# * rotate the log file if its size is > 100Mb OR if one day has elapsed
# * save rotated logs into a gzipped timestamped backup
# * log file timestamp (controlled by 'dateformat') includes seconds too. this
# ensures that logrotate can generate unique logfiles during each rotation
# (otherwise it skips rotation if 'maxsize' is reached multiple times in a
# day).
# * keep only 5 old (rotated) logs, and will discard older logs.
local logrotate_files=( "kube-scheduler" "kube-proxy" "kube-apiserver" "kube-controller-manager" "kube-addons" )
for file in "${logrotate_files[@]}" ; do
cat > /etc/logrotate.d/${file} <<EOF
Expand All @@ -92,8 +101,10 @@ EOF
missingok
notifempty
compress
size 100M
maxsize 100M
daily
dateext
dateformat -%Y%m%d-%s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a comment about the significance (i.e collisions, and that exceeding maxsize within a second will not rotate). A meta comment about how we expect logorate to happen will also help, something like: "logrotate daily or if the size exceeds maxsize, into a gzipped timestamped backup. Backups are discarded after ."

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we going backwards on maxsize?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. From my testing, this is the only combination (maxsize with dateformat) that gives us expected behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. So this will stamp out >100M files every 1h on busy clusters?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I tried explain a bit in the comment. PTAL.

create 0644 root root
}
EOF
Expand Down
4 changes: 3 additions & 1 deletion cluster/saltbase/salt/logrotate/conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@
missingok
notifempty
compress
size 100M
maxsize 100M
daily
dateext
dateformat -%Y%m%d-%s
create 0644 root root
}