Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] adding vpa support #308

Merged
merged 1 commit into from
Jan 20, 2023
Merged

[FEAT] adding vpa support #308

merged 1 commit into from
Jan 20, 2023

Conversation

nicolastakashi
Copy link
Contributor

Adding vpa support as requested: #307

@stevehipwell
Copy link
Collaborator

@nicolastakashi this PR would introduce a breaking change so might be tricky to get through as is. Out of interest what is your use case for using the VPA for a Deployment over the HPA?

There are currently discussions about introducing focused charts specifically for collection (DaemonSet) and aggregation (StatefulSet/Deployment) with strong SemVer guarantees. The changes you've suggested here are similar to what I was thinking of for the aggregator although I was also going to support using the addon-resizer when in StatefulSet mode in addition to the VPA (not the HPA with the current volume behaviour).

@nicolastakashi
Copy link
Contributor Author

Hi, @stevehipwell when I changed it I thought that could be tricky because of the breaking change, but we can try a different approach to avoid this breaking change.

Regarding the use case for VPA for Deployments in my view, there are two:

  1. VPA has a recommendation mode, that only provides recommendations about resource usage, it's nice if you don't want to automatically scale your resources or you just want to give insights to later change it manually.

  2. You can combine VPA with HPA using a custom metric or keda, I don't know if fluent-bit is a metric that measures the amount of logs collected, if so the user can horizontally scale the number of replicas by this metric, while using VPA to scale CPU and Memory (I know this could not be useful on the fluent-bit use case)

@stevehipwell
Copy link
Collaborator

@nicolastakashi if you're using a deployment (so no persistence) then I'm not convinced about the use cases other than to recommend on the resource settings. The other scenarios are better suited to the HPA especially as restarting the Fluent Bit pod could cause data loss so probably shouldn't be used in a feedback loop situation.

If Fluent Bit is a stateful set with persistence then I can see more value from the VPA as it couldn't use the HPA and would need scaling vertically.

If you really want to land this and you make the changes non-breaking we could add it.

@nicolastakashi
Copy link
Contributor Author

@stevehipwell I'll change the implementation to avoid the breaking change 😄

@nicolastakashi nicolastakashi force-pushed the main branch 4 times, most recently from 195246a to 2dd6e23 Compare January 20, 2023 09:23
@nicolastakashi
Copy link
Contributor Author

@stevehipwell can you take a second look pls?

Copy link
Collaborator

@stevehipwell stevehipwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicolastakashi I've reviewed the changes and added some suggestions.

charts/fluent-bit/templates/vpa.yaml Outdated Show resolved Hide resolved
charts/fluent-bit/templates/vpa.yaml Show resolved Hide resolved
charts/fluent-bit/templates/vpa.yaml Outdated Show resolved Hide resolved
charts/fluent-bit/values.yaml Outdated Show resolved Hide resolved
charts/fluent-bit/values.yaml Outdated Show resolved Hide resolved
@nicolastakashi nicolastakashi force-pushed the main branch 3 times, most recently from afe6ccf to deb4dfb Compare January 20, 2023 11:06
@nicolastakashi nicolastakashi requested review from stevehipwell and removed request for edsiper and naseemkullah January 20, 2023 11:09
Copy link
Collaborator

@stevehipwell stevehipwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicolastakashi could you bump the version in Chart.yaml and update the annotations to cover your change?

@nicolastakashi
Copy link
Contributor Author

@stevehipwell can you check if I did it properly?

Copy link
Collaborator

@stevehipwell stevehipwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of minor suggestions.

charts/fluent-bit/Chart.yaml Outdated Show resolved Hide resolved
charts/fluent-bit/Chart.yaml Outdated Show resolved Hide resolved
@stevehipwell
Copy link
Collaborator

@nicolastakashi it looks like you've lost your code from the commit?

@nicolastakashi
Copy link
Contributor Author

Hey @stevehipwell yeah I made a mistake but I was able to recovery hehe.
Could you take a look again?

@stevehipwell
Copy link
Collaborator

@nicolastakashi have you tested the functionality you've added locally?

Signed-off-by: Nicolas Takashi <nicolas.takashi@coralogix.com>
@nicolastakashi
Copy link
Contributor Author

@stevehipwell yup, just to evidence.

image

@stevehipwell
Copy link
Collaborator

@nicolastakashi assuming the CI passes are you happy for this to be released?

@nicolastakashi
Copy link
Contributor Author

@stevehipwell yeah, let's party 🥳 😄

Copy link
Collaborator

@stevehipwell stevehipwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stevehipwell stevehipwell merged commit 1b148fe into fluent:main Jan 20, 2023
Matiasmct pushed a commit to giffgaff/fluent-helm that referenced this pull request Oct 24, 2023
Signed-off-by: Nicolas Takashi <nicolas.takashi@coralogix.com>

Signed-off-by: Nicolas Takashi <nicolas.takashi@coralogix.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants