[v1.16] ctmap/gc: don't clamp conntrack scan timeout in CI #37380
+2
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, the agent panics if the initial ctmap/nat GC scan does not complete within an hard-coded timeout of 30 seconds. Additionally, this timeout is further reduced to 5 seconds in the Ginkgo tests. There, we are observing quite often failures in the datapath-misc matrix entry, due to agents crashes caused by this timeout being hit (most likely due to resource contention in the CI environment). Let's try avoiding clamping this value in CI, and see if flakiness decreases or there's a more serious issue under the hood. Note that this issue does not affect v1.17 and later, as refactored by fea19ec ("ctmap/gc: do not terminate agent fatal on ctmap gc init after timeout"). Nor it affects v1.14, as the time wrapper had not been introduced yet there. Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
giorio94
added
release-note/ci
maintainer-s-little-helper
bot
added
backport/1.16
giorio94
changed the title
|
/test |
julianwiedmann
requested review from
a team and
jibi
and removed request for
a team
julianwiedmann
approved these changes
Feb 10, 2025
There was a problem hiding this comment.
makes sense, thank you!
@joestringer fyi, as it's "fallout" from #27253.
|
Curious do we have any theories why this tends to hit |
I didn't check too much the details, but one explanation could be simply higher resource contention, which makes everything significantly slower. AFAIR that test entry creates a significant number of pods on the test cluster. |
2 tasks
nbusseneau
added
backport-pending/1.15
github-actions
bot
added
backport-done/1.15
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the agent panics if the initial ctmap/nat GC scan does not complete within an hard-coded timeout of 30 seconds. Additionally, this timeout is further reduced to 5 seconds in the Ginkgo tests. There, we are observing quite often failures in the datapath-misc matrix entry, due to agents crashes caused by this timeout being hit (most likely due to resource contention in the CI environment). Let's try avoiding clamping this value in CI, and see if flakiness decreases or there's a more serious issue under the hood.
Note that this issue does not affect v1.17 and later, as refactored by fea19ec ("ctmap/gc: do not terminate agent fatal on ctmap gc init after timeout"). Nor it affects v1.14, as the time wrapper had not been introduced yet there.