cc @eparis @roberthbailey

Labelling this PR as size/S

@zmerlynn expressed some strong concerns about how well this is going to work over the lifetime of the release-1.2 branch. Right after the branch and during code slush, all commits apply cleanly to the release branch. As soon as we start refactoring the master branch, this is no longer going to be the case.

@zmerlynn estimated that somewhere up to 30% of cherry picks need to be manually edited when we applied them to the release-1.1 branch (after the initial release -- prior to that the number would have been much lower as the release-1.1 branch closely tracked the master branch). If you do cherry picks on a one-by-one basis, then this pain is foisted onto the individual who nominates the cherry pick. If you batch up multiple cherry picks into a single change, then you exponentially increase the chance of having a conflict. Automation isn't going to be able to automatically resolve these types of conflicts.

I think that the batch cherry pick process is great for the time between the first beta release and the first official release on a branch (and maybe the first patch release), but we should think hard about whether we should revert to individual cherry picks after that.

Having a centralized set of branch reviewers and a dashboard to look at cherry picks is awesome (and a huge improvement over what we had for the release-1.1 branch) but that doesn't imply that we shouldn't just pick changes in one-by-one.

Labelling this PR as size/M

GCE e2e build/test passed for commit 9a6ff3c72fdb90f476dd614378c8c3a9950f50e4.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test passed for commit 1f5c34ae19274ddfbdff99b3376e8049afbb4065.

• Build Log
• Test Artifacts
• Internal Jenkins Results

@roberthbailey @zmerlynn These are good points. Certainly there will be times when patches don't apply cleanly even if the order is controlled via a central person/mechanism. Both of those methods (out of order individual or ordered refactored) can and will produce conflicts.

One more automated way to solve this is by having a release-1.2-patch-list file maintained in the tree that ensures the correct ordering of the patches coupled with a continuous patch/build process on a copy of the release branch that excepts and notifies individuals when their patch doesn't apply. Order issues can be solved by editing the release-1.2-patch-list file and refactor/conflict issues can be resolved by the notified author applying a modified patch to the parallel tree and then updating the release-1.2-patch-list with that hash. The continuous process can ensure the final release-1.2-patch-list will apply cleanly and then just Do It at release time. Git probably has more nifty tricks we could use, but that's the basic idea.
This hoists all the work onto machines except for the one bit that needs an author. The author corrects when needed and the machine continues merrily on all the way through the final (or several batches) patch into the real release branch.

There's work obviously to get us to automation nirvana, but in the meantime, as you said, the branch owners doing the batching will find these errors and either solve the issues or notify the authors if more work needs to be done.

In lieu of the automation being in place, if we want to continue down the batching path, we'll see firsthand how much pain is experienced by the branch owner in using that method and that may accelerate the desire to automate, or not.

I'm against patch reordering/replay every patch release. The point cherry pick process has extreme value in continuous testing, and this batching process doesn't seem to solve any real problems but does seem to change things unnecessarily.

Without point cherry picks and CI on release branches, there's a lot of stuff we need to retool and assumptions we need to revisit.

I really liked batching for releases up to and including 1.2.0. It was useful to understand what was not being cherrypicked as well as what was being cherrypicked.

However, I think I agree with @roberthbailey that batching may be more trouble than it's worth after the rate of cherrypicks subsides and the divergence between the release branch and master increases.

In the week since 1.2.0, we've had 17 PRs merge with cherrypick-candidate. 2-4 were maybe dubious, but probably safe to cherrypick, and could be reviewed by the release owner as cherrypick PRs.

Cherrypicking out of order may still be a problem for generated code/docs, but it should be easy to regenerate in the release branch.

I'd like an easy way to figure out what has been cherrypicked since the previous release, but we need to solve that problem for release notes, anyway.

Speaking of release notes, we should probably require release notes for all cherrypick PRs.

For the 17 cherrypick candidates we currently have, I see a few options:

1. Batch them using the semi-manual process used prior to 1.2.0. Move release-1.2 forward to 3d58b9c07fdd2bf8cb063c30d791b94649b794e5 #22710 (comment)
2. Batch them using the existing cherrypick script
3. Ask the PR authors to move forward with creating cherrypick PRs

Any preferences?

GCE e2e build/test passed for commit 6e254ff5c972ad1cda0b5e51a8a213466f6e88fd.

• Build Log
• Test Artifacts
• Internal Jenkins Results

It sounds like we have some consensus here:

1. Batching during the branch point up to the first major X.X.0 release
2. Individual cherry-picking post X.X.0

I'll update the doc.

After a brief chat with @bgrant0607, we want to first get the initial 17 PRs that have piled up in using the @eparis method. Eric can you help with this?
Then I'll draft up an email to kubernetes-dev@ that we'd like to do a 1.2.1 next week and to get anything else in ASAP, using the individual cherry-pick method.

If you want to continue using @eparis's cherry-pick detection method as well, you're going to have to update the script as well, I think. (@eparis, for batching you were doing a merge commit so the original commits were reflected, right?) That's not that hard, it's just a bit obnoxious.

Then the script becomes "micro-batches".

I don't actually have a strong preference how we get those 17 PRs in.

GCE e2e build/test passed for commit 1b85546.

• Build Log
• Test Artifacts
• Internal Jenkins Results

I will do another batch tomorrow.

The bot which automatically clears 'cherrypick-candidate' looks for the same log message in the release branch is in the MERGE commit wihch went to master.

The hack/cherry-pick scripts use git am on the patch series. So the release branch gets all of the commit messages but does not get the merge commit.

The existing heuristic(s) is here:

https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/cherrypick-clear-after-merge.go#L103

If this is extended to look for the same commit message as EVERY commit which was part of the merge to master we should be able to get the bot and the hack script playing together.

I'm not certain when I'll really get to look at that.

Hm. Since the cherry-pick script was modified to use hub, which I didn't initially use when I first wrote it, it wouldn't be too hard to mock something the munger could cue off of. That would mean just modifying the cherry-pick script to copy the commit message from the original pull and blasting it into hub in some form, which is a little gross but would be a hacky workaround for now.

@zmerlynn this isn't so hard afterall, we can just have 'hub' explicitly set some string in the commit message which the bot queues off of. Should be a tiny change in the script and a tiny change in the bot, no?

although I am worried how to handle the case where someone picks 2+ PRs at once and uses --skip during a conflict. How would the script know to not include the skipped PR/commit in the list of things we mention in the hub commit message...

This PR is good to go. Can I get a LGTM? This is a doc change explaining the process for pre and post X.X.0. The discussion on how to proceed on the current set of 1.2.0 patches and how to do this going forward can happen on the other issue discussing the automation.

LGTM