Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gengo performance: Speed up update-codegen.sh by switching to FormatOnly in gengo #127011

Merged
merged 4 commits into from
Sep 4, 2024

Conversation

jpbetz
Copy link
Contributor

@jpbetz jpbetz commented Aug 30, 2024

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:

Avoid expensive per-target calls out to goimports from gengo.

This improve codegen update and verify times by about 25%.

Special notes for your reviewer:

This is based on f63343f from @thockin.

This depends on a gengo/v2 change: kubernetes/gengo#277

The main changes here are to fix sloppy import handling that was masked by gengo's import fixup, disable gengo's import fix up with kubernetes/gengo#277, and then generate with the import name changes that result.

The main import changes are:

  • Before, conversion-gen would import packages under the same package alias as the local package. For example, the conversion v1 package would import a types package using the v1 package alias. This is legal go code, but is less clear than an import like corev1 or metav1, so we've added a rule that imported type aliases must not match the local package alias exactly.
  • Before, generated code like "fmt.Errorf" could be used without importing "fmt" and goimports would fix up the imports by adding "fmt". Now, our gengo adds all imports and follows the gengo convention of redundantly defining the import as fmt "fmt". We could further change gengo to avoid redundant import aliases, but for this change we're just switching the codegen to be consistent for all types, using gengo's conventions, instead of a mix or gengo and goimport conventions.

Does this PR introduce a user-facing change?

NONE

/sig api-machinery
/assign @thockin @skitt

@k8s-ci-robot k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Aug 30, 2024
@k8s-ci-robot k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/apiserver area/cloudprovider area/code-generation area/dependency Issues or PRs related to dependency changes area/kube-proxy area/kubeadm area/kubectl area/kubelet kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. labels Aug 30, 2024
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 80a4306af57ec10c22e7574d3b301fa5369be70d

@thockin
Copy link
Member

thockin commented Sep 3, 2024

Ouch. There's no good answer here, is there. I guess this is a better result, so if we're going to churn, this seems overall preferable.

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2024
Copy link
Member

@thockin thockin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missed sending this

@@ -213,7 +213,6 @@ func (g *listerGenerator) Namers(c *generator.Context) namer.NameSystems {

func (g *listerGenerator) Imports(c *generator.Context) (imports []string) {
imports = append(imports, g.imports.ImportLines()...)
imports = append(imports, "k8s.io/apimachinery/pkg/api/errors")
imports = append(imports, "k8s.io/apimachinery/pkg/labels")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these also disappear if we are being explicit everywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mea culpa. I saw these in passing but didn't clean them up. I can do that now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@jpbetz
Copy link
Contributor Author

jpbetz commented Sep 3, 2024

/hold

@jpbetz jpbetz force-pushed the format-only-imports branch from 10e0a35 to b14b6dc Compare September 3, 2024 17:31
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2024
@k8s-ci-robot k8s-ci-robot requested a review from thockin September 3, 2024 17:31
@jpbetz
Copy link
Contributor Author

jpbetz commented Sep 3, 2024

/hold cancel
#127011 (comment) is fixed

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 3, 2024
…t generation of import to package being generated

Co-authored-by: Tim Hockin <thockin@google.com>
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Sep 3, 2024

@jpbetz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-kind-classic-dra a7c68f5 link false /test pull-kubernetes-kind-classic-dra
pull-kubernetes-e2e-kind-kms a7c68f5 link false /test pull-kubernetes-e2e-kind-kms
pull-kubernetes-kind-dra a7c68f5 link false /test pull-kubernetes-kind-dra
pull-kubernetes-node-e2e-crio-cgrpv1-dra a7c68f5 link false /test pull-kubernetes-node-e2e-crio-cgrpv1-dra
pull-kubernetes-node-e2e-crio-cgrpv2-dra a7c68f5 link false /test pull-kubernetes-node-e2e-crio-cgrpv2-dra
pull-kubernetes-node-e2e-containerd-1-7-dra a7c68f5 link false /test pull-kubernetes-node-e2e-containerd-1-7-dra

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@jpbetz jpbetz force-pushed the format-only-imports branch from b14b6dc to 2595aa1 Compare September 3, 2024 18:26
@k8s-triage-robot
Copy link

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@Jefftree
Copy link
Member

Jefftree commented Sep 3, 2024

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 3, 2024
@jpbetz
Copy link
Contributor Author

jpbetz commented Sep 3, 2024

@thockin @skitt I think this is ready.

@thockin
Copy link
Member

thockin commented Sep 3, 2024

Thanks!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: b34028ab5ea762e475779de1fa0f0153c224ecc0

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 9595667 into kubernetes:master Sep 4, 2024
26 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Sep 4, 2024
@@ -115,7 +115,7 @@ func GetTargets(context *generator.Context, args *args.Args) []generator.Target
gv: gv,
typesToGenerate: typesToRegister,
outputPackage: pkg.Path,
imports: generator.NewImportTracker(),
imports: generator.NewImportTrackerForPackage(pkg.Path),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xref breakage in register-gen associated with this gengo bump... #129290

unnoticed because nothing in-tree apparently uses register-gen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/cloudprovider area/code-generation area/dependency Issues or PRs related to dependency changes area/kube-proxy area/kubeadm area/kubectl area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Archived in project
Archived in project
Archived in project
Development

Successfully merging this pull request may close these issues.

7 participants