Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Something about the go and toolchain directive in the go.mod file #9489

Closed
StefMa opened this issue Aug 20, 2024 · 8 comments
Closed

Something about the go and toolchain directive in the go.mod file #9489

StefMa opened this issue Aug 20, 2024 · 8 comments
Labels
enhancement a request to improve CLI needs-triage needs to be reviewed needs-user-input

Comments

@StefMa
Copy link

StefMa commented Aug 20, 2024

Hey together,

I just came across your go.mod file. I have to admit I am a bit confused how you decided the values of the go and toolchain directive and I think those should be changed... 🤔

Let me explain this.

Note

I am not an go developer by day. Also the toolchain feature is fairly new in the go ecosystem.
Therefore it might be the case that Iam totally wrong with all of this here. If so, please let me know 🙃.

Starting with go 1.21 the behavior of the go directive in the go.mod file changed. Before that version, the statments acts more like an "information". It just indicates what go version that module needs to be compiled with. However, go was not very strict with that. If you used an go version lower than that, go just tried to compile the module. If its succeeded, it's fine, otherwise it's throws an (compile) error.

In go 1.21 the go statement indicates the minimum go version required to build that module. Same as before, but now more strict.

See the docs here
https://go.dev/ref/mod#go-mod-file-go

In go 1.21 they also introduce the toolchain directive. With that you can create reproducible builds across the developer (and CI) toolchain. As go will make sure it will always use that exactt go version to compile the module. It does this by downloading the specified go version if your local go version is lower than that.

See the docs here
https://go.dev/doc/toolchain

With that change, they also changed their versioning. In the past a major.minor version was released. They changed that to release a full semver version instead. With that said, to a major.minor.patch version. (Notice the .patch 😉)

See the docs here
https://go.dev/doc/toolchain#version

I guess using a non-patch version for higher go versions than 1.21 will only work because of backwards compatibility.
But in fact, it is "wrong". Or at least not considered as best practise anymore.


How does this relates to this project?

In this PR @matthewhughes934 wanted to fix your go.mod file by introducing the patch version. He also provided a lot of useful links there.
(Note: I don't know why he faced the docker issue, I am not an expert in that field. But regardless of that error, using a patch version is more correct!)
You can checkout his first commit in this PR to see his change:
9fae21f

For reasons he decided together with @williammartin to introduce the toolchain directive.

That also fixed that issue. But not because "it is more correct" (to fix that specific error). But instead it would solve that problem because with the toolchain feature, go will look at this (and download the correct version) and not at the (still wrong) go directive.
I guess 🫠


Later this PR were opened.
Besides of other changes, the go directive was changed to 1.22.0 (so basically what the first PR also wanted to do 😁 ), AND the toolchain was changed to 1.22.5 (from 1.22.2).
Here is also a nice comment by @williammartin on that PR, saying "we should report upstream that using a patch version is "wrong"". However, as already pointed out above, using a patch version is actually there more correct usage now. There is no release in go without a patch version anymore!

So actually using 1.22.0 is correct, while using 1.22 for the go directive is just wrong.


Something about CI.

Right now you are using the setup-go action together with the go-version-file: 'go.mod'.
Because you also declared the toolchain directive, you download right now 2 versions on your CI.
First, the setup-action will download 1.22.0 (based on your go.mod go directive), afterwards go (version 1.22.0 you just downloaded) will download 1.22.5. This is because of your toolchain directive in your go.mod file.

Since 1.21 is the default go version on github hosted runners, I guess you could avoid downloading 1.22.0 and just rely on the toolchain directive. But I'm not 100% sure with that 🤷 At least, we could save a few seconds on CI 🤓

Screenshot 2024-08-20 at 2 16 47 PM

To summarize:

  • The go directive declares the minimum version now. Nothing more.
  • The go directive should be a valid semver version. (major.minor.patch)
  • The toolchain is the go version that will be used to compile the module

Because this CLI project here is a "final end product" and not a module that others use, I guess you are fine by just using the go directive that uses always the latest go version.
Probably "no one" (assuption) depends on this project and therefore you might not need to declare any backward compatibily with others.
When you decide that, you might want to remove the toolchain directive. Why? Because starting with go 1.21, go will automatically download the version declared in the go directive. Without an toolchain, it will use automatically the go declared version as a toolchain version and hence download it.
So there is no need for this.

On the other side, if you think that other go projects depends on you, then it make sense to have both, go and toolchain.
While go should be "as low as possible", while the toolchain can be always be on the edge to enjoy bug fixes, faster compile time (assuming a greater go version is faster 😅 ), new features, and a streamlined go version across developers (and CI).

What do you think? 🤔

@StefMa StefMa added the enhancement a request to improve CLI label Aug 20, 2024
@cliAutomation cliAutomation added the needs-triage needs to be reviewed label Aug 20, 2024
@williammartin
Copy link
Member

williammartin commented Aug 20, 2024

What do you think? 🤔

I think this topic is slowly killing me. 😅

Thank you for your comprehensive write up. I think what we need to definitely iron out here is what the valid versions are for the go directive.

My position (loosely held because I find the Go docs for this hard to read) is that 1.22 is valid, where you believe we need 1.22.x.

Let's start with this statement:

A go directive indicates that a module was written assuming the semantics of a given version of Go. The version must be a valid Go version, such as 1.9, 1.14, or 1.21rc1.

Jumping over to the go version doc:

The syntax ‘1.N’ is called a “language version”. It denotes the overall family of Go releases implementing that version of the Go language and standard library.

And:

Any two Go versions can be compared to decide whether one is less than, greater than, or equal to the other
...
For example, 1.21 < 1.21rc1 < 1.21rc2 < 1.21.0 < 1.21.1 < 1.21.2.

From this paragraph, 1.21 is being listed as a comparable Go version. This leads me to believe that all language versions are go versions but only some go versions are language versions.

As you called out:

While go should be "as low as possible"

Since I'm not aware of any language changes (like range-over-func) coming in patch releases, it makes the most sense in my mind to use the language version.

This has the advantage of not forcing every downstream consumer of your module to bump every time you change the patch version of your go.mod for no real reason (I understand you're not saying that people should do this, only that you think a patch version is required).

As for the CI stuff, thank you very much for bringing it up. I'll definitely have a think about it but it feels like addressing this patch disagreement first seems the best.

If you have any further reading about the necessity of patch versions in the go directive, please, please, please point me to it. Each of your well-researched comments of mine above were born out from me searching for this information. My decision to pin the go directive to the language version was primarily driven by looking at other large Go projects and seeing at least some of them behaved the same as I anticipated.

Cheers!

@StefMa
Copy link
Author

StefMa commented Aug 20, 2024

I think this topic is slowly killing me. 😅

Hehe, can fully understand that 🙃.

In regards of the patch version as go directive, I decided to ask the expects.
👉 golang/go#68971
Lets see what they say about that.
In fact, a simple go 1.23 will fail, while go 1.23.0 will not.
Assuming (to me) that the docs are outdated (of course, we are devs, docs are always outdated).

Since I'm not aware of any language changes (like range-over-func) coming in patch releases, it makes the most sense in my mind to use the language version.

What would you expect from go in case you use the language version?
Should it always download the latest patch version (based on the language version)?
Therefore, "each time" you run any go command, it wil do an HTTP request, checking if you're up-to-date? 🤔
("Each time" = maybe this can be smarter like caching this value for 24 hours or so).

My decision to pin the go directive to the language version was primarily driven by looking at other large Go projects and seeing at least some of them behaved the same as I anticipated.

Did you looked at projects that are actual modules, so a small piece of software that can be included by someone? 🤔
If so, maybe you looked at projects that have a go directive lower than 1.21 and therefore "working correctly", as versions before 1.21 are allowed to use a non-patched version... 💭
And of course, they use an lower go version because this defines the minimum required go version now. Increasnig that, will limited to consumers of this library quite a lot...

Small update:

Seems that is related (haven't read it yet):
golang/go#65631 that leads to golang/go#62278

@williammartin
Copy link
Member

In regards of the patch version as go directive, I decided to ask the expects.

Thanks! I appreciate you asking this question. I understand why go: 1.22 and no toolchain results in an error (because the toolchain is required to be a toolchain name as per the syntax):

ToolchainDirective = "toolchain" ToolchainName newline .
ToolchainName = string | ident .  /* valid toolchain name; see “Go toolchains” */

What would you expect from go in case you use the language version?

That's a good question. In our particular usage, I would only expect the toolchain to be downloaded and it has a specific patch version. The go directive would only be used to ensure language feature usage is valid.

In the case of go: 1.22 and no toolchain, I'm not sure what reasonable behaviour is.

Did you looked at projects that are actual modules, so a small piece of software that can be included by someone? 🤔

I think in an earlier comment I linked to etcd but you can look through https://github.com/search?q=path%3A**%2Fgo.mod+%2F%5Ego+1.22%24%2F&type=code

@StefMa
Copy link
Author

StefMa commented Aug 21, 2024

I understand why go: 1.22 [..] results in an error

Well, they changed that behaviour in 1.23, 1.22.4, and 1.21.9 🙈
This is not an error and using a language version in those (and future) versions is not an error.
They decided that in case you specify and langauge version as go, but not toolchain, to download the 0 (zero) patch version.

See golang/go#68971 (comment)
that leads to golang/go#62278 (comment) and golang/go#62278 (comment)

So using a language version in go is not possible in 1.21.0 -1.21.8 and 1.22.0 - 1.22.4.
In other versions it works again and download the 0 (zero) patch version if no toolchain is declared.

This also answers my question:

What would you expect from go in case you use the language version?

It will download the 0 (zero) patch version as toolchain.

Because, in fact, if no toolchain is declared, it will use the go declared version as toolchain.

Still, the question is "using a patch version YES or NO?".
I asked this in my golang issue and they answered:

The release versions are expected and generated by go tooling.

The release versions (starting with 1.21) including patch versions.
There is no 1.21, 1.22, and 1.23 release versions.
So they want that people using a patch version!


In general the issue golang/go#62278 is quite interesting to read.
On mentioned (can't find it right now) somehting like:

The confusing is clear, before 1.21 they forced you to use a non-patch version, after that release, they forced you to use a patch version

This is of course 100% confusing for everyone.


With that being said, I guess the github/cli/cli go.mod file is doing fine by declaring 1.22.0 🙃

cli/go.mod

Line 3 in 95a2f95

go 1.22.0

Agree?

I'm happy to discuss the next chapter of that toolchain 🤓
But I'll leave it out for now to not mix stuff.

@matthewhughes934
Copy link
Contributor

As for the CI stuff, thank you very much for bringing it up. I'll definitely have a think about it but it feels like addressing this patch disagreement first seems the best.

I've also spent too much time thinking about this 🙃, here's a relevant issue in setup-go actions/setup-go#457, the CI output in this repo also includes a bunch of tar errors which are also related to the toolchain handling actions/setup-go#424

@williammartin
Copy link
Member

Thanks so much for the investigation @StefMa. If the go tooling produces patch versions I'm happy to know that the CLI is fine as it is. I think the most important thing generally is that the go version isn't bumped simply to "move to the latest version of Go" because of the effect it has on all downstream consumers.

So with regards CI, I guess the primary complaint is that we're downloading 1.22.0 because setup-go is using the go directive in the go.mod, and then immediately after, go is respecting the toolchain directive and downloading 1.22.5. This seems to be the issue @matthewhughes934 has raised here: actions/setup-go#457

Was there anything I missed?

@StefMa
Copy link
Author

StefMa commented Aug 21, 2024

I think the most important thing generally is that the go version isn't bumped simply to "move to the latest version of Go" because of the effect it has on all downstream consumers.

This is something I mentioned in my initial description of this issue 🤓.
I think the CLI itself can "always use the latest version" as this is an "end-product".
The CLI act as a, well, CLI and not as a module that can(should?, is intended to?) be imported in another go module.

So in my hoenstly opinion, youre fine to use the latest version in the go line.

If you don't want/like this (because maybe someone else uses this CLI as a module and you don't want to annoy them), you can keep it like it is of course.

But this leads to another question/suggestions/improvement in my initial description issue.

If you want to stay at 1.22.0 to keep backwards compatibily, but still want to enjoy reproducible builds as well as a streamlined developer (and CI) workflow, pointing the toolchain "always moved to the latest version".
With that you can make sure that you enjoy faster compile times, using the same go version across the ecosystem, etc.

Being said, toolchain can now be moved to 1.23.0 on the CLI 🤓.

Or, you can simply say "I don't want to turn my head around toolchain", this drives me crazy, you can also go ahead and remove the toolchain line in the go.mod file. As using 1.22.0 in the go statement will automatically use 1.22.0 as a toolchain as well.

To summarize:

  • Either only go "pointing always to the latest release"
    • Benefit: You enjoy always the latest and greatest
    • Downside: You don't maintain backward compatibilty with consumers of the CLI.
  • Use go pointing "to your minimal support go version" PLUS using toolchain
    • Benefit: You enjoy always the latest and greates in regards of compiling, you have a streamlined go version across the ecosystem
    • Downside: You can't always use the latest and greatest features as you have to maintain go 1.22 as the minimal support version

Final summary:
I guess youre fine what you have right now.
Maybe toolchain can be bumped to 1.23.0 or removed completely 🤷
I would prever to bump it, but that is not my decision.

So with regards CI, I guess the primary complaint is that we're downloading 1.22.0 because setup-go is using the go directive in the go.mod, and then immediately after, go is respecting the toolchain directive and downloading 1.22.5. This seems to be the issue @matthewhughes934 has raised here: actions/setup-go#457

That is exactly the point, yes 🙃
But unfourtnaly it seems no one from github takes care of this 😭
Maybe you can raise this problem to someone at github 😉


Feel free to close this issue aftwards.
And thank you for this nice discussion and back and worth.
Much appreacheated it.
I learned a lot ❤️

@williammartin
Copy link
Member

The CLI act as a, well, CLI and not as a module that can(should?, is intended to?) be imported in another go module.

Sometimes people do important parts of the module: https://github.com/search?type=code&q=path%3A**%2Fgo.mod+%22github.com%2Fcli%2Fcli%22

There's not really an intention to support or not to support this. Personally, I wouldn't have a problem making breaking changes in the cli module to improve our lives, but I also don't want to make life harder for anyone else unnecessarily.

Being said, toolchain can now be moved to 1.23.0 on the CLI 🤓.

I'd be happy to have a PR for that, please open one if you'd like to. I don't want to bump the go directive because ^

But unfourtnaly it seems no one from github takes care of this 😭

I'll see what I can do here. I'm sure people care but I know the folks in actions have a lot on their plate.

Thanks for all the conversation and investigation here to bring clarity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement a request to improve CLI needs-triage needs to be reviewed needs-user-input
Projects
None yet
Development

No branches or pull requests

4 participants