tinygo: -compiler command line argument (e.g. tinygo) #121

archie2x · 2024-08-06T17:00:24Z

help:

~~I haven't figured out how to propagate this to mkuimage.~~
it may make more sense as a buildopt instead of environ in build.go(?)
- but the BuildOpts isn't always available
name of flag? -go-compiler(?)
any feedback!

This allows specifying the compiler explicitly in makebb and handles peculiarities of tinygo. It works by resolving and caching the path to the go-compiler then queries tinygo for its hidden go-build-tags and propagates them to package lookup / install. This allows the pruning algorithm to correctly add / remove files when syncing dependent packages.

Usage:

% makebb -h 
Usage of makebb:
  -compiler string
    	override go compiler to use (e.g. "/path/to/tinygo")
[...]

Resulting busybox is significantly smaller. I compiled a slightly more complicated project than e.g. u-root/cmds/core/{true,false} to test package resolution.

Standard go:

u-root % GOOS=linux makebb cmds/core/init
10:10:54 Build environment: GOARCH=arm64 GOOS=linux GOPATH=/Users/roger/go CGO_ENABLED=0 GO111MODULE= GOROOT=/opt/homebrew/Cellar/go/1.22.2/libexec PATH=/opt/homebrew/Cellar/go/1.22.2/libexec/bin
10:10:54 Compiler: go version go1.22.2 darwin/arm64
10:10:59 Successfully built "bb" (size 2556039 bytes -- 2.4 MiB). <============

tinygo:

u-root % GOOS=linux makebb -compiler tinygo cmds/core/init
10:10:06 Build environment: GOARCH=arm64 GOOS=linux GOPATH=/Users/roger/go CGO_ENABLED=0 GO111MODULE= GOROOT=/opt/homebrew/Cellar/go/1.22.2/libexec PATH=/opt/homebrew/Cellar/go/1.22.2/libexec/bin
10:10:06 Compiler: tinygo version 0.33.0-dev-6184a6cd darwin/arm64 (using go version go1.22.2 and LLVM version 18.1.2)
10:10:20 Successfully built "bb" (size 1408888 bytes -- 1.3 MiB).  <============

Needs:

tinygo dev branch (for syscall fixes supplied by @leongross )
this patch to u-root to compile (link) with tinygo
(rebased) Full Go "net" package port, WIP tinygo-org/tinygo#4273

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go
index 850ce067..f979aaea 100644
--- a/cmds/core/init/fns_other.go
+++ b/cmds/core/init/fns_other.go
@@ -2,8 +2,8 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && !(darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
-// +build !tinygo,!darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
+//go:build !(darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
+// +build !darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
 
 package main
 
diff --git a/cmds/core/init/fns_unix.go b/cmds/core/init/fns_unix.go
index 4bcac715..4db8d9d0 100644
--- a/cmds/core/init/fns_unix.go
+++ b/cmds/core/init/fns_unix.go
@@ -2,8 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && (darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
-// +build !tinygo
+//go:build (darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris)
 // +build darwin dragonfly freebsd linux netbsd openbsd solaris
 
 package main
diff --git a/cmds/core/init/init_plan9.go b/cmds/core/init/init_plan9.go
index 4fb93f94..6b88f159 100644
--- a/cmds/core/init/init_plan9.go
+++ b/cmds/core/init/init_plan9.go
@@ -2,8 +2,8 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !tinygo && plan9
-// +build !tinygo,plan9
+//go:build plan9
+// +build plan9
 
 package main
 
diff --git a/cmds/core/init/init_tinygo.go b/cmds/core/init/init_tinygo.go
new file mode 100644
index 00000000..1d670023
--- /dev/null
+++ b/cmds/core/init/init_tinygo.go
@@ -0,0 +1,29 @@
+//go:build tinygo
+
+package main
+
+/*
+ld.lld: error: undefined symbol: syscall.runtime_BeforeExec
+>>> referenced by exec_unix.go:282 ([...]/go/1.22.2/libexec/src/syscall/exec_unix.go:282)
+>>>               /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main.lto.main.o:(runtime.run$1$gowrapper)
+
+ld.lld: error: undefined symbol: syscall.runtime_AfterExec
+>>> referenced by exec_unix.go:308 ([...]/go/1.22.2/libexec/src/syscall/exec_unix.go:308)
+>>>               /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main.lto.main.o:(runtime.run$1$gowrapper)
+failed to run tool: ld.lld
+failed to link /var/folders/7f/_6cnvk8j475gw2rk8q3hv8q00000gn/T/tinygo3894403787/main: exit status 1
+*/
+
+import (
+	_ "unsafe" // for go:linkname
+)
+
+//go:linkname runtime_BeforeExec syscall.runtime_BeforeExec
+func runtime_BeforeExec() {
+
+}
+
+//go:linkname runtime_AfterExec syscall.runtime_AfterExec
+func runtime_AfterExec() {
+
+}

leongross

Nice work, I anticipate this will help a lot to enable u-root builds with tinygo

src/pkg/golang/build.go

leongross · 2024-08-07T11:41:40Z

@archie2x if you need said changes

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go

in u-root, do you mind opening a pr? at some point it needs to be merged anyway, right?

archie2x · 2024-08-07T12:40:02Z

@archie2x if you need said changes

diff --git a/cmds/core/init/fns_other.go b/cmds/core/init/fns_other.go

in u-root, do you mind opening a pr? at some point it needs to be merged anyway, right?

I was going to to, and actually, maybe a larger patch for enabling tinygo build of all cmds in u-root but I saw you already have a PR to make init in u-root/u-root#3062 or am I misunderstanding?

I think you're on a better track: putting the link fixes for tinygo in a shared package.

For the sake of managing all this, I just created a patch to tinygo to help us identify versions in the sources: tinygo-org/tinygo#4389

leongross · 2024-08-09T07:51:55Z

@archie2x in this project you have to sign your commits (see here). Without that, the CI is not run and with that reviewing changes might take longer.

leongross · 2024-08-09T08:02:25Z

@rminnich @hugelgupf what do you think of this?

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

archie2x · 2024-08-09T09:24:59Z

@rminnich @hugelgupf what do you think of this?

I marked it as draft because I want to work through the GOROOT. I think, unless overridden by env, it has to be the same as the tinygo 'cached-GOROOT'

src/pkg/golang/build.go

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

codecov · 2024-09-18T05:16:43Z

Codecov Report

Attention: Patch coverage is 56.28415% with 80 lines in your changes missing coverage. Please review.

Project coverage is 57.27%. Comparing base (f4e63be) to head (2765048).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/pkg/golang/compiler.go	55.02%	76 Missing ⚠️
src/cmd/makebb/makebb.go	66.66%	2 Missing ⚠️
src/pkg/golang/build.go	75.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #121      +/-   ##
==========================================
- Coverage   58.25%   57.27%   -0.98%     
==========================================
  Files          16       17       +1     
  Lines        1636     1753     +117     
==========================================
+ Hits          953     1004      +51     
- Misses        683      749      +66

Flag	Coverage Δ
1.20	`77.22% <56.28%> (-3.35%)`	⬇️
1.21.x	`77.22% <56.28%> (-3.35%)`	⬇️
1.22.x	`57.27% <56.28%> (-0.98%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/cmd/makebb/makebb.go

src/pkg/golang/compiler.go

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

leongross · 2024-11-22T10:12:33Z

@hugelgupf I think @archie2x addressed all the requested changes, should we get this in?

leongross · 2024-11-22T13:32:10Z

Another thing: To enable tinygo support in u-root we had do move from go assembly to CGO. I tested this branch and realized that gobusybox does not support Cgo (also mentioned in the docs, I know). I tried to hack on this for a while but could not get it running yet.
I tried to build exp/esxiboot and got the following error:

14:27:07 Preserving bb generated source directory at /tmp/bb-2772405532 due to error: collecting and putting dependencies in place failed: writing package github.com/u-root/u-root/
pkg/boot/multiboot/internal/trampoline failed: $HOME/u-root/pkg/boot/multiboot/internal/
trampoline/trampoline_linux_amd64_tinygo.go:28:8: could not import C (no metadata for C)

I tracked the error down into go/packages/packages.go:1219, but I am not sure where in the busybox builder I can check for the "C" package and ignore it.

rminnich · 2024-12-27T07:04:05Z

Leon approved these, it never merged, but if it passes, and looks reasonable to me, I'll merge.

archie2x force-pushed the tinygo branch from 2f2097c to 5f108d5 Compare August 6, 2024 17:18

archie2x added a commit to archie2x/mkuimage that referenced this pull request Aug 6, 2024

add option to specify go -compiler, e.g. tinygo see u-root/gobusybox#121

81690d3

archie2x mentioned this pull request Aug 6, 2024

add option to specify go -compiler, e.g. tinygo see u-root/gobusybox#121 u-root/mkuimage#34

Open

leongross reviewed Aug 7, 2024

View reviewed changes

src/pkg/golang/build.go Outdated Show resolved Hide resolved

src/pkg/golang/build.go Outdated Show resolved Hide resolved

archie2x mentioned this pull request Aug 7, 2024

tinygo version build-tags (duplicate go1.X build-tag functionality) tinygo-org/tinygo#4391

Closed

archie2x marked this pull request as draft August 8, 2024 21:20

archie2x added 2 commits August 9, 2024 02:19

-compiler command line argument (e.g. tinygo)

b37fc01

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

add golang.WithCompiler() to set compiler on existing env

ee82c9a

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

archie2x force-pushed the tinygo branch from fc24790 to ee82c9a Compare August 9, 2024 09:20

archie2x mentioned this pull request Aug 29, 2024

tinygo #2979 update tinygo build constraints u-root/u-root#3071

Closed

hugelgupf reviewed Sep 14, 2024

View reviewed changes

src/pkg/golang/build.go Outdated Show resolved Hide resolved

hugelgupf reviewed Sep 14, 2024

View reviewed changes

src/pkg/golang/build.go Outdated Show resolved Hide resolved

archie2x requested review from hugelgupf and leongross September 16, 2024 23:17

archie2x marked this pull request as ready for review September 16, 2024 23:17

Address CRs

81f0c36

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

archie2x force-pushed the tinygo branch from 9d34271 to 81f0c36 Compare September 16, 2024 23:32

leongross reviewed Sep 19, 2024

View reviewed changes

More CR.

be6e468

Signed-off-by: Roger Standridge <9526806+archie2x@users.noreply.github.com>

archie2x force-pushed the tinygo branch from 38a92a0 to be6e468 Compare September 19, 2024 18:45

leongross self-requested a review November 22, 2024 10:11

leongross approved these changes Nov 22, 2024

View reviewed changes

This was referenced Nov 22, 2024

Experiment with u-root (u-root-builder booted from network, and tinygoized u-root) linuxboot/heads#1559

Open

Consider moving away of busybox linuxboot/heads#1484

Open

Merge branch 'main' into tinygo

2765048

rminnich merged commit 1318b4a into u-root:main Dec 27, 2024
20 of 22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tinygo: -compiler command line argument (e.g. tinygo) #121

tinygo: -compiler command line argument (e.g. tinygo) #121

archie2x commented Aug 6, 2024 •

edited

Loading

leongross left a comment

leongross commented Aug 7, 2024

archie2x commented Aug 7, 2024

leongross commented Aug 9, 2024

leongross commented Aug 9, 2024

archie2x commented Aug 9, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

leongross commented Nov 22, 2024

leongross commented Nov 22, 2024

rminnich commented Dec 27, 2024

tinygo: -compiler command line argument (e.g. tinygo) #121

tinygo: -compiler command line argument (e.g. tinygo) #121

Conversation

archie2x commented Aug 6, 2024 • edited Loading

leongross left a comment

Choose a reason for hiding this comment

leongross commented Aug 7, 2024

archie2x commented Aug 7, 2024

leongross commented Aug 9, 2024

leongross commented Aug 9, 2024

archie2x commented Aug 9, 2024

codecov bot commented Sep 18, 2024 • edited Loading

Codecov Report

leongross commented Nov 22, 2024

leongross commented Nov 22, 2024

rminnich commented Dec 27, 2024

archie2x commented Aug 6, 2024 •

edited

Loading

codecov bot commented Sep 18, 2024 •

edited

Loading