Add simple cross compile mechanism for flisp/libsupport (+ dependencies) #30526

Keno · 2018-12-28T06:50:36Z

This makes some incremental progress towards #30338 in the build system.
In particular, it allows compiling a separate boostrap version of flisp
and its dependencies for an architecture different from the target.
The immediate use case here is to build libjulia for wasm. Of course,
there's still the enormous problem of building the system image itself,
but luckily wasm is largely memory-layout-identical to linux32, so doing
that limited bootstrap is fairly easy after this change (though the subject
of a different PR).

The mechanism here is to use the out-of-tree build support to create
separate host/ subdirectories for the build directories of interest.
These have a special make variable set that tell it to include a
different Make.user (or in the absence thereof use the host defaults).

cc @SimonDanisch

Needs a bit more testing, but seems to do what I need it to in preliminary evaluation.

Make.inc

deps/tools/common.mk

Keno · 2018-12-30T04:53:26Z

Is there a way we could run the full barrage of buildbots on this branch? There's a good number of mechanism changes here and I'd like to make sure I don't break anything.

staticfloat · 2018-12-30T05:12:39Z

Yes. Go to this page, login by authentication through github in the top right, then cmd-click on each builder name starting with package_ to open a tab into each of our builders. Finally, click the blue "package" button in the top right, and paste in the gitsha (or branch name) that you're interested in building.

I've gone ahead and done this for you; here are the in-progress builds.

freebsd
linux32
linux64
aarchf64
armv7l (this one running on the testing buildbots because I haven't hooked the armv7l buildbots up to main build.julialang.org)
ppc64le
osx64
win32
win64

This makes me miss @jlbuild. :'(

vtjnash · 2018-12-30T06:01:48Z

The mechanism here is to use the out-of-tree build support to create

That was secretly one of the goals of that work—I'm glad to see it's finally getting put to its intended use :) I've only reviewed fairly briefly, but the general direction of all of the changes looked good to me.

Could we avoid most of the MAYBE_HOST changes to deps by just altering the BUILDDIR and build_staging variables, instead of modifying all uses of them?

Keno · 2018-12-30T21:51:49Z

Could we avoid most of the MAYBE_HOST changes to deps by just altering the BUILDDIR and build_staging variables, instead of modifying all uses of them?

Ok, done.

vtjnash · 2018-12-30T22:29:00Z

deps/tools/common.mk

-$$(build_prefix)/manifest/$(strip $1): $$(build_staging)/$2.tgz | $(build_prefix)/manifest
-	mkdir -p $$(build_prefix)
+$$(build_prefix)/manifest/$(strip $1): $$(build_staging)/$2.tgz
+	mkdir -p $$(build_prefix)/manifest


Seems unnecessary? We usually use the gmake extention to do mkdir, instead of each separate recipe being involved

Didn't work for me for some reason. Is there something special I need to do?

It appears to have been working for everyone else

vtjnash · 2018-12-30T22:34:05Z

src/flisp/Makefile

-$(BUILDDIR)/flisp.boot: flisp.boot | $(BUILDDIR)
+$(BUILDDIR)/host/Makefile:
+	mkdir -p $(BUILDDIR)/host
+	@# add Makefiles to the build directories for convenience (pointing back to the source location of each)


Not the right file source in the comment. Is it worth trying to use the existing code to do this?

Fixing the comment. I don't think it's worth trying to deduplicate, esp since this adds a couple extra lines.

vtjnash · 2018-12-30T22:42:10Z

src/flisp/Makefile

+$(LLT_release): $(LLTSRCDIR)/*.h $(LLTSRCDIR)/*.c
+	$(MAKE) -C $(NATIVE_BUILDDIR)/$(LLTDIR) $(subst $(abspath $(NATIVE_BUILDDIR)/$(LLTDIR))/,,$(abspath $(LLT_release)))
+$(LLT_debug): $(LLTSRCDIR)/*.h $(LLTSRCDIR)/*.c
+	$(MAKE) -C $(NATIVE_BUILDDIR)/$(LLTDIR) $(subst $(abspath $(NATIVE_BUILDDIR)/$(LLTDIR))/,,$(abspath $(LLT_debug)))


Isn’t this slightly conflicted? The directory seems to be for native, but the subst looks for a slightly different prefix? Maybe we should add the abspath rule to the LLT Makefile, so we can remove the subst from here?

I don't think there's much to be gained. Keeping the targets appropriately relative seems simplest to me. This just turns the absolute path back into a relative path.

sure, just seemed like it wasn't relative to anything obviously meaningful

Well, it's just "relative" to the LLT build dir, which seems reasonably meaningful.

vtjnash · 2018-12-30T22:42:42Z

src/support/Makefile

+	$(MAKE) -C $(BUILDDIR)/host libsupport.a
+
+$(BUILDDIR)/host/libsupport-debug.a: $(BUILDDIR)/host/Makefile
+	$(MAKE) -C $(BUILDDIR)/host libsupport.a


This makes some incremental progress towards #30338 in the build system. In particular, it allows compiling a separate boostrap version of flisp and its dependencies for an architecture different from the target. The immediate use case here is to build libjulia for wasm. Of course, there's still the enormous problem of building the system image itself, but luckily wasm is largely memory-layout-identical to linux32, so doing that limited bootstrap is fairly easy after this change (though the subject of a different PR). The mechanism here is to use the out-of-tree build support to create separate host/ subdirectories for the build directories of interest. These have a special make variable set that tell it to include a different Make.user (or in the absence thereof use the host defaults).

Keno · 2019-07-01T20:16:58Z

Rebased and addressed review comments (as separate commits for easier review - will squash down to merge).

Keno · 2019-07-01T23:24:46Z

@staticfloat why did the buildbots not show up to CI this? Is it because the PR was created before those were a thing? Will the show up if I close and re-open?

vtjnash · 2019-07-04T06:14:41Z

src/flisp/Makefile

+$(BUILDDIR)/host/$(EXENAME): $(BUILDDIR)/host/Makefile
+	make -C $(BUILDDIR)/host $(EXENAME)
+
+ifneq ($(BUILDDIR)$(BUILDING_HOST_TOOLS),.)


Seems like this change may have introdced a minor build error?

make[2]: Circular /data/vtjnash/julia/src/flisp/flisp.boot <- /data/vtjnash/julia/src/flisp/flisp.boot dependency dropped.

Keno requested a review from staticfloat December 28, 2018 06:50

Keno force-pushed the kf/flispcross branch 2 times, most recently from 5a0bbad to 44d3854 Compare December 28, 2018 19:22

ararslan added the building Build system, or building Julia or its dependencies label Dec 28, 2018

staticfloat reviewed Dec 28, 2018

View reviewed changes

Make.inc Show resolved Hide resolved

deps/tools/common.mk Outdated Show resolved Hide resolved

Keno force-pushed the kf/flispcross branch from 44d3854 to 9f7c924 Compare December 28, 2018 23:15

Keno force-pushed the kf/flispcross branch from 9f7c924 to 6eaee67 Compare December 30, 2018 21:51

vtjnash reviewed Dec 30, 2018

View reviewed changes

Keno added 5 commits July 1, 2019 15:54

Proper cross handling for flisp dependencies

1e29862

Fixup support target

ab6ae59

Undo the manifest change

428d87d

Fix the comment in the generated Makefile

6b645b5

Keno force-pushed the kf/flispcross branch from 6eaee67 to 6b645b5 Compare July 1, 2019 20:16

Keno closed this Jul 1, 2019

Keno reopened this Jul 1, 2019

Keno merged commit 1fc51b7 into master Jul 2, 2019

martinholters deleted the kf/flispcross branch July 2, 2019 20:42

vtjnash reviewed Jul 4, 2019

View reviewed changes

ararslan mentioned this pull request Aug 12, 2019

circular dependency in build #32852

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add simple cross compile mechanism for flisp/libsupport (+ dependencies) #30526

Add simple cross compile mechanism for flisp/libsupport (+ dependencies) #30526

Keno commented Dec 28, 2018 •

edited

Loading

Keno commented Dec 30, 2018

staticfloat commented Dec 30, 2018 •

edited

Loading

vtjnash commented Dec 30, 2018

Keno commented Dec 30, 2018

vtjnash Dec 30, 2018

Keno Dec 31, 2018

vtjnash Dec 31, 2018

vtjnash Dec 30, 2018

Keno Jul 1, 2019

vtjnash Dec 30, 2018

Keno Jul 1, 2019

vtjnash Jul 2, 2019

Keno Jul 2, 2019

vtjnash Dec 30, 2018

Keno commented Jul 1, 2019

Keno commented Jul 1, 2019

vtjnash Jul 4, 2019

Add simple cross compile mechanism for flisp/libsupport (+ dependencies) #30526

Add simple cross compile mechanism for flisp/libsupport (+ dependencies) #30526

Conversation

Keno commented Dec 28, 2018 • edited Loading

Keno commented Dec 30, 2018

staticfloat commented Dec 30, 2018 • edited Loading

vtjnash commented Dec 30, 2018

Keno commented Dec 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Keno commented Jul 1, 2019

Keno commented Jul 1, 2019

Choose a reason for hiding this comment

Keno commented Dec 28, 2018 •

edited

Loading

staticfloat commented Dec 30, 2018 •

edited

Loading