Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: macOS toolchain simplification and bump #19240

Merged
merged 8 commits into from
Jun 23, 2020

Conversation

dongcarl
Copy link
Contributor

@dongcarl dongcarl commented Jun 10, 2020

This PR achieves 3 main things:

  1. It simplifies the macOS SDK generation by putting the logic inside a (semi-)portable python3 script gen-sdk
  2. It transitions us to using libc++ headers extracted from the Xcode.app, which is more correct as those headers better match the .tbd library stubs we use from the MacOSX.sdk (located under the same Xcode.app). Previously, we used libc++ headers copied from our downloaded, pinned clang (see native_cctools.mk).
  3. It bumps the macOS toolchain in a way that fulfills all of the following constraints:
    1. The new SDK should support compiling with C++17 (our current one doesn't)
    2. The new toolchain should not change our minimum supported macOS version (-mmacosx-version-min)
    3. The new toolchain should expect to use a version of cctools that is supported by https://github.com/tpoechtrager/cctools-port

For the constraints in (3), you can reference this chart to see that the newest toolchain we can use with our cctools-port is 11.3.1, and the rest of the constraints were tested with local builds.

But the other Wikipedia chart says that the "min macOS to run" for Xcode 11.3.1 is 10.14.4, doesn't that violate constraint (ii)?

This confused me at first too, but the "min macOS to run" is for the Xcode.app App itself. The SDK still supports 10.12, as evident in a few plist files and as proven through local builds.

Why bundle all of this together in a single PR?

We need (1) and (2) together, because if we don't, manually adding the libc++ headers and writing that out in a README.md is going to result in a lot of user error, so it's great to have these together to be more correct and also make it easier on the user at the same time.

We need (3) together with everything else because bumping (or in the case of (1), renaming) the SDK requires some human coordination and may break some builds. And since it's not that complicated a change, it makes sense to do it together with the rest.

@dongcarl dongcarl changed the title 2020 06 macos sdkgen simplify Simplify macOS SDK generation and include libc++ headers Jun 10, 2020
@dongcarl dongcarl force-pushed the 2020-06-macos-sdkgen-simplify branch from 9b52985 to ac1ded1 Compare June 10, 2020 20:32
@laanwj
Copy link
Member

laanwj commented Jun 11, 2020

For anyone who's looked at the last PR, this one just has a simpler sdk-gen.sh script that I wrote myself, and includes some .xip unpacking innovations thanks to theuni.

Thanks for working on this. This is great news. Maybe can manage to do it myself this time without a Mac 👍

contrib/macdeploy/README.md Outdated Show resolved Hide resolved
@dongcarl dongcarl force-pushed the 2020-06-macos-sdkgen-simplify branch from 94da331 to 66aca4b Compare June 11, 2020 18:55
Copy link
Member

@theuni theuni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this. Testing now.

contrib/macdeploy/README.md Outdated Show resolved Hide resolved
die() { cat 1>&2 ; exit 1; }

# Defaults to our current SDK's name, but overridable
SDKNAME="${SDKNAME:-MacOSX10.14.sdk}"
Copy link
Member

@theuni theuni Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we name this something to indicate that we've included the headers as well?

Edit: expanding on this: since builders presumably already have a different, incompatible MacOSX10.14.sdk in sources, I think it makes sense to rename the unpacked dir (SDK in hosts/darwin.mk) as well as the tarball to avoid clashes/confusion.

contrib/macdeploy/README.md Outdated Show resolved Hide resolved
@theuni
Copy link
Member

theuni commented Jun 12, 2020

I'm seeing a Qt build failure due to some symlink/hardlink issue:

kernel/qdnslookup_unix.cpp:52:10: fatal error: 'arpa/nameser.h' file not found
#include <arpa/nameser.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
Makefile:12337: recipe for target '.obj/qdnslookup_unix.o' failed

The link is broken hardlink in the generated sdk:

 $ ls -al MacOSX10.14.sdk/usr/include/arpa/nameser.h
lrwxrwxrwx 1 cory cory 26 Jun 12 14:28 MacOSX10.14.sdk/usr/include/arpa/nameser.h -> MacOSX10.14.sdk./nameser.h

Whereas in the original SDK it's a working symlink:

 $ ls -al MacOSX10.14.sdk/usr/include/arpa/nameser.h
lrwxrwxrwx 1 cory cory 12 Jun 12 14:28 MacOSX10.14.sdk/usr/include/arpa/nameser.h -> ../nameser.h

Edit: For reference, here's my extraction output from Ubuntu 18.04.2:

cory@desktop:15:41:55:~/dev/bitcoin2 (19240)
 $ ./contrib/macdeploy/gen-sdk.sh /tmp/apple-sdk-tools/Xcode.app
Going to attempt to extract the MacOSX10.14.sdk SDK from the Xcode.app located at:
	'/tmp/apple-sdk-tools/Xcode.app'

And place the resulting gzipped tarball at:
	'/home/cory/dev/bitcoin2/MacOSX10.14.sdk.tar.gz'

Going to use tar command: 'tar'
Detected tar style: 'GNU'

Adding MacOSX10.14.sdk SDK files...
Adding libc++ headers...
Compressing using gzip...

Done! Find the resulting gzipped tarball at:
	'/home/cory/dev/bitcoin2/MacOSX10.14.sdk.tar.gz'

@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 13, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link
Member

@fanquake fanquake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK.

Cool. Looks like Travis is happy now. Can you squash the fixup commits?

I ran some gitian builds locally to compare the binaries produced by master (195822f) and master + this diff:

diff --git a/contrib/gitian-descriptors/gitian-osx.yml b/contrib/gitian-descriptors/gitian-osx.yml
index bbae7201e..936d097d2 100644
--- a/contrib/gitian-descriptors/gitian-osx.yml
+++ b/contrib/gitian-descriptors/gitian-osx.yml
@@ -32,7 +32,7 @@ remotes:
 - "url": "https://github.com/bitcoin/bitcoin.git"
   "dir": "bitcoin"
 files:
-- "MacOSX10.14.sdk.tar.gz"
+- "MacOSX10.14-with-libcxx-headers.sdk.tar.gz"
 script: |
   set -e -o pipefail
 
@@ -90,7 +90,7 @@ script: |
   BASEPREFIX="${PWD}/depends"
 
   mkdir -p ${BASEPREFIX}/SDKs
-  tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.14.sdk.tar.gz
+  tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.14-with-libcxx-headers.sdk.tar.gz
 
   # Build dependencies for each host
   for i in $HOSTS; do
diff --git a/depends/hosts/darwin.mk b/depends/hosts/darwin.mk
index 82e086a32..70baeae90 100644
--- a/depends/hosts/darwin.mk
+++ b/depends/hosts/darwin.mk
@@ -1,6 +1,6 @@
 OSX_MIN_VERSION=10.12
 OSX_SDK_VERSION=10.14
-OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk
+OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION)-with-libcxx-headers.sdk
 darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK)
 darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++
 
diff --git a/depends/packages/native_cctools.mk b/depends/packages/native_cctools.mk
index 4195230b4..19fcf54f9 100644
--- a/depends/packages/native_cctools.mk
+++ b/depends/packages/native_cctools.mk
@@ -73,6 +73,5 @@ define $(package)_stage_cmds
   cp lib/libLTO.so $($(package)_staging_prefix_dir)/lib/ && \
   cp -rf lib/clang/$($(package)_clang_version)/include/* $($(package)_staging_prefix_dir)/lib/clang/$($(package)_clang_version)/include/ && \
   cp bin/llvm-dsymutil $($(package)_staging_prefix_dir)/bin/$(host)-dsymutil && \
-  if `test -d include/c++/`; then cp -rf include/c++/ $($(package)_staging_prefix_dir)/include/; fi && \
   if `test -d lib/c++/`; then cp -rf lib/c++/ $($(package)_staging_prefix_dir)/lib/; fi

The binaries produced by the "new" SDK are all slightly larger (except for bitcoin-wallet) than the binaries currently produced by master:

Commit bitcoind bitcoin-cli bitcoin-qt bitcoin-tx bitcoin-wallet test_bitcoin
Master 9084960 399464 24920412 1042272 2721068 15010424
This PR 9135728 407680 24958868 1050504 2717020 15021368

I did a quick comparison of bitcoind, there is a single symbols difference:

--- 19.txt
+++ ea.txt
@@ -26,14 +26,15 @@
 U std::bad_exception::what() const
 U std::runtime_error::what() const
 U std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::find(char, unsigned long) const
 U std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::rfind(char, unsigned long) const
 U std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::compare(char const*) const
 U std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::compare(unsigned long, unsigned long, char const*) const
 U std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::compare(unsigned long, unsigned long, char const*, unsigned long) const
+T std::__1::enable_if<__can_be_converted_to_string_view<char, std::__1::char_traits<char>, std::__1::basic_string_view<char, std::__1::char_traits<char> > >::value, int>::type std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::compare<std::__1::basic_string_view<char, std::__1::char_traits<char> > >(std::__1::basic_string_view<char, std::__1::char_traits<char> > const&) const
 U std::__1::__shared_weak_count::__get_deleter(std::type_info const&) const
 U std::__1::__vector_base_common<true>::__throw_length_error() const
 U std::__1::__vector_base_common<true>::__throw_out_of_range() const

There is also one less initializer function called in the new bitcoind, however I haven't looked into that yet.

@dongcarl dongcarl force-pushed the 2020-06-macos-sdkgen-simplify branch from b424774 to 1944c5f Compare June 15, 2020 13:14
@dongcarl
Copy link
Contributor Author

Squashed in the fixups!

Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. I did some testing with 1944c5f on macOS and Ubuntu. Code looks reasonable, but mostly above my head.

I found extract_xcode.py to be deterministic, but gen-sdk.sh produces a different checksum depending on the OS.


```bash
# Install/clone tools needed for extracting Xcode.app
apt install cpio
git clone https://github.com/theuni/apple-sdk-tools.git
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we merge this, I'd like to move that repo over to the bitcoin-core org.

find Xcode.app -type d -name MacOSX.sdk -exec sh -c 'tar --transform="s/MacOSX.sdk/MacOSX10.14.sdk/" -c -C$(dirname {}) MacOSX.sdk/ | gzip -9n > MacOSX10.14.sdk.tar.gz' \;
# Unpack Xcode_10.2.1.xip and place the resulting Xcode.app in your current
# working directory
python3 apple-sdk-tools/extract_xcode.py -f Xcode_10.2.1.xip | cpio -d -i
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When running the extract tool 41bfaca0db3a900c52fb66b6eae794901b9510cb on macOS 10.15.5 I'm flooded with:

../apple-sdk-tools/extract_xcode.py -f Xcode_10.2.1.xip| cpio -d -i
./Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/_ctype.h: Can't create 'Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/_ctype.h'
...
28524433 blocks

The shasum's of the resulting directories, after sorting, obtained with incantation:

find Xcode.app -type f -print0  | xargs -0 shasum -a 256 > shasum_list_unsorted.txt
sort shasum_list_unsorted.txt > shasum_list_sorted.txt
shasum -a 256 shasum_list_sorted.txt 
ae267efecbfe750e4193903fdf1044af8c7ba263072a11ad99143a34a7e2b1a0  shasum_list_sorted.txt

When using xip -x I get the same checksum, nice!

I get the same shasum on Ubuntu 20.04 (LC_ALL=C sort)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

contrib/macdeploy/extract-osx-sdk.sh
rm -rf 5.hfs MacOSX10.11.sdk
# Generate a MacOSX10.14-with-libcxx-headers.sdk.tar.gz from the supplied Xcode.app
./contrib/macdeploy/gen-sdk.sh '/path/to/Xcode.app'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0e7fe352b806ed3cd5a735e1aa476edd0b0c7a24918e49c4c4992203dab1ce28 MacOSX10.14-with-libcxx-headers.sdk.tar.gz when run on macOS, but 6c6319005fe579d0dd174e15ef8d3beab63f1aed4d15e73305eba8c63ffebebb on Ubuntu.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's somewhat expected, as bsdtar and gnutar (and even different versions of gnutar) might lay out the archive in different ways that all conform to standard. ☺️

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can live with a reproducibility issue, it must be fine :-)

@theuni
Copy link
Member

theuni commented Jun 16, 2020

I discussed this today with @fanquake, and we realized that after this change, we'll also need to bump our SDK in the near-future so that we can begin testing with c++17. And when we do that, it's also a good time to bump our clang/cctools to match.

I hacked that together and tested quickly, and all seems to be working as expected:
https://github.com/theuni/bitcoin/tree/19240

At the risk of slowing down this PR, we could potentially integrate the rest of the toolchain bump in here as well. The upside of that would be that builders only have to download and generate a new sdk tarball once, whereas if we do this in stages, they'll have do it twice.

But then again, Carl has been maintaining these changes for quite a while now and is probably tired of waiting.

@dongcarl Your call :)

@@ -73,6 +73,5 @@ define $(package)_stage_cmds
cp lib/libLTO.so $($(package)_staging_prefix_dir)/lib/ && \
cp -rf lib/clang/$($(package)_clang_version)/include/* $($(package)_staging_prefix_dir)/lib/clang/$($(package)_clang_version)/include/ && \
cp bin/llvm-dsymutil $($(package)_staging_prefix_dir)/bin/$(host)-dsymutil && \
if `test -d include/c++/`; then cp -rf include/c++/ $($(package)_staging_prefix_dir)/include/; fi && \
if `test -d lib/c++/`; then cp -rf lib/c++/ $($(package)_staging_prefix_dir)/lib/; fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be deleted now too, we know where to expect the c++ libs now.

@Sjors
Copy link
Member

Sjors commented Jun 17, 2020

Let's merge this first. As long as the other stuff gets in before 0.21 most people won't be impacted anyway.

@dongcarl
Copy link
Contributor Author

dongcarl commented Jun 17, 2020

Thought a bit about what's needed for the steps that are required before 0.21, and it seems like I'll have to redo the entire gen-sdk.sh in python for the tarfile and plistlib modules, so it probably doesn't make sense to merge this in the meantime. Will push up a new version in this PR when I have it.

Many thanks to laanwj, theuni, fanquake, and especially Sjors for reviewing.

@dongcarl dongcarl marked this pull request as draft June 17, 2020 17:37
@theuni
Copy link
Member

theuni commented Jun 17, 2020

Let's merge this first.

Sounds good.

My only remaining ask for this PR is something @dongcarl and I discussed yesterday: Adding the xcode version string to the (already lengthy!) sdk tarball/dirname. This is to compensate for the fact that Apple updates the SDK in new XCode versions, so unfortunately "MacOSX10.xy.sdk" alone doesn't mean much. Adding this string will prevent future clashes on our end.

Edit: Whoops, look like Carl and I commented here at the same time. Sorry for throwing another wrench in this :(

@dongcarl dongcarl force-pushed the 2020-06-macos-sdkgen-simplify branch 3 times, most recently from b05b140 to 538bb0b Compare June 18, 2020 23:24
Fuzzbawls pushed a commit to Fuzzbawls/PIVX that referenced this pull request Apr 5, 2021
Without this clang fails to add any newly-added linker features.

Removing this in ca5055a was likely a
regression.

See bitcoin#19240 (comment)
for more discussion.
furszy added a commit to PIVX-Project/PIVX that referenced this pull request May 25, 2021
e1b89ac Fix QPainter non-determinism on macOS (Andrew Chow)
831c317 macOS deploy: use the new plistlib API (Jonas Schnelli)
5857aaf doc: Document ALLOW_HOST_PACKAGES dependency option (skmcontrib)
2329e08 build: Fix behavior when ALLOW_HOST_PACKAGES unset (Hennadii Stepanov)
1768870 depends: native_ds_store 1.3.0 (fanquake)
3f9f3e5 depends: pull upstream libdmg-hfsplus changes (fanquake)
f7606dc depends: latest config.guess & config.sub (fanquake)
cc3ae74 depends: bump native_cctools for fixed lto with external clang (Cory Fields)
b26c648 depends: enable lto support for Apple's ld64 (Cory Fields)
50933d7 depends: Add documentation for FORCE_USE_SYSTEM_CLANG make flag (Carl Dong)
ba3ddf2 depends: Reformat make options as definition list (Carl Dong)
3b855a7 depends: Add justifications for macOS clang flags (Carl Dong)
4104de0 depends: specify libc++ header location for darwin (Cory Fields)
cd4335f depends: force a new host id string if FORCE_USE_SYSTEM_CLANG is in use (Cory Fields)
d30e1af depends: Allow building with system clang (Carl Dong)
234828b depends: Decouple toolchain + binutils (Carl Dong)
1dd3a5a doc: explain why passing -mlinker-version is required (fanquake)
5cc0d0f darwin: pass mlinker-version so that clang enables new features (Cory Fields)
813a552 macos: Bump to xcode 11.3.1 and 10.15 SDK (Cory Fields)
ee7085f depends: bump MacOS toolchain (Cory Fields)
e5b092b contrib: macdeploy: Remove historical extraction notes (Carl Dong)
5893caf contrib: macdeploy: Use apple-sdk-tools instead of xar+pbzx (Carl Dong)
9f2d4ba native_cctools: Don't use libc++ from pinned clang (Carl Dong)
0c8d217 Adapt rest of tooling to new SDK naming scheme (Carl Dong)
bdacfa8 contrib: macdeploy: Correctly generate macOS SDK (Carl Dong)
f7eee2c Fix naming of macOS SDK and clarify version (Andrew Chow)
62f9e23 build: use macOS 10.14 SDK (fanquake)
bc2e1af depends: native_cctools 921, ld64 409.12, libtapi 1000.10.8 (fanquake)
a296d87 depends: clang 6.0.1 (fanquake)
8f6c475 build: Set minimum supported macOS to 10.12 (Fuzzbawls)

Pull request description:

  This backports the following upstream PRs to update the macOS cross-compiling tools:

  bitcoin#17550
  bitcoin#16392
  bitcoin#18589
  bitcoin#19240
  bitcoin#19407
  bitcoin#17919
  bitcoin#19530
  bitcoin#17057
  bitcoin#20333
  bitcoin#18051
  bitcoin#19124
  bitcoin#20298
  bitcoin#20447

  The tools being updated are

  ### Clang
  Upgraded from `3.7.1` to `8.0.0`

  ### cctools

  * cctools `877.8` -> `949.0.1`
  * LD64 `253.9` -> `530`
  * TAPI `1000.10.8`

  ### DSStore
  Upgraded from `1.1.2` to `1.3.0` (this removes the biplist dependency)

  This also effectively bumps our minimum supported macOS version to 10.12 (Sierra).

ACKs for top commit:
  furszy:
    tested ACK e1b89ac
  random-zebra:
    utACK e1b89ac

Tree-SHA512: f5cec8db57e07d8855070646b9e1400d48aac1d01e3c2c3b3e134665c6372d6535f3328888bb9a75087f7b3d5231ecb4b509723bfa51bd40770ffe2810c67f65
kwvg added a commit to kwvg/dash that referenced this pull request Jul 15, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 15, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 20, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 20, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 20, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 20, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 24, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 25, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 26, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 27, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 30, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021
lyricidal added a commit to PRCYCoin/PRCYCoin that referenced this pull request Oct 29, 2021
Without this clang fails to add any newly-added linker features.

Removing this in PIVX-Project/PIVX@ca5055a was likely a regression.

See bitcoin/bitcoin#19240 (comment) for more discussion.
lyricidal added a commit to PRCYCoin/PRCYCoin that referenced this pull request Oct 31, 2021
Without this clang fails to add any newly-added linker features.

Removing this in PIVX-Project/PIVX@ca5055a was likely a regression.

See bitcoin/bitcoin#19240 (comment) for more discussion.
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants