Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

depends: expat 2.4.8 & fix building with -flto #25697

Merged
merged 2 commits into from
Jul 27, 2022

Conversation

fanquake
Copy link
Member

@fanquake fanquake commented Jul 25, 2022

Currently, when building the expat package in depends, using -flto (LTO=1), the configure check can fail, because it cannot determine the system endianess:

configure:18718: result: unknown
configure:18733: error: unknown endianness
 presetting ac_cv_c_bigendian=no (or yes) will help

Fix that by defining _DEFAULT_SOURCE, which in turn defines __USE_MISC (features.h):

#if defined _DEFAULT_SOURCE
# define __USE_MISC	1
#endif

which exposes additional definitions in endian.h:

#include <features.h>

/* Get the definitions of __*_ENDIAN, __BYTE_ORDER, and __FLOAT_WORD_ORDER.  */
#include <bits/endian.h>

#ifdef __USE_MISC
# define LITTLE_ENDIAN	__LITTLE_ENDIAN
# define BIG_ENDIAN	__BIG_ENDIAN
# define PDP_ENDIAN	__PDP_ENDIAN
# define BYTE_ORDER	__BYTE_ORDER
#endif

and gives us a working configure.

You could test building this change with Guix + LTO with this branch. Note that that build may fail for other reasons (on x86_64), unrelated to this change.

Some related upstream discussion:
https://bugs.gentoo.org/757681
https://forums.gentoo.org/viewtopic-t-1013786.html

@DrahtBot
Copy link
Contributor

DrahtBot commented Jul 25, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #25391 (guix: Use LTO to build releases by fanquake)
  • #24283 (build: Add show-% target for multi-line variables and debug info by hebasto)
  • #22552 (build: Improve depends build system robustness by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@jarolrod
Copy link
Member

jarolrod commented Jul 25, 2022

GUIX hashes for lto_in_guix

x86:

$ env HOSTS='aarch64-linux-gnu arm-linux-gnueabihf powerpc64-linux-gnu powerpc64le-linux-gnu riscv64-linux-gnu x86_64-linux-gnu' ./contrib/guix/guix-build 
$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

5004a59842f8f206f4847dde50598670d99874da9fd0d028e99a946f3b99f96d  guix-build-75abb99603cc/output/aarch64-linux-gnu/SHA256SUMS.part
ce0028199f70ac1bd9fcb6068e19f0a9c4c4bb29daddcd149513e3a7cc255eca  guix-build-75abb99603cc/output/aarch64-linux-gnu/bitcoin-75abb99603cc-aarch64-linux-gnu-debug.tar.gz
68c3953c69b28ea6a07fce6fb948dd94d90b3838fc225732a60beca9364a0f0b  guix-build-75abb99603cc/output/aarch64-linux-gnu/bitcoin-75abb99603cc-aarch64-linux-gnu.tar.gz
dfef8e0a7ef0fdf0aa6ab45f04268eb7f5fce8013f6e93d3c41f6e79ef46ec34  guix-build-75abb99603cc/output/arm-linux-gnueabihf/SHA256SUMS.part
1aba24c79463b7ca02b264070cd37e86fc371c7ba1d217f51c537aef887c88b5  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf-debug.tar.gz
efc6aed030f9261c0dddf09f6855236123ff26ec3a741dfea474b78f80831a9e  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf.tar.gz
ca9294f428c3e50fe9db9e3b25a056577956eafd84bbc148e422c1ce421a0540  guix-build-75abb99603cc/output/dist-archive/bitcoin-75abb99603cc.tar.gz
a7e52ac891968f0659e32e146b423a0b473a91fed6c103b121934bb014a90f32  guix-build-75abb99603cc/output/powerpc64-linux-gnu/SHA256SUMS.part
94b63ff49b908b6202ce78c94213581197df9655fd99631dcfac43d1ba6c7533  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu-debug.tar.gz
e6bb1de2fc4fc37e232b982b69841486355c7fbac603159995b52684a2198e79  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu.tar.gz
bb40740f3d3d7db728f6234e263fb22252176856ef5850b7b8c343664608b6c7  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/SHA256SUMS.part
078970d4dd9084f928065074feb52fefcc7795c332d066bdc90a2495a674255c  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu-debug.tar.gz
7260e272084140e2d1b2c78d3786f363aaaea16f967274bb804289849c8f0351  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu.tar.gz
7239351d62b1db37ef9d45ca562c165f458f8dd1420cb80473323e4a4d0d0476  guix-build-75abb99603cc/output/riscv64-linux-gnu/SHA256SUMS.part
d9db73da46e36dfde19b207f809189e657002b717d2da08a754e7ee2d364b161  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu-debug.tar.gz
30bd2e0878a602cdfeafe9f3770e8d327cd3357a9b6cd0943f3c03fd5b1ddc02  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu.tar.gz
9eecd92f5964fdcac87f2337c7fa88faa58feedb095c4b07395bffcc1421a627  guix-build-75abb99603cc/output/x86_64-linux-gnu/SHA256SUMS.part
241934184478155d1919d80b67db01ce2659f3c9ca5960b1c8a0bee34281b24d  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu-debug.tar.gz
db57a69879d2b8efedaaf8b9254d61465cdd0606e06a490b599bc56d1d7e3931  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu.tar.gz

arm64:

$ env HOSTS='arm-linux-gnueabihf powerpc64-linux-gnu powerpc64le-linux-gnu riscv64-linux-gnu x86_64-linux-gnu' ./contrib/guix/guix-build 
$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

080237f918c928a2e13c4644075ddcbd1011f3bf243b2096ec16e489e7ea9279  guix-build-75abb99603cc/output/arm-linux-gnueabihf/SHA256SUMS.part
106e71d5aaa6569896cf97cfbc98f1f3cdeb9392a2313f2414c3c51012e8d4c9  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf-debug.tar.gz
1af28e6fda591c83ee9ab8a35be3ff4c82e46a96fb3ea4afc2ee947b329479c3  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf.tar.gz
ca9294f428c3e50fe9db9e3b25a056577956eafd84bbc148e422c1ce421a0540  guix-build-75abb99603cc/output/dist-archive/bitcoin-75abb99603cc.tar.gz
f47d609deda0768a1ae2e69e44b02545d0ae0baff4be7ccad07eeab3d637f298  guix-build-75abb99603cc/output/powerpc64-linux-gnu/SHA256SUMS.part
72f5eedc3cbea81706a6f47f974760cc9db5be536514dea41d105a049a999959  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu-debug.tar.gz
4bea76a954edfecef6b7db0fd27dba4cd272b57b388bb821687c369ff7d6f581  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu.tar.gz
5d68c496dee71dfa32c53002dca93eadc33f93cc49ea97d67f65d59a0cb8a287  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/SHA256SUMS.part
4532e72989706a46b1da733c1624842ad4cc58c73290445c25c02e4166f71ff1  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu-debug.tar.gz
cdab8486ea2b294f4fba7a4c8bd89d6fb17acf03b9fa690d7ba3c9d414d7ac31  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu.tar.gz
c7b0872824132c0f1d77bf40d9e37ffa9c89cff05ebf4c6e374944f57266daa8  guix-build-75abb99603cc/output/riscv64-linux-gnu/SHA256SUMS.part
23a90bc3073366ac8755f92f47f5b7e30a91f3bdcc90b04ccb04dba11a76406c  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu-debug.tar.gz
9a55baaaab65088447654ef7a93b5264d8840cee427944286d7f02eee6e15358  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu.tar.gz
bd9b2c120df3c73d4c81f4f5016995cff7595b1a928197eba1e54971c8259014  guix-build-75abb99603cc/output/x86_64-linux-gnu/SHA256SUMS.part
4d93c7aa1ede282eff25bc52942e3317b4f148af4ced7978ac8b15d160ed4dfc  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu-debug.tar.gz
f3fc20822a49b1372091021efa5e3a8a28a934e5235b4d9d43581f66df1486aa  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu.tar.gz

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, when building the expat package in depends, using -flto (LTO=1), inside Guix (and only inside Guix from what I've seen), the configure check fails...

It fails not only inside Guix, but for every cross-compiling, e.g.,:

$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf

Considering that the _DEFAULT_SOURCE macro implies a wider feature set than the _POSIX_C_SOURCE one, is it safe to define the former for a dependency while the latter been used in

#define _POSIX_C_SOURCE 200112L

?

depends/packages/expat.mk Show resolved Hide resolved
@fanquake
Copy link
Member Author

Considering that the _DEFAULT_SOURCE macro implies a wider feature set than the _POSIX_C_SOURCE one, is it safe to define the former for a dependency while the latter been used in

I don't really think that makes much of a difference here. _DEFAULT_SOURCE is still only used for the expat build.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 790ae82, I have reviewed the code and it looks OK, I agree it can be merged.

It's worth to mention that in the span from v2.4.1 to v2.4.8 a few CVEs have been fixed.

I've verified that no other adjustments of our depends build system required.

Successfully can cross-compile the expat package in depends with LTO=1.

depends/packages/expat.mk Outdated Show resolved Hide resolved
@fanquake fanquake force-pushed the expat_2_4_8_with_lto branch from 790ae82 to e838a98 Compare July 26, 2022 10:38
@fanquake
Copy link
Member Author

Addressed the suggestion and updated the PR description.

@fanquake fanquake changed the title depends: expat 2.4.8 & fix building with -flto when using Guix depends: expat 2.4.8 & fix building with -flto Jul 26, 2022
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK e838a98, only suggested changes since my recent review.

Copy link
Member

@jarolrod jarolrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code review ACK e838a98

@fanquake
Copy link
Member Author

It fails not only inside Guix, but for every cross-compiling, e.g.,:
$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf

This doesn't fail for me. What system is this on? What compiler are you using etc?

@hebasto
Copy link
Member

hebasto commented Jul 27, 2022

It fails not only inside Guix, but for every cross-compiling, e.g.,:
$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf

This doesn't fail for me. What system is this on? What compiler are you using etc?

$ git rev-parse HEAD
31c1b14754574b0b0a54587e937fab66b3b85e39
$ git diff
diff --git a/depends/packages/expat.mk b/depends/packages/expat.mk
index 349319138e..50791ebc6e 100644
--- a/depends/packages/expat.mk
+++ b/depends/packages/expat.mk
@@ -9,7 +9,6 @@ define $(package)_set_vars
   $(package)_config_opts += --disable-dependency-tracking --enable-option-checking
   $(package)_config_opts += --without-xmlwf
   $(package)_config_opts_linux=--with-pic
-  $(package)_cflags += -fno-lto
 endef
 
 define $(package)_config_cmds
$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf
...
configure: error: unknown endianness
 presetting ac_cv_c_bigendian=no (or yes) will help
...
$ arm-linux-gnueabihf-g++ --version
arm-linux-gnueabihf-g++ (Ubuntu 11.2.0-17ubuntu1) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

@fanquake
Copy link
Member Author

I've opened an issue upstream: libexpat/libexpat#618.

@fanquake fanquake merged commit 207a228 into bitcoin:master Jul 27, 2022
@fanquake fanquake deleted the expat_2_4_8_with_lto branch July 27, 2022 11:56
@fanquake
Copy link
Member Author

Not an expat bug, so I've submitted a report further upstream to autoconf: https://savannah.gnu.org/support/index.php?110687.

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 27, 2022
e838a98 depends: re-enable using -flto when building expat (fanquake)
3044525 depends: expat 2.4.8 (fanquake)

Pull request description:

  Currently, when building the expat package in depends, using `-flto` (`LTO=1`), the configure check can fail, because it cannot determine the system endianess:
  ```bash
  configure:18718: result: unknown
  configure:18733: error: unknown endianness
   presetting ac_cv_c_bigendian=no (or yes) will help
  ```

  Fix that by defining `_DEFAULT_SOURCE`, which in turn defines `__USE_MISC` (`features.h`):
  ```c
  #if defined _DEFAULT_SOURCE
  # define __USE_MISC1
  #endif
  ```
  which exposes additional definitions in `endian.h`:
  ```c
  #include <features.h>

  /* Get the definitions of __*_ENDIAN, __BYTE_ORDER, and __FLOAT_WORD_ORDER.  */
  #include <bits/endian.h>

  #ifdef __USE_MISC
  # define LITTLE_ENDIAN__LITTLE_ENDIAN
  # define BIG_ENDIAN__BIG_ENDIAN
  # define PDP_ENDIAN__PDP_ENDIAN
  # define BYTE_ORDER__BYTE_ORDER
  #endif
  ```
  and gives us a working configure.

  You could test building this change with Guix + LTO with [this branch](https://github.com/fanquake/bitcoin/tree/lto_in_guix). Note that that build may fail for other reasons (on x86_64), unrelated to this change.

  Some related upstream discussion:
  https://bugs.gentoo.org/757681
  https://forums.gentoo.org/viewtopic-t-1013786.html

ACKs for top commit:
  hebasto:
    re-ACK e838a98, only [suggested](bitcoin#25697 (comment)) changes since my recent [review](bitcoin#25697 (review)).
  jarolrod:
    code review ACK e838a98

Tree-SHA512: 9dbf64c9bd1fd995a4d1addc011ffeff83d50df736030012346c97605e63aed4b5bac390a81abe646c1be28ad6fd600f64560dcb26bbc2edf5d513ca3b180bfa
@fanquake
Copy link
Member Author

This has now also been fixed in autoconf upstream: https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=234fc6c86613ed3f366dd1d88996e4d5d85ee222.

@bitcoin bitcoin locked and limited conversation to collaborators Jul 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants