fix: add support for AIX #988

bhuntsman · 2023-05-24T05:54:21Z

No description provided.

henryiii · 2023-05-24T14:54:35Z

Is there a reason this was closed? The Fedora failure has nothing to do with this (@LecrisUT).

LecrisUT · 2023-05-24T18:41:50Z

No idea need to test it with mock. Can at leaat try to rebuild.
/packit build

packit-as-a-service · 2023-05-24T18:42:00Z

Account LecrisUT has no write access nor is author of PR!

LecrisUT · 2023-05-24T20:32:21Z

A few notes on the failed tests.

cmake--build --target install and cmake --install are technically different. I think install component is more robust than target install unless you are working with a huge custom environment like llvm
My concern with install targets is that it is dangerous when escalating privileges for installing. These should be separate tasks, explicitly, and I think that's what cmake tries to do with --install, among other things.
Did the cmake install reporting changed recently that the grep failed? This is on rawhide so it should be testable in a container

bhuntsman · 2023-05-24T22:19:41Z

Is there a reason this was closed? The Fedora failure has nothing to do with this (@LecrisUT).

I withdrew it as it needs a few other adjustments that I missed. I will reopen another PR after I get the rest worked out.

Specifically, XL C doesn't support the version script functionality in skbuild/resources/cmake/FindPythonExtensions.cmake, and I'll need to add some logic to skip it on the aix platform.

henryiii · 2023-05-24T22:30:23Z

Sounds good, thanks! Wanted to make sure you weren't discouraged by the unrelated failure.

cmake --install is faster and lighter-weight than cmake--build --target install, as it doesn't need make/ninja/whatever to run. I was planning to allow the install target be customizable, and an empty target would trigger --install, while setting it would trigger --target - so a user could specify "install" which would give --target install.

I don't think --install has changed reporting, but I haven't tracked closely. Do we have a diff somewhere?

henryiii · 2023-05-24T22:35:24Z

Okay, great, just wanted to make sure you weren't discouraged by the unrelated failures.

cmake --build --target install and cmake --install

Ahh, this is scikti-build. My brain moved to scikit-build-core at some point, and was confused since it uses --install. Scikit-build can't use --install since it doesn't require a version of CMake that has it. Looking at the logs I see failures talking about the system time being not set, is that the place where I can find the error you are looking at? Is there a specific test that is failing with a changed diff?

LecrisUT · 2023-05-25T03:08:09Z

I was looking at the reported error (test_build) .

bhuntsman · 2023-05-25T13:59:33Z

Upon further review, I am mistaken, with XLC, CMAKE_C_COMPILER_ID should equal "XL", and therefore the whole code block in skbuild/resources/cmake/FindPythonExtensions.cmake dealing with the version script is not applicable. I am re-opening this PR. This should be fine as-is.

codecov · 2023-05-25T14:00:35Z

Codecov Report

Merging #988 (4e34b8a) into main (d9d795a) will increase coverage by 0.28%.
The diff coverage is 90.90%.

❗ Current head 4e34b8a differs from pull request most recent head d02cefc. Consider uploading reports for the commit d02cefc to get more accurate results

@@            Coverage Diff             @@
##             main     #988      +/-   ##
==========================================
+ Coverage   85.97%   86.26%   +0.28%     
==========================================
  Files          32       33       +1     
  Lines        1576     1587      +11     
  Branches      347      350       +3     
==========================================
+ Hits         1355     1369      +14     
+ Misses        157      155       -2     
+ Partials       64       63       -1

Impacted Files	Coverage Δ
skbuild/platform_specifics/aix.py	`87.50% <87.50%> (ø)`
skbuild/platform_specifics/platform_factory.py	`100.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

bhuntsman · 2023-05-25T14:03:40Z

It is worth noting, the IBM-supplied cmake 3.22 has issues. Building cmake 3.26.4 from source seemed to resolve.

henryiii · 2023-05-25T14:27:23Z

Any idea what issues? Might be worth a note in docs somewhere?

(I'm probably going to mostly take your word for what works on AIX :) )

bhuntsman · 2023-05-25T14:34:55Z

Any idea what issues? Might be worth a note in docs somewhere?

Here is an example:

$ cat t.c
#include <stdio.h>
 
void main(void)
{
        printf("hello\n");
}
$ cat CMakeLists.txt
project(HelloWorld C)
cmake_minimum_required(VERSION 3.0)
 
add_executable(t t.c)
 
$ cmake .
-- The C compiler identification is unknown
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /opt/IBM/xlC/16.1.0/bin/cc
-- Check for working C compiler: /opt/IBM/xlC/16.1.0/bin/cc - broken
CMake Error at /opt/freeware/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler
 
    "/opt/IBM/xlC/16.1.0/bin/cc"
 
  is not able to compile a simple test program.
 
  It fails with the following output:
 
    Change Dir: /home/ben/cmaketest/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make -f Makefile cmTC_33606/fast &&   make  -f CMakeFiles/cmTC_33606.dir/build.make CMakeFiles/cmTC_33606.dir/build
    Building C object CMakeFiles/cmTC_33606.dir/testCCompiler.c.o
        /opt/IBM/xlC/16.1.0/bin/cc    -o CMakeFiles/cmTC_33606.dir/testCCompiler.c.o -c /home/ben/cmaketest/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_33606
        /opt/freeware/bin/cmake -E cmake_link_script CMakeFiles/cmTC_33606.dir/link.txt --verbose=1
    /opt/IBM/xlC/16.1.0/bin/cc CMakeFiles/cmTC_33606.dir/testCCompiler.c.o -o cmTC_33606  /usr/lib /lib 
    ld: 0711-168 SEVERE ERROR: Input file: /usr/lib
        Input files must be regular files.
    make: 1254-004 The error code from the last command is 12.
    
    
    Stop.
    make: 1254-004 The error code from the last command is 2.
    
    
    Stop.
    
    
 
  
 
  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:1 (project)
 
 
-- Configuring incomplete, errors occurred!
See also "/home/ben/cmaketest/CMakeFiles/CMakeOutput.log".
See also "/home/ben/cmaketest/CMakeFiles/CMakeError.log".

The cmake . step works fine under cmake 3.26.4 without any other changes. This is worth noting, but is entirely not our problem here.

LecrisUT · 2023-05-26T07:17:53Z

Thanks for the debug info

Could you confirm that this is

independent of camke compatibility (bump cmake_minimum_required)
independent of build tool (try with ninja-build)
independent of cmake installation (try PyPI version or compiled from source version)
if it's compatible with lower or higher cmake versions

We could also check CMakeTestCCompiler.cmake history for any clues.

henryiii · 2023-05-26T07:21:00Z

Can't try PyPI version, there are no AIX wheels & scikit-build is required to build CMake wheels. :)

LecrisUT · 2023-05-26T07:24:27Z

Can't try PyPI version, there are no AIX wheels & scikit-build is required to build CMake wheels. :)

Lol, that's a fun chicken-and-egg problem. Well, at least points 1, 2, 4 can hel narrow down

bhuntsman · 2023-05-26T15:06:16Z

Thanks for the debug info

Could you confirm that this is

independent of camke compatibility (bump cmake_minimum_required)

independent of build tool (try with ninja-build)

independent of cmake installation (try PyPI version or compiled from source version)

if it's compatible with lower or higher cmake versions

We could also check CMakeTestCCompiler.cmake history for any clues.

I believe the issue is with the IBM-supplied cmake 3.22. I built cmake 3.26.4 from source and it works just fine with the exact same CMakeLists.txt listed earlier.

I'm just trying to use this to build ninja-python-distributions. After installing this into my venv with the commit here, I can try to build ninja-python-distributions as follows:

pip install . --no-build-isolation -vvv

The --no-build-isolation is needed to force it to use my modified version of scikit-build.

It fails, but it gets further than the old message about not supporting platform aix. Here's the output:

Using pip 23.1.2 from /home/ben/PythonDevenv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-3dpb03y5
Initialized build tracking at /tmp/pip-build-tracker-3dpb03y5
Created build tracker: /tmp/pip-build-tracker-3dpb03y5
Entered build tracker: /tmp/pip-build-tracker-3dpb03y5
Created temporary directory: /tmp/pip-install-szwolucq
Created temporary directory: /tmp/pip-ephem-wheel-cache-8dzmc2zn
Processing /home/ben/ninja-python-distributions
  Added file:///home/ben/ninja-python-distributions to build tracker '/tmp/pip-build-tracker-3dpb03y5'
  Created temporary directory: /tmp/pip-modern-metadata-qgw6dmyu
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info
  writing /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*' found under directory '_skbuild'
  warning: no previously-included files matching '*' found under directory 'Ninja-src'
  warning: no files found matching 'versioneer.py'
  adding license file 'LICENSE_Apache_20'
  adding license file 'AUTHORS.rst'
  writing manifest file '/tmp/pip-modern-metadata-qgw6dmyu/ninja.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-qgw6dmyu/ninja.dist-info'
  Preparing metadata (pyproject.toml) ... done
  Source in /home/ben/ninja-python-distributions has version 0.0.0, which satisfies requirement ninja==0.0.0 from file:///home/ben/ninja-python-distributions
  Removed ninja==0.0.0 from file:///home/ben/ninja-python-distributions from build tracker '/tmp/pip-build-tracker-3dpb03y5'
Created temporary directory: /tmp/pip-unpack-qz710_3r
Building wheels for collected packages: ninja
  Created temporary directory: /tmp/pip-wheel-xsv9xip7
  Destination directory: /tmp/pip-wheel-xsv9xip7
  Running command Building wheel for ninja (pyproject.toml)
  Target "CMakeFiles/download_ninja_source.dir/build" is up to date.
  [ 44%] Built target download_ninja_source
  [ 50%] Performing build step for 'build_ninja'
  Target "CMakeFiles/libninja-re2c.dir/build" is up to date.
  [  2%] Built target libninja-re2c
  [  4%] Building CXX object CMakeFiles/libninja.dir/src/build_log.cc.o
  "/home/ben/ninja-python-distributions/Ninja-src/src/hash_map.h", line 94.10: 1540-0836 (S) The #include file <ext/hash_map> is not found.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  Traceback (most recent call last):
    File "/home/ben/PythonDevenv/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
      cmkr.make(make_args, install_target=cmake_install_target, env=env)
    File "/home/ben/PythonDevenv/lib/python3.9/site-packages/skbuild/cmaker.py", line 697, in make
      self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
    File "/home/ben/PythonDevenv/lib/python3.9/site-packages/skbuild/cmaker.py", line 742, in make_impl
      raise SKBuildError(msg)

  An error occurred while building with CMake.
    Command:
      /usr/local/bin/cmake --build . --target install --config Release --
    Install target:
      install
    Source directory:
      /home/ben/ninja-python-distributions
    Working directory:
      /home/ben/ninja-python-distributions/_skbuild/aix-7301-9988-64-3.9/cmake-build
  Please check the install target is valid and see CMake's output for more information.

  error: subprocess-exited-with-error
  
  Building wheel for ninja (pyproject.toml) did not run successfully.
  exit code: 1
  
  See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/ben/PythonDevenv/bin/python3 /home/ben/PythonDevenv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpxe88ld8j
  cwd: /home/ben/ninja-python-distributions
  Building wheel for ninja (pyproject.toml) ... error
  ERROR: Failed building wheel for ninja
Failed to build ninja
ERROR: Could not build wheels for ninja, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/home/ben/PythonDevenv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/ben/PythonDevenv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/home/ben/PythonDevenv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 426, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for ninja, which is required to install pyproject.toml-based projects
Remote version of pip: 23.1.2
Local version of pip:  23.1.2
Was pip installed by pip? True
Removed build tracker: '/tmp/pip-build-tracker-3dpb03y5'

The compiler is XL C 16.1. IBM supplies <ext/hash_map> in /opt/IBM/xlC/16.1.0/include2/c++/ext/hash_map. To me this looks like a cmake problem where it is not adding in /opt/IBM/xlC/16.1.0/include2/c++/ to the list of include paths. It also seems that ninja supports AIX but not XLC and assumes gcc on AIX. That's fine, but that's a problem with ninja, ninja-python-distributions, or cmake, not with scikit-build. But adding support to scikit-build is the first step.

bhuntsman · 2023-05-30T18:12:59Z

Hi @henryiii and @LecrisUT, do I need to do anything else at this time to get this reviewed and accepted? Are we in agreement that the issue above (which doesn't really have anything to do with scikit-build) is entirely upstream and therefore, not our problem to work around?

Thank you!

henryiii · 2023-05-30T18:24:24Z

I'm working on the testing issues in #879. Once that's in, this should be unblocked.

henryiii · 2023-05-30T18:25:15Z

Also, I want to add a little note about AIX requiring a more recent CMake than it ships with, but I can probably do that.

bhuntsman · 2023-05-30T18:27:06Z

Also, I want to add a little note about AIX requiring a more recent CMake than it ships with, but I can probably do that.

FYI, I also brought this issue to the attention of the IBM open source builds team. They have acknowledged the problem and have committed to providing an official build of a more recent release of CMake.

AIX support requires a version of CMake newer than the 3.22.0 version that IBM supplies in their AIX Toolbox for Open Source Software. IBM is aware of the issue and will be providing a newer build soon, but for now we should note that users will need to build their own copy of a newer CMake on AIX at this time.

henryiii · 2023-05-30T23:45:18Z

Thanks!

bhuntsman closed this May 24, 2023

bhuntsman reopened this May 25, 2023

bhuntsman added 2 commits May 30, 2023 16:31

fix: add support for AIX

24d213b

henryiii force-pushed the aix-support branch from d02cefc to f7aaf17 Compare May 30, 2023 20:31

henryiii merged commit 36c7a9c into scikit-build:main May 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: add support for AIX #988

fix: add support for AIX #988

bhuntsman commented May 24, 2023

henryiii commented May 24, 2023

LecrisUT commented May 24, 2023

packit-as-a-service bot commented May 24, 2023

LecrisUT commented May 24, 2023

bhuntsman commented May 24, 2023 •

edited

Loading

henryiii commented May 24, 2023

henryiii commented May 24, 2023

LecrisUT commented May 25, 2023

bhuntsman commented May 25, 2023

codecov bot commented May 25, 2023 •

edited

Loading

bhuntsman commented May 25, 2023

henryiii commented May 25, 2023

bhuntsman commented May 25, 2023

LecrisUT commented May 26, 2023

henryiii commented May 26, 2023

LecrisUT commented May 26, 2023

bhuntsman commented May 26, 2023

bhuntsman commented May 30, 2023

henryiii commented May 30, 2023

henryiii commented May 30, 2023

bhuntsman commented May 30, 2023 •

edited

Loading

henryiii commented May 30, 2023

fix: add support for AIX #988

fix: add support for AIX #988

Conversation

bhuntsman commented May 24, 2023

henryiii commented May 24, 2023

LecrisUT commented May 24, 2023

packit-as-a-service bot commented May 24, 2023

LecrisUT commented May 24, 2023

bhuntsman commented May 24, 2023 • edited Loading

henryiii commented May 24, 2023

henryiii commented May 24, 2023

LecrisUT commented May 25, 2023

bhuntsman commented May 25, 2023

codecov bot commented May 25, 2023 • edited Loading

Codecov Report

bhuntsman commented May 25, 2023

henryiii commented May 25, 2023

bhuntsman commented May 25, 2023

LecrisUT commented May 26, 2023

henryiii commented May 26, 2023

LecrisUT commented May 26, 2023

bhuntsman commented May 26, 2023

bhuntsman commented May 30, 2023

henryiii commented May 30, 2023

henryiii commented May 30, 2023

bhuntsman commented May 30, 2023 • edited Loading

henryiii commented May 30, 2023

bhuntsman commented May 24, 2023 •

edited

Loading

codecov bot commented May 25, 2023 •

edited

Loading

bhuntsman commented May 30, 2023 •

edited

Loading