How to rebuild the _ext.cpp when developing? #302

kawaho · 2024-06-28T18:19:24Z

kawaho
Jun 28, 2024

Sorry this is a really basic question. I am trying to add some features into _ext.cpp. How do I rebuild the package after modifying the file?

In the README.md, it says

bash python -m pip install '.[test]'

but I assume this is just for the first time building the package. I imagine there is a way to only recompile the changes I did in the .cpp files?

Answered by matthewfeickert

Sep 13, 2024

Between PR #311 and #279 (comment) I think this is now fully resolved so that one can now just run

python -m pip install --upgrade .

from the top level directory and get a working rebuild. Until Issue #279 is fully resolved, to get a totally clean slate for all of the Git submodules you can run #279 (comment) in advance of install.

View full answer

matthewfeickert · 2024-06-28T18:34:05Z

matthewfeickert
Jun 28, 2024
Maintainer

I am trying to add some features into _ext.cpp.

@kawaho Please open up an Issue (https://github.com/scikit-hep/fastjet/issues) that describes the features that you want to add, so these can get discussed in advance.

How do I rebuild the package after modifying the file?

Building happens during the install, so just install your modified version with

python -m pip install --upgrade .

0 replies

kawaho · 2024-09-09T20:22:35Z

kawaho
Sep 9, 2024
Author

Hi @matthewfeickert , coming back to this question, I was not able to rebuild the package using

python -m pip install --upgrade .

When I do so, I get the following error:

  building 'fastjet._ext' extension
  /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -fPIC -I/isilon/export/home/kho29/fastjet_dev/fastjet/src/fastjet/_fastjet_core/include -I/tmp/pip-build-env-1urvtoly/overlay/lib/python3.12/site-packages/pybind11/include -I/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include/python3.12 -c src/_ext.cpp -o build/temp.linux-x86_64-cpython-312/src/_ext.o -fvisibility=hidden -g0 -std=c++11
  /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -pthread -B /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/compiler_compat -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -Wl,-rpath-link,/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -L/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include build/temp.linux-x86_64-cpython-312/src/_ext.o -L/isilon/export/home/kho29/fastjet_dev/fastjet/src/fastjet/_fastjet_core/lib -Wl,-R$ORIGIN/_fastjet_core/lib -lfastjet -lfastjettools -lsiscone -lsiscone_spherical -lfastjetplugins -lfastjetcontribfragile -o build/lib.linux-x86_64-cpython-312/fastjet/_ext.cpython-312-x86_64-linux-gnu.so
  installing to build/bdist.linux-x86_64/wheel
  running install
  error: [Errno 17] File exists: 'build/lib.linux-x86_64-cpython-312/fastjet/_fastjet_core'
  error: subprocess-exited-with-error
  
  × Building wheel for fastjet (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/python /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpgrzugkax
  cwd: /isilon/export/home/kho29/fastjet_dev/fastjet
  Building wheel for fastjet (pyproject.toml) ... error
  ERROR: Failed building wheel for fastjet
Failed to build fastjet
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (fastjet)

If I manually remove build and src/fastjet/_fastjet_core/, everything gets built from scratch instead of just updating my modifications. Any suggestions on this? Thanks!

9 replies

kawaho Sep 10, 2024
Author

Hi @matthewfeickert ,

You should not be removing the FastJet C++ source code. This is definitely not correct.

This is the part where I got confused because there are fastjet-core/ and src/fastjet/_fastjet_core. The latter is created after the build so I thought it's ok to remove it as pip complains something like src/fastjet/_fastjet_core already exists .

Please explain what it is you're actually trying to do and what the state of the repository is in. Though again, if you want to contribute additions to the source code open an Issue first so that there can be a discussion in advance.

Yes, I have opened an issue some time ago: #303

matthewfeickert Sep 11, 2024
Maintainer

The latter is created after the build so I thought it's ok to remove it as pip complains something like src/fastjet/_fastjet_core already exists .

@kawaho My apologies, you're absolutely right, and even though you pointed everything out clearly and I said I carefully reread things I still misread the paths. Sorry.

You're correct that do do rebuilds you would need to remove this transient directory. This, again, is not something that you as a developer should be responsible for manually having to keep track of, so we again see the need for Issue #279 and Issue #310 to get dealt with.

matthewfeickert Sep 11, 2024
Maintainer

I've opened PR #311 to partially address parts of this.

matthewfeickert Sep 13, 2024
Maintainer

Between PR #311 and #279 (comment) I think this is now fully resolved so that one can now just run

python -m pip install --upgrade .

from the top level directory and get a working rebuild. Until Issue #279 is fully resolved, to get a totally clean slate for all of the Git submodules you can run #279 (comment) in advance of install.

Answer selected by matthewfeickert

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to rebuild the _ext.cpp when developing? #302

{{title}}

Replies: 2 comments 9 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

How to rebuild the _ext.cpp when developing? #302

kawaho Jun 28, 2024

Replies: 2 comments · 9 replies

matthewfeickert Jun 28, 2024 Maintainer

kawaho Sep 9, 2024 Author

kawaho Sep 10, 2024 Author

matthewfeickert Sep 11, 2024 Maintainer

matthewfeickert Sep 11, 2024 Maintainer

matthewfeickert Sep 13, 2024 Maintainer

kawaho
Jun 28, 2024

Replies: 2 comments 9 replies

matthewfeickert
Jun 28, 2024
Maintainer

kawaho
Sep 9, 2024
Author

kawaho Sep 10, 2024
Author

matthewfeickert Sep 11, 2024
Maintainer

matthewfeickert Sep 11, 2024
Maintainer

matthewfeickert Sep 13, 2024
Maintainer