Enable and Fix Environment Examples in CMake CI #642

mostafaelhoushi · 2022-03-31T13:54:07Z

Fixes #631

ensured that the example scripts are invoked in CMake CI by creating example_without_bazel_test.py scripts
ensured that unit tests in env_tests.py are invoked in CMake CI by copying them into env_without_bazel_test.py
disabled the tests cases in Linux and MacOS bazel builds for now (hopefully will fix that with How to build and access with loop_unroller and opt_loops binaries in CI #707 )

codecov-commenter · 2022-03-31T14:15:57Z

Codecov Report

Merging #642 (52f5d04) into development (71b1b77) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           development     #642      +/-   ##
===============================================
- Coverage        88.68%   88.63%   -0.06%     
===============================================
  Files              131      131              
  Lines             7934     7934              
===============================================
- Hits              7036     7032       -4     
- Misses             898      902       +4

Impacted Files	Coverage Δ
compiler_gym/service/connection.py	`77.66% <0.00%> (-0.67%)`	⬇️
...ompiler_gym/service/client_service_compiler_env.py	`90.43% <0.00%> (-0.42%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71b1b77...52f5d04. Read the comment docs.

ChrisCummins · 2022-04-01T20:28:58Z

Looks like the CI is broken on this

mostafaelhoushi · 2022-05-25T14:19:50Z

I am revisiting this pull request.
So the CI runs the tests by calling make -C example tests which in turn runs the Python tests without bazel.
The tests run fine if I use bazel, e.g.,

bazel test //examples/example_compiler_gym_service/...

but running the Python files direcrtly doesn't work.

So I had to follow the approach we did when we copied demo.py to demo_without_bazel.py, so here I am copying env_test.py to env_without_bazel_test.py and made some changes accordingly.

However, I am now stuck with this error:

$ cd examples
$ python example_compiler_gym_service/env_without_bazel_test.py                            
======================================================================= test session starts ========================================================================
platform darwin -- Python 3.8.11, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /Users/melhoushi/opt/anaconda3/envs/compiler_gym/bin/python
cachedir: .pytest_cache
rootdir: /Users/melhoushi/CompilerGym, configfile: tox.ini
plugins: flaky-3.7.0, hydra-core-1.1.0
collected 0 items / 1 error                                                                                                                                        
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1210, in pytest_collection
INTERNALERROR>     self._validate_config_options()
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1233, in _validate_config_options
INTERNALERROR>     self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1269, in _warn_or_fail_if_strict
INTERNALERROR>     self.issue_config_time_warning(PytestConfigWarning(message), stacklevel=3)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1321, in issue_config_time_warning
INTERNALERROR>     warnings.warn(warning, stacklevel=stacklevel)
INTERNALERROR> pytest.PytestConfigWarning: Unknown config option: timeout

========================================================================= 1 error in 0.16s =========================================================================

ChrisCummins · 2022-05-25T15:02:42Z

That's a strange one. Looks like maybe a problem with missing / bad plugin?

If you disable env_without_bazel_test.py (by renaming it foo.py), does the problem go away?

mostafaelhoushi · 2022-05-25T15:15:21Z

That's a strange one. Looks like maybe a problem with missing / bad plugin?

If you disable env_without_bazel_test.py (by renaming it foo.py), does the problem go away?

Got the same error :(

$ mv example_compiler_gym_service/env_without_bazel_test.py example_compiler_gym_service/foo.py
$ python example_compiler_gym_service/foo.py 

======================================================================= test session starts ========================================================================
platform darwin -- Python 3.8.11, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /Users/melhoushi/opt/anaconda3/envs/compiler_gym/bin/python
cachedir: .pytest_cache
rootdir: /Users/melhoushi/CompilerGym, configfile: tox.ini
plugins: flaky-3.7.0, hydra-core-1.1.0
collected 0 items / 1 error                                                                                                                                        
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1210, in pytest_collection
INTERNALERROR>     self._validate_config_options()
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1233, in _validate_config_options
INTERNALERROR>     self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1269, in _warn_or_fail_if_strict
INTERNALERROR>     self.issue_config_time_warning(PytestConfigWarning(message), stacklevel=3)
INTERNALERROR>   File "/Users/melhoushi/opt/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1321, in issue_config_time_warning
INTERNALERROR>     warnings.warn(warning, stacklevel=stacklevel)
INTERNALERROR> pytest.PytestConfigWarning: Unknown config option: timeout

========================================================================= 1 error in 0.15s =========================================================================

mostafaelhoushi · 2022-05-25T15:43:17Z

Never mind.. when I installed:

$ pip install pytest-timeout==1.4.2

I no longer got the error....

Now fixing other errors :)

mostafaelhoushi · 2022-05-25T16:48:52Z

@ChrisCummins : do you know why the pre-commit job is failing? It seems to be not related to this PR:

ChrisCummins · 2022-05-25T17:07:48Z

Copy the examples/sensitivity_analysis dir from latest development head

I broke it yesterday, should be fixed now 🙂

ChrisCummins · 2022-05-25T17:22:31Z

Ah, sorry, I was confused! The development branch is still broken. I'll merge an update now so that you can rebase this branch on it. Stay tuned ...

ChrisCummins · 2022-05-25T17:24:15Z

Okay, development is now un-broken. Off to the races!

mostafaelhoushi · 2022-05-25T20:38:01Z

The CI is failing with WARNING: Overriding environment example-v0.

I was able to mitigate that on my local machine by adding those arguments when calling main (full context here):

if __name__ == "__main__":
    main(
        extra_pytest_args=[
            "-W",
            "ignore::UserWarning",
        ]
    )

I am not sure why it still fails on CI.

Any idea @ChrisCummins ?

ChrisCummins · 2022-05-27T03:53:48Z

Hmm, not sure about this one. Will have to take a look on Tuesday.

Cheers,
Chris

mostafaelhoushi · 2022-05-27T04:54:04Z

OK. I fixed it in this commit 0077632 by choosing a different name for the registered environment in env_without_bazel_test.py from example-v0 to example-v1.

(Perhaps the error is due to demo_without_bazel_test.py running before env_without_bazel_test.py creating an environment with the same name example-v0).

ChrisCummins · 2022-05-31T22:12:26Z

Ah, that makes sense. Every environment you register must have a unique name. So if two tests register example-v0, one will clobber the other when all of the tests are run in a single python session.

Cheers,
Chris

try Path instead of runfiles_path to see if CI passes add test without bazel revert commits to bazel tests and update without_bazel test revert name change dont throw errors on UserWarning enable import tests by adding __init__ file avoid importing tests as it causes pytest to throw an error rename env to example-v1 to avoid warning fix test_observation_space get example_unrolling_service to work without bazel avoid 'WARNING: unrolling-v1 environment already exists' error build loop_unroller from within python if it is not found don't include the CMake for standalone build mv clang-10 to clang sometimes clang-10 doesn't exist so handle that

mostafaelhoushi · 2022-06-13T16:13:56Z

@ChrisCummins , test-examples-macos failed with:

        if done:
            # Computing an observation should never cause a terminal state since
            # no action has been applied.
            msg = f"Failed to compute observation '{observation_space.id}'"
            if info.get("error_details"):
                msg += f": {info['error_details']}"
>           raise ServiceError(msg)
E           compiler_gym.errors.service_errors.ServiceError: Failed to compute observation 'runtime': RPC call returned status code StatusCode.INTERNAL and error `Compilation job failed with returncode 1 (SIGHUP)
E           Command: /Users/runner/.local/share/compiler_gym/llvm-v0/bin/clang /Users/runner/.local/cache/compiler_gym/s/0613T160607-720121-389c/benchmark.o -O3 -o /Users/runner/.local/cache/compiler_gym/s/0613T160607-720121-389c/benchmark.exe
E           Stderr: ld: library not found for -lSystem
E           clang: error: linker command failed with exit code 1 (use -v to see invocation)`

do you know a fix for that?

(Full log should be here: https://github.com/facebookresearch/CompilerGym/runs/6865005685?check_suite_focus=true)

ChrisCummins · 2022-06-13T16:34:51Z

Yep. The LLVM binaries we use for CompilerGym don't know about the system libraries, we need to figure them out ourselves. To do this, call compiler_gym.envs.llvm.get_system_library_flags() and append the list of flags it returns to your clang invocation(s).

Cheers,
Chris

…st-examples-macos till we build them in bazel

mostafaelhoushi · 2022-06-13T21:48:29Z

I have removed the explicit run of python examples in CI, and ensured that they run in pytest instead.

I have disabled the tests cases in Linux and MacOS bazel builds for now (hopefully will fix that with #707 ).

The Pull Request should be ready now for review @ChrisCummins

ChrisCummins · 2022-06-13T22:19:15Z

LGTM. Many thanks for pushing on this @mostafaelhoushi.

Feel free to merge when you're happy with it.

Cheers,
Chris

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 31, 2022

mostafaelhoushi force-pushed the enable-examples-tests-in-ci branch from fa20351 to 6f1ef6b Compare May 25, 2022 14:13

mostafaelhoushi force-pushed the enable-examples-tests-in-ci branch from 5ac4fea to 4c1009f Compare May 25, 2022 18:15

mostafaelhoushi mentioned this pull request May 31, 2022

Run examples in cmake CI #695

Merged

mostafaelhoushi force-pushed the enable-examples-tests-in-ci branch from 5ce9316 to e229d5f Compare June 13, 2022 15:16

remove lazy build of loop_unroller

a79e4cf

mostafaelhoushi added 6 commits June 13, 2022 12:47

disable loop_unroller or opt_loops for bazel builds in CI

1bce46a

add get_system_library_flags() to pass cmake test

baccb7f

ignore example_unrolling_service and loop_optimizations_service in te…

c910349

…st-examples-macos till we build them in bazel

no need to run examples in CI as they're covered in pytest

0b25c50

update loop_optimizations_service to run in cmake ci

bfdacbb

update loop_optimizations_service to run in cmake ci

52f5d04

mostafaelhoushi changed the title ~~Enable and Fix Environment Examples in CI~~ Enable and Fix Environment Examples in CMake CI Jun 13, 2022

mostafaelhoushi merged commit d67ce92 into facebookresearch:development Jun 13, 2022

This was referenced Nov 1, 2022

CompilerGym v0.2.5 #767

Merged

CompilerGym v0.2.5 #773

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable and Fix Environment Examples in CMake CI #642

Enable and Fix Environment Examples in CMake CI #642

mostafaelhoushi commented Mar 31, 2022 •

edited

Loading

codecov-commenter commented Mar 31, 2022 •

edited

Loading

ChrisCummins commented Apr 1, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 25, 2022

mostafaelhoushi commented May 25, 2022

mostafaelhoushi commented May 25, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 25, 2022 •

edited

Loading

ChrisCummins commented May 25, 2022

ChrisCummins commented May 25, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 27, 2022

mostafaelhoushi commented May 27, 2022

ChrisCummins commented May 31, 2022

mostafaelhoushi commented Jun 13, 2022

ChrisCummins commented Jun 13, 2022

mostafaelhoushi commented Jun 13, 2022

ChrisCummins commented Jun 13, 2022

Enable and Fix Environment Examples in CMake CI #642

Enable and Fix Environment Examples in CMake CI #642

Conversation

mostafaelhoushi commented Mar 31, 2022 • edited Loading

codecov-commenter commented Mar 31, 2022 • edited Loading

Codecov Report

ChrisCummins commented Apr 1, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 25, 2022

mostafaelhoushi commented May 25, 2022

mostafaelhoushi commented May 25, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 25, 2022 • edited Loading

ChrisCummins commented May 25, 2022

ChrisCummins commented May 25, 2022

mostafaelhoushi commented May 25, 2022

ChrisCummins commented May 27, 2022

mostafaelhoushi commented May 27, 2022

ChrisCummins commented May 31, 2022

mostafaelhoushi commented Jun 13, 2022

ChrisCummins commented Jun 13, 2022

mostafaelhoushi commented Jun 13, 2022

ChrisCummins commented Jun 13, 2022

mostafaelhoushi commented Mar 31, 2022 •

edited

Loading

codecov-commenter commented Mar 31, 2022 •

edited

Loading

ChrisCummins commented May 25, 2022 •

edited

Loading