Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add system tests workflow #2052

Open
wants to merge 24 commits into
base: develop
Choose a base branch
from
Open

Add system tests workflow #2052

wants to merge 24 commits into from

Conversation

uekerman
Copy link
Member

Main changes of this PR

Adds a workflow to run the "release" testsuite, which gets triggered by the trigger-system-tests label.

Motivation and additional information

We have the system tests now. Let's use them.

Author's checklist

  • I used the pre-commit hook to prevent dirty commits and used pre-commit run --all to format old commits.
  • I squashed / am about to squash all commits that should be seen as one.

Reviewers' checklist

  • Do you understand the code changes?

@MakisH
Copy link
Member

MakisH commented Jul 11, 2024

Related documentation PR: precice/tutorials#550

@uekerman
Copy link
Member Author

mmh, seems there is still a problem. @MakisH any quick guess?

@MakisH
Copy link
Member

MakisH commented Jul 16, 2024

mmh, seems there is still a problem. @MakisH any quick guess?

You can click on the Summary and from the bottom of that page download the runs.
Knowing that a specific case fails (in this case, all the FEniCS and Nutils cases), you can look into the stdout.log, stderr.log, and the logs of each participant.

The issue is independent of the system tests: it's the familiar NumPy 2.0 incompatibility.

To fix it, we would need a bugfix release in the tutorials (which I cannot prepare this week).

@uekerman
Copy link
Member Author

To fix it, we would need a bugfix release in the tutorials (which I cannot prepare this week).

I switched to develop for the moment such that we can test if everything else works.

Copy link
Member

@MakisH MakisH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still get numpy 2. From the stdout.log of the flow-over-heated-plate OpenFOAM-FEniCS:

#25 [solid-fenics fenics_adapter 3/4] RUN pip3 install --user fenics-ufl
#25 0.911 Collecting fenics-ufl
#25 0.948   Downloading fenics_ufl-2024.1.0.post1-py3-none-any.whl.metadata (2.6 kB)
#25 0.961 Requirement already satisfied: numpy in /home/precice/.local/lib/python3.10/site-packages (from fenics-ufl) (2.0.0)

I can fix all these issues myself, just not right now.

.github/workflows/system-tests.yaml Outdated Show resolved Hide resolved
@MakisH
Copy link
Member

MakisH commented Sep 8, 2024

Current state:

  • Elastic tube 1D (fluid-cpp, solid-python): ✔️
  • Flow over heated plate (fluid-openfoam, solid-nutils): ✔️
  • Flow over heated plate (fluid-openfoam, solid-fenics): ✔️
  • Flow over heated plate (fluid-openfoam, solid-openfoam): ✔️
  • Perpendicular flap (fluid-openfoam, solid-calculix): ❌ Results regression (!)
  • Perpendicular flap (fluid-su2, solid-fenics): ❌ Results regression (!)

@MakisH
Copy link
Member

MakisH commented Sep 22, 2024

Looking into the failing job, trying to understand the regression in the perpendicular flap, which is now also raised in precice/tutorials#573.

The reference results used have been created with (download the runs, look into the SU2-FEniCS reference_results.metadata:

name value
PRECICE_REF v3.1.1
OPENFOAM_EXECUTABLE openfoam2312
OPENFOAM_ADAPTER_REF v1.3.0
PYTHON_BINDINGS_REF v3.1.0
FENICS_ADAPTER_REF v2.1.0
TUTORIALS_REF 340b447
PLATFORM ubuntu_2204
CALULIX_VERSION 2.20
CALULIX_ADAPTER_REF v2.20.1
SU2_VERSION 7.5.1
SU2_ADAPTER_REF 64d4aff

Watchpoint for OpenFOAM-CalculiX:

watchpoint-solid-calculix

Watchpoint for SU2-FEniCS:

watchpoint-solid-fenics

How it should be: https://precice.org/tutorials-perpendicular-flap.html#post-processing

Related commits in the tutorials repository: https://github.com/precice/tutorials/commits/develop/perpendicular-flap

@MakisH
Copy link
Member

MakisH commented Nov 7, 2024

Now that precice/tutorials#573 is fixed, one test is still failing (Perpendicular flap (fluid-openfoam, solid-calculix)), which completes but reports deviations above tolerance.

Note that I had to manually trigger the system tests to use the released preCICE v3.1.2, because the current develop has changed the names of the exported files, as discussed in #2053 (comment):

gh workflow run run_testsuite_manual.yml -f suites=release_test -f build_args=PLATFORM:ubuntu_2204,PRECICE_REF:v3.1.2,PYTHON_BINDINGS_REF:b6b9ee5,CALCULIX_VERSION:2.20,CALCULIX_ADAPTER_REF:2a5a73a,FENICS_ADAPTER_REF:3de561d,OPENFOAM_EXECUTABLE:openfoam2406,OPENFOAM_ADAPTER_REF:20b4617,SU2_VERSION:7.5.1,SU2_ADAPTER_REF:64d4aff,TUTORIALS_REF:a10e4a6

I will now investigate which of the components causes the deviation and update this comment. Update:

@MakisH
Copy link
Member

MakisH commented Nov 8, 2024

The reason for the still failing test was the different OpenFOAM version. I downgraded to v2312 and waiting for the follow-up to #2053 (comment)

@MakisH
Copy link
Member

MakisH commented Nov 8, 2024

Now all the Python-related system tests fail with the same segfault, but I cannot reproduce this locally. I suspect this has something to do with the layer caching.

Full output of an example: stdout.log
Same file with only the output of solid-python: solid-python.log

I will now clear all the Docker cache and try again. -> did not help

(potentially related issue: precice/tutorials#584)

@MakisH
Copy link
Member

MakisH commented Nov 11, 2024

After debugging a bit with @fsimonis, we found out that the issue currently is in preCICE, since the migration of the exporters to samples (e414c76).

Commit a41d1a2 (one before) does not show the segfault.

The issue is not related to Python, as it also happens in OpenFOAM and in the elastic_tube_1d C++. What the failing cases have in common: nearest-neighbor read-consistent mapping and serial-implicit scheme. One case has data initialization, but the other does not.

@MakisH
Copy link
Member

MakisH commented Nov 12, 2024

Another system tests run that reproduces the issue, with debug and trace information.

@uekerman
Copy link
Member Author

We are getting closer:

  <participant name="Solid">
    <provide-mesh name="Solid-Nodes-Mesh" />
    <receive-mesh name="Fluid-Nodes-Mesh" from="Fluid" />
    <write-data name="CrossSectionLength" mesh="Solid-Nodes-Mesh" />
    <read-data name="Pressure" mesh="Solid-Nodes-Mesh" />
    <mapping:nearest-neighbor
      direction="read"
      from="Fluid-Nodes-Mesh"
      to="Solid-Nodes-Mesh"
      constraint="consistent" />
      <export:vtk />
  </participant>

We assert here:

for (const mesh::PtrData &data : mesh.data()) { // Plot vertex data
const Eigen::VectorXd &values = data->timeStepsStorage().last().sample.values;

For Fluid-Nodes-Mesh and CrossSectionLength. This is data that we neither write/read nor receive. It actually makes no sense to export this data. We could either write zeros or skip the export.

@MakisH MakisH mentioned this pull request Nov 14, 2024
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants