Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The test in debug_test.py / quick_test.bat, that diffs against correctout.txt randomly fails. Stochastic behaviour (flakey test) or bug? #21

Open
JamesParrott opened this issue Feb 14, 2024 · 2 comments

Comments

@JamesParrott
Copy link
Collaborator

Sometimes:

ERROR: Formula evaluation gave negative result for link 12

or link 2 is outputted, but the test requires

ERROR: Formula evaluation gave negative result for link 10

I assumed Windows was downloading a huge update during office hours again, or something else particular to my local test environment was causing the weird behaviour. But this test behaves very strangely in CI too. So I'm not sure if the test is flakey, overly reliant on the test environment, if it relies on a stochastic function within sDNA, or if there really is a bug (if so, how important is this part of the test? - do tests need to fail if error messages are wrong, failing for failing wrongly?).

I was able to get all the other tests to pass, by simply persevering, and rerunning any failed jobs. But not the Python 3.8 test, yet:

image

https://github.com/JamesParrott/sdna_plus/actions/runs/7907698164/job/21586911903

However it passed fine in Python 3.8.10 locally just now, so I doubt there's a Python 3.8 only bug:

(pyenv_3.8.10_pytest_numpy) C:\...\sdna_plus\sDNA\sdna_vs2008\tests\pytest>set DONT_TEST_ONE_LINK_SUBSYSTEMS_ORDER=1 & pytest
================================================= test session starts =================================================
platform win32 -- Python 3.8.10, pytest-8.0.0, pluggy-1.4.0
rootdir: C:\...\repos\sdna_plus\sDNA\sdna_vs2008\tests\pytest
collected 10 items

test_gathering_windows_command_line_tests.py ..........                                                          [100%]

================================================= 10 passed in 26.09s =================================================


@JamesParrott JamesParrott changed the title The test in debug_test.py / quick_test.bat, that diffes against correctout.txt randomly fails. Stochastic behaviour (flakey test) or bug? The test in debug_test.py / quick_test.bat, that diffs against correctout.txt randomly fails. Stochastic behaviour (flakey test) or bug? Feb 14, 2024
@JamesParrott
Copy link
Collaborator Author

debug_test.py passed on all Python versions (2.7, 3.7, ..., 3.12) on this attempt:

image

https://github.com/JamesParrott/sdna_plus/actions/runs/7908696185/job/21588978919

@JamesParrott
Copy link
Collaborator Author

Temporary work around. This issue is currently ignored in the regression tests by setting the environment variable ALLOW_NEGATIVE_FORMULA_ERROR_ON_ANY_LINK_PRESENT

# Configure allowing regression tests to pass despite issue #21

The calling workflows set this env variable, by setting the input to the custom Action

ALLOW_NEGATIVE_FORMULA_ERROR_ON_ANY_LINK_PRESENT:

JamesParrott added a commit that referenced this issue Jun 17, 2024
Update tests to ignore tiny floating point diffrences
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant