In b9eebb3, I tried intentionally causing a unit test failure, and this change:

# before
Run-R-Code-Redirect-Stderr "source('testthat.R')" ; Check-Output $?

# after
Rscript.exe --vanilla "testthat.R" ; Check-Output $?

Saw that both CI jobs failed and reported test failures in logs (build link).

Then, in 7ce18e2, tried reverting the intentional test failure. Interestingly, the windows-2022 job passed and the windows-2019 job failed. Both should have passed.

• ❌ windows-2019, MSVC, R 3.6, cmake (build link)
• ✅ windows-2022, MSVC, R 4.1, cmake (build link)

I'll investigate this more tomorrow.

iiiiinterresting!

commenting out all of the dataset.R tests results in the windows-2019 job succeeding!

• commit: a075579
• job: https://github.com/microsoft/LightGBM/runs/8219900714?check_suite_focus=true

Alright, this is ready for review!

I believe the root cause of the original issue (#5439) is something in the intersection between:

• how the library compiled with MSVC interacts with stdout + stderr + R's output streams
• how {testthat} interacts with R's output streams
• the way that powershell sometimes treats output to stderr as a failure (see discussion from [ci] Move R Windows CI to GitHub Actions #3168)

This PR proposes two changes to ensure that MSVC R-package CI jobs actually fail when the tests fail:

• using Rscript.exe testthat.R directly to run the tests, instead of the Run-R-Code-Redirect-Stderr powershell function
• skipping one test in test_dataset.R that only fails on windows-2019, MSVC, R 3.6, cmake job

Why I think it's ok to skip that one test only on that one job

• that test is passing on all other operating systems and versions of R tested in CI
  • including, especially, the combination of R 3.6 + Windows + cmake build (just with MinGW instead)
• the time spent investigating this one test on an old Visual Studio version + old powershell version + old R version could be better spent working on other items for the 4.0.0 release ([RFC] 4.0.0 Release #5153)
• re-enabling these two CI jobs will give LightGBM coverage of R + cmake + MSVC in CI again (which it currently doesn't really have, since all jobs silently pass)

Evidence that the jobs will fail as of these changes

See the results from this commit which intentionally made one R unit test fail: aff9acf.

• ❌ windows-2019, MSVC, R 3.6, cmake (build link)
• ❌ windows-2022, MSVC, R 4.1, cmake (build link)

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.