Description

The R-package CI jobs which compile the library with CMake + MSVC do not fail if the R package's unit tests fail.

• r-package (windows-2019, MSVC, R 3.6, cmake)
• r-package (windows-2022, MSVC, R 4.1, cmake)

Reproducible example

I noticed this while looking at test failures on #5437 (build link), but I suspect that this would happen for any change that causes the R unit tests to fail.

For example, I suspect (haven't confirmed) that adding something like that following anywhere in an R test case would reproduce this issue.

stop("intentionally failing")

But at the end of the tests, I see logs like the following.

Error: Test failures
Execution halted

== Failed ======================================================================
-- 1. Failure (test_basic.R:93:3): train and predict binary classification
...
-- 10. Failure (test_basic.R:3190:5): lgb.train() only prints eval metrics when 

Maximum number of 10 failures reached, some test results may be missing.

Additional Comments

I haven't investigated this much yet, just creating this issue so it isn't forgotten.