Description
Related problem
I think that Nushell has huge potential as a testing tool given it's data manipulation and integration capabilities. Both testing external systems and data as well as for Nushell scripts. Just one example, if you've ever used Nushell with WireMock, using records and merge to mock the expected http requests and responses, varying data for different use cases, is just beautiful. I'm very impressed with Nushell's practical focus and how the functionally fits together so well.
Given the importance of testing, it would be great to see the out of the box experience improved with a runner, reporting and test coverage. One reason I'd reach for Nushell over Python for example is, other than being a great integration tool, not having to deal with virtual environments and external dependencies for the 80% case makes it really easy to pick up and run with.
Describe the solution you'd like
Here's what is ideally like to see:
- A basic out the box test runner. Something like testing.nu used to test the standard library but with output as a table that can then be reported in more flexible ways.
- A default reporter into some common format like Cobertura that can be used in CI/CD systems or sucked into some other converter tool.
- Line-based test coverage metrics.
I can implement the first two points. I'd probably start with the runner for the standard library, put some tests around it, and add support for scripts that aren't modules, all in the hopes that it could be promoted to the Nushell standard library. However, I wonder if that's a bit short sighted. The bit I'd have trouble implementing is the line+file based test coverage, which would likely need to be implemented in the core (or bundled plugin?) and likely needs a far bit of knowledge of Nushell internals.
Describe alternatives you've considered
Given the ideal scenario of having coverage information, it seems that the strategic option would be to implement some kind of introspection/debugging functionality into the core that can be used to understand the coverage, and a runner and reporter as a plugin or nushell script. Or perhaps just a debugging (coverage-gathering) run mode in the core perhaps behind a nu command switch.
A more tactical solution could be done in Nushell without coverage, and perhaps later promoted to core functionality where code coverage could then be added. If we do that right, we could minimise changes to test files, e.g. a use std testing statement becomes optional.
Perhaps the core team can provide some guidance / discuss what they would like to do before I start down the tactical road?
Additional context and details
Prior art:
- The docs I added for writing a simple test runner.
- The test runner in nupm here.
- The testing.nu test runner used to run tests for the standard library.
- A test runner I wrote to run the container tests here, as none of the above was practical to use in that context. You can see the report output for that in PR Tests for new Alpine and Debian image builds #14225 .
Perhaps improvements in this area can also help encourage testing nushell scripts in the core. Here are some related issues: