You may want to see this gist for my original proposal.

Also, this PR is apparently 1.4 mil new lines... sorry not sorry. :)

I'm going with a zip archive. These tests have to be done manually anyway.

Purpose

This is a major insurance PR. I want to make some pretty significant changes to the lexer/parser to solve some longstanding issues in SC, and obviously don't want to be doing it in the dark. This provides regression tests and related utilities that will allow us to examine the behavior of the lexer/parser/compiler under a wide and thorough (in some cases, complete) set of conditions. It will also be useful for anyone else who wants to make changes to the language in the future. These tests may seem excessive, but actually they are excessive, and that's OK.

Summary of additions

The main additions are (1) LexerParserCompilerTestUtils.sc, which provides the bulk of test code, file formatting, and IO functionality; and (2) TestLPCTestUtils.sc, which provides some basic functionality tests for (1). Whether or not that stays in this repository is up to you, although I like having tests (obviously).

The rest of the files are either test files or validation files. The test files are short bits of code that actually run the tests, and can also be used to generate the validation files. The validation files are the real regression data that future changes will be tested against. The test utilities provide the means to clearly display the list of differences, additions, and subtractions between test and validation outputs.

Test format

Generally, the operation is formalized as a series of interpretations or bytecodeifications of "all possible strings", where each string is made of a sequence, of fixed length, of symbols from a predefined alphabet. For a given all-possible-strings test, an optional prefix and/or suffix may be provided.

For instance, if the alphabet is all lowercase letters [a-z], the prefix "hello_", the suffix "_world", the sequence length 3, then the tests would cover:

hello_aaa_world
hello_aab_world
hello_aac_world
...
hello_mqx_world
...
hello_zzy_world
hello_zzz_world

Generally, two methods are used:

• Compile the string. If that succeeds, interpret it. Record the resulting object using .asString, and if applicable, the resulting object's class.
• Compile the string. If that succeeds, record its FunctionDef's bytecodes.

If any step fails, a unique ID indicating that either a compile-time or run-time error occurred is recorded. In this way, we can compare the output of the parser/lexer/compiler across versions of SC to see what input is legal, what is illegal, and how certain expressions compile (or don't compile).

Summary of tests

• Lexer:
  • strings up to length 3
  • prefix/suffix combos for line and block comments, { }, code bits added before and after
• Parser:
  • strings up to length 8 with an alphabet consisting of parser-important things like "var", "arg", ";"
  • a prefix/suffix combo for { }

See test files for full details, or I could add a .md file to this PR to explain what I've already explained here.

TODO

• Thorough review of .sc files
• Is this the right folder? I don't want to run these tests every time in Travis; they take almost an hour, and rightly so. They should really only apply when deeper layers of sclang have been modified.
• Validate across platforms (unlikely to conflict) (I was so naïve)
• Validate across locales (possible conflicts)
• Add tests targeting specific subroutines of the lexer (accidentals, floats, radix/hex notation) - that's why this is a work-in-progress
• Add tests that target the compiler by repurposing the lexer tests
• Finish documentation via readme
• Add notes elsewhere about the existence of these tests