This directory contains scripts for CI testing.

• regtest.sh -- This is a Bash-based script to test any target. To run, cd to grammar-vs/, then type bash regtest.sh <target> where <target> is CSharp, Java, Cpp, Dart, Go, or one of the other targets. To test the target, you will need the NET SDK installed, access to the internet, and to toolchain for the target you want to test. For CSharp, it will download the Antlr4 tool and runtime. For the other targets, you will need to download the Antlr4 tool antlr-4.9.3-complete.jar and place it in /tmp.

• test.ps1 -- this is a Powershell script for testing, similar to regtest.sh.

Both scripts use trgen to generate a program from the pom.xml file in the grammar source. The tool generates a parser program for one of the target langauges, currently CSharp, Java, Cpp, Dart, Go, and Python. The generated code contains makefiles and Powershell files to present an idealized interface to build and run the parse with the examples, so you do not need to memorize how to build and run the parser for the given target.

To build and run the parser using the makefiles, you must be in an environment that provides make and Bash. Ubuntu provides that out of the box. For Windows, you must use MinGW64, then "pacman install make" and provide a symbolic linked file for "make" once installed. I don't know why, but the package doesn't define just a plain old "make.exe".

cd to a grammar directory, e.g., `cd abnf`.
trgen -t CSharp --template-sources-directory _scripts/templates/ --antlr-tool-path /tmp/antlr-4.9.3-complete.jar
make
make test

Please note the parameters to trgen: "--template-sources-directory" is used to say where to find the templates used in this repo for the driver programs. By default, trgen has a collection of generic templates, and while they are generic, you might want to test the grammars in this repo with the templates used in CI builds.

For Windows or Ubuntu, you may want to provide a "~/.trgen.rc" file to override defaults for that program. For example,

{
    "antlr_tool_path" : "c:/users/kenne/downloads/antlr-4.9.3-complete.jar"
}

For the trwdog tool, you may want to override the default 5 minute timeout for parse tests to complete, e.g., if you are on a slow machine. To do that, create a "~/.trwdog.rc" file.

{
    "Timeout" : 2
}

The home directory is computed via the C# runtime: Environment.GetFolderPath(Environment.SpecialFolder.UserProfile).

These are the templates for testing the grammars, which are used by the above scripts and trgen.

If you add or delete any files from the templates/ directory you should regenerate the file templates/files using cd templates/; find . -type f > files. Otherwise, the trgen tool may not add the new template files into your generated code.

You should take care to specify a version when installing the trgen tool. I change the tool often, and there may be regressions. Currently version 0.13.2 works well.

All the templates are in the "per-target directory" standard:

• A template directory contains files that are instantiated in the generated output directory.
• Contained in the template directory are sub-directories named after the target, e.g., CSharp, Java, JavaScript, etc. Those files are copied to the generated output directory, but only for the specified target for trgen.

Here is the structure of the default templates:

templates/
    Antlr4cs/
        Program.cs
        Test.csproj
        makefile
    CSharp/
        ErrorListener.cs
        Program.cs
        Test.csproj
        makefile
    Cpp/
    Dart/
        analysis_options.yaml
        cli.dart
        makefile
        pubspec.yaml
    Go/
        Program.go
        makefile
    Java/
        ErrorListener.java
        Program.java
        makefile
    JavaScript/
        index.js
        makefile
        package.json
    Python3/
        Program.py
        makefile
    files
    Arithmetic.g4

Driver code is generated by merging the code containing the grammar and sources in grammars-v4 with the template source code. For example, for trgen -t CSharp in grammars-v4/abb/, the tool will extract from the pom.xml file details about the grammar, copy the source files from grammars-v4/abb/ to Generated/, then copy the template files from templates/CSharp to Generated/. The end result is:

Generated/
    abbParser.g4       (from the source abb/abbParser.g4)
    abbLexer.g4       (from the source abb/abbLexer.g4)
    ErrorListener.cs  (from the source templates/CSharp/ErrorListener.cs)
    makefile   (from the source templates/CSharp/makefile)
    Program.cs   (from the source templates/CSharp/Program.cs)
    Test.csproj   (from the source templates/CSharp/Test.csproj)

All files in templates/ are StringTemplate files. To see how it is used, I suggest looking at the default set of templates for the tool here. Look closely for template attribute references, which begin with '<' and end with '>'. (To avoid template attribute replacement, escape '<' with a backslash, e.g., '<'. You only need to do the '<', not the closing '>' symbol.)

Trgen will construct and pass to the template evaluator the following attributes:

To generate the driver code from templates for a grammar, you will need to have the NET SDK installed. Afterwards, install trgen:

dotnet tool install -g trgen --version 0.5.3

To create a driver program:

1. Change directory to grammar directory, e.g., cd abb.
2. trgen -t Java --template-sources-directory ../_scripts/templates/

You will need to pass to trgen --template-sources-directory ../_scripts/templates/ to indicate the path to the templates. If you do not pass the --template-sources-directory option, the tool will use the default templates contained with the tool. You can generate a different target using -t. Acceptable targets are Antlr4cs, Cpp, CSharp, Dart, Go, Java, JavaScript, Python3. The "files" directory in the templates directory indicates what files are included in generation, and obtained via "find . -type f > files".