Note: This package is a work-in-progress.

This package aims to provide an official and comprehensive test suite that checks that the behaviour of Plutus conforms with the specified behaviour. We run all the tests in our CI to ensure continuous conformance. Users are encouraged to contribute test cases to this collection of tests and utilize the suite (e.g., running the tests on alternative implementations of Plutus Core).

This suite tests the latest version of Plutus. Testing of older versions may be added in the future.

The tests currently cover or will cover the Haskell and Agda implementation of:

• Untyped Plutus Core (UPLC) evaluation
• Typechecking for Typed Plutus Core (TPLC), including checking of alpha equivalence. (tplc-typecheck-test)
• TPLC evaluation
• Erasure of TPLC to UPLC
• Coverage test
• CPU/memory costing of scripts

The tests mostly take the form of golden tests of fairly simple UPLC programs. The input files for the tests are organised in a tree of directories under the test-cases directory. If a directory in this tree contains one or more subdirectories then any other files in the directory are ignored and the subdirectories are recursively searched for test cases. If a directory <name> has no subdirectories then it is expected to contain a file called <name>.uplc and no other files with the .uplc extension. The file <name>.uplc should contain textual source code for a UPLC program, and the directory should also contain a file called <name>.uplc.expected containing the expected output of the program and a file called <name>.uplc.budget.expectedcontaining the expected CPU and memory budgets. Any other files (for example README files) in the directory are ignored. See the addInteger-01 for an example of the expected format. To avoid difficulties with case-insensitive filesystems no two subdirectories of a test directory should have names which differ only by case (eg True and true).

To update or add test outputs, use the accept test option of the tests. E.g., to have the test results overwrite the .expected files in the Haskell implementation test suite (haskell-conformance) , run:

cabal test haskell-conformance --test-options=--accept

The library provides functions that users can import and run conformance tests with their own implementation. At the moment the tests can only be run against another Haskell implementation. More support will be added later. Of course, one can wrap an arbitrary executable in Haskell. See an explanation here and the related documentation.

The UPLC evaluation tests ensure conformance of evaluation of untyped plutus core programs.

Currently we have tested the conformance of UPLC evaluation against our Haskell and Agda implementations of the CEK machine. Note that we are not testing conformance of a reducer. We are testing an evaluator. One noticeable difference between a reducer and an evaluator is that a reducer reduces an ill-defined term to an error term while an evaluator has a special error state. See section 6.1 of our specification for more details.

The expected output may contain:

The input files are expected to have the concrete syntax. The expected output will show "parse error" when the parser fails to parse the input file.

If evaluation fails with an error, the expected output will show "evaluation failure".

This means the input file successfully evaluates to the output program as per the specification. The evaluated program is represented in the concrete syntax.

In the library we provide a function named runUplcEvalTests with the following signature:

import UntypedPlutusCore.Core.Type qualified as UPLC

type UplcProg = UPLC.Program Name DefaultUni DefaultFun ()

type UplcEvaluatorFun res = UplcProg -> Maybe res

data UplcEvaluator =
  UplcEvaluatorWithoutCosting (UplcEvaluatorFun UplcProg)
  | UplcEvaluatorWithCosting (CostModelParams -> UplcEvaluatorFun (UplcProg, ExBudget))

runUplcEvalTests :: UplcEvaluator -> (FilePath -> Bool) -> (FilePath -> Bool) -> IO ()

Users can call this function with their own UplcEvaluatorFun, which should evaluate a UPLC program and return a Maybe UplcProg, or a Maybe (UplcProg, ExBudget) if the budget tests are to be performed as well. Given a UPLC program, the runner should return the evaluated program. In case of evaluation failure, the runner should return Nothing. The two arguments of type FilePath -> Bool allow selected evaluation and budget tests (the ones for which the function returns True) to be ignored if desired.

We welcome contributions and comments to this repository. Feel free to open an issue.

If we add the tests you share, we will acknowledge your contribution and post a link back to your repository.

We are grateful to these external partners for their contributions:

• Runtime Verification Inc. (runtimeverification/plutus-core-semantics)
• Hachi Security (HachiSecurity/plc-llvm/tree/main/compiler/test-data/untyped)