The test harness consists of a variety of standard tests to ensure the database-specific interactions within Liquibase work against specific versions and configurations

The test harness logically consists of three parts:

1. Test logic
2. Configuration files containing inputs for the test logic
3. Configuration files containing outputs/expectations for the test logic.

The built-in tests are designed to test an overall functional flow, iterating over all configured connections. For each connection, it will run each applicable input configuration and compare it to the expected output/expectations.

Both the input and output configuration files can be defined in a way that makes them apply to all databases or to specific types and/or specific versions.

The general pattern is that for each directory containing configuration files:

• Files directly in that root apply to all databases. Example: liquibase/harness/change/changelogs
• Files in a subdirectory named for the database type apply to only that type of database. Example: liquibase/harness/change/changelogs/mysql
• Files in a subdirectory with a version apply only to this version of the database. Example: liquibase/harness/change/changelogs/mysql/8

At each level in that hierarchy, new configurations can be added and/or can override configurations from a lower level.

Currently, there are five test types defined in the test harness:

• Base Compatibility test
• Change Object Tests
• Change Data Tests
• Snapshot Command Test
• Diff Command Test

This repository is configured to run against databases supported by Liquibase Core.

Extensions that add support for additional databases and/or define additional functionality can add this framework as a dependency and use the existing tests to:

• More easily verify their new functionality works
• And that it also doesn't break existing logic

The test harness will look for a file called harness-config.yml in the root of your classpath.

That file contains a list of the database connections to test against, as well as an ability to control which subsets of tests to run.

See src/test/resources/harness-config.yml to see what this repository is configured to use.

For more information on using the test harness in your extension, see [README.extensions.md]

This test validates work of basic Liquibase functions.

1. runs Liquibase validate command to ensure the changelog is valid;
2. runs changelog (all supported formats: XML, YAML, JSON, SQL) with basic metadata decorations (labels, contexts, comments) using Liquibase update command;
3. runs Liquibase tag command;
4. runs select query from DATABASECHANGELOG table using jdbc to ensure contexts, labels, comments and tags are present in metadata;
5. runs verification query using jdbc to ensure a test object was actually created or modified during Liquibase update command by comparing it to JSON-formatted expected result set (Note! Result set for your database may differ from existing result set if it is not present in test)
6. runs Liquibase history command;
7. runs Liquibase status command;
8. runs Liquibase rollback command;
9. runs verification query to ensure a test object was actually removed during Liquibase rollback command;

As far as this test validates work of Basic Liquibase functions it is essential to keep its configuration as simple as possible:

1. If you have your database instance up and running you need to just add appropriate configuration details to src/test/resources/harness-config.yml file. Following the example:

   • name: database_name (mandatory)
     version: database_version (optional)
     prefix: local (optional parameter required for CI/CD tests, leave it empty or set local)
     url: db_connection_url (mandatory)
     username: username (optional if your database authentication config doesn't require it)
     password: password (optional if your database authentication config doesn't require it)
     

2. Add driver dependency for you database to POM.xml file

3. To run the test go to you IDE run configurations and add new JUnit configuration. Add liquibase.harness.base.BaseCompatibilityTest as target class and use -DdbName, -DdbVersion to set up appropriate parameters. Or you may just comment out/delete all existing configurations in harness-config.yml file leaving just your configuration and run BaseCompatibilityTest directly from the class file.

In case you want to set up your database instance using docker image then you may use src/test/resources/docker/docker-compose.yml file for configuration.

Checks if your database doesn't "choke" while Liquibase tries to deploy very long queries (inserts and updates with 10k rows).

The test-harness validates most of the Data Definition Language related Change Types as listed on Home Page. The primary focus is on add, create, drop & rename database objects.

The groovy/liquibase/harness/ChangeObjectsTests.groovy test executes changelogs against the database and validates the SQL generated by them as well as whether they make the expected changes.

• The test behavior is as follows:
  • It reads the changesets from the changelogs provided in src/main/resources/liquibase/harness/change/changelogs folders (recursively)
  • Runs Liquibase 'updateSql' command to generate query
  • Compares generated query with expected query (provided in src/main/resources/liquibase/harness/change/expectedSql)
  • If the query generation is correct, the test then runs liquibase update to deploy the changeset to the DB
  • The test takes a snapshot of the database after deployment by running Liquibase 'snapshot' command
  • Actual DB snapshot is compared to expected DB snapshot (provided in src/main/resources/liquibase/harness/change/expectedSnapshot)
  • Finally, deployed changes are then rolled back by either using rollbackToDate (default) or rollback by tag (test-harness-tag). See -DrollbackStrategy option below for more information.

• The tests work with 4 types of input files that are supported by Liquibase itself - xml, yaml, json, sql. Thus files with extensions 'xml', 'sql', 'json', 'yml', 'yaml' are taken into account, but not all formats together in the same run.
• The default format is xml, so by default only changelogs with xml file extension are executed. To change it to another format, like 'sql' for instance, specify -DinputFormat=sql as the command line argument for Maven or as VM option to your JUnit test run config.

1. Go to src/main/resources/liquibase/harness/change/changelogs and add the xml (or other) changeset for the change type you want to test.

• The framework tries to rollback changes after deploying it to DB. If Liquibase knows how to do a rollback for that particular changeset, it will automatically do that. If not, you will need to provide the rollback by yourself. To learn more about rollbacks read Rolling back changesets article.

2. Go to src/main/resources/liquibase/harness/change/expectedSql and add expected query.

• You will need to add this under the database specific folder.
• NOTE: If your changeSet will generate multiple SQL statements, you should add each SQL statement as a separate line. (See renameTable.sql in the postgres folder for an example.)
• If you would like to test another DB type, please add the requisite folder.

3. Go to src/main/resources/liquibase/harness/change/expectedSnapshot and add expected DB Snapshot results.

• To verify the absence of an object in a snapshot (such as with drop* commands) add "_noMatch": true, to that tree level where the missing object should be verified. See dropSequence.json as an example. Additionally the _noMatchField parameter can be used to define the exact property which should be absent or different for that particular database object (for example Column, Table etc.) see createTableWithNumericColumn.json
• You will need to add this under the database specific folder.
• If you would like to test another DB type, please add the requisite folder.

4. Go to your IDE and run the test class ChangeObjectTests.groovy (You can also choose to run BaseTestHarnessSuite, or LiquibaseHarnessSuiteTest -- at present they all work the same).

The primary goal of these tests is to validate change types related to DML (Data Manipulation Language) aspect. Generally it is similar to ChangeObjectTests except it doesn't use Liquibase snapshot to verify data but obtains result set via JDBC.

• src/main/resources/liquibase/harness/data/changelogs - add DML related changelogs here;
• src/main/resources/liquibase/harness/data/checkingSql - add select query which will obtain a result set from required DB object;
• src/main/resources/liquibase/harness/data/expectedResultSet - add JSON formatted expected result set from required DB object; where left part of a JSON node is the name of a change type and right part is JSON Array with a result set;
• src/main/resources/liquibase/harness/data/expectedSql - add query which is expected to be generated by Liquibase;

This test executes the following steps:

• Reads src/test/resources/harness-config.yml and src/main/resources/liquibase/harness/diff/diffDatabases.yml to locate the databases that need to be compared
• Creates a diff based on 2 databases (targetDatabase and referenceDatabase) from diffDatabases.yml
• Generates the changelog based on diff
• Applies the generated changelog to the targetDatabase
• Checks the diff between the target and reference databases again
• If some diffs still exist, then they are matched with the expected diff from liquibase/harness/diff/expectedDiff folder

This test validates work of Liquibase 'snapshot' command by comparing expected and generated snapshots after a DB object was created.

Java 1.8

1. Make sure you have a docker container up and running first
2. Go to src/test/resources/docker and run docker-compose up -d. Wait until the databases start up.
3. Open src/test/groovy/liquibase/harness/LiquibaseHarnessSuiteTest.groovy in your IDE of choice and run it

Build the project first by running mvn clean install -DskipTests

Execute mvn test with the (optional) flags outlined below:

• -DinputFormat=xml or select from the other inputFormats listed in Types of input files
• -DchangeObjects=createTable,dropTable flag allows you to run specific changeObjects rather than all. Use comma separated lists.
• -DchangeData=insert,delete flag that allows to run specific changeData through ChangeDataTests. Use comma separated list
• -DconfigFile=customConfigFile.yml enables to override default config file which is(src/test/resources/harness-config.yml)
• -Dprefix=docker filters database from config file by some common platform identifier. E.g. all AWS based platforms, all Titan managed platforms, all from default docker file.
• -DdbName=mysql overrides the database type. This is only a single value property for now.
• -DdbVersion overrides the database version. Works in conjunction with -DdbName flag.
• -DdbUsername=myUsername overrides the database login username. Providing placeholder username in config.yml file is still required.
• -DdbPassword=myPassword overrides the database login password. Providing placeholder password in config.yml file is still required.
• -DdbUrl=myUrl overrides the database url. Providing placeholder url in config.yml file is still required.
• -DrollbackStrategy overrides the default rollback strategy of rollbackToDate where we create a timestamp in UTC timezone and then try to rollback to that point in time. But this rollback strategy might not work well in some cases like cloud databases for instance -- cloud databases are often in different timezones than the test-harness runners, so the rollback command can be used instead in conjunction with the test-harness-tag tag. To do so, use -DrollbackStrategy=rollbackByTag.
• -Dliquibase-core.version overrides default version of liquibase-core.

To run the test suite itself, you can execute mvn -Dtest=LiquibaseHarnessSuiteTest test

When you are done with test execution, run docker-compose down --volumes to stop the docker containers gracefully and to allow the tests to start from a clean slate on the next run.