Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] #10868

hubert-da · 2021-09-13T16:28:51Z

CHANGELOG_BEGIN
CHANGELOG_END

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

nicu-da

LGTM! Really nice work!
Just 2 small questions

nicu-da · 2021-09-13T16:44:17Z

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala

+        if (pendingCommands.contains(TrackedCommandKey(submissionId, commandId))) {
          // TODO return an error identical to the server side duplicate command error once that's defined.
          throw new IllegalStateException(
-            s"A command with id $commandId is already being tracked. CommandIds submitted to the CommandTracker must be unique."
+            s"A command $commandId from a submission $submissionId is already being tracked. CommandIds submitted to the CommandTracker must be unique."
          ) with NoStackTrace


Do you think we have any gaps in testing though for this? Wondering how come conformance tests were green considering this code. Shouldn't they throw an error for 2 submissions with the same command id currently?

Also that TODO, should we throw an ApiException for duplicate command from here?

Thanks for the feedback!

Do you think we have any gaps in testing though for this? Wondering how come conformance tests were green considering this code. Shouldn't they throw an error for 2 submissions with the same command id currently?

Not yet sure, I was going to think about this tomorrow, as I'm also not sure about some of these changes at all. Partially because I don't know Akka Streams Graph API 😄 I submitted it mostly as an FYI so far.

Also that TODO, should we throw an ApiException for duplicate command from here?

Was also thinking about this. Will keep you posted.

Sounds good, let me know if you need any help with the Graph API, spent a lot of time writing akka graphs.

Also that TODO, should we throw an ApiException for duplicate command from here?

Not sure what to do, as this is a Unit method. Maybe let's keep it as is for now and I can do something about this in a separate PR.

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala

andreaslochbihler-da

Thanks for pulling this together so quickly. I think we should add the following changes:

The TrackingKey should be (submissionId, changeId) rather than (submissionId, commandId) so that submissions from different submitters or different applications are isolated against each other.
The CommandService should set the submission ID to a UUID if the application hasn't set a submission ID. I haven't seen such a change here and I haven't checked whether the is already happening elsewhere, but I'd be positively surprised if this is already implemented.

AFAICS there are not test cases for the new functionality. Can you make sure we've covered the following cases:

Several parallel submissions of the same change ID with different submission IDs
Defaulting of submission IDs in the CommandService (also for parallel submissions) if the application has omitted them.
Parallel submissions from different applications / submitters (actAs) for the same command ID.

hubert-da · 2021-09-14T09:54:58Z

The TrackingKey should be (submissionId, changeId) rather than (submissionId, commandId) so that submissions from different submitters or different applications are isolated against each other.

As discussed in a different comment, the rest of the changeId is covered by the Tracking.Key which is a different thing. It makes the Trackers for different applications (and parties) isolated. Then, the TrackedCommandKey makes submissions distinguishable within a Tracker.

The CommandService should set the submission ID to a UUID if the application hasn't set a submission ID. I haven't seen such a change here and I haven't checked whether the is already happening elsewhere, but I'd be positively surprised if this is already implemented.

As also discussed in another comment, it's implemented but doesn't have proper test coverage. So, moving smoothly to...

AFAICS there are not test cases for the new functionality.

The test coverage is something that I have to revisit. Thanks!

…ng `submissionId` [KVL-1104] CHANGELOG_BEGIN CHANGELOG_END

CHANGELOG_BEGIN CHANGELOG_END

hubert-da · 2021-09-14T17:59:17Z

@andreaslochbihler-da , in regard to the test cases:

Several parallel submissions of the same change ID with different submission IDs

Different change IDs are covered by TrackerMapSpec. Different/same submission IDs are now covered by CommandTrackerFlowTest and are going to be covered by the conformance test: #10869

Defaulting of submission IDs in the CommandService (also for parallel submissions) if the application has omitted them.

This one is covered by most of the conformance tests and SubmitRequestValidatorTest as with this change: #10882, the validation requires a submission ID.

Parallel submissions from different applications / submitters (actAs) for the same command ID.

This one is covered on the level of TrackerMap and tested in TrackerMapSpec.

Anyway, the test coverage is probably not perfect but if we want to release it tomorrow, this is what we have. I think at least a minimal set of test cases already exists.

CHANGELOG_BEGIN CHANGELOG_END

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala

nicu-da

LGMT!
Nice work, only have a few things that i pointed out.

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala

hubert-da · 2021-09-14T23:24:35Z

Trigger-related tests are failing and I don't know how to fix them yet, for some reason sometimes there are no pendingCommands.

andreaslochbihler-da · 2021-09-15T07:06:24Z

Different/same submission IDs are now covered by CommandTrackerFlowTest and are going to be covered by the conformance test: #10869

AFAICT #10869 tests only the CommandSubmissionService. But it doesn't test that the command service doesn't get confused about the incoming command completions that differ only by their submission ID. It would also be great to mix parallel submissions to the CommandService and the CommandSubmissionService so that the CommandService is not aware of concurrent submissions with the same change ID.

Anyway, the test coverage is probably not perfect but if we want to release it tomorrow, this is what we have. I think at least a minimal set of test cases already exists.

I agree. We should somewhere keep a list of the tests that we want to add so that we don't forget. Is there already a ticket for that?

CHANGELOG_BEGIN CHANGELOG_END

hubert-da · 2021-09-15T08:55:04Z

We should somewhere keep a list of the tests that we want to add so that we don't forget. Is there already a ticket for that?

I'll create a story to improve the test coverage here with a list of suggested test cases.

CHANGELOG_BEGIN CHANGELOG_END

fabiotudone-da

Minor clarity improvement suggestions, still OK to go IMO.

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala

...st/suite/scala/com/digitalasset/ledger/client/services/commands/CommandTrackerFlowTest.scala

hubert-da · 2021-09-15T10:16:01Z

Thanks @fabiotudone-da for the comments, will address in a separate PR.

CHANGELOG_BEGIN CHANGELOG_END

@akrmn

Manual release process. @akrmn is in charge of this release. Commit log: ``` b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) b4328b3 ledger-api-test-tool - Add conformance test for parallel command deduplication using CommandSubmissionService [KVL-1099] (#10869) 0c32e3b Fix Parallel Indexer initialization issue [DPP-542] (#10889) b3e4975 Chore slow migration error removal (#10886) e4cce53 Create a new grpc exception for each duplicate result [KVL-1099] (#10887) a939594 Sandbox on H2 - performance improvements for the append-only schema [DPP-600] (#10888) 9a1a101 Increase timeout for heavy tests in ParticipantPruningIT (#10894) 9093c6c Improve wording for the active contracts service description (#10880) c12f546 Document #10780 (#10781) 5814f6a update NOTICES file (#10893) 38227a8 [Ledger API error codes] ErrorCode enrichments [DPP-591] (#10874) e7c443a enable json index for all fields that are queried with JSON_EXISTS (#10658) 6c1c02a document complete authorized auth0 setup (#10881) e4230dc Do not drop generated `submissionId`s in `GrpcCommandService` [KVL-1104] (#10882) b4750a4 trigger reach auth on internal network (#10844) 7908083 add Auth0 support to create-daml-app (#10673) b86490c Add @adriaanm-da to the release rotation (#10872) 9e918c3 Update trigger-service docs to use --dar option in the corresponding example (#10877) 49a9556 [docs] Fix minor typo in doc of exerciseByKey in TS. (#10863) f7c07ea interfaces: scala protobuf encoder (#10878) be4e064 Ledger API Test Tool: support `--additional` tests [KVL-1100] (#10829) 97e14de [Ledger API error codes] ErrorCode interfaces and generator [DPP-591] (#10836) 6dcdaa4 [DPP-589] Add CLI flag to select minimum enabled TLS version (#10854) 1fc58d9 Navigator customviews highlight and choices button, apply custom theme on the login screen (#10859) 6faddc9 Update Daml Documentation to reflect command deduplication related changes [KVL-1094] (#10852) 7c29eee Cleanup normalize from svalue (#10873) 053f22a Convert SValue to Value, and normalize, in a single code pass. (#10828) 37a1cb2 compatibility-tests - Exclude CommandDeduplicationIT from running for existing 1.17 snapshots (#10866) dfae9f6 Command deduplication - better support for different deduplication modes in conformance tests [KVL-1099] (#10864) 6f151e2 save kibana exports (#10861) 99f0362 [JSON-API] drop package token doc changes (#10865) b50bb8e Populate `definite_answer` in `ApiException` [KVL-1004] (#10832) a471225 LF: Add missing collision check for type synonyms (#10841) 1e1c452 LF: drop ad-hoc FrontStack builders (#10839) 8f5b4fa interfaces: protobuf encoder haskell side (#10850) 63f6678 ParticipantPruningIT divulgence test fixes to avoid flakiness on canton (#10860) 8a9d19a Command deduplication - KV conformance test for usage of max deduplication duration [KVL-1098] (#10846) 24fff88 LF: Simplify TransactionBuilder (#10753) 9a4c9df Implement LF desugaring of interface definitions (#10834) 2aaf601 interfaces: protobuf decoder haskell side (#10849) 6dc769b interfaces: lf typechecker implementation (#10843) d9178d2 Clarify version usage in test tool exclusion docs (#10858) c113954 Clarify docs for test tool exclusions (#10855) 8c9edd8 es cluster tweaks (#10853) 842c5b1 Drop early access notice from profiler docs (#10856) 7c47aca Improvements to wording in ledger-api protobuf docs (#10851) cff0358 ledger-api: Remove unimplemented fields [KVL-1094] (#10822) dcec6ea kvutils: Populate `definite_answer` in rejections [KVL-1004] (#10801) 1c4f173 Command deduplication - kvutils - Always use max deduplication duration as deduplication period [KVL-1098] (#10824) 567fe43 tweak trigger-service docs (#10845) fb5ab5d setvar doesn't like new lines in assignment, refactor (#10842) 7225c04 [docs] Replace AdoptOpenJDK suggestion by Adoptium (#10837) 6a9c8a6 release 1.17.0-snapshot.20210910.7786.0.976ca400 (#10838) 6ed2124 LF: clean up useless version tests. (#10833) 85f6f36 Modify the name of the secrets-url CLI flag to tls-secrets-url [DPP-604] (#10840) d809fd9 [JSON-API] surrogate template id cache (#10806) ``` Changelog: ``` - [Sandbox] - Added a CLI parameter for configuring the number of connections in the database connection pool used for serving ledger API requests [Docs] Improved description of the purpose and usage of the active contracts service [Docs/JSON API] documented 256B limitation of Oracle query store - The Trigger Service can now accept separate `--auth-internal` and `--auth-external` CLI arguments, where `--auth-internal` is the address used by the Trigger Service to reach the Auth Middleware directly, and `--auth-external` is the address the Trigger Service uses in generated URLs sent back to the client. The `--auth` option remains and keeps working as before, setting both internal and external addresses to the same given value. - The `create-daml-app` template now includes support for a third authentication scheme (in addition to the existing "dev mode" and Daml Hub support): Auth0. Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server. - [Navigator] The currently selected custom view is now highlighted on the sidebar kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests. Modify the name of the secrets-url CLI flag to tls-secrets-url. ``` changelog_begin changelog_end

…ng `submissionId` [KVL-1104] (#10868) CHANGELOG_BEGIN CHANGELOG_END

* Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) CHANGELOG_BEGIN CHANGELOG_END * Throw an exception if the submission id is empty in `CommandTracker` CHANGELOG_BEGIN CHANGELOG_END * Generate a submission ID in the `CommandClient` if it's empty CHANGELOG_BEGIN CHANGELOG_END

hubert-da · 2021-09-20T12:14:00Z

AFAICT #10869 tests only the CommandSubmissionService. But it doesn't test that the command service doesn't get confused about the incoming command completions that differ only by their submission ID. It would also be great to mix parallel submissions to the CommandService and the CommandSubmissionService so that the CommandService is not aware of concurrent submissions with the same change ID.

@andreaslochbihler-da, AppendOnlyCommandDeduplicationParallelIT now supports both CommandSubmissionService and CommandService. I also added a unit test for command completions that differ only by their submission ID here: #10939. I think it now makes sense to just add a Ledger API test that mixes both services, so I created a story for that KVL-1106.

hubert-da · 2021-09-20T14:49:51Z

@fabiotudone-da, your comments are addressed by #10868, which will be ready to review soon, so I can finally leave this PR.

@garyverhaegen-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @garyverhaegen-da is in charge of this release. Commit log: ``` 1b473c2 Switch trigger service time provider default to wall-clock time (#10967) 4acf34b Add conversions from/to interfaces in Daml parser (#10954) d50df11 interface: Add to/from interface in scala ast (#10960) 9f88e09 Drop jdk8 override (#10963) e0587bc Add a bit of detail to RegisteredTemplates documentation, motivated by: (#10962) cebe6ed Refactor safetyStep in the simplifier. (#10948) ac192fc interfaces: Add to/from interface in Haskell typechecker (#10951) 4b4d7a3 Remove update normalizer which was too aggresive (#10925) 66e1098 Add tests for the `CompletionFromTransaction` converter [KVL-1104] (#10885) 724e50d interfaces: Add to/from interface in Haskell AST (#10945) 9a8d55a Change slack link to discuss wiki. Fixes #10946 (#10947) 8b3b033 LF: Test preprocessor resuming (#10936) 2edfc06 ifaces: name collision, typecheck fetch/exercise (#10896) 5dc15c6 LF: rename language Interface to PackageInterface (#10938) 054c6ab Upgrade canton to a more recent version (#10944) b8533d5 [JSON-API] Production/HA documentation. (#10903) a331762 Clarify `CommandTracker` [KVL-1104] (#10943) 5244643 Changes to increase timeout and remove assertions around mock CommandSubmissionServiceImpl (#10942) 6cc42ee rotate release duty after 1.17.0-snapshot.20210915.7832.0.38227a8e (#10892) 88ef05e sandbox-classic: Only allow `--max-parallel-submissions` here. (#10941) ac02dbd LF: Exhaustive test for valueTranslator. (#10927) 409c0b4 interfaces: Add to/from_interface in proto (#10937) 02c8a9d Split CommonStorageBackend (#10871) dc71a6a update NOTICES file (#10932) 0ba54a4 Add a missing test case to the `CommandTrackerFlowTest` (#10939) 855ecdf [DPP-572] Add ledger API test case for verifying conformance to `--min-tls-version` flag. (#10898) 3e13e3d Switch to stable urls in scoop python manifest (#10933) fc2c87d ledger-api-client: Generate a submission id if it's empty in the `CommandClient` [KVL-1104] (#10926) 04d8f75 Clean up sandbox-on-x conformance tests. (#10766) b4541b5 Logging delay of submitted commands (#10912) b8e21d8 Fix takeFilter for the test StreamConsumer (#10918) 906368d LF: exhaustive test for CommandPreprocessor (#10914) 61d214e Add fetch, exercise implementations for interfaces in speedy. (#10911) d01f8e1 Fix flaky ApiConfigManagementServiceSpec test (#10922) 50291ed interfaces: scala typechecker implementation (#10867) cac8391 Bump ghc-lib to include daml interfaces parser (#10747) b6a6bf7 [Ledger API error codes] Extracted common errors and groups [DPP-607] (#10890) 308f938 Dpp 494 unit testing ha coordinator (#10862) 30f74ad Mark Extractor's VeryLargeArchiveSpec test as flaky (#10916) 9582e01 LF: Refactor PreprocessorSpec test (#10909) 9b0fa29 Separate exercise & fetch for interfaces from templates (#10908) f4adee9 Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883) 8a39118 Rename Completion.deduplication_time to deduplication_duration [KVL-1057] (#10900) 8e22bb6 Drop ContractId typeparameter from Value (#10827) 4543705 Drop cocreature as a codeowner from runtime things (#10906) e6e8147 Release 1.17.0 RC from second to last commit (#10904) f08ac5f Desugar interface implements declarations (#10895) b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) ``` Changelog: ``` [Triggers Service] The service now starts by default using wall-clock time instead of static time. If you want to run using static time, you need to do so explicitly using the new '-s' or '--static-time' CLI option. If you were already using '-w' or '--wall-clock-time' the flag has no effect. It's anyway safe to leave it there. Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server. ``` CHANGELOG_BEGIN CHANGELOG_END

@garyverhaegen-da

* release 1.17.0-snapshot.20210921.7889.0.1b473c2b This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @garyverhaegen-da is in charge of this release. Commit log: ``` 1b473c2 Switch trigger service time provider default to wall-clock time (#10967) 4acf34b Add conversions from/to interfaces in Daml parser (#10954) d50df11 interface: Add to/from interface in scala ast (#10960) 9f88e09 Drop jdk8 override (#10963) e0587bc Add a bit of detail to RegisteredTemplates documentation, motivated by: (#10962) cebe6ed Refactor safetyStep in the simplifier. (#10948) ac192fc interfaces: Add to/from interface in Haskell typechecker (#10951) 4b4d7a3 Remove update normalizer which was too aggresive (#10925) 66e1098 Add tests for the `CompletionFromTransaction` converter [KVL-1104] (#10885) 724e50d interfaces: Add to/from interface in Haskell AST (#10945) 9a8d55a Change slack link to discuss wiki. Fixes #10946 (#10947) 8b3b033 LF: Test preprocessor resuming (#10936) 2edfc06 ifaces: name collision, typecheck fetch/exercise (#10896) 5dc15c6 LF: rename language Interface to PackageInterface (#10938) 054c6ab Upgrade canton to a more recent version (#10944) b8533d5 [JSON-API] Production/HA documentation. (#10903) a331762 Clarify `CommandTracker` [KVL-1104] (#10943) 5244643 Changes to increase timeout and remove assertions around mock CommandSubmissionServiceImpl (#10942) 6cc42ee rotate release duty after 1.17.0-snapshot.20210915.7832.0.38227a8e (#10892) 88ef05e sandbox-classic: Only allow `--max-parallel-submissions` here. (#10941) ac02dbd LF: Exhaustive test for valueTranslator. (#10927) 409c0b4 interfaces: Add to/from_interface in proto (#10937) 02c8a9d Split CommonStorageBackend (#10871) dc71a6a update NOTICES file (#10932) 0ba54a4 Add a missing test case to the `CommandTrackerFlowTest` (#10939) 855ecdf [DPP-572] Add ledger API test case for verifying conformance to `--min-tls-version` flag. (#10898) 3e13e3d Switch to stable urls in scoop python manifest (#10933) fc2c87d ledger-api-client: Generate a submission id if it's empty in the `CommandClient` [KVL-1104] (#10926) 04d8f75 Clean up sandbox-on-x conformance tests. (#10766) b4541b5 Logging delay of submitted commands (#10912) b8e21d8 Fix takeFilter for the test StreamConsumer (#10918) 906368d LF: exhaustive test for CommandPreprocessor (#10914) 61d214e Add fetch, exercise implementations for interfaces in speedy. (#10911) d01f8e1 Fix flaky ApiConfigManagementServiceSpec test (#10922) 50291ed interfaces: scala typechecker implementation (#10867) cac8391 Bump ghc-lib to include daml interfaces parser (#10747) b6a6bf7 [Ledger API error codes] Extracted common errors and groups [DPP-607] (#10890) 308f938 Dpp 494 unit testing ha coordinator (#10862) 30f74ad Mark Extractor's VeryLargeArchiveSpec test as flaky (#10916) 9582e01 LF: Refactor PreprocessorSpec test (#10909) 9b0fa29 Separate exercise & fetch for interfaces from templates (#10908) f4adee9 Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883) 8a39118 Rename Completion.deduplication_time to deduplication_duration [KVL-1057] (#10900) 8e22bb6 Drop ContractId typeparameter from Value (#10827) 4543705 Drop cocreature as a codeowner from runtime things (#10906) e6e8147 Release 1.17.0 RC from second to last commit (#10904) f08ac5f Desugar interface implements declarations (#10895) b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) ``` Changelog: ``` [Triggers Service] The service now starts by default using wall-clock time instead of static time. If you want to run using static time, you need to do so explicitly using the new '-s' or '--static-time' CLI option. If you were already using '-w' or '--wall-clock-time' the flag has no effect. It's anyway safe to leave it there. Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server. ``` CHANGELOG_BEGIN CHANGELOG_END * 1.18 not 1.17 changelog_begin changelog_end Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com> Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

@garyverhaegen-da

@garyverhaegen-da is in charge of this release. This supersedes the failed #10971 and #10993. Commit log: ``` ced4a27 Include concurrency info in output (#10970) a0b6800 Limit maximum number of concurrent tests (#10969) a0fe0f6 release 1.18.0-snapshot.20210921.7889.1.1b473c2b (#10993) 58c82b6 LF: reduce usage of NodeCreate.coinst (#10988) f56ce2a kvutils: Add structured error data to the error metadata [KVL-1032] (#10964) 3f4dbf6 interfaces: Syntax highlighting for interfaces (#10987) d52a285 interface: Add test for conversion functions (#10991) 28b8d9a bump dotnet (#10979) fe10ffb Use ValueEnricher in ScenarioRunner. (#10897) bdad7d5 Upgrade Java on Windows (#10965) 34390f7 interfaces: Implemented conversions, added test. (#10982) 7740b60 participant-integration-api: Improve the CommandConfiguration docs. (#10978) 04f322e Bump resources for daml-ledger tests (#10984) 6d9922e Retry upsert of command deduplication on oracle and h2 [DPP-609] (#10976) eb4f1b2 LF: Move lookup tests out of EngineTest (#10973) 99836d2 Handle fetchByKey callback correctly in scenario runner (#10980) 23e6a2d Improve feedback from failing match in AuthMatchers (#10981) e299103 [Short] Small test tool's CLI parser refactor (#10923) fc153a9 release 1.18.0-snapshot.20210921.7889.0.1b473c2b (#10971) 1b473c2 Switch trigger service time provider default to wall-clock time (#10967) 4acf34b Add conversions from/to interfaces in Daml parser (#10954) d50df11 interface: Add to/from interface in scala ast (#10960) 9f88e09 Drop jdk8 override (#10963) e0587bc Add a bit of detail to RegisteredTemplates documentation, motivated by: (#10962) cebe6ed Refactor safetyStep in the simplifier. (#10948) ac192fc interfaces: Add to/from interface in Haskell typechecker (#10951) 4b4d7a3 Remove update normalizer which was too aggresive (#10925) 66e1098 Add tests for the `CompletionFromTransaction` converter [KVL-1104] (#10885) 724e50d interfaces: Add to/from interface in Haskell AST (#10945) 9a8d55a Change slack link to discuss wiki. Fixes #10946 (#10947) 8b3b033 LF: Test preprocessor resuming (#10936) 2edfc06 ifaces: name collision, typecheck fetch/exercise (#10896) 5dc15c6 LF: rename language Interface to PackageInterface (#10938) 054c6ab Upgrade canton to a more recent version (#10944) b8533d5 [JSON-API] Production/HA documentation. (#10903) a331762 Clarify `CommandTracker` [KVL-1104] (#10943) 5244643 Changes to increase timeout and remove assertions around mock CommandSubmissionServiceImpl (#10942) 6cc42ee rotate release duty after 1.17.0-snapshot.20210915.7832.0.38227a8e (#10892) 88ef05e sandbox-classic: Only allow `--max-parallel-submissions` here. (#10941) ac02dbd LF: Exhaustive test for valueTranslator. (#10927) 409c0b4 interfaces: Add to/from_interface in proto (#10937) 02c8a9d Split CommonStorageBackend (#10871) dc71a6a update NOTICES file (#10932) 0ba54a4 Add a missing test case to the `CommandTrackerFlowTest` (#10939) 855ecdf [DPP-572] Add ledger API test case for verifying conformance to `--min-tls-version` flag. (#10898) 3e13e3d Switch to stable urls in scoop python manifest (#10933) fc2c87d ledger-api-client: Generate a submission id if it's empty in the `CommandClient` [KVL-1104] (#10926) 04d8f75 Clean up sandbox-on-x conformance tests. (#10766) b4541b5 Logging delay of submitted commands (#10912) b8e21d8 Fix takeFilter for the test StreamConsumer (#10918) 906368d LF: exhaustive test for CommandPreprocessor (#10914) 61d214e Add fetch, exercise implementations for interfaces in speedy. (#10911) d01f8e1 Fix flaky ApiConfigManagementServiceSpec test (#10922) 50291ed interfaces: scala typechecker implementation (#10867) cac8391 Bump ghc-lib to include daml interfaces parser (#10747) b6a6bf7 [Ledger API error codes] Extracted common errors and groups [DPP-607] (#10890) 308f938 Dpp 494 unit testing ha coordinator (#10862) 30f74ad Mark Extractor's VeryLargeArchiveSpec test as flaky (#10916) 9582e01 LF: Refactor PreprocessorSpec test (#10909) 9b0fa29 Separate exercise & fetch for interfaces from templates (#10908) f4adee9 Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883) 8a39118 Rename Completion.deduplication_time to deduplication_duration [KVL-1057] (#10900) 8e22bb6 Drop ContractId typeparameter from Value (#10827) 4543705 Drop cocreature as a codeowner from runtime things (#10906) e6e8147 Release 1.17.0 RC from second to last commit (#10904) f08ac5f Desugar interface implements declarations (#10895) b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) ``` Changelog: ``` [Triggers Service] The service now starts by default using wall-clock time instead of static time. If you want to run using static time, you need to do so explicitly using the new '-s' or '--static-time' CLI option. If you were already using '-w' or '--wall-clock-time' the flag has no effect. It's anyway safe to leave it there. Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server. ``` CHANGELOG_BEGIN CHANGELOG_END

@garyverhaegen-da

@garyverhaegen-da is in charge of this release. This supersedes the failed #10971 and #10993. Commit log: ``` ced4a27 Include concurrency info in output (#10970) a0b6800 Limit maximum number of concurrent tests (#10969) a0fe0f6 release 1.18.0-snapshot.20210921.7889.1.1b473c2b (#10993) 58c82b6 LF: reduce usage of NodeCreate.coinst (#10988) f56ce2a kvutils: Add structured error data to the error metadata [KVL-1032] (#10964) 3f4dbf6 interfaces: Syntax highlighting for interfaces (#10987) d52a285 interface: Add test for conversion functions (#10991) 28b8d9a bump dotnet (#10979) fe10ffb Use ValueEnricher in ScenarioRunner. (#10897) bdad7d5 Upgrade Java on Windows (#10965) 34390f7 interfaces: Implemented conversions, added test. (#10982) 7740b60 participant-integration-api: Improve the CommandConfiguration docs. (#10978) 04f322e Bump resources for daml-ledger tests (#10984) 6d9922e Retry upsert of command deduplication on oracle and h2 [DPP-609] (#10976) eb4f1b2 LF: Move lookup tests out of EngineTest (#10973) 99836d2 Handle fetchByKey callback correctly in scenario runner (#10980) 23e6a2d Improve feedback from failing match in AuthMatchers (#10981) e299103 [Short] Small test tool's CLI parser refactor (#10923) fc153a9 release 1.18.0-snapshot.20210921.7889.0.1b473c2b (#10971) 1b473c2 Switch trigger service time provider default to wall-clock time (#10967) 4acf34b Add conversions from/to interfaces in Daml parser (#10954) d50df11 interface: Add to/from interface in scala ast (#10960) 9f88e09 Drop jdk8 override (#10963) e0587bc Add a bit of detail to RegisteredTemplates documentation, motivated by: (#10962) cebe6ed Refactor safetyStep in the simplifier. (#10948) ac192fc interfaces: Add to/from interface in Haskell typechecker (#10951) 4b4d7a3 Remove update normalizer which was too aggresive (#10925) 66e1098 Add tests for the `CompletionFromTransaction` converter [KVL-1104] (#10885) 724e50d interfaces: Add to/from interface in Haskell AST (#10945) 9a8d55a Change slack link to discuss wiki. Fixes #10946 (#10947) 8b3b033 LF: Test preprocessor resuming (#10936) 2edfc06 ifaces: name collision, typecheck fetch/exercise (#10896) 5dc15c6 LF: rename language Interface to PackageInterface (#10938) 054c6ab Upgrade canton to a more recent version (#10944) b8533d5 [JSON-API] Production/HA documentation. (#10903) a331762 Clarify `CommandTracker` [KVL-1104] (#10943) 5244643 Changes to increase timeout and remove assertions around mock CommandSubmissionServiceImpl (#10942) 6cc42ee rotate release duty after 1.17.0-snapshot.20210915.7832.0.38227a8e (#10892) 88ef05e sandbox-classic: Only allow `--max-parallel-submissions` here. (#10941) ac02dbd LF: Exhaustive test for valueTranslator. (#10927) 409c0b4 interfaces: Add to/from_interface in proto (#10937) 02c8a9d Split CommonStorageBackend (#10871) dc71a6a update NOTICES file (#10932) 0ba54a4 Add a missing test case to the `CommandTrackerFlowTest` (#10939) 855ecdf [DPP-572] Add ledger API test case for verifying conformance to `--min-tls-version` flag. (#10898) 3e13e3d Switch to stable urls in scoop python manifest (#10933) fc2c87d ledger-api-client: Generate a submission id if it's empty in the `CommandClient` [KVL-1104] (#10926) 04d8f75 Clean up sandbox-on-x conformance tests. (#10766) b4541b5 Logging delay of submitted commands (#10912) b8e21d8 Fix takeFilter for the test StreamConsumer (#10918) 906368d LF: exhaustive test for CommandPreprocessor (#10914) 61d214e Add fetch, exercise implementations for interfaces in speedy. (#10911) d01f8e1 Fix flaky ApiConfigManagementServiceSpec test (#10922) 50291ed interfaces: scala typechecker implementation (#10867) cac8391 Bump ghc-lib to include daml interfaces parser (#10747) b6a6bf7 [Ledger API error codes] Extracted common errors and groups [DPP-607] (#10890) 308f938 Dpp 494 unit testing ha coordinator (#10862) 30f74ad Mark Extractor's VeryLargeArchiveSpec test as flaky (#10916) 9582e01 LF: Refactor PreprocessorSpec test (#10909) 9b0fa29 Separate exercise & fetch for interfaces from templates (#10908) f4adee9 Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883) 8a39118 Rename Completion.deduplication_time to deduplication_duration [KVL-1057] (#10900) 8e22bb6 Drop ContractId typeparameter from Value (#10827) 4543705 Drop cocreature as a codeowner from runtime things (#10906) e6e8147 Release 1.17.0 RC from second to last commit (#10904) f08ac5f Desugar interface implements declarations (#10895) b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) ``` Changelog: ``` [Triggers Service] The service now starts by default using wall-clock time instead of static time. If you want to run using static time, you need to do so explicitly using the new '-s' or '--static-time' CLI option. If you were already using '-w' or '--wall-clock-time' the flag has no effect. It's anyway safe to leave it there. Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server. ``` CHANGELOG_BEGIN CHANGELOG_END

@garyverhaegen-da

@garyverhaegen-da is in charge of this release. This supersedes the failed #10971 and #10993. Commit log: ``` ced4a27 Include concurrency info in output (#10970) a0b6800 Limit maximum number of concurrent tests (#10969) a0fe0f6 release 1.18.0-snapshot.20210921.7889.1.1b473c2b (#10993) 58c82b6 LF: reduce usage of NodeCreate.coinst (#10988) f56ce2a kvutils: Add structured error data to the error metadata [KVL-1032] (#10964) 3f4dbf6 interfaces: Syntax highlighting for interfaces (#10987) d52a285 interface: Add test for conversion functions (#10991) 28b8d9a bump dotnet (#10979) fe10ffb Use ValueEnricher in ScenarioRunner. (#10897) bdad7d5 Upgrade Java on Windows (#10965) 34390f7 interfaces: Implemented conversions, added test. (#10982) 7740b60 participant-integration-api: Improve the CommandConfiguration docs. (#10978) 04f322e Bump resources for daml-ledger tests (#10984) 6d9922e Retry upsert of command deduplication on oracle and h2 [DPP-609] (#10976) eb4f1b2 LF: Move lookup tests out of EngineTest (#10973) 99836d2 Handle fetchByKey callback correctly in scenario runner (#10980) 23e6a2d Improve feedback from failing match in AuthMatchers (#10981) e299103 [Short] Small test tool's CLI parser refactor (#10923) fc153a9 release 1.18.0-snapshot.20210921.7889.0.1b473c2b (#10971) 1b473c2 Switch trigger service time provider default to wall-clock time (#10967) 4acf34b Add conversions from/to interfaces in Daml parser (#10954) d50df11 interface: Add to/from interface in scala ast (#10960) 9f88e09 Drop jdk8 override (#10963) e0587bc Add a bit of detail to RegisteredTemplates documentation, motivated by: (#10962) cebe6ed Refactor safetyStep in the simplifier. (#10948) ac192fc interfaces: Add to/from interface in Haskell typechecker (#10951) 4b4d7a3 Remove update normalizer which was too aggresive (#10925) 66e1098 Add tests for the `CompletionFromTransaction` converter [KVL-1104] (#10885) 724e50d interfaces: Add to/from interface in Haskell AST (#10945) 9a8d55a Change slack link to discuss wiki. Fixes #10946 (#10947) 8b3b033 LF: Test preprocessor resuming (#10936) 2edfc06 ifaces: name collision, typecheck fetch/exercise (#10896) 5dc15c6 LF: rename language Interface to PackageInterface (#10938) 054c6ab Upgrade canton to a more recent version (#10944) b8533d5 [JSON-API] Production/HA documentation. (#10903) a331762 Clarify `CommandTracker` [KVL-1104] (#10943) 5244643 Changes to increase timeout and remove assertions around mock CommandSubmissionServiceImpl (#10942) 6cc42ee rotate release duty after 1.17.0-snapshot.20210915.7832.0.38227a8e (#10892) 88ef05e sandbox-classic: Only allow `--max-parallel-submissions` here. (#10941) ac02dbd LF: Exhaustive test for valueTranslator. (#10927) 409c0b4 interfaces: Add to/from_interface in proto (#10937) 02c8a9d Split CommonStorageBackend (#10871) dc71a6a update NOTICES file (#10932) 0ba54a4 Add a missing test case to the `CommandTrackerFlowTest` (#10939) 855ecdf [DPP-572] Add ledger API test case for verifying conformance to `--min-tls-version` flag. (#10898) 3e13e3d Switch to stable urls in scoop python manifest (#10933) fc2c87d ledger-api-client: Generate a submission id if it's empty in the `CommandClient` [KVL-1104] (#10926) 04d8f75 Clean up sandbox-on-x conformance tests. (#10766) b4541b5 Logging delay of submitted commands (#10912) b8e21d8 Fix takeFilter for the test StreamConsumer (#10918) 906368d LF: exhaustive test for CommandPreprocessor (#10914) 61d214e Add fetch, exercise implementations for interfaces in speedy. (#10911) d01f8e1 Fix flaky ApiConfigManagementServiceSpec test (#10922) 50291ed interfaces: scala typechecker implementation (#10867) cac8391 Bump ghc-lib to include daml interfaces parser (#10747) b6a6bf7 [Ledger API error codes] Extracted common errors and groups [DPP-607] (#10890) 308f938 Dpp 494 unit testing ha coordinator (#10862) 30f74ad Mark Extractor's VeryLargeArchiveSpec test as flaky (#10916) 9582e01 LF: Refactor PreprocessorSpec test (#10909) 9b0fa29 Separate exercise & fetch for interfaces from templates (#10908) f4adee9 Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883) 8a39118 Rename Completion.deduplication_time to deduplication_duration [KVL-1057] (#10900) 8e22bb6 Drop ContractId typeparameter from Value (#10827) 4543705 Drop cocreature as a codeowner from runtime things (#10906) e6e8147 Release 1.17.0 RC from second to last commit (#10904) f08ac5f Desugar interface implements declarations (#10895) b5648c0 Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] (#10868) ``` Changelog: ``` [Triggers Service] The service now starts by default using wall-clock time instead of static time. If you want to run using static time, you need to do so explicitly using the new '-s' or '--static-time' CLI option. If you were already using '-w' or '--wall-clock-time' the flag has no effect. It's anyway safe to leave it there. Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server. ``` CHANGELOG_BEGIN CHANGELOG_END

hubert-da requested review from andreaslochbihler-da and nicu-da September 13, 2021 16:28

nicu-da reviewed Sep 13, 2021

View reviewed changes

hubert-da force-pushed the hubert-da/api-command-service-track-by-submission-id branch from 6c61df8 to 832a1e8 Compare September 13, 2021 16:49

andreaslochbihler-da reviewed Sep 14, 2021

View reviewed changes

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Show resolved Hide resolved

andreaslochbihler-da reviewed Sep 14, 2021

View reviewed changes

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Show resolved Hide resolved

andreaslochbihler-da reviewed Sep 14, 2021

View reviewed changes

hubert-da mentioned this pull request Sep 14, 2021

Add tests for the CompletionFromTransaction converter [KVL-1104] #10885

Merged

6 tasks

hubert-da added 2 commits September 14, 2021 19:20

Make CommandTracker distinguish submissions of the same command usi…

158b0b7

…ng `submissionId` [KVL-1104] CHANGELOG_BEGIN CHANGELOG_END

Fix the Completions handling for the mutating schema

a8c575e

CHANGELOG_BEGIN CHANGELOG_END

hubert-da force-pushed the hubert-da/api-command-service-track-by-submission-id branch from 832a1e8 to a8c575e Compare September 14, 2021 17:51

Fix logging

4f94f9b

CHANGELOG_BEGIN CHANGELOG_END

hubert-da marked this pull request as ready for review September 14, 2021 18:12

hubert-da requested review from meiersi-da and a team as code owners September 14, 2021 18:12

hubert-da requested review from a team and fabiotudone-da September 14, 2021 18:13

fabiotudone-da approved these changes Sep 14, 2021

View reviewed changes

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Outdated Show resolved Hide resolved

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Outdated Show resolved Hide resolved

nicu-da approved these changes Sep 14, 2021

View reviewed changes

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Outdated Show resolved Hide resolved

...src/main/scala/com/digitalasset/ledger/client/services/commands/tracker/CommandTracker.scala Outdated Show resolved Hide resolved

Address review comments

33ed17e

CHANGELOG_BEGIN CHANGELOG_END

Fix compilation error

c6ca50a

CHANGELOG_BEGIN CHANGELOG_END

hubert-da enabled auto-merge (squash) September 15, 2021 08:58

Change everything to a Seq and fix a test

5597322

CHANGELOG_BEGIN CHANGELOG_END

fabiotudone-da approved these changes Sep 15, 2021

View reviewed changes

Add explicit conversion for Scala 2.12

e6b71d9

CHANGELOG_BEGIN CHANGELOG_END

hubert-da merged commit b5648c0 into main Sep 15, 2021

hubert-da deleted the hubert-da/api-command-service-track-by-submission-id branch September 15, 2021 14:15

hubert-da added a commit that referenced this pull request Sep 17, 2021

Make CommandTracker distinguish submissions of the same command usi…

1b25e48

…ng `submissionId` [KVL-1104] (#10868) CHANGELOG_BEGIN CHANGELOG_END

hubert-da mentioned this pull request Sep 17, 2021

Backport of #10868 and #10926 #10928

Merged

6 tasks

hubert-da added the needs-backport Candidate fix for backporting to the latest release branch label Sep 20, 2021

hubert-da mentioned this pull request Sep 20, 2021

ledger-api-client: Clarify CommandTracker [KVL-1104] #10943

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] #10868

Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] #10868

hubert-da commented Sep 13, 2021 •

edited

Loading

nicu-da left a comment

nicu-da Sep 13, 2021

hubert-da Sep 13, 2021

nicu-da Sep 13, 2021

hubert-da Sep 14, 2021

andreaslochbihler-da left a comment

hubert-da commented Sep 14, 2021 •

edited

Loading

hubert-da commented Sep 14, 2021

nicu-da left a comment

hubert-da commented Sep 14, 2021 •

edited

Loading

andreaslochbihler-da commented Sep 15, 2021

hubert-da commented Sep 15, 2021

fabiotudone-da left a comment

hubert-da commented Sep 15, 2021

hubert-da commented Sep 20, 2021 •

edited

Loading

hubert-da commented Sep 20, 2021

Make CommandTracker distinguish submissions of the same command using submissionId [KVL-1104] #10868

Make CommandTracker distinguish submissions of the same command using submissionId [KVL-1104] #10868

Conversation

hubert-da commented Sep 13, 2021 • edited Loading

Pull Request Checklist

nicu-da left a comment

Choose a reason for hiding this comment

nicu-da Sep 13, 2021

Choose a reason for hiding this comment

hubert-da Sep 13, 2021

Choose a reason for hiding this comment

nicu-da Sep 13, 2021

Choose a reason for hiding this comment

hubert-da Sep 14, 2021

Choose a reason for hiding this comment

andreaslochbihler-da left a comment

Choose a reason for hiding this comment

hubert-da commented Sep 14, 2021 • edited Loading

hubert-da commented Sep 14, 2021

nicu-da left a comment

Choose a reason for hiding this comment

hubert-da commented Sep 14, 2021 • edited Loading

andreaslochbihler-da commented Sep 15, 2021

hubert-da commented Sep 15, 2021

fabiotudone-da left a comment

Choose a reason for hiding this comment

hubert-da commented Sep 15, 2021

hubert-da commented Sep 20, 2021 • edited Loading

hubert-da commented Sep 20, 2021

Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] #10868

Make `CommandTracker` distinguish submissions of the same command using `submissionId` [KVL-1104] #10868

hubert-da commented Sep 13, 2021 •

edited

Loading

hubert-da commented Sep 14, 2021 •

edited

Loading

hubert-da commented Sep 14, 2021 •

edited

Loading

hubert-da commented Sep 20, 2021 •

edited

Loading