Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command dedup: add columns to completions in append-only schemas [KVL-1057] #10652

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
8dfedb4
Augment completion.proto with deduplication-related info
fabiotudone-da Aug 19, 2021
9768373
Merge branch 'main' into fabiotudone-da/command-dedup/ledger-api-comp…
fabiotudone-da Aug 23, 2021
e43f2fc
Explicitly specify fields not yet filled in when building Completion
fabiotudone-da Aug 23, 2021
d566b97
Time-based deduplication periods are measured in record time of compl…
fabiotudone-da Aug 23, 2021
89336ab
Add deduplication_offset as a deduplication_period option
fabiotudone-da Aug 23, 2021
eb14dd9
Don't skip proto field numbers
fabiotudone-da Aug 23, 2021
1ba2045
CompletionFromTransaction: use default Completion constructor
fabiotudone-da Aug 23, 2021
efa01ff
submission_rank: reserve proto field for future use
fabiotudone-da Aug 23, 2021
73fd7a2
Add comment about reserved proto field
fabiotudone-da Aug 23, 2021
8e54bb5
Add command deduplication columns to completions in append-only schemas
fabiotudone-da Aug 24, 2021
35c6ded
Merge branch 'main' into fabiotudone-da/command-dedup/index-db-add-in…
fabiotudone-da Aug 24, 2021
8079673
Remove duplicated `application_id` field from Postgres and H2 schemas
fabiotudone-da Aug 24, 2021
3e5fa10
Make `application_id` and `submission_id` nullable
fabiotudone-da Aug 24, 2021
f68a130
Re-generate migrations' sha256
fabiotudone-da Aug 24, 2021
abf65e3
Merge branch 'main' into fabiotudone-da/command-dedup/index-db-add-in…
fabiotudone-da Aug 24, 2021
0cb77e4
Tidy-up
fabiotudone-da Aug 25, 2021
1cab32b
Add RW logic for new command dedup columns in completions
fabiotudone-da Aug 25, 2021
156dae8
Merge branch 'main' into fabiotudone-da/command-dedup/index-db-add-in…
fabiotudone-da Aug 25, 2021
6db6a86
Declare `deduplication_start` as a nullable column
fabiotudone-da Aug 25, 2021
484db4a
Fix UpdateToDbDtoSpec
fabiotudone-da Aug 25, 2021
8dbe592
Fix merge from main
fabiotudone-da Aug 25, 2021
19856a2
Replace Either3 with an ADT in StorageBackendTestValues
fabiotudone-da Aug 25, 2021
2ba045d
Test with non-zero nanos
fabiotudone-da Aug 25, 2021
477e219
Fix UpdateToDbDtoSpec
fabiotudone-da Aug 25, 2021
1432308
Also change Schema.scala
fabiotudone-da Aug 26, 2021
0e27dbe
Simplify DbDto.CommandCompletion construction
fabiotudone-da Aug 26, 2021
b7c86d7
Merge branch 'main' into fabiotudone-da/command-dedup/index-db-add-in…
fabiotudone-da Aug 26, 2021
39ded6d
Fix merge from `main` (migration SHAs and filename)
fabiotudone-da Aug 26, 2021
15223de
Fix UpdateToDbDto: restore transactionId
fabiotudone-da Aug 30, 2021
24d1d6e
Fix completions' application_id: it must be non-nullable in all schem…
fabiotudone-da Aug 30, 2021
eb44d3c
Actually use newly read columns
fabiotudone-da Aug 30, 2021
5643b95
Add DB round-trip tests for new fields in StorageBackendTestValues
fabiotudone-da Aug 30, 2021
51ab254
Format
fabiotudone-da Aug 30, 2021
87da1dd
StorageBackendTestValues: property type someSubmissionId
fabiotudone-da Aug 30, 2021
482891e
UpdateToDbDto: remove duplicated logic
fabiotudone-da Aug 30, 2021
804c99e
Simplify UpdateToDbDto.commandCompletion
fabiotudone-da Aug 30, 2021
f32346e
Fix comment in CompletionFromTransaction.toApiDeduplicationPeriod
fabiotudone-da Aug 30, 2021
4ebd1e0
Fix CompletionFromTransaction.toApiCompletion
fabiotudone-da Aug 30, 2021
b1bc3d4
Fix deduplication_offset in Schema
fabiotudone-da Aug 30, 2021
85ed113
Make parameters explicit in CompletionFromTransaction and CompletionS…
fabiotudone-da Aug 30, 2021
8da5af0
Shorten error message in CompletionFromTransaction
fabiotudone-da Aug 30, 2021
166cd83
Fix StorageBackendTestsCompletions
fabiotudone-da Aug 30, 2021
3b7a9e5
Fix CompletionFromTransaction
fabiotudone-da Aug 30, 2021
bde3d1e
Separate command dedup-related test cases in UpdateToDbDtoSpec
fabiotudone-da Aug 31, 2021
4a94562
Simplify further UpdateToDbDto.commandCompletion
fabiotudone-da Aug 31, 2021
2749c75
Simplify CompletionFromTransaction.toApiCompletion
fabiotudone-da Aug 31, 2021
db74dd0
Format
fabiotudone-da Aug 31, 2021
9c305fc
Merge branch 'main' into fabiotudone-da/command-dedup/index-db-add-in…
fabiotudone-da Aug 31, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add command deduplication columns to completions in append-only schemas
CHANGELOG_BEGIN
CHANGELOG_END
  • Loading branch information
fabiotudone-da committed Aug 24, 2021
commit 8e54bb52b8f7f1f97b38799a45293fdea1d30306
Original file line number Diff line number Diff line change
@@ -1 +1 @@
aae0b43e4735b3ffbdb782a61bad75f741c58a282327f3d3e18b0e41da5c69f6
c9d6d032d1371e29bb84bf9e6dff92cb0dc162594c89c73a72786d2c41ec15cc
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,19 @@ CREATE TABLE participant_command_completions (
command_id VARCHAR NOT NULL,
-- The transaction ID is `NULL` for rejected transactions.
transaction_id VARCHAR,
-- The application ID has to be provided by the application.
application_id VARCHAR NOT NULL,
fabiotudone-da marked this conversation as resolved.
Show resolved Hide resolved
-- The submission ID will be provided by the participant or driver if the application didn't provide one.
submission_id VARCHAR NOT NULL,
-- The three alternatives below are mutually exclusive, i.e. the deduplication
-- interval could have specified by the application as one of:
-- 1. an initial offset
-- 2. an initial timestamp
-- 3. a duration (split into two columns, seconds and nanos, mapping protobuf's 1:1)
deduplication_offset VARCHAR,
deduplication_start TIMESTAMP,
deduplication_time_seconds BIGINT,
fabiotudone-da marked this conversation as resolved.
Show resolved Hide resolved
deduplication_time_nanos INT,
-- The three columns below are `NULL` if the completion is for an accepted transaction.
-- The `rejection_status_details` column contains a Protocol-Buffers-serialized message of type
-- `daml.platform.index.StatusDetails`, containing the code, message, and further details
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
c9148396eec01471c1135ff384d0b83442442ada1d6ca12d731f8e84b6f4869f
882dbf2fb1f48e5d240d21a5ba6bab7d60d6cc9306cbc0466ee9db4e0cf72458
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,19 @@ CREATE TABLE participant_command_completions
record_time TIMESTAMP NOT NULL,

application_id NVARCHAR2(1000) NOT NULL,
-- The submission ID will be provided by the participant or driver if the application didn't provide one.
submission_id NVARCHAR2(1000) NOT NULL,

-- The three alternatives below are mutually exclusive, i.e. the deduplication
-- interval could have specified by the application as one of:
-- 1. an initial offset
-- 2. an initial timestamp
-- 3. a duration (split into two columns, seconds and nanos, mapping protobuf's 1:1)
deduplication_offset VARCHAR2(4000),
deduplication_start TIMESTAMP,
deduplication_time_seconds NUMBER,
deduplication_time_nanos NUMBER,

submitters CLOB NOT NULL CONSTRAINT ensure_json_submitters CHECK (submitters IS JSON),
command_id NVARCHAR2(1000) NOT NULL,

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
6701a42bcb0229b873317f62e424297116bb79fcfbb3cb465076b81108dc6aac
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- Copyright (c) 2021 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
-- SPDX-License-Identifier: Apache-2.0

-- Add columns to store command deduplication information in completions.

ALTER TABLE participant_command_completions
-- The application ID has to be provided by the application.
ADD COLUMN application_id text NOT NULL,
fabiotudone-da marked this conversation as resolved.
Show resolved Hide resolved
-- The submission ID will be provided by the participant or driver if the application didn't provide one.
ADD COLUMN submission_id text NOT NULL,
fabiotudone-da marked this conversation as resolved.
Show resolved Hide resolved
-- The three alternatives below are mutually exclusive, i.e. the deduplication
-- interval could have specified by the application as one of:
-- 1. an initial offset
-- 2. an initial timestamp
-- 3. a duration (split into two columns, seconds and nanos, mapping protobuf's 1:1)
ADD COLUMN deduplication_offset text,
ADD COLUMN deduplication_start timestamp,
ADD COLUMN deduplication_time_seconds bigint,
ADD COLUMN deduplication_time_nanos integer;