Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): table data TTL #5234

Merged
merged 119 commits into from
Jan 9, 2025
Merged

feat(core): table data TTL #5234

merged 119 commits into from
Jan 9, 2025

Conversation

bluestreak01
Copy link
Member

@bluestreak01 bluestreak01 commented Dec 9, 2024

New syntax:

CREATE TABLE ... TTL n HOUR(S)/DAY(S)/WEEK(S)/MONTH(S)/YEAR(S)
CREATE TABLE ... TTL nH/D/W/M/Y
ALTER TABLE ... SET TTL ...

Data whose TTL has expired will be deleted, at the granularity of full partitions.

A row's lifetime is measured as max_timestamp - row_timestamp, so it doesn't relate to wall clock time. Whether we do it this way or by wall clock time is something we can discuss further.

TTL is persisted as either the number of hours (for the HOUR, DAY, WEEK units), or the number of months (for the MONTH and YEAR units). A positive value implies hours, negative implies months. Zero value means "unbounded TTL".

Using PGWireInsertSelectBenchmark shows no performance impact for the base case where no partitions need to be dropped. Raw perf data:

1 JDBC inserters, 1 selectors. 10 seconds runtime.

batch size = 1:

vi_ttl, no TTL:
JDBC: 12,483 inserts per second, 14,071 selects per second
ILP: 7,994 inserts per second, 14,691 selects per second

vi_ttl, huge TTL:
JDBC: 12,621 inserts per second, 14,090 selects per second
ILP: 7,779 inserts per second, 15,361 selects per second

master:
JDBC: 13,156 inserts per second, 14,785 selects per second
ILP: 8,326 inserts per second, 15,636 selects per second


batch size = 10:

vi_ttl:
JDBC: 120,721 inserts per second, 16,125 selects per second
ILP: 79,141 inserts per second, 15,248 selects per second

vi_ttl, huge TTL:
JDBC: 119,435 inserts per second, 15,497 selects per second
ILP: 78,679 inserts per second, 15,415 selects per second

master:
JDBC: 109,914 inserts per second, 14,353 selects per second
ILP: 78,835 inserts per second, 15,377 selects per second


batch size = 100:

vi_ttl, no TTL:
JDBC: 542,487 inserts per second, 14,565 selects per second
ILP: 669,285 inserts per second, 15,384 selects per second

vi_ttl, huge TTL:
JDBC: 522,093 inserts per second, 13,816 selects per second
ILP: 659,954 inserts per second, 14,897 selects per second

master:
JDBC: 514,009 inserts per second, 14,604 selects per second
ILP: 660,710 inserts per second, 15,548 selects per second

TODO:

  • add TTL column to tables()
  • ensure TTL works with replication
  • add TTL to WalWriterFuzzTest

FUTURE WORK:

Make writing expired data a no-op. If it's behind TTL, it never gets written.

@bluestreak01 bluestreak01 changed the title feat(core): table data DDL feat(core): table data TTL Dec 9, 2024
@mtopolnik mtopolnik marked this pull request as ready for review December 11, 2024 13:10
@glasstiger
Copy link
Contributor

[PR Coverage check]

😍 pass : 251 / 282 (89.01%)

file detail

path covered line new line coverage
🔵 io/questdb/cairo/wal/seq/MetadataServiceStub.java 0 3 00.00%
🔵 parquet_write/varchar.rs 0 1 00.00%
🔵 io/questdb/std/datetime/microtime/Timestamps.java 14 23 60.87%
🔵 io/questdb/griffin/engine/ops/AlterOperation.java 8 11 72.73%
🔵 io/questdb/cairo/TableWriter.java 46 56 82.14%
🔵 io/questdb/cairo/CairoTable.java 11 12 91.67%
🔵 io/questdb/cairo/PartitionBy.java 28 30 93.33%
🔵 io/questdb/griffin/SqlParser.java 49 51 96.08%
🔵 io/questdb/cairo/MetadataCache.java 2 2 100.00%
🔵 io/questdb/griffin/engine/table/ShowCreateTableRecordCursorFactory.java 23 23 100.00%
🔵 io/questdb/griffin/engine/ops/AlterOperationBuilder.java 6 6 100.00%
🔵 io/questdb/cairo/TableReaderMetadata.java 3 3 100.00%
🔵 io/questdb/cairo/TableUtils.java 1 1 100.00%
🔵 io/questdb/griffin/engine/ops/CreateTableOperationBuilderImpl.java 4 4 100.00%
🔵 io/questdb/cairo/TxReader.java 3 3 100.00%
🔵 io/questdb/griffin/engine/ops/CreateTableOperationImpl.java 4 4 100.00%
🔵 io/questdb/cairo/TableWriterMetadata.java 4 4 100.00%
🔵 io/questdb/cairo/TableStructure.java 1 1 100.00%
🔵 io/questdb/griffin/SqlCompilerImpl.java 17 17 100.00%
🔵 io/questdb/griffin/engine/functions/catalogue/TablesFunctionFactory.java 22 22 100.00%
🔵 io/questdb/cutlass/line/tcp/TableStructureAdapter.java 1 1 100.00%
🔵 io/questdb/griffin/SqlKeywords.java 4 4 100.00%

@bluestreak01 bluestreak01 merged commit 30efb5e into master Jan 9, 2025
34 checks passed
@bluestreak01 bluestreak01 deleted the vi_ttl branch January 9, 2025 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants