Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): table data TTL #5234

Merged
merged 119 commits into from
Jan 9, 2025
Merged
Changes from 1 commit
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
60a4fa3
feat(core): table data DDL
bluestreak01 Dec 9, 2024
ebdb036
Implement enforceTTL()
mtopolnik Dec 9, 2024
6061865
Auto-reorder code
mtopolnik Dec 9, 2024
1345282
Treat TTL = 0 as "no TTL"
mtopolnik Dec 9, 2024
b48b039
Use max timestamp instead of current time
mtopolnik Dec 9, 2024
8598dd0
Remove this. from method calls, add @NotNull
mtopolnik Dec 10, 2024
e8b33f4
Add ttlHours to CairoTable
mtopolnik Dec 10, 2024
53380fc
Rename "ttl" to "ttlHours"
mtopolnik Dec 10, 2024
f41b713
Read ttlHours in TableReaderMetadata.load()
mtopolnik Dec 10, 2024
c038221
Validate TTL in TableUtils.validateMeta()
mtopolnik Dec 10, 2024
73f1183
Fix writing isWalEnabled in TableUtils.writeMetadata()
mtopolnik Dec 10, 2024
6a52b1d
Add TTL <number> to CREATE TABLE syntax
mtopolnik Dec 10, 2024
61d2261
Fix existing test
mtopolnik Dec 10, 2024
65e266c
TtlTest
mtopolnik Dec 10, 2024
21bee4a
Finish renaming "ttl" to "ttlHours"
mtopolnik Dec 10, 2024
cd9cbfe
Small fix to unrelated code
mtopolnik Dec 11, 2024
23e00b4
Auto-cleanup code
mtopolnik Dec 11, 2024
8e9395a
Implement TTL time units
mtopolnik Dec 11, 2024
755d413
More testing
mtopolnik Dec 11, 2024
459d6f2
Merge branch 'master' into vi_ttl
mtopolnik Dec 11, 2024
f40bcac
Fix test assertion
mtopolnik Dec 11, 2024
0ec9be8
Commit to trigger build
mtopolnik Dec 11, 2024
d191093
Delete anemic class
mtopolnik Dec 12, 2024
47efc8b
Auto-improve code
mtopolnik Dec 12, 2024
53825e9
Small error message fix
mtopolnik Dec 12, 2024
a729a88
Break long lines
mtopolnik Dec 12, 2024
a7375cf
Add test for negative TTL value
mtopolnik Dec 12, 2024
dd923b9
Implement ALTER TABLE ... SET TTL ...
mtopolnik Dec 12, 2024
04dc627
Fix test assertion
mtopolnik Dec 12, 2024
67d58f0
Rename method
mtopolnik Dec 12, 2024
d61fd73
Proper implementation of MONTH/YEAR time unit
mtopolnik Dec 13, 2024
8f6562c
Break long lines
mtopolnik Dec 13, 2024
78970d8
Add TODO
mtopolnik Dec 13, 2024
6e56f8a
Improve TTL eviction algorithm
mtopolnik Dec 13, 2024
b1ab8d4
Adapt tests to improved algo
mtopolnik Dec 13, 2024
263b568
Add concise time unit syntax
mtopolnik Dec 16, 2024
393cd47
Complete renaming of ttlHours->ttlHoursOrMonths
mtopolnik Dec 16, 2024
8edac4c
Avoid TimeUnit API
mtopolnik Dec 16, 2024
5ffa1fc
reformat line
mtopolnik Dec 16, 2024
cbd841d
extract tok.length()
mtopolnik Dec 16, 2024
1a18ec4
Add TODO to eagerly enforce TTL upon ALTER TABLE
mtopolnik Dec 16, 2024
61184ef
Merge branch 'master' into vi_ttl
mtopolnik Dec 16, 2024
cd0a258
Improve error reporting
mtopolnik Dec 17, 2024
de810d3
Fix param name
mtopolnik Dec 17, 2024
2f931fa
Fix possible long->int conversion overflow
mtopolnik Dec 17, 2024
d68ad1c
Validate that the table is partitioned
mtopolnik Dec 17, 2024
90f5914
Add test for invalid ALTER TABLE SET TTL syntax
mtopolnik Dec 17, 2024
80075b5
Rename ttlHours -> ttlHoursOrMonths
mtopolnik Dec 17, 2024
02f8b07
Add TTL to tables() function
mtopolnik Dec 17, 2024
73ffbe9
Auto-imports
mtopolnik Dec 18, 2024
d3df5c4
Update test to new number of cols from tables()
mtopolnik Dec 18, 2024
d9384f3
Update more tests
mtopolnik Dec 18, 2024
95cef21
Align parameter order in setFuzzProbabilities
mtopolnik Dec 18, 2024
fb4de71
Align parameter order
mtopolnik Dec 18, 2024
4931c12
Add setTtlProb to setFuzzProbabilities()
mtopolnik Dec 18, 2024
65aecd4
Clean up param order in generateSet()
mtopolnik Dec 18, 2024
3b1335d
Add log statement when evicting a partition
mtopolnik Dec 19, 2024
4c175da
Fix TTL enforcement algorithm
mtopolnik Dec 20, 2024
562da96
Provisional code for TTL Enforcer Job
mtopolnik Dec 20, 2024
d6996a1
Add test with many partitions
mtopolnik Dec 20, 2024
be4704d
ALTER TABLE SET TTL takes effect immediately
mtopolnik Dec 20, 2024
5dcb341
Merge branch 'master' into vi_ttl
mtopolnik Dec 20, 2024
ff67973
Add random insertion test
mtopolnik Dec 20, 2024
0d1d20c
Merge branch 'master' into vi_ttl
mtopolnik Dec 23, 2024
4765136
Simplify benchmark code
mtopolnik Dec 23, 2024
23c25c7
Merge fallout
mtopolnik Dec 23, 2024
8fd64d6
Undo breaking a line
mtopolnik Dec 23, 2024
9e19b82
Add some Javadoc
mtopolnik Dec 23, 2024
6bab262
Remove unused TtlEnforcerJob
mtopolnik Dec 23, 2024
8fb7916
Undo changes to fuzz test code
mtopolnik Dec 23, 2024
759437e
Remove stray method call
mtopolnik Dec 23, 2024
1868c9a
Reduce scope of variable
mtopolnik Dec 23, 2024
115ab6f
Fix metadata field write order
mtopolnik Dec 23, 2024
7421e01
Fix tests
mtopolnik Dec 27, 2024
90f78aa
Fix test
mtopolnik Dec 27, 2024
8c6151c
Merge branch 'master' into vi_ttl
mtopolnik Dec 27, 2024
1e23a18
Cargo lint
mtopolnik Dec 27, 2024
cabb350
cargo fmt
mtopolnik Dec 27, 2024
084b98f
Adjust constant name
mtopolnik Dec 29, 2024
4442211
Clear TTL in CreateTableOperationBuilder.clear()
mtopolnik Dec 29, 2024
652477a
Simplify partition eviction code
mtopolnik Dec 29, 2024
b638bf2
Reduce test runtime
mtopolnik Dec 29, 2024
cb4e4e3
Auto-arrange code
mtopolnik Jan 3, 2025
60caca0
Add TTL to SHOW CREATE TABLE
mtopolnik Jan 3, 2025
729e73c
Merge branch 'master' into vi_ttl
mtopolnik Jan 3, 2025
22810d0
Return correct TTL unit from tables()
mtopolnik Jan 7, 2025
8d8e87d
Merge branch 'master' into vi_ttl
mtopolnik Jan 7, 2025
c574c5f
Remove TODO which is a WON'T DO
mtopolnik Jan 7, 2025
8b0d601
Remove TODO which is a WON'T DO
mtopolnik Jan 7, 2025
850298a
Remove unused method in test class TableModel
mtopolnik Jan 7, 2025
2fe5682
Remove unused constant
mtopolnik Jan 7, 2025
1dfe896
clean up
bluestreak01 Jan 7, 2025
3f87cff
Merge branch 'vi_ttl' of https://github.com/questdb/questdb into vi_ttl
bluestreak01 Jan 7, 2025
be5e0c3
merge fallout
bluestreak01 Jan 7, 2025
a88ad90
Eliminate creating substrings
mtopolnik Jan 8, 2025
ac6cfe9
Parameterize TtlTest with WAL
mtopolnik Jan 8, 2025
8759a14
Merge branch 'master' into vi_ttl
mtopolnik Jan 8, 2025
2027241
Add bock ttl() method in test class TableModel
mtopolnik Jan 8, 2025
4824363
Add test that calls enforceTtl() on uncommitted writer
mtopolnik Jan 8, 2025
faba01a
Add test for CREATE TABLE LIKE tango
mtopolnik Jan 8, 2025
6011d44
Align fuzz probabilities order
mtopolnik Jan 8, 2025
b8a1970
Put equalTsRowsProb in correct place
mtopolnik Jan 8, 2025
cca9581
Add setTtlProb to AbstractFuzzTest, FuzzRunner
mtopolnik Jan 8, 2025
ef1d667
Align param order in FuzzTransactionGenerator
mtopolnik Jan 8, 2025
603fa5e
Add setTtlProb to FuzzTransactionGenerator
mtopolnik Jan 8, 2025
966c729
Activate TTL in fuzz test
mtopolnik Jan 8, 2025
f0b87f9
Merge remote-tracking branch 'origin/master' into vi_ttl
bluestreak01 Jan 8, 2025
053c7ac
fix bug in fuzz test runner (test bug)
bluestreak01 Jan 8, 2025
fd9a6ba
Auto-reorder code
mtopolnik Jan 9, 2025
907a04e
Add table name to log statement
mtopolnik Jan 9, 2025
a3e7ab1
Fix bug: drop higher parts of evicted split partition
mtopolnik Jan 9, 2025
5ac65f2
Randomize iteration where TTL is set
mtopolnik Jan 9, 2025
365bde9
Merge branch 'master' into vi_ttl
mtopolnik Jan 9, 2025
cd263ce
Randomize TTL value
mtopolnik Jan 9, 2025
5f728b8
Activate TTL on all WalWriter fuzz tests
mtopolnik Jan 9, 2025
1c326b8
Merge branch 'master' into vi_ttl
bluestreak01 Jan 9, 2025
a1b6393
Code style
mtopolnik Jan 9, 2025
080165a
Validate TTL not finer than partition size
mtopolnik Jan 9, 2025
afef858
more aggressive TTL fuzzing
bluestreak01 Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix bug: drop higher parts of evicted split partition
  • Loading branch information
mtopolnik committed Jan 9, 2025
commit a3e7ab10a34eea0fab546209ca79975d67fbb8c6
14 changes: 13 additions & 1 deletion core/src/main/java/io/questdb/cairo/TableWriter.java
Original file line number Diff line number Diff line change
Expand Up @@ -1927,9 +1927,20 @@ public void enforceTtl() {
return;
}
long maxTimestamp = getMaxTimestamp();
long evictedPartitionTimestamp = -1;
do {
long partitionTimestamp = getPartitionTimestamp(0);
assert partitionTimestamp == floorFn.floor(partitionTimestamp) : "Partition 0 timestamp weirdness";
long floorTimestamp = floorFn.floor(partitionTimestamp);
if (evictedPartitionTimestamp != -1 && floorTimestamp == evictedPartitionTimestamp) {
assert partitionTimestamp != floorTimestamp : "Expected a higher part of a split partition";
dropPartitionByExactTimestamp(partitionTimestamp);
continue;
}
assert partitionTimestamp == floorTimestamp :
String.format("partitionTimestamp %s != floor(partitionTimestamp) %s, evictedPartitionTimestamp %s",
Timestamps.toString(partitionTimestamp),
Timestamps.toString(floorTimestamp),
Timestamps.toString(evictedPartitionTimestamp));
long partitionCeiling = ceilFn.ceil(partitionTimestamp);
// TTL < 0 means it's in months
boolean shouldEvict = ttl > 0
Expand All @@ -1941,6 +1952,7 @@ public void enforceTtl() {
.$(" partitionTs=").microTime(partitionTimestamp)
.$();
dropPartitionByExactTimestamp(partitionTimestamp);
evictedPartitionTimestamp = partitionTimestamp;
} else {
// Partitions are sorted by timestamp, no need to check the rest
break;
Expand Down