-
Notifications
You must be signed in to change notification settings - Fork 333
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: refactor PgStore
#5309
Conversation
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis pull request introduces a series of modifications across multiple files in the GrepTime project, focusing on enhancing PostgreSQL store configuration, error handling, and test infrastructure. The changes primarily involve updating method signatures, improving SQL query handling, and refactoring key-value backend interactions. The modifications aim to provide more flexible and robust database interactions, with a particular emphasis on PostgreSQL-related functionality. Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
…atable (#4) * fix: election use bytea as well * fix: use Serializable to avoid read unrepeatable
a88bda6
to
14c8a60
Compare
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
🔭 Outside diff range comments (1)
src/meta-srv/src/election/postgres.rs (1)
Line range hint
347-358
: Improve Error Handling When Retrieving Database ValuesIn the
get_value_with_lease
method, usingtry_get().unwrap_or_default()
may mask underlying errors by defaulting to an empty string if the retrieval fails:let current_time_str = res[0].try_get(1).unwrap_or_default();Consider explicitly handling potential errors returned by
try_get()
to ensure that unexpected issues are surfaced and handled appropriately, rather than silently defaulting and potentially leading to incorrect behavior.
🧹 Nitpick comments (1)
src/common/meta/src/kv_backend/postgres.rs (1)
173-202
: Consider PostgreSQL Parameter Limit in Batch OperationsIn the methods
generate_batch_upsert_query
,generate_batch_delete_query
, andgenerate_batch_get_query
, SQL queries are generated with a number of parameters proportional to the batch size. PostgreSQL has a limit on the number of parameters in a prepared statement (typically around 32767). For very large batches, this limit might be exceeded, causing runtime errors.Please ensure that batch sizes are appropriately limited to avoid exceeding PostgreSQL's parameter limit. If necessary, implement logic to split large batches into smaller chunks that stay within the parameter limit.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
src/cli/src/bench.rs
(1 hunks)src/common/meta/src/error.rs
(3 hunks)src/common/meta/src/kv_backend/etcd.rs
(3 hunks)src/common/meta/src/kv_backend/memory.rs
(2 hunks)src/common/meta/src/kv_backend/postgres.rs
(14 hunks)src/common/meta/src/kv_backend/test.rs
(10 hunks)src/common/meta/src/rpc/store.rs
(4 hunks)src/log-store/src/raft_engine/backend.rs
(3 hunks)src/meta-srv/src/bootstrap.rs
(1 hunks)src/meta-srv/src/election/postgres.rs
(9 hunks)
🧰 Additional context used
🪛 GitHub Check: Check (ubuntu-20.04)
src/common/meta/src/error.rs
[failure] 817-817:
failed to resolve: use of undeclared crate or module tokio_postgres
[failure] 815-815:
no variant named PostgresTransaction
found for enum error::Error
⏰ Context from checks skipped due to timeout of 90000ms (5)
- GitHub Check: Toml Check
- GitHub Check: Check License Header
- GitHub Check: Clippy
- GitHub Check: Build GreptimeDB binaries (ubuntu-20.04)
- GitHub Check: Build GreptimeDB binary (profile-CI) (ubuntu-20.04)
🔇 Additional comments (14)
src/common/meta/src/kv_backend/postgres.rs (1)
88-133
: Enhancement: Introduction ofSqlTemplateFactory
Improves ModularityThe addition of
SqlTemplateFactory
andSqlTemplateSet
encapsulates SQL template generation, enhancing code modularity and readability. This refactoring simplifies the creation and management of SQL statements and reduces redundancy.src/meta-srv/src/election/postgres.rs (3)
333-337
: Improved Function Signatures by Using&str
Instead of&String
Changing the parameter types from
&String
to&str
in methods likeget_value_with_lease
simplifies the code and aligns with Rust best practices by reducing unnecessary indirection.
Line range hint
65-70
: Ensure Safe SQL Concatenation inCAS_WITH_EXPIRE_TIME
In the
CAS_WITH_EXPIRE_TIME
SQL statement, concatenation is performed using parameters:v=convert_to($3 || $4 || TO_CHAR(CURRENT_TIMESTAMP + INTERVAL '1 second' * $5, 'YYYY-MM-DD HH24:MI:SS.MS'), 'UTF8')Please ensure that the concatenation of parameters is secure against SQL injection and that all parameters (
$3
,$4
,$5
) are properly sanitized, especially if they include any untrusted input.✅ Verification successful
SQL Concatenation in
CAS_WITH_EXPIRE_TIME
is SecureThe implementation is safe against SQL injection as it properly uses PostgreSQL's parameterized queries with type-safe parameter binding through Rust's
ToSql
trait. The string concatenation occurs at the database level after parameter binding, and the timestamp formatting uses a hardcoded format string.🏁 Scripts executed
The following scripts were executed for the analysis:
Error: No shell scripts found
Script:
#!/bin/bash # Find the complete implementation containing these SQL statements ast-grep --pattern 'const CAS_WITH_EXPIRE_TIME: &str = r#"' # Find where these constants are used rg "CAS_WITH_EXPIRE_TIME" -A 5Length of output: 982
54-57
: Verify Safe Handling of SQL Parameter ConcatenationIn the
PUT_IF_NOT_EXISTS_WITH_EXPIRE_TIME
SQL statement, values are concatenated using the||
operator with parameters$2
,$3
, and a formatted date:VALUES($1, convert_to($2 || $3 || TO_CHAR(CURRENT_TIMESTAMP + INTERVAL '1 second' * $4, 'YYYY-MM-DD HH24:MI:SS.MS'), 'UTF8'))While parameterization is used, please verify that concatenating parameters in this manner does not introduce SQL injection vulnerabilities, especially if any of the parameters (
$2
,$3
,$4
) can contain untrusted data.✅ Verification successful
SQL Parameter Concatenation is Secure
The implementation is safe from SQL injection as it:
- Uses proper parameter binding via Postgres's client library
- Performs string concatenation within Postgres after parameter binding, not through string interpolation
- Has strongly typed parameters (bytes for key, fixed separator constant, and numeric TTL)
🏁 Scripts executed
The following scripts were executed for the analysis:
Error: No shell scripts found
Script:
#!/bin/bash # Find the complete SQL statement and its usage rg -A 10 -B 10 "PUT_IF_NOT_EXISTS_WITH_EXPIRE_TIME" src/ # Look for the function/method that uses this SQL statement ast-grep --pattern 'const PUT_IF_NOT_EXISTS_WITH_EXPIRE_TIME: &str'Length of output: 4267
src/cli/src/bench.rs (1)
81-83
: Update toPgStore::with_url
Reflects API ChangesThe call to
PgStore::with_url
has been updated to include thetable_name
parameter:PgStore::with_url(postgres_addr, "greptime_metakv", 128)This change aligns with the updated method signature in
PgStore
. Please ensure that all usages ofwith_url
are updated accordingly throughout the codebase to prevent compilation errors.src/common/meta/src/kv_backend/memory.rs (1)
358-358
: LGTM! Improved memory efficiency in tests.The changes correctly modify the test functions to use references instead of taking ownership, which is more efficient and aligns with Rust's ownership model.
Also applies to: 379-379, 386-386, 391-397
src/common/meta/src/kv_backend/etcd.rs (1)
594-594
: LGTM! Consistent test improvements.The changes correctly:
- Use references in test functions for better memory efficiency
- Add proper cleanup after tests
Also applies to: 621-622, 631-632, 639-644
src/common/meta/src/kv_backend/test.rs (2)
64-75
: LGTM! Improved error reporting.The enhanced assertion message now includes the prefix in UTF-8 format, which will help with debugging test failures.
355-355
: LGTM! Consistent use of references in test functions.The changes consistently modify function signatures to use
&impl KvBackend
instead of taking ownership, which:
- Improves memory efficiency
- Follows Rust's ownership best practices
- Maintains consistency across the test suite
Also applies to: 359-359, 410-410, 414-414, 454-454, 479-479, 509-509, 538-538, 578-578, 618-618
src/log-store/src/raft_engine/backend.rs (1)
647-647
: LGTM! Improved efficiency by passing backend by reference.The changes consistently update test function calls to pass the backend by reference instead of by value, which is more efficient and aligns with Rust's ownership model.
Also applies to: 674-674, 683-683
src/common/meta/src/error.rs (2)
652-654
: LGTM! Enhanced error context for Postgres execution failures.Including the SQL statement in the error message improves debuggability by providing more context about the failing operation.
812-821
: LGTM! Added helper method to identify Postgres serialization errors.The
is_serialization_error
method provides a clean way to check for specific Postgres serialization failures, which is useful for handling transient errors in transaction retries.🧰 Tools
🪛 GitHub Check: Check (ubuntu-20.04)
[failure] 817-817:
failed to resolve: use of undeclared crate or moduletokio_postgres
[failure] 815-815:
no variant namedPostgresTransaction
found for enumerror::Error
src/common/meta/src/rpc/store.rs (2)
269-269
: LGTM! Added Default trait implementations.Adding the Default trait to these response types improves ergonomics by allowing them to be created with default values.
Also applies to: 428-428, 512-512
757-768
: LGTM! Enhanced DeleteRangeResponse API.The new constructor and setter methods provide a more ergonomic way to create and modify DeleteRangeResponse instances.
Rest LGTM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super Weny!
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5309 +/- ##
==========================================
- Coverage 84.15% 83.92% -0.23%
==========================================
Files 1200 1200
Lines 225396 225386 -10
==========================================
- Hits 189684 189162 -522
- Misses 35712 36224 +512 |
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
minor refactoring
PgStore
Bench results:
Serializable
Read repeatable
etcd
PR Checklist
Please convert it to a draft if some of the following conditions are not met.
Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Chores
Default
trait implementations for response structs