From 2b077d5236c3c8fc8827c4d763ebcd46d7a616e9 Mon Sep 17 00:00:00 2001 From: Alexander Tobi Fashakin Date: Mon, 19 Jun 2023 19:34:02 +0100 Subject: [PATCH 01/46] Add blogpost on FerretDB v1.4.0 (#2858) --- ...2023-06-19-ferretdb-v-1-4-0-new-release.md | 178 ++++++++++++++++++ website/static/img/blog/ferretdb-v1.4.0.jpg | 3 + 2 files changed, 181 insertions(+) create mode 100644 website/blog/2023-06-19-ferretdb-v-1-4-0-new-release.md create mode 100644 website/static/img/blog/ferretdb-v1.4.0.jpg diff --git a/website/blog/2023-06-19-ferretdb-v-1-4-0-new-release.md b/website/blog/2023-06-19-ferretdb-v-1-4-0-new-release.md new file mode 100644 index 000000000000..dd1c8be170c9 --- /dev/null +++ b/website/blog/2023-06-19-ferretdb-v-1-4-0-new-release.md @@ -0,0 +1,178 @@ +--- +slug: ferretdb-v-1-4-0-new-release +title: FerretDB v1.4.0 New Release +authors: [alex] +description: > + We’re delighted to announce the latest FerretDB release – v1.4.0, which now includes options for creating unique indexes, and support for more aggregation pipeline stages, among other interesting updates. +image: /img/blog/ferretdb-v1.4.0.jpg +tags: [release] +--- + +![FerretDB v.1.4.0](/img/blog/ferretdb-v1.4.0.jpg) + +We're delighted to announce the latest [FerretDB](https://www.ferretdb.io/) release – v1.4.0, which now includes options for creating unique indexes, and support for more aggregation pipeline stages, among other interesting updates. + + + +Building on the [list of currently supported commands](https://docs.ferretdb.io/reference/supported-commands/), we've been working to improve compatibility and enable more real-world applications to use FerretDB as their open-source MongoDB alternative. +To facilitate this, FerretDB has seen several feature updates, enhancements, and bug fixes since the [first GA announcement](https://blog.ferretdb.io/ferretdb-1-0-ga-opensource-mongodb-alternative/). + +All of these have led to a great deal of interest in FerretDB from different communities, most importantly the open source community and contributors willing to help us with these improvements; for example, one of our newest contributors, [@shibasisp](https://github.com/shibasisp), was instrumental in implementing the `$unset`, `$set`, and `$addField` aggregation pipeline stages in this release. +We thank everyone that has contributed to FerretDB – you've all been amazing! + +### New Features + +One of the notable features in this release is the implementation of the `createIndexes` command to support unique indexes; this should help ensure uniqueness in your indexes so that no two indexed fields have the same values. + +You can create a unique index by setting the `unique` option in the `createIndexes` command as `true`. + +```js +db.collection.createIndexes({ indexedfield: 1 }, { unique: `true` }) +``` + +Read more about [unique indexes in our documentation](https://docs.ferretdb.io/indexes/#unique-indexes). + +Beyond this, we've added `$type` operator support in aggregation `$project` stage. +You can use the `$type` operator in a `$project` stage to return the BSON data type of a specified field in the documents. + +Suppose you have the following document: + +```json5 +[ + { + _id: 1, + name: 'John', + age: 35, + salary: 5000 + }, + { + _id: 2, + name: 'Robert', + age: 42, + salary: 7000 + } +] +``` + +To return the BSON type of the `age` field in the documents, you can use the following query: + +```js +db.employees.aggregate([ + { + $project: { + name: 1, + age: 1, + ageType: { $type: '$age' } + } + } +]) +``` + +The output will be: + +```json5 +[ + { _id: 1, name: 'John', age: 35, ageType: 'int' }, + { _id: 2, name: 'Robert', age: 42, ageType: 'int' } +] +``` + +In addition to already supported aggregation pipeline stages, we have now added support for the `$unset`, `$set`, and `$addFields` stages. +The `$set` stage modifies the values of existing fields, while the`$addFields` stage adds new fields or updates the values of existing ones in the aggregation pipeline; the `$unset` stage excludes or removes specific fields from documents passed to the next stage of the aggregation process. + +:::tip +The `$set` stage can only update the values of existing fields, while the `$addFields` stage can add new fields or update the values of existing ones in a document. +::: + +Use the `$addFields` stage to add the `department` and `employmentType` fields to all documents: + +```js +db.employees.aggregate([ + { + $addFields: { + department: 'HR', + employmentType: 'Full-time' + } + } +]) +``` + +The output will be: + +```json5 +[ + { + _id: 1, + name: 'John', + age: 35, + salary: 5000, + department: 'HR', + employmentType: 'Full-time' + }, + { + _id: 2, + name: 'Robert', + age: 42, + salary: 7000, + department: 'HR', + employmentType: 'Full-time' + } +] +``` + +Let's use the `$set` stage to update the `department` field: + +```js +db.employees.aggregate([ + { + $set: { + department: 'Sales' + } + } +]) +``` + +The output: + +```json5 +[ + { _id: 1, name: 'John', age: 35, salary: 5000, department: 'Sales' }, + { + _id: 2, + name: 'Robert', + age: 42, + salary: 7000, + department: 'Sales' + } +] +``` + +Use the `$unset` stage to remove the `salary` field from documents passed to the next stage in a pipeline: + +```js +db.employees.aggregate([ + { + $unset: 'salary' + } +]) +``` + +Output: + +```json5 +[ + { _id: 1, name: 'John', age: 35 }, + { _id: 2, name: 'Robert', age: 42 } +] +``` + +### Other Changes + +To improve our documentation and content, we've included textlint rules for en-dashes and em-dashes so we don't wrongly document them. +We've also updated our contribution guidelines to include description of our current test naming conventions – [have a look at that here](https://github.com/FerretDB/FerretDB/blob/main/CONTRIBUTING.md#integration-tests-naming-guidelines). +To learn about other changes in this release, please see the [FerretDB release notes for v1.4.0](https://github.com/FerretDB/FerretDB/releases/tag/v1.4.0). + +We're always looking to improve the features of FerretDB, so any suggestions, feedback, or questions you might have would be greatly appreciated. +[Reach out to us here](https://docs.ferretdb.io/#community). + +[**Be a part of the FerretDB community - check us out on GitHub**](https://github.com/FerretDB/FerretDB/). diff --git a/website/static/img/blog/ferretdb-v1.4.0.jpg b/website/static/img/blog/ferretdb-v1.4.0.jpg new file mode 100644 index 000000000000..b23ebe3cd4f7 --- /dev/null +++ b/website/static/img/blog/ferretdb-v1.4.0.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b6def407b974eef301b628605792d90d9d74b79d1dde497731f1df0bd0c254d7 +size 63282 From 4d40b2b2174c99e3ce3c25d8ec2817a24b1b93d4 Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Tue, 20 Jun 2023 03:49:52 +0900 Subject: [PATCH 02/46] Add more validation and tests for `$unset` (#2853) Closes #1432. --- integration/aggregate_documents_test.go | 211 +++++++++++++++++- integration/findandmodify_test.go | 30 +++ .../common/aggregations/stages/unset.go | 117 ++++++++-- internal/handlers/common/update.go | 38 +--- internal/handlers/commonerrors/error.go | 6 + .../handlers/commonerrors/errorcode_string.go | 70 +++--- .../types/documentpatherrorcode_string.go | 6 +- internal/types/path.go | 50 +++++ 8 files changed, 435 insertions(+), 93 deletions(-) diff --git a/integration/aggregate_documents_test.go b/integration/aggregate_documents_test.go index 310e1b4f5938..2a3edb96c066 100644 --- a/integration/aggregate_documents_test.go +++ b/integration/aggregate_documents_test.go @@ -403,13 +403,7 @@ func TestAggregateProjectErrors(t *testing.T) { ctx, collection := setup.Setup(t) _, err := collection.Aggregate(ctx, tc.pipeline) - - if tc.altMessage != "" { - AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) - return - } - - AssertEqualCommandError(t, *tc.err, err) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) }) } } @@ -460,13 +454,208 @@ func TestAggregateSetErrors(t *testing.T) { ctx, collection := setup.Setup(t) _, err := collection.Aggregate(ctx, tc.pipeline) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) + }) + } +} - if tc.altMessage != "" { - AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) - return +func TestAggregateUnsetErrors(t *testing.T) { + t.Parallel() + + for name, tc := range map[string]struct { //nolint:vet // used for test only + pipeline bson.A // required, aggregation pipeline stages + + err *mongo.CommandError // required + altMessage string // optional, alternative error message + skip string // optional, skip test with a specified reason + }{ + "EmptyString": { + pipeline: bson.A{ + bson.D{{"$unset", ""}}, + }, + err: &mongo.CommandError{ + Code: 40352, + Name: "Location40352", + Message: "Invalid $unset :: caused by :: FieldPath cannot be constructed with empty string", + }, + }, + "InvalidType": { + pipeline: bson.A{ + bson.D{{"$unset", bson.D{}}}, + }, + err: &mongo.CommandError{ + Code: 31002, + Name: "Location31002", + Message: "$unset specification must be a string or an array", + }, + }, + "PathEmptyKey": { + pipeline: bson.A{ + bson.D{{"$unset", "v..foo"}}, + }, + err: &mongo.CommandError{ + Code: 15998, + Name: "Location15998", + Message: "Invalid $unset :: caused by :: FieldPath field names may not be empty strings.", + }, + }, + "PathEmptySuffixKey": { + pipeline: bson.A{ + bson.D{{"$unset", "v."}}, + }, + err: &mongo.CommandError{ + Code: 40353, + Name: "Location40353", + Message: "Invalid $unset :: caused by :: FieldPath must not end with a '.'.", + }, + }, + "PathEmptyPrefixKey": { + pipeline: bson.A{ + bson.D{{"$unset", ".v"}}, + }, + err: &mongo.CommandError{ + Code: 15998, + Name: "Location15998", + Message: "Invalid $unset :: caused by :: FieldPath field names may not be empty strings.", + }, + }, + "PathDollarPrefix": { + pipeline: bson.A{ + bson.D{{"$unset", "$v"}}, + }, + err: &mongo.CommandError{ + Code: 16410, + Name: "Location16410", + Message: "Invalid $unset :: caused by :: FieldPath field names may not start with '$'. Consider using $getField or $setField.", + }, + }, + "ArrayEmpty": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{}}}, + }, + err: &mongo.CommandError{ + Code: 31119, + Name: "Location31119", + Message: "$unset specification must be a string or an array with at least one field", + }, + altMessage: "$unset specification must be a string or an array with at least one field", + }, + "ArrayInvalidType": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"field1", 1}}}, + }, + err: &mongo.CommandError{ + Code: 31120, + Name: "Location31120", + Message: "$unset specification must be a string or an array containing only string values", + }, + }, + "ArrayEmptyString": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{""}}}, + }, + err: &mongo.CommandError{ + Code: 40352, + Name: "Location40352", + Message: "Invalid $unset :: caused by :: FieldPath cannot be constructed with empty string", + }, + }, + "ArrayPathDuplicate": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v", "v"}}}, + }, + err: &mongo.CommandError{ + Code: 31250, + Name: "Location31250", + Message: "Invalid $unset :: caused by :: Path collision at v", + }, + }, + "ArrayPathOverwrites": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v", "v.foo"}}}, + }, + err: &mongo.CommandError{ + Code: 31249, + Name: "Location31249", + Message: "Invalid $unset :: caused by :: Path collision at v.foo remaining portion foo", + }, + }, + "ArrayPathOverwritesRemaining": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v", "v.foo.bar"}}}, + }, + err: &mongo.CommandError{ + Code: 31249, + Name: "Location31249", + Message: "Invalid $unset :: caused by :: Path collision at v.foo.bar remaining portion foo.bar", + }, + }, + "ArrayPathCollision": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v.foo", "v"}}}, + }, + err: &mongo.CommandError{ + Code: 31250, + Name: "Location31250", + Message: "Invalid $unset :: caused by :: Path collision at v", + }, + }, + "ArrayPathEmptyKey": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v..foo"}}}, + }, + err: &mongo.CommandError{ + Code: 15998, + Name: "Location15998", + Message: "Invalid $unset :: caused by :: FieldPath field names may not be empty strings.", + }, + }, + "ArrayPathEmptySuffixKey": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"v."}}}, + }, + err: &mongo.CommandError{ + Code: 40353, + Name: "Location40353", + Message: "Invalid $unset :: caused by :: FieldPath must not end with a '.'.", + }, + }, + "ArrayPathEmptyPrefixKey": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{".v"}}}, + }, + err: &mongo.CommandError{ + Code: 15998, + Name: "Location15998", + Message: "Invalid $unset :: caused by :: FieldPath field names may not be empty strings.", + }, + }, + "ArrayPathDollarPrefix": { + pipeline: bson.A{ + bson.D{{"$unset", bson.A{"$v"}}}, + }, + err: &mongo.CommandError{ + Code: 16410, + Name: "Location16410", + Message: "Invalid $unset :: caused by :: FieldPath field names may not start with '$'. Consider using $getField or $setField.", + }, + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) } - AssertEqualCommandError(t, *tc.err, err) + t.Parallel() + + require.NotNil(t, tc.pipeline, "pipeline must not be nil") + require.NotNil(t, tc.err, "err must not be nil") + + ctx, collection := setup.Setup(t) + + _, err := collection.Aggregate(ctx, tc.pipeline) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) }) } } diff --git a/integration/findandmodify_test.go b/integration/findandmodify_test.go index 2212f41a8e69..20c2a95504cd 100644 --- a/integration/findandmodify_test.go +++ b/integration/findandmodify_test.go @@ -397,6 +397,36 @@ func TestFindAndModifyCommandErrors(t *testing.T) { Message: "The update path 'v.' contains an empty field name, which is not allowed.", }, }, + "ConflictCollision": { + command: bson.D{ + {"query", bson.D{{"_id", bson.D{{"$exists", false}}}}}, + {"update", bson.D{ + {"$set", bson.D{{"v", "val"}}}, + {"$min", bson.D{{"v.foo", "val"}}}, + }}, + }, + err: &mongo.CommandError{ + Code: 40, + Name: "ConflictingUpdateOperators", + Message: "Updating the path 'v.foo' would create a conflict at 'v'", + }, + altMessage: "Updating the path 'v' would create a conflict at 'v'", + }, + "ConflictOverwrite": { + command: bson.D{ + {"query", bson.D{{"_id", bson.D{{"$exists", false}}}}}, + {"update", bson.D{ + {"$set", bson.D{{"v.foo", "val"}}}, + {"$min", bson.D{{"v", "val"}}}, + }}, + }, + err: &mongo.CommandError{ + Code: 40, + Name: "ConflictingUpdateOperators", + Message: "Updating the path 'v' would create a conflict at 'v'", + }, + altMessage: "Updating the path 'v.foo' would create a conflict at 'v.foo'", + }, } { name, tc := name, tc t.Run(name, func(t *testing.T) { diff --git a/internal/handlers/common/aggregations/stages/unset.go b/internal/handlers/common/aggregations/stages/unset.go index 62553e86708f..1750b1605149 100644 --- a/internal/handlers/common/aggregations/stages/unset.go +++ b/internal/handlers/common/aggregations/stages/unset.go @@ -17,6 +17,8 @@ package stages import ( "context" "errors" + "fmt" + "strings" "github.com/FerretDB/FerretDB/internal/handlers/common/aggregations" "github.com/FerretDB/FerretDB/internal/handlers/common/aggregations/stages/projection" @@ -33,18 +35,15 @@ import ( // // or { $unset: [ "", "", ... ] } type unset struct { - field *types.Document + exclusion *types.Document } // newUnset validates unset document and creates a new $unset stage. func newUnset(stage *types.Document) (aggregations.Stage, error) { - fields, err := stage.Get("$unset") - if err != nil { - return nil, lazyerrors.Error(err) - } + fields := must.NotFail(stage.Get("$unset")) - // fieldsToUnset contains keys with `false` values. They are used to specify projection exclusion later. - fieldsToUnset := must.NotFail(types.NewDocument()) + // exclusion contains keys with `false` values to specify projection exclusion later. + exclusion := must.NotFail(types.NewDocument()) switch fields := fields.(type) { case *types.Array: @@ -56,11 +55,13 @@ func newUnset(stage *types.Document) (aggregations.Stage, error) { ) } - fieldIter := fields.Iterator() - defer fieldIter.Close() + iter := fields.Iterator() + defer iter.Close() + + var visitedPaths []types.Path for { - _, field, err := fieldIter.Next() + _, v, err := iter.Next() if err != nil { if errors.Is(err, iterator.ErrIteratorDone) { break @@ -69,7 +70,7 @@ func newUnset(stage *types.Document) (aggregations.Stage, error) { return nil, lazyerrors.Error(err) } - fieldStr, ok := field.(string) + field, ok := v.(string) if !ok { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrStageUnsetArrElementInvalidType, @@ -78,18 +79,52 @@ func newUnset(stage *types.Document) (aggregations.Stage, error) { ) } - fieldsToUnset.Set(fieldStr, false) + path, err := validateUnsetField(field) + if err != nil { + return nil, err + } + + err = types.IsConflictPath(visitedPaths, *path) + var pathErr *types.DocumentPathError + + if errors.As(err, &pathErr) { + if pathErr.Code() == types.ErrDocumentPathConflictOverwrite { + // the path overwrites one of visitedPaths. + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrUnsetPathOverwrite, + fmt.Sprintf("Invalid $unset :: caused by :: Path collision at %s", field), + "$unset (stage)", + ) + } + + if pathErr.Code() == types.ErrDocumentPathConflictCollision { + // the path creates collision at one of visitedPaths. + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrUnsetPathCollision, + fmt.Sprintf( + "Invalid $unset :: caused by :: Path collision at %s remaining portion %s", + path.String(), + pathErr.Error(), + ), + "$unset (stage)", + ) + } + } + + if err != nil { + return nil, lazyerrors.Error(err) + } + + visitedPaths = append(visitedPaths, *path) + + exclusion.Set(field, false) } case string: - if fields == "" { - return nil, commonerrors.NewCommandErrorMsgWithArgument( - commonerrors.ErrEmptyFieldPath, - "Invalid $unset :: caused by :: FieldPath cannot be constructed with empty string", - "$unset (stage)", - ) + if _, err := validateUnsetField(fields); err != nil { + return nil, err } - fieldsToUnset.Set(fields, false) + exclusion.Set(fields, false) default: return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrStageUnsetInvalidType, @@ -99,13 +134,14 @@ func newUnset(stage *types.Document) (aggregations.Stage, error) { } return &unset{ - field: fieldsToUnset, + exclusion: exclusion, }, nil } // Process implements Stage interface. func (u *unset) Process(_ context.Context, iter types.DocumentsIterator, closer *iterator.MultiCloser) (types.DocumentsIterator, error) { //nolint:lll // for readability - return projection.ProjectionIterator(iter, closer, u.field) + // Use $project to unset fields, $unset is alias for $project exclusion. + return projection.ProjectionIterator(iter, closer, u.exclusion) } // Type implements Stage interface. @@ -113,6 +149,45 @@ func (u *unset) Type() aggregations.StageType { return aggregations.StageTypeDocuments } +// validateUnsetField returns error on invalid field value. +func validateUnsetField(field string) (*types.Path, error) { + if field == "" { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrEmptyFieldPath, + "Invalid $unset :: caused by :: FieldPath cannot be constructed with empty string", + "$unset (stage)", + ) + } + + if strings.HasPrefix(field, "$") { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrFieldPathInvalidName, + "Invalid $unset :: caused by :: FieldPath field names may not start with '$'. "+ + "Consider using $getField or $setField.", + "$unset (stage)", + ) + } + + if strings.HasSuffix(field, ".") { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidFieldPath, + "Invalid $unset :: caused by :: FieldPath must not end with a '.'.", + "$unset (stage)", + ) + } + + path, err := types.NewPathFromString(field) + if err != nil { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrPathContainsEmptyElement, + "Invalid $unset :: caused by :: FieldPath field names may not be empty strings.", + "$unset (stage)", + ) + } + + return &path, nil +} + // check interfaces var ( _ aggregations.Stage = (*unset)(nil) diff --git a/internal/handlers/common/update.go b/internal/handlers/common/update.go index a6988419c588..566141273595 100644 --- a/internal/handlers/common/update.go +++ b/internal/handlers/common/update.go @@ -902,7 +902,7 @@ func newUpdateError(code commonerrors.ErrorCode, msg, command string) error { // validateOperatorKeys returns error if any key contains empty path or // the same path prefix exists in other key or other document. func validateOperatorKeys(command string, docs ...*types.Document) error { - visitedPaths := []*types.Path{} + var visitedPaths []types.Path for _, doc := range docs { for _, key := range doc.Keys() { @@ -918,8 +918,12 @@ func validateOperatorKeys(command string, docs ...*types.Document) error { ) } - for _, oldPath := range visitedPaths { - if checkSlicePrefix(oldPath.Slice(), nextPath.Slice()) { + err = types.IsConflictPath(visitedPaths, nextPath) + var pathErr *types.DocumentPathError + + if errors.As(err, &pathErr) { + if pathErr.Code() == types.ErrDocumentPathConflictOverwrite || + pathErr.Code() == types.ErrDocumentPathConflictCollision { return newUpdateError( commonerrors.ErrConflictingUpdateOperators, fmt.Sprintf( @@ -929,34 +933,16 @@ func validateOperatorKeys(command string, docs ...*types.Document) error { ) } } - visitedPaths = append(visitedPaths, &nextPath) - } - } - - return nil -} -// checkSlicePrefix returns true if one slice is the beginning of another slice. -// The example of slice prefix: arr1 = ["a","b","c"] arr2 = ["a","b"]; -// If both slices are empty, it returns true. If one slice is empty and another slice is not, it returns false. -func checkSlicePrefix(arr1, arr2 []string) bool { - target, prefix := arr1, arr2 - - if len(target) < len(prefix) { - target, prefix = prefix, target - } - - if len(prefix) == 0 { - return len(target) == 0 - } + if err != nil { + return lazyerrors.Error(err) + } - for i := range prefix { - if prefix[i] != target[i] { - return false + visitedPaths = append(visitedPaths, nextPath) } } - return true + return nil } // extractValueFromUpdateOperator gets operator "op" value and returns CommandError for `findAndModify` diff --git a/internal/handlers/commonerrors/error.go b/internal/handlers/commonerrors/error.go index ba67671434e3..a4353993fa97 100644 --- a/internal/handlers/commonerrors/error.go +++ b/internal/handlers/commonerrors/error.go @@ -189,6 +189,12 @@ const ( // ErrStageUnwindNoPrefix indicates that $unwind aggregation stage doesn't include '$' prefix. ErrStageUnwindNoPrefix = ErrorCode(28818) // Location28818 + // ErrUnsetPathCollision indicates that an $unset path creates collision at another path in arguments. + ErrUnsetPathCollision = ErrorCode(31249) // Location31249 + + // ErrUnsetPathOverwrite indicates that an $unset path have overwrites another path in arguments. + ErrUnsetPathOverwrite = ErrorCode(31250) // Location31250 + // ErrProjectionInEx for $elemMatch indicates that inclusion statement found // while projection document already marked as exclusion. ErrProjectionInEx = ErrorCode(31253) // Location31253 diff --git a/internal/handlers/commonerrors/errorcode_string.go b/internal/handlers/commonerrors/errorcode_string.go index 9b3c0d85fbac..401403ee9865 100644 --- a/internal/handlers/commonerrors/errorcode_string.go +++ b/internal/handlers/commonerrors/errorcode_string.go @@ -61,6 +61,8 @@ func _() { _ = x[ErrStageUnsetInvalidType-31002] _ = x[ErrStageUnwindNoPath-28812] _ = x[ErrStageUnwindNoPrefix-28818] + _ = x[ErrUnsetPathCollision-31249] + _ = x[ErrUnsetPathOverwrite-31250] _ = x[ErrProjectionInEx-31253] _ = x[ErrProjectionExIn-31254] _ = x[ErrAggregatePositionalProject-31324] @@ -95,7 +97,7 @@ func _() { _ = x[ErrStageCollStatsInvalidArg-5447000] } -const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" +const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" var _ErrorCode_map = map[ErrorCode]string{ 0: _ErrorCode_name[0:5], @@ -150,38 +152,40 @@ var _ErrorCode_map = map[ErrorCode]string{ 31002: _ErrorCode_name[729:742], 31119: _ErrorCode_name[742:755], 31120: _ErrorCode_name[755:768], - 31253: _ErrorCode_name[768:781], - 31254: _ErrorCode_name[781:794], - 31324: _ErrorCode_name[794:807], - 31325: _ErrorCode_name[807:820], - 31394: _ErrorCode_name[820:833], - 31395: _ErrorCode_name[833:846], - 40156: _ErrorCode_name[846:859], - 40157: _ErrorCode_name[859:872], - 40158: _ErrorCode_name[872:885], - 40160: _ErrorCode_name[885:898], - 40234: _ErrorCode_name[898:911], - 40237: _ErrorCode_name[911:924], - 40238: _ErrorCode_name[924:937], - 40272: _ErrorCode_name[937:950], - 40323: _ErrorCode_name[950:963], - 40352: _ErrorCode_name[963:976], - 40353: _ErrorCode_name[976:989], - 40414: _ErrorCode_name[989:1002], - 40415: _ErrorCode_name[1002:1015], - 50840: _ErrorCode_name[1015:1028], - 51024: _ErrorCode_name[1028:1041], - 51075: _ErrorCode_name[1041:1054], - 51091: _ErrorCode_name[1054:1067], - 51108: _ErrorCode_name[1067:1080], - 51246: _ErrorCode_name[1080:1093], - 51247: _ErrorCode_name[1093:1106], - 51270: _ErrorCode_name[1106:1119], - 51272: _ErrorCode_name[1119:1132], - 4822819: _ErrorCode_name[1132:1147], - 5107200: _ErrorCode_name[1147:1162], - 5107201: _ErrorCode_name[1162:1177], - 5447000: _ErrorCode_name[1177:1192], + 31249: _ErrorCode_name[768:781], + 31250: _ErrorCode_name[781:794], + 31253: _ErrorCode_name[794:807], + 31254: _ErrorCode_name[807:820], + 31324: _ErrorCode_name[820:833], + 31325: _ErrorCode_name[833:846], + 31394: _ErrorCode_name[846:859], + 31395: _ErrorCode_name[859:872], + 40156: _ErrorCode_name[872:885], + 40157: _ErrorCode_name[885:898], + 40158: _ErrorCode_name[898:911], + 40160: _ErrorCode_name[911:924], + 40234: _ErrorCode_name[924:937], + 40237: _ErrorCode_name[937:950], + 40238: _ErrorCode_name[950:963], + 40272: _ErrorCode_name[963:976], + 40323: _ErrorCode_name[976:989], + 40352: _ErrorCode_name[989:1002], + 40353: _ErrorCode_name[1002:1015], + 40414: _ErrorCode_name[1015:1028], + 40415: _ErrorCode_name[1028:1041], + 50840: _ErrorCode_name[1041:1054], + 51024: _ErrorCode_name[1054:1067], + 51075: _ErrorCode_name[1067:1080], + 51091: _ErrorCode_name[1080:1093], + 51108: _ErrorCode_name[1093:1106], + 51246: _ErrorCode_name[1106:1119], + 51247: _ErrorCode_name[1119:1132], + 51270: _ErrorCode_name[1132:1145], + 51272: _ErrorCode_name[1145:1158], + 4822819: _ErrorCode_name[1158:1173], + 5107200: _ErrorCode_name[1173:1188], + 5107201: _ErrorCode_name[1188:1203], + 5447000: _ErrorCode_name[1203:1218], } func (i ErrorCode) String() string { diff --git a/internal/types/documentpatherrorcode_string.go b/internal/types/documentpatherrorcode_string.go index 62ad1ef03c0f..0ae1f29c7586 100644 --- a/internal/types/documentpatherrorcode_string.go +++ b/internal/types/documentpatherrorcode_string.go @@ -14,11 +14,13 @@ func _() { _ = x[ErrDocumentPathIndexOutOfBound-4] _ = x[ErrDocumentPathCannotCreateField-5] _ = x[ErrDocumentPathEmptyKey-6] + _ = x[ErrDocumentPathConflictOverwrite-7] + _ = x[ErrDocumentPathConflictCollision-8] } -const _DocumentPathErrorCode_name = "ErrDocumentPathKeyNotFoundErrDocumentPathCannotAccessErrDocumentPathArrayInvalidIndexErrDocumentPathIndexOutOfBoundErrDocumentPathCannotCreateFieldErrDocumentPathEmptyKey" +const _DocumentPathErrorCode_name = "ErrDocumentPathKeyNotFoundErrDocumentPathCannotAccessErrDocumentPathArrayInvalidIndexErrDocumentPathIndexOutOfBoundErrDocumentPathCannotCreateFieldErrDocumentPathEmptyKeyErrDocumentPathConflictOverwriteErrDocumentPathConflictCollision" -var _DocumentPathErrorCode_index = [...]uint8{0, 26, 53, 85, 115, 147, 170} +var _DocumentPathErrorCode_index = [...]uint8{0, 26, 53, 85, 115, 147, 170, 202, 234} func (i DocumentPathErrorCode) String() string { i -= 1 diff --git a/internal/types/path.go b/internal/types/path.go index 224ae322a34b..8901ead8b6b0 100644 --- a/internal/types/path.go +++ b/internal/types/path.go @@ -52,6 +52,12 @@ const ( // ErrDocumentPathEmptyKey indicates that provided path contains empty key. ErrDocumentPathEmptyKey + + // ErrDocumentPathConflictOverwrite indicates a path overwrites another path. + ErrDocumentPathConflictOverwrite + + // ErrDocumentPathConflictCollision indicates a path creates collision at another path. + ErrDocumentPathConflictCollision ) // DocumentPathError describes an error that could occur on document path related operations. @@ -167,6 +173,50 @@ func RemoveByPath[T CompositeTypeInterface](comp T, path Path) { removeByPath(comp, path) } +// IsConflictPath returns DocumentPathError error if adding a path creates conflict at any of paths. +// Returned DocumentPathError error codes: +// +// - ErrDocumentPathConflictOverwrite when path overwrites any paths: paths = []{{"a","b"}} path = {"a"}; +// - ErrDocumentPathConflictCollision when path creates collision: paths = []{{"a"}} path = {"a","b"}; +func IsConflictPath(paths []Path, path Path) error { + for _, p := range paths { + target, prefix := p.Slice(), path.Slice() + + if len(target) < len(prefix) { + target, prefix = prefix, target + } + + if len(prefix) == 0 { + panic("path cannot be empty string") + } + + var different bool + + for i := range prefix { + if prefix[i] != target[i] { + different = true + break + } + } + + if different { + continue + } + + if p.Len() >= path.Len() { + return newDocumentPathError(ErrDocumentPathConflictOverwrite, errors.New("path overwrites previous path")) + } + + // collisionPart is part of the path which creates collision, used in command error message. + // If visitedPath is `a.b` and path is `a.b.c`, collisionPart is `b.c`. + collisionPart := strings.Join(target[len(prefix):], ".") + + return newDocumentPathError(ErrDocumentPathConflictCollision, errors.New(collisionPart)) + } + + return nil +} + // getByPath returns a value by path - a sequence of indexes and keys. func getByPath[T CompositeTypeInterface](comp T, path Path) (any, error) { var next any = comp From bc14e2422b3e2c3c89ada059d8fd33425464ffa1 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Tue, 20 Jun 2023 17:02:29 +0400 Subject: [PATCH 03/46] Make it easier to debug GitHub Actions (#2860) --- .github/workflows/go-trust.yml | 1 + .github/workflows/go.yml | 8 ++++++++ integration/setup/startup.go | 6 ++++++ 3 files changed, 15 insertions(+) diff --git a/.github/workflows/go-trust.yml b/.github/workflows/go-trust.yml index dd6597a23d3f..5dc8c72ebc12 100644 --- a/.github/workflows/go-trust.yml +++ b/.github/workflows/go-trust.yml @@ -93,6 +93,7 @@ jobs: - name: Run ${{ matrix.task }} tests run: bin/task test-integration-${{ matrix.task }} env: + GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} FERRETDB_HANA_URL: ${{ secrets.FERRETDB_HANA_URL }} # The token is not required but should make uploads more stable. diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 6542f6bbf5aa..62825511caeb 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -55,6 +55,8 @@ jobs: - name: Run short unit tests run: bin/task test-unit-short + env: + GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} # we don't want them on CI - name: Clean test and fuzz caches @@ -106,6 +108,8 @@ jobs: - name: Run unit tests run: bin/task test-unit + env: + GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} # The token is not required but should make uploads more stable. # If secrets are unavailable (for example, for a pull request from a fork), it fallbacks to the tokenless uploads. @@ -193,6 +197,8 @@ jobs: - name: Run ${{ matrix.name }} tests (${{ matrix.shard_index }}/${{ matrix.shard_total }}) run: bin/task test-integration-${{ matrix.task }} SHARD_INDEX=${{ matrix.shard_index }} SHARD_TOTAL=${{ matrix.shard_total }} + env: + GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} # The token is not required but should make uploads more stable. # If secrets are unavailable (for example, for a pull request from a fork), it fallbacks to the tokenless uploads. @@ -275,6 +281,8 @@ jobs: - name: Run env-data run: bin/task env-data + env: + GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} golangci-lint: name: golangci-lint diff --git a/integration/setup/startup.go b/integration/setup/startup.go index 7b523f42015e..124fb45714e8 100644 --- a/integration/setup/startup.go +++ b/integration/setup/startup.go @@ -49,6 +49,12 @@ var sqliteDir = filepath.Join("..", "tmp", "sqlite-tests") func Startup() { logging.Setup(zap.DebugLevel, "") + // https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables + if os.Getenv("RUNNER_DEBUG") == "1" { + zap.S().Info("Enabling setup debug logging on GitHub Actions.") + *debugSetupF = true + } + // use any available port to allow running different configuration in parallel go debug.RunHandler(context.Background(), "127.0.0.1:0", prometheus.DefaultRegisterer, zap.L().Named("debug")) From e3c51507c43b6b06c154a5a6a15bf16943e3d69c Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Wed, 21 Jun 2023 00:29:54 +0900 Subject: [PATCH 04/46] Support `find` `singleBatch` and validate `getMore` parameters (#2855) Closes #2005. --- Taskfile.yml | 1 + integration/commands_diagnostic_test.go | 4 +- integration/commands_freemonitoring_test.go | 12 +- integration/query_test.go | 154 +++++++++++++--- integration/update_field_test.go | 8 +- internal/handlers/common/getmore.go | 71 +++++++- internal/handlers/commonerrors/error.go | 3 + .../handlers/commonerrors/errorcode_string.go | 168 +++++++++--------- internal/handlers/pg/msg_find.go | 4 +- website/docs/reference/supported-commands.md | 4 +- 10 files changed, 299 insertions(+), 130 deletions(-) diff --git a/Taskfile.yml b/Taskfile.yml index dc7adcd72bde..80897dbd5024 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -199,6 +199,7 @@ tasks: -target-tls -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb -compat-url='mongodb://username:password@127.0.0.1:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem' + -enable-cursors vars: SHARD_RUN: sh: go run -C .. ./cmd/envtool tests shard --index={{.SHARD_INDEX}} --total={{.SHARD_TOTAL}} diff --git a/integration/commands_diagnostic_test.go b/integration/commands_diagnostic_test.go index 4b5ba4d26ed1..672351dbb785 100644 --- a/integration/commands_diagnostic_test.go +++ b/integration/commands_diagnostic_test.go @@ -125,9 +125,9 @@ func TestCommandsDiagnosticGetLog(t *testing.T) { ctx, collection := res.Ctx, res.Collection for name, tc := range map[string]struct { - command bson.D // required, command to run - expected map[string]any // optional, expected keys of response + command bson.D // required, command to run + expected map[string]any // optional, expected keys of response err *mongo.CommandError // optional, expected error from MongoDB altMessage string // optional, alternative error message for FerretDB, ignored if empty skip string // optional, skip test with a specified reason diff --git a/integration/commands_freemonitoring_test.go b/integration/commands_freemonitoring_test.go index 1c39a2bf4d70..68ada6fd00c8 100644 --- a/integration/commands_freemonitoring_test.go +++ b/integration/commands_freemonitoring_test.go @@ -57,13 +57,13 @@ func TestCommandsFreeMonitoringSetFreeMonitoring(t *testing.T) { }) for name, tc := range map[string]struct { - command bson.D // required, command to run - expectedRes bson.D // optional, expected response - expectedStatus string // optional, expected status + command bson.D // required, command to run - err *mongo.CommandError // optional, expected error from MongoDB - altMessage string // optional, alternative error message for FerretDB, ignored if empty - skip string // optional, skip test with a specified reason + expectedRes bson.D // optional, expected response + expectedStatus string // optional, expected status + err *mongo.CommandError // optional, expected error from MongoDB + altMessage string // optional, alternative error message for FerretDB, ignored if empty + skip string // optional, skip test with a specified reason }{ "Enable": { command: bson.D{{"setFreeMonitoring", 1}, {"action", "enable"}}, diff --git a/integration/query_test.go b/integration/query_test.go index 7ab4202be163..af115d1968a0 100644 --- a/integration/query_test.go +++ b/integration/query_test.go @@ -625,9 +625,10 @@ func TestQueryCommandBatchSize(t *testing.T) { require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only - batchSize any // optional, nil to leave batchSize unset - firstBatch primitive.A // optional, expected firstBatch + filter any // optional, nil to leave filter unset + batchSize any // optional, nil to leave batchSize unset + firstBatch primitive.A // optional, expected firstBatch err *mongo.CommandError // optional, expected error from MongoDB altMessage string // optional, alternative error message for FerretDB, ignored if empty skip string // optional, skip test with a specified reason @@ -635,12 +636,10 @@ func TestQueryCommandBatchSize(t *testing.T) { "Int": { batchSize: 1, firstBatch: docs[:1], - skip: "https://github.com/FerretDB/FerretDB/issues/2005", }, "Long": { batchSize: int64(2), firstBatch: docs[:2], - skip: "https://github.com/FerretDB/FerretDB/issues/2005", }, "LongZero": { batchSize: int64(0), @@ -655,6 +654,10 @@ func TestQueryCommandBatchSize(t *testing.T) { }, altMessage: "BSON field 'batchSize' value must be >= 0, actual value '-1'", }, + "DoubleZero": { + batchSize: float64(0), + firstBatch: bson.A{}, + }, "DoubleNegative": { batchSize: -1.1, err: &mongo.CommandError{ @@ -666,7 +669,6 @@ func TestQueryCommandBatchSize(t *testing.T) { "DoubleFloor": { batchSize: 1.9, firstBatch: docs[:1], - skip: "https://github.com/FerretDB/FerretDB/issues/2005", }, "Bool": { batchSize: true, @@ -676,18 +678,21 @@ func TestQueryCommandBatchSize(t *testing.T) { Name: "TypeMismatch", Message: "BSON field 'FindCommandRequest.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double']", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2005", + altMessage: "BSON field 'find.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double]'", }, "Unset": { // default batchSize is 101 when unset batchSize: nil, firstBatch: docs[:101], - skip: "https://github.com/FerretDB/FerretDB/issues/2005", }, "LargeBatchSize": { batchSize: 102, firstBatch: docs[:102], - skip: "https://github.com/FerretDB/FerretDB/issues/2005", + }, + "LargeBatchSizeFilter": { + filter: bson.D{{"_id", bson.D{{"$in", bson.A{0, 1, 2, 3, 4, 5}}}}}, + batchSize: 102, + firstBatch: docs[:6], }, } { name, tc := name, tc @@ -699,6 +704,10 @@ func TestQueryCommandBatchSize(t *testing.T) { t.Parallel() var rest bson.D + if tc.filter != nil { + rest = append(rest, bson.E{Key: "filter", Value: tc.filter}) + } + if tc.batchSize != nil { rest = append(rest, bson.E{Key: "batchSize", Value: tc.batchSize}) } @@ -736,6 +745,96 @@ func TestQueryCommandBatchSize(t *testing.T) { } } +func TestQueryCommandSingleBatch(t *testing.T) { + t.Parallel() + ctx, collection := setup.Setup(t) + + docs := generateDocuments(0, 5) + _, err := collection.InsertMany(ctx, docs) + require.NoError(t, err) + + for name, tc := range map[string]struct { //nolint:vet // used for testing only + batchSize any // optional, nil to leave batchSize unset + singleBatch any // optional, nil to leave singleBatch unset + + cursorClosed bool // optional, set true for expecting cursor to be closed + err *mongo.CommandError // optional, expected error from MongoDB + altMessage string // optional, alternative error message for FerretDB, ignored if empty + skip string // optional, skip test with a specified reason + }{ + "True": { + singleBatch: true, + batchSize: 3, + cursorClosed: true, + }, + "False": { + singleBatch: false, + batchSize: 3, + cursorClosed: false, + }, + "Int": { + singleBatch: int32(1), + batchSize: 3, + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "Field 'singleBatch' should be a boolean value, but found: int", + }, + altMessage: "BSON field 'find.singleBatch' is the wrong type 'int', expected type 'bool'", + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Parallel() + + var rest bson.D + if tc.batchSize != nil { + rest = append(rest, bson.E{Key: "batchSize", Value: tc.batchSize}) + } + + if tc.singleBatch != nil { + rest = append(rest, bson.E{Key: "singleBatch", Value: tc.singleBatch}) + } + + command := append( + bson.D{{"find", collection.Name()}}, + rest..., + ) + + var res bson.D + err := collection.Database().RunCommand(ctx, command).Decode(&res) + if tc.err != nil { + assert.Nil(t, res) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) + + return + } + + require.NoError(t, err) + + v, ok := res.Map()["cursor"] + require.True(t, ok) + + cursor, ok := v.(bson.D) + require.True(t, ok) + + cursorID := cursor.Map()["id"] + assert.NotNil(t, cursorID) + + if !tc.cursorClosed { + assert.NotZero(t, cursorID) + return + } + + assert.Equal(t, int64(0), cursorID) + }) + } +} + func TestQueryBatchSize(t *testing.T) { t.Parallel() ctx, collection := setup.Setup(t) @@ -746,8 +845,6 @@ func TestQueryBatchSize(t *testing.T) { require.NoError(t, err) t.Run("SetBatchSize", func(t *testing.T) { - t.Skip("https://github.com/FerretDB/FerretDB/issues/2005") - t.Parallel() // set BatchSize to 2 @@ -798,8 +895,6 @@ func TestQueryBatchSize(t *testing.T) { }) t.Run("DefaultBatchSize", func(t *testing.T) { - t.Skip("https://github.com/FerretDB/FerretDB/issues/2005") - t.Parallel() // leave batchSize unset, firstBatch uses default batchSize 101 @@ -826,7 +921,7 @@ func TestQueryBatchSize(t *testing.T) { require.Equal(t, 118, cursor.RemainingBatchLength()) }) - t.Run("SingleBatch", func(t *testing.T) { + t.Run("NegativeLimit", func(t *testing.T) { t.Parallel() // set limit to negative, it ignores batchSize and returns single document in the firstBatch. @@ -854,8 +949,6 @@ func TestQueryBatchSize(t *testing.T) { } func TestQueryCommandGetMore(t *testing.T) { - t.Skip("https://github.com/FerretDB/FerretDB/issues/2005") - t.Parallel() ctx, collection := setup.Setup(t) @@ -865,13 +958,13 @@ func TestQueryCommandGetMore(t *testing.T) { require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only - findBatchSize any // optional, nil to leave findBatchSize unset - getMoreBatchSize any // optional, nil to leave getMoreBatchSize unset - collection any // optional, nil to leave collection unset - cursorID any // optional, defaults to cursorID from find() - firstBatch primitive.A // required, expected find firstBatch - nextBatch primitive.A // optional, expected getMore nextBatch + findBatchSize any // optional, nil to leave findBatchSize unset + getMoreBatchSize any // optional, nil to leave getMoreBatchSize unset + collection any // optional, nil to leave collection unset + cursorID any // optional, defaults to cursorID from find() + firstBatch primitive.A // required, expected find firstBatch + nextBatch primitive.A // optional, expected getMore nextBatch err *mongo.CommandError // optional, expected error from MongoDB altMessage string // optional, alternative error message for FerretDB, ignored if empty skip string // optional, skip test with a specified reason @@ -893,7 +986,6 @@ func TestQueryCommandGetMore(t *testing.T) { Name: "Location51024", Message: "BSON field 'batchSize' value must be >= 0, actual value '-1'", }, - altMessage: "BSON field 'batchSize' value must be >= 0, actual value '-1'", }, "IntZero": { findBatchSize: 1, @@ -980,6 +1072,7 @@ func TestQueryCommandGetMore(t *testing.T) { Name: "TypeMismatch", Message: "BSON field 'getMore.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double']", }, + altMessage: "BSON field 'getMore.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double]'", }, "Unset": { findBatchSize: 1, @@ -1007,6 +1100,20 @@ func TestQueryCommandGetMore(t *testing.T) { Name: "TypeMismatch", Message: "BSON field 'getMore.getMore' is the wrong type 'string', expected type 'long'", }, + altMessage: "BSON field 'getMore.getMore' is the wrong type, expected type 'long'", + }, + "Int32CursorID": { + findBatchSize: 1, + getMoreBatchSize: 1, + collection: collection.Name(), + cursorID: int32(1111), + firstBatch: docs[:1], + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "BSON field 'getMore.getMore' is the wrong type 'int', expected type 'long'", + }, + altMessage: "BSON field 'getMore.getMore' is the wrong type, expected type 'long'", }, "NotFoundCursorID": { findBatchSize: 1, @@ -1072,7 +1179,8 @@ func TestQueryCommandGetMore(t *testing.T) { t.Skip(tc.skip) } - // Do not run tests in parallel, MongoDB throws error that session and cursor does not match. + // TODO: https://github.com/FerretDB/FerretDB/issues/1807 + // Do not run tests in parallel, MongoDB throws error that session and cursor do not match. // > Location50738 // > Cannot run getMore on cursor 2053655655200551971, // > which was created in session 2926eea5-9775-41a3-a563-096969f1c7d5 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= - - , diff --git a/integration/update_field_test.go b/integration/update_field_test.go index 9cee979f1a96..39b8a91ee7d5 100644 --- a/integration/update_field_test.go +++ b/integration/update_field_test.go @@ -33,12 +33,12 @@ func TestUpdateFieldSet(t *testing.T) { t.Parallel() for name, tc := range map[string]struct { - id string // optional, defaults to empty - update bson.D // required, used for update parameter + id string // optional, defaults to empty + update bson.D // required, used for update parameter + res *mongo.UpdateResult // optional, expected response from update findRes bson.D // optional, expected response from find - - skip string // optional, skip test with a specified reason + skip string // optional, skip test with a specified reason }{ "ArrayNil": { id: "string", diff --git a/internal/handlers/common/getmore.go b/internal/handlers/common/getmore.go index 369d4ddefc92..a04fa431714e 100644 --- a/internal/handlers/common/getmore.go +++ b/internal/handlers/common/getmore.go @@ -16,9 +16,12 @@ package common import ( "context" + "fmt" "github.com/FerretDB/FerretDB/internal/clientconn/conninfo" "github.com/FerretDB/FerretDB/internal/clientconn/cursor" + "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" + "github.com/FerretDB/FerretDB/internal/handlers/commonparams" "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/iterator" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" @@ -38,33 +41,83 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* return nil, err } - collection, err := GetRequiredParam[string](document, "collection") + // TODO: Use ExtractParam https://github.com/FerretDB/FerretDB/issues/2859 + v, err := document.Get("collection") if err != nil { - return nil, err + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrMissingField, + "BSON field 'getMore.collection' is missing but a required field", + document.Command(), + ) + } + + collection, ok := v.(string) + if !ok { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrTypeMismatch, + fmt.Sprintf( + "BSON field 'getMore.collection' is the wrong type '%s', expected type 'string'", + commonparams.AliasFromType(v), + ), + document.Command(), + ) + } + + if collection == "" { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidNamespace, + "Collection names cannot be empty", + document.Command(), + ) } cursorID, err := GetRequiredParam[int64](document, document.Command()) if err != nil { - return nil, err + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrTypeMismatch, + "BSON field 'getMore.getMore' is the wrong type, expected type 'long'", + document.Command(), + ) } // TODO maxTimeMS, comment username, _ := conninfo.Get(ctx).Auth() + // TODO: Use ExtractParam https://github.com/FerretDB/FerretDB/issues/2859 cursor := registry.Cursor(username, cursorID) if cursor == nil { - return nil, lazyerrors.Errorf("no cursor %d", cursorID) + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrCursorNotFound, + fmt.Sprintf("cursor id %d not found", cursorID), + document.Command(), + ) + } + + v, _ = document.Get("batchSize") + if v == nil || types.Compare(v, int32(0)) == types.Equal { + // TODO: Use 16MB batchSize limit https://github.com/FerretDB/FerretDB/issues/2824 + // Unlimited default batchSize is used for missing batchSize and zero values, + // set 250 assuming it is small enough not to crash FerretDB. + v = int32(250) } - // TODO this logic should be tested - batchSize, _ := GetOptionalParam(document, "batchSize", cursor.BatchSize) - if batchSize < 0 { - batchSize = 101 + batchSize, err := commonparams.GetValidatedNumberParamWithMinValue(document.Command(), "batchSize", v, 0) + if err != nil { + return nil, err } if cursor.DB != db || cursor.Collection != collection { - return nil, lazyerrors.Errorf("cursor %d is for %s.%s, not %s.%s", cursorID, cursor.DB, cursor.Collection, db, collection) + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrUnauthorized, + fmt.Sprintf("Requested getMore on namespace '%s.%s', but cursor belongs to a different namespace %s.%s", + db, + collection, + cursor.DB, + cursor.Collection, + ), + document.Command(), + ) } resDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor.Iter), int(batchSize)) diff --git a/internal/handlers/commonerrors/error.go b/internal/handlers/commonerrors/error.go index a4353993fa97..d619aff1fa50 100644 --- a/internal/handlers/commonerrors/error.go +++ b/internal/handlers/commonerrors/error.go @@ -41,6 +41,9 @@ const ( // ErrFailedToParse indicates user input parsing failure. ErrFailedToParse = ErrorCode(9) // FailedToParse + // ErrUnauthorized indicates that cursor is not authorized to access another namespace. + ErrUnauthorized = ErrorCode(13) // Unauthorized + // ErrTypeMismatch for $sort indicates that the expression in the $sort is not an object. ErrTypeMismatch = ErrorCode(14) // TypeMismatch diff --git a/internal/handlers/commonerrors/errorcode_string.go b/internal/handlers/commonerrors/errorcode_string.go index 401403ee9865..a54647fcf5d7 100644 --- a/internal/handlers/commonerrors/errorcode_string.go +++ b/internal/handlers/commonerrors/errorcode_string.go @@ -12,6 +12,7 @@ func _() { _ = x[errInternalError-1] _ = x[ErrBadValue-2] _ = x[ErrFailedToParse-9] + _ = x[ErrUnauthorized-13] _ = x[ErrTypeMismatch-14] _ = x[ErrAuthenticationFailed-18] _ = x[ErrIllegalOperation-20] @@ -97,95 +98,96 @@ func _() { _ = x[ErrStageCollStatsInvalidArg-5447000] } -const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" +const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseUnauthorizedTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" var _ErrorCode_map = map[ErrorCode]string{ 0: _ErrorCode_name[0:5], 1: _ErrorCode_name[5:18], 2: _ErrorCode_name[18:26], 9: _ErrorCode_name[26:39], - 14: _ErrorCode_name[39:51], - 18: _ErrorCode_name[51:71], - 20: _ErrorCode_name[71:87], - 26: _ErrorCode_name[87:104], - 27: _ErrorCode_name[104:117], - 28: _ErrorCode_name[117:130], - 40: _ErrorCode_name[130:156], - 43: _ErrorCode_name[156:170], - 48: _ErrorCode_name[170:185], - 52: _ErrorCode_name[185:208], - 53: _ErrorCode_name[208:217], - 56: _ErrorCode_name[217:231], - 59: _ErrorCode_name[231:246], - 66: _ErrorCode_name[246:260], - 67: _ErrorCode_name[260:277], - 72: _ErrorCode_name[277:291], - 73: _ErrorCode_name[291:307], - 85: _ErrorCode_name[307:327], - 86: _ErrorCode_name[327:348], - 96: _ErrorCode_name[348:363], - 121: _ErrorCode_name[363:388], - 168: _ErrorCode_name[388:411], - 197: _ErrorCode_name[411:442], - 238: _ErrorCode_name[442:456], - 11000: _ErrorCode_name[456:469], - 15947: _ErrorCode_name[469:482], - 15948: _ErrorCode_name[482:495], - 15955: _ErrorCode_name[495:508], - 15958: _ErrorCode_name[508:521], - 15959: _ErrorCode_name[521:534], - 15969: _ErrorCode_name[534:547], - 15973: _ErrorCode_name[547:560], - 15974: _ErrorCode_name[560:573], - 15975: _ErrorCode_name[573:586], - 15976: _ErrorCode_name[586:599], - 15981: _ErrorCode_name[599:612], - 15998: _ErrorCode_name[612:625], - 16020: _ErrorCode_name[625:638], - 16410: _ErrorCode_name[638:651], - 16872: _ErrorCode_name[651:664], - 17276: _ErrorCode_name[664:677], - 28667: _ErrorCode_name[677:690], - 28724: _ErrorCode_name[690:703], - 28812: _ErrorCode_name[703:716], - 28818: _ErrorCode_name[716:729], - 31002: _ErrorCode_name[729:742], - 31119: _ErrorCode_name[742:755], - 31120: _ErrorCode_name[755:768], - 31249: _ErrorCode_name[768:781], - 31250: _ErrorCode_name[781:794], - 31253: _ErrorCode_name[794:807], - 31254: _ErrorCode_name[807:820], - 31324: _ErrorCode_name[820:833], - 31325: _ErrorCode_name[833:846], - 31394: _ErrorCode_name[846:859], - 31395: _ErrorCode_name[859:872], - 40156: _ErrorCode_name[872:885], - 40157: _ErrorCode_name[885:898], - 40158: _ErrorCode_name[898:911], - 40160: _ErrorCode_name[911:924], - 40234: _ErrorCode_name[924:937], - 40237: _ErrorCode_name[937:950], - 40238: _ErrorCode_name[950:963], - 40272: _ErrorCode_name[963:976], - 40323: _ErrorCode_name[976:989], - 40352: _ErrorCode_name[989:1002], - 40353: _ErrorCode_name[1002:1015], - 40414: _ErrorCode_name[1015:1028], - 40415: _ErrorCode_name[1028:1041], - 50840: _ErrorCode_name[1041:1054], - 51024: _ErrorCode_name[1054:1067], - 51075: _ErrorCode_name[1067:1080], - 51091: _ErrorCode_name[1080:1093], - 51108: _ErrorCode_name[1093:1106], - 51246: _ErrorCode_name[1106:1119], - 51247: _ErrorCode_name[1119:1132], - 51270: _ErrorCode_name[1132:1145], - 51272: _ErrorCode_name[1145:1158], - 4822819: _ErrorCode_name[1158:1173], - 5107200: _ErrorCode_name[1173:1188], - 5107201: _ErrorCode_name[1188:1203], - 5447000: _ErrorCode_name[1203:1218], + 13: _ErrorCode_name[39:51], + 14: _ErrorCode_name[51:63], + 18: _ErrorCode_name[63:83], + 20: _ErrorCode_name[83:99], + 26: _ErrorCode_name[99:116], + 27: _ErrorCode_name[116:129], + 28: _ErrorCode_name[129:142], + 40: _ErrorCode_name[142:168], + 43: _ErrorCode_name[168:182], + 48: _ErrorCode_name[182:197], + 52: _ErrorCode_name[197:220], + 53: _ErrorCode_name[220:229], + 56: _ErrorCode_name[229:243], + 59: _ErrorCode_name[243:258], + 66: _ErrorCode_name[258:272], + 67: _ErrorCode_name[272:289], + 72: _ErrorCode_name[289:303], + 73: _ErrorCode_name[303:319], + 85: _ErrorCode_name[319:339], + 86: _ErrorCode_name[339:360], + 96: _ErrorCode_name[360:375], + 121: _ErrorCode_name[375:400], + 168: _ErrorCode_name[400:423], + 197: _ErrorCode_name[423:454], + 238: _ErrorCode_name[454:468], + 11000: _ErrorCode_name[468:481], + 15947: _ErrorCode_name[481:494], + 15948: _ErrorCode_name[494:507], + 15955: _ErrorCode_name[507:520], + 15958: _ErrorCode_name[520:533], + 15959: _ErrorCode_name[533:546], + 15969: _ErrorCode_name[546:559], + 15973: _ErrorCode_name[559:572], + 15974: _ErrorCode_name[572:585], + 15975: _ErrorCode_name[585:598], + 15976: _ErrorCode_name[598:611], + 15981: _ErrorCode_name[611:624], + 15998: _ErrorCode_name[624:637], + 16020: _ErrorCode_name[637:650], + 16410: _ErrorCode_name[650:663], + 16872: _ErrorCode_name[663:676], + 17276: _ErrorCode_name[676:689], + 28667: _ErrorCode_name[689:702], + 28724: _ErrorCode_name[702:715], + 28812: _ErrorCode_name[715:728], + 28818: _ErrorCode_name[728:741], + 31002: _ErrorCode_name[741:754], + 31119: _ErrorCode_name[754:767], + 31120: _ErrorCode_name[767:780], + 31249: _ErrorCode_name[780:793], + 31250: _ErrorCode_name[793:806], + 31253: _ErrorCode_name[806:819], + 31254: _ErrorCode_name[819:832], + 31324: _ErrorCode_name[832:845], + 31325: _ErrorCode_name[845:858], + 31394: _ErrorCode_name[858:871], + 31395: _ErrorCode_name[871:884], + 40156: _ErrorCode_name[884:897], + 40157: _ErrorCode_name[897:910], + 40158: _ErrorCode_name[910:923], + 40160: _ErrorCode_name[923:936], + 40234: _ErrorCode_name[936:949], + 40237: _ErrorCode_name[949:962], + 40238: _ErrorCode_name[962:975], + 40272: _ErrorCode_name[975:988], + 40323: _ErrorCode_name[988:1001], + 40352: _ErrorCode_name[1001:1014], + 40353: _ErrorCode_name[1014:1027], + 40414: _ErrorCode_name[1027:1040], + 40415: _ErrorCode_name[1040:1053], + 50840: _ErrorCode_name[1053:1066], + 51024: _ErrorCode_name[1066:1079], + 51075: _ErrorCode_name[1079:1092], + 51091: _ErrorCode_name[1092:1105], + 51108: _ErrorCode_name[1105:1118], + 51246: _ErrorCode_name[1118:1131], + 51247: _ErrorCode_name[1131:1144], + 51270: _ErrorCode_name[1144:1157], + 51272: _ErrorCode_name[1157:1170], + 4822819: _ErrorCode_name[1170:1185], + 5107200: _ErrorCode_name[1185:1200], + 5107201: _ErrorCode_name[1200:1215], + 5447000: _ErrorCode_name[1215:1230], } func (i ErrorCode) String() string { diff --git a/internal/handlers/pg/msg_find.go b/internal/handlers/pg/msg_find.go index 98ba554c738f..e8a874b33870 100644 --- a/internal/handlers/pg/msg_find.go +++ b/internal/handlers/pg/msg_find.go @@ -133,7 +133,9 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er var cursorID int64 - if h.EnableCursors { + if h.EnableCursors && !params.SingleBatch && int64(len(resDocs)) > params.BatchSize { + // Cursor is not created for singleBatch, and when resDocs is less than batchSize, + // all response fits in the firstBatch. iter := iterator.Values(iterator.ForSlice(resDocs)) c := cursor.New(&cursor.NewParams{ Iter: iter, diff --git a/website/docs/reference/supported-commands.md b/website/docs/reference/supported-commands.md index ee642f8ff71a..a555eeb40efb 100644 --- a/website/docs/reference/supported-commands.md +++ b/website/docs/reference/supported-commands.md @@ -32,8 +32,8 @@ Use ❌ for commands and arguments that are not implemented at all. | | `hint` | ⚠️ | Ignored | | | `skip` | ⚠️ | | | | `limit` | ✅ | | -| | `batchSize` | ⚠️ | [Issue](https://github.com/FerretDB/FerretDB/issues/2005) | -| | `singleBatch` | ⚠️ | [Issue](https://github.com/FerretDB/FerretDB/issues/2005) | +| | `batchSize` | ✅ | | +| | `singleBatch` | ✅ | | | | `comment` | ⚠️ | Not implemented in Tigris | | | `maxTimeMS` | ✅ | | | | `readConcern` | ⚠️ | Ignored | From 6d3967c44d1efdf8e3545273e642f2cc5f8aa9bc Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 05:39:08 +0400 Subject: [PATCH 05/46] Unify tests for indexes (#2866) --- integration/indexes_compat_test.go | 85 +++++++++++++++++++++--------- 1 file changed, 60 insertions(+), 25 deletions(-) diff --git a/integration/indexes_compat_test.go b/integration/indexes_compat_test.go index 182aa0c71517..03cd3a2951b9 100644 --- a/integration/indexes_compat_test.go +++ b/integration/indexes_compat_test.go @@ -44,14 +44,21 @@ func TestListIndexesCompat(t *testing.T) { t.Helper() t.Parallel() - targetCur, targetErr := targetCollection.Indexes().List(ctx) - compatCur, compatErr := compatCollection.Indexes().List(ctx) + targetCursor, targetErr := targetCollection.Indexes().List(ctx) + compatCursor, compatErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } - require.NoError(t, compatErr) require.NoError(t, targetErr) + require.NoError(t, compatErr) - targetRes := FetchAll(t, ctx, targetCur) - compatRes := FetchAll(t, ctx, compatCur) + targetRes := FetchAll(t, ctx, targetCursor) + compatRes := FetchAll(t, ctx, compatCursor) assert.Equal(t, compatRes, targetRes) @@ -252,14 +259,21 @@ func TestCreateIndexesCompat(t *testing.T) { } // List indexes to check they are identical after creation. - targetCur, targetErr := targetCollection.Indexes().List(ctx) - compatCur, compatErr := compatCollection.Indexes().List(ctx) + targetCursor, targetErr := targetCollection.Indexes().List(ctx) + compatCursor, compatErr := compatCollection.Indexes().List(ctx) + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + + require.NoError(t, targetErr) require.NoError(t, compatErr) - assert.Equal(t, compatErr, targetErr) - targetIndexes := FetchAll(t, ctx, targetCur) - compatIndexes := FetchAll(t, ctx, compatCur) + targetIndexes := FetchAll(t, ctx, targetCursor) + compatIndexes := FetchAll(t, ctx, compatCursor) assert.Equal(t, compatIndexes, targetIndexes) @@ -564,14 +578,21 @@ func TestDropIndexesCompat(t *testing.T) { } // List indexes to see they are identical after drop. - targetCur, targetErr := targetCollection.Indexes().List(ctx) - compatCur, compatErr := compatCollection.Indexes().List(ctx) + targetCursor, targetErr := targetCollection.Indexes().List(ctx) + compatCursor, compatErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + require.NoError(t, targetErr) require.NoError(t, compatErr) - require.Equal(t, compatErr, targetErr) - targetIndexes := FetchAll(t, ctx, targetCur) - compatIndexes := FetchAll(t, ctx, compatCur) + targetIndexes := FetchAll(t, ctx, targetCursor) + compatIndexes := FetchAll(t, ctx, compatCursor) require.Equal(t, compatIndexes, targetIndexes) }) @@ -695,14 +716,21 @@ func TestDropIndexesCommandCompat(t *testing.T) { require.NoError(t, targetErr) // List indexes to see they are identical after creation. - targetCur, targetListErr := targetCollection.Indexes().List(ctx) - compatCur, compatListErr := compatCollection.Indexes().List(ctx) + targetCursor, targetListErr := targetCollection.Indexes().List(ctx) + compatCursor, compatListErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } - require.NoError(t, compatListErr) require.NoError(t, targetListErr) + require.NoError(t, compatListErr) - targetList := FetchAll(t, ctx, targetCur) - compatList := FetchAll(t, ctx, compatCur) + targetList := FetchAll(t, ctx, targetCursor) + compatList := FetchAll(t, ctx, compatCursor) require.Equal(t, compatList, targetList) } @@ -746,14 +774,21 @@ func TestDropIndexesCommandCompat(t *testing.T) { } // List indexes to see they are identical after deletion. - targetCur, targetListErr := targetCollection.Indexes().List(ctx) - compatCur, compatListErr := compatCollection.Indexes().List(ctx) + targetCursor, targetListErr := targetCollection.Indexes().List(ctx) + compatCursor, compatListErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + require.NoError(t, targetListErr) require.NoError(t, compatListErr) - assert.Equal(t, compatListErr, targetListErr) - targetList := FetchAll(t, ctx, targetCur) - compatList := FetchAll(t, ctx, compatCur) + targetList := FetchAll(t, ctx, targetCursor) + compatList := FetchAll(t, ctx, compatCursor) assert.Equal(t, compatList, targetList) }) From e653182d202b37f4f3825935fc9593c322bc1ad3 Mon Sep 17 00:00:00 2001 From: Alexander Tobi Fashakin Date: Wed, 21 Jun 2023 07:11:01 +0100 Subject: [PATCH 06/46] Add blog post on "Meet FerretDB at Percona University in Casablanca and Belgrade" (#2870) --- ...-percona-university-casablanca-belgrade.md | 38 +++++++++++++++++++ .../static/img/blog/percona-university.jpg | 3 ++ 2 files changed, 41 insertions(+) create mode 100644 website/blog/2023-06-21-meet-ferretdb-percona-university-casablanca-belgrade.md create mode 100644 website/static/img/blog/percona-university.jpg diff --git a/website/blog/2023-06-21-meet-ferretdb-percona-university-casablanca-belgrade.md b/website/blog/2023-06-21-meet-ferretdb-percona-university-casablanca-belgrade.md new file mode 100644 index 000000000000..739450bfa1ed --- /dev/null +++ b/website/blog/2023-06-21-meet-ferretdb-percona-university-casablanca-belgrade.md @@ -0,0 +1,38 @@ +--- +slug: meet-ferretdb-percona-university-casablanca-belgrade +title: 'Meet FerretDB at Percona University in Casablanca and Belgrade' +author: Peter Benei +description: > + FerretDB joins Percona University to bring a truly open-source MongoDB alternative to the Casablanca, Morocco, and Belgrade, Serbia database community. +image: /img/blog/percona-university.jpg +keywords: [percona university, percona events] +tags: [events] +--- + +![Meet FerretDB at Percona University in Casablanca and Belgrade](/img/blog/percona-university.jpg) + +[FerretDB](https://www.ferretdb.io) joins [Percona University](https://www.percona.com/blog/percona-university-is-back-in-business/) to bring a truly open-source MongoDB alternative to the Casablanca, Morocco, and Belgrade, Serbia database community. +Our CEO, Peter Farkas, will share insights on transitioning from MongoDB to FerretDB, the open-source MongoDB alternative using PostgreSQL as the backend. + + + +We are excited to support database communities worldwide with open-source database solutions. +FerretDB's goal is to prove a truly open-source solution for MongoDB users, helping them transition their workloads using Postgres. + +That's why we joined forces with Percona University to meet with community members in two amazing locations in the upcoming weeks. + +First, on the 27th of June, we will visit Casablanca, Morocco. +It will be a great opportunity to discover not just FerretDB's solution but also learn more about migrating to MySQL8, benchmarking open-source databases with Manticore Search, switching to Kubernetes with Ivinco, or more details about Kafka from EDC4IT and Zen Networks. + +Next, we will visit Belgrade, Serbia, on the 5th of July. +Community members in Belgrade can learn more about Percona, how to work with Percona distributions and making a career out of open source. +On top of it, we will also present our solution for the community. + +Both Percona events are free to attend. +You can grab your tickets here: + +- Casablanca, Technopark, 27th of June - [https://www.eventbrite.com/e/percona-university-morocco-2023-tickets-643139326037](https://www.eventbrite.com/e/percona-university-morocco-2023-tickets-643139326037) +- Belgrade, Singidunum University, 5th of July - [https://www.eventbrite.com/e/percona-university-serbia-2023-tickets-657635093267](https://www.eventbrite.com/e/percona-university-serbia-2023-tickets-657635093267) + +Register to save your spot. +Excited to meet you there! diff --git a/website/static/img/blog/percona-university.jpg b/website/static/img/blog/percona-university.jpg new file mode 100644 index 000000000000..5a7ca61090e6 --- /dev/null +++ b/website/static/img/blog/percona-university.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6dac24d7813041503c5821f2da152c4474437efe02b72ba63ec6143d288c218f +size 57805 From 9d75840bea854c84b766c9a8de0266439a394934 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 12:43:11 +0400 Subject: [PATCH 07/46] Implement `count` for SQLite (#2865) --- internal/handlers/common/count_iterator.go | 2 +- internal/handlers/pg/msg_count.go | 18 +++++-- internal/handlers/sqlite/msg_count.go | 58 +++++++++++++++++++++- internal/handlers/sqlite/msg_update.go | 4 +- internal/types/array.go | 4 ++ internal/types/document.go | 5 ++ internal/types/document_test.go | 4 ++ 7 files changed, 88 insertions(+), 7 deletions(-) diff --git a/internal/handlers/common/count_iterator.go b/internal/handlers/common/count_iterator.go index ac225bba9bf4..c277ae7fc8dd 100644 --- a/internal/handlers/common/count_iterator.go +++ b/internal/handlers/common/count_iterator.go @@ -24,7 +24,7 @@ import ( ) // CountIterator returns an iterator that returns a single document containing -// the number of input documents in the specified field: {field: count}. +// the number of input documents (as int32) in the specified field: {field: count}. // It will be added to the given closer. // // Next method returns that document, subsequent calls return ErrIteratorDone. diff --git a/internal/handlers/pg/msg_count.go b/internal/handlers/pg/msg_count.go index 85a9b59d0787..3d972a96b819 100644 --- a/internal/handlers/pg/msg_count.go +++ b/internal/handlers/pg/msg_count.go @@ -16,6 +16,7 @@ package pg import ( "context" + "errors" "github.com/jackc/pgx/v5" @@ -55,7 +56,7 @@ func (h *Handler) MsgCount(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, e qp.Filter = params.Filter } - var resDocs []*types.Document + var n int32 err = dbPool.InTransaction(ctx, func(tx pgx.Tx) error { var iter types.DocumentsIterator @@ -73,11 +74,22 @@ func (h *Handler) MsgCount(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, e iter = common.LimitIterator(iter, closer, params.Limit) - resDocs, err = iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) + iter = common.CountIterator(iter, closer, "count") + + var res *types.Document + + _, res, err = iter.Next() + if errors.Is(err, iterator.ErrIteratorDone) { + err = nil + } + if err != nil { return lazyerrors.Error(err) } + count, _ := res.Get("count") + n, _ = count.(int32) + return nil }) @@ -88,7 +100,7 @@ func (h *Handler) MsgCount(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, e var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ Documents: []*types.Document{must.NotFail(types.NewDocument( - "n", int32(len(resDocs)), + "n", n, "ok", float64(1), ))}, })) diff --git a/internal/handlers/sqlite/msg_count.go b/internal/handlers/sqlite/msg_count.go index 505eb939e132..891d5b77de9f 100644 --- a/internal/handlers/sqlite/msg_count.go +++ b/internal/handlers/sqlite/msg_count.go @@ -16,12 +16,68 @@ package sqlite import ( "context" + "errors" + "github.com/FerretDB/FerretDB/internal/handlers/common" + "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" "github.com/FerretDB/FerretDB/internal/util/must" "github.com/FerretDB/FerretDB/internal/wire" ) // MsgCount implements HandlerInterface. func (h *Handler) MsgCount(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return nil, notImplemented(must.NotFail(msg.Document()).Command()) + document, err := msg.Document() + if err != nil { + return nil, lazyerrors.Error(err) + } + + params, err := common.GetCountParams(document, h.L) + if err != nil { + return nil, err + } + + db := h.b.Database(params.DB) + defer db.Close() + + queryRes, err := db.Collection(params.Collection).Query(ctx, nil) + if err != nil { + return nil, lazyerrors.Error(err) + } + + iter := queryRes.Iter + + closer := iterator.NewMultiCloser(iter) + defer closer.Close() + + iter = common.FilterIterator(iter, closer, params.Filter) + + iter = common.SkipIterator(iter, closer, params.Skip) + + iter = common.LimitIterator(iter, closer, params.Limit) + + iter = common.CountIterator(iter, closer, "count") + + _, res, err := iter.Next() + if errors.Is(err, iterator.ErrIteratorDone) { + err = nil + } + + if err != nil { + return nil, lazyerrors.Error(err) + } + + count, _ := res.Get("count") + n, _ := count.(int32) + + var reply wire.OpMsg + must.NoError(reply.SetSections(wire.OpMsgSection{ + Documents: []*types.Document{must.NotFail(types.NewDocument( + "n", n, + "ok", float64(1), + ))}, + })) + + return &reply, nil } diff --git a/internal/handlers/sqlite/msg_update.go b/internal/handlers/sqlite/msg_update.go index 878dacd6feff..135601b32916 100644 --- a/internal/handlers/sqlite/msg_update.go +++ b/internal/handlers/sqlite/msg_update.go @@ -66,7 +66,7 @@ func (h *Handler) MsgUpdate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, // updateDocument iterate through all documents in collection and update them. func (h *Handler) updateDocument(ctx context.Context, params *common.UpdatesParams) (int32, int32, *types.Array, error) { var matched, modified int32 - var upserted *types.Array + var upserted types.Array db := h.b.Database(params.DB) defer db.Close() @@ -178,5 +178,5 @@ func (h *Handler) updateDocument(ctx context.Context, params *common.UpdatesPara } } - return matched, modified, upserted, nil + return matched, modified, &upserted, nil } diff --git a/internal/types/array.go b/internal/types/array.go index 072f857df180..1b23cc102273 100644 --- a/internal/types/array.go +++ b/internal/types/array.go @@ -93,6 +93,10 @@ func (a *Array) Set(index int, value any) error { // Append appends given values to the array. func (a *Array) Append(values ...any) { + if a == nil { + panic("types.Array.Append: nil array") + } + if a.s == nil { a.s = values return diff --git a/internal/types/document.go b/internal/types/document.go index 2bc60b1526b6..d34bdb649980 100644 --- a/internal/types/document.go +++ b/internal/types/document.go @@ -236,6 +236,7 @@ func (d *Document) Has(key string) bool { // Get returns a value at the given key. // If the key is duplicated, it panics. +// It returns nil for nil Document. // // The only possible error is returned for a missing key. // In that case, Get returns nil for the value. @@ -246,6 +247,10 @@ func (d *Document) Has(key string) bool { // // The error value will be removed in the future. func (d *Document) Get(key string) (any, error) { + if d == nil { + return nil, fmt.Errorf("types.Document.Get: key not found: %q (nil document)", key) + } + if d.isKeyDuplicate(key) { panic(fmt.Sprintf("types.Document.Get: key is duplicated: %s", key)) } diff --git a/internal/types/document_test.go b/internal/types/document_test.go index c306f81cc122..b3cb476001b5 100644 --- a/internal/types/document_test.go +++ b/internal/types/document_test.go @@ -34,6 +34,10 @@ func TestDocument(t *testing.T) { assert.Nil(t, doc.Map()) assert.Nil(t, doc.Keys()) assert.Equal(t, "", doc.Command()) + + value, err := doc.Get("foo") + assert.Error(t, err) + assert.Nil(t, value) }) t.Run("ZeroValues", func(t *testing.T) { From f5edff7af1715671f9d563555edf1015e41b29af Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 14:49:52 +0400 Subject: [PATCH 08/46] Bump deps (#2875) --- go.mod | 6 +- go.sum | 12 +- integration/go.mod | 6 +- integration/go.sum | 12 +- tools/go.mod | 60 +++---- tools/go.sum | 379 +++++++++++++-------------------------------- 6 files changed, 155 insertions(+), 320 deletions(-) diff --git a/go.mod b/go.mod index 1bfd3fe93331..a7ab24d69d2a 100644 --- a/go.mod +++ b/go.mod @@ -4,12 +4,12 @@ go 1.20 require ( github.com/AlekSi/pointer v1.2.0 - github.com/SAP/go-hdb v1.3.6 + github.com/SAP/go-hdb v1.3.8 github.com/alecthomas/kong v0.7.1 github.com/google/uuid v1.3.0 github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f - github.com/jackc/pgx/v5 v5.4.0 + github.com/jackc/pgx/v5 v5.4.1 github.com/pmezard/go-difflib v1.0.0 github.com/prometheus/client_golang v1.16.0 github.com/prometheus/client_model v0.4.0 @@ -52,7 +52,7 @@ require ( github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect - github.com/prometheus/procfs v0.10.1 // indirect + github.com/prometheus/procfs v0.11.0 // indirect github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect go.opentelemetry.io/otel/metric v1.16.0 // indirect go.uber.org/atomic v1.10.0 // indirect diff --git a/go.sum b/go.sum index 1a8b7294e93e..3a6791a68f0c 100644 --- a/go.sum +++ b/go.sum @@ -9,8 +9,8 @@ github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tS github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= -github.com/SAP/go-hdb v1.3.6 h1:8j4FLYwp6s6JV/bjfs7P15rBqKBofGHKX0KLCV9BmiI= -github.com/SAP/go-hdb v1.3.6/go.mod h1:oHuCjMonQNRgjWLPaHkDZPlJhjBUMciFB6mDNQaPN1s= +github.com/SAP/go-hdb v1.3.8 h1:o1HyjliL9SrrGLN+P+yI4K0vJFhn/SWiXuaQhak/TgY= +github.com/SAP/go-hdb v1.3.8/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= github.com/alecthomas/assert/v2 v2.1.0 h1:tbredtNcQnoSd3QBhQWI7QZ3XHOVkw1Moklp2ojoH/0= github.com/alecthomas/kong v0.7.1 h1:azoTh0IOfwlAX3qN9sHWTxACE2oV8Bg2gAwBsMwDQY4= github.com/alecthomas/kong v0.7.1/go.mod h1:n1iCIO2xS46oE8ZfYCNDqdR0b0wZNrXAIAqro/2132U= @@ -109,8 +109,8 @@ github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/ github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f h1:ahoGnXfh4wiCisojvzq1PzgxzFwJEUHMI26pUY6oluk= github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f/go.mod h1:m9tCxmy1PSUQa5o0aL4rQTowmJD1BK2Zc7dgnK/IrXc= -github.com/jackc/pgx/v5 v5.4.0 h1:BSr+GCm4N6QcgIwv0DyTFHK9ugfEFF9DzSbbzxOiXU0= -github.com/jackc/pgx/v5 v5.4.0/go.mod h1:q6iHT8uDNXWiFNOlRqJzBTaSH3+2xCXkokxHZC5qWFY= +github.com/jackc/pgx/v5 v5.4.1 h1:oKfB/FhuVtit1bBM3zNRRsZ925ZkMN3HXL+LgLUM9lE= +github.com/jackc/pgx/v5 v5.4.1/go.mod h1:q6iHT8uDNXWiFNOlRqJzBTaSH3+2xCXkokxHZC5qWFY= github.com/jackc/puddle/v2 v2.2.0 h1:RdcDk92EJBuBS55nQMMYFXTxwstHug4jkhT5pq8VxPk= github.com/jackc/puddle/v2 v2.2.0/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jhump/protoreflect v1.15.1 h1:HUMERORf3I3ZdX05WaQ6MIpd/NJ434hTp5YiKgfCL6c= @@ -150,8 +150,8 @@ github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUo github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU= github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY= github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY= -github.com/prometheus/procfs v0.10.1 h1:kYK1Va/YMlutzCGazswoHKo//tZVlFpKYh+PymziUAg= -github.com/prometheus/procfs v0.10.1/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM= +github.com/prometheus/procfs v0.11.0 h1:5EAgkfkMl659uZPbe9AS2N68a7Cc1TJbPEuGzFuRbyk= +github.com/prometheus/procfs v0.11.0/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM= github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= diff --git a/integration/go.mod b/integration/go.mod index d518df63efc8..472c709e8aaf 100644 --- a/integration/go.mod +++ b/integration/go.mod @@ -22,7 +22,7 @@ require ( require ( cloud.google.com/go/compute v1.19.1 // indirect cloud.google.com/go/compute/metadata v0.2.3 // indirect - github.com/SAP/go-hdb v1.3.6 // indirect + github.com/SAP/go-hdb v1.3.8 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/benbjohnson/clock v1.1.0 // indirect github.com/beorn7/perks v1.0.1 // indirect @@ -43,7 +43,7 @@ require ( github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f // indirect - github.com/jackc/pgx/v5 v5.4.0 // indirect + github.com/jackc/pgx/v5 v5.4.1 // indirect github.com/jackc/puddle/v2 v2.2.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect @@ -57,7 +57,7 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.4.0 // indirect github.com/prometheus/common v0.44.0 // indirect - github.com/prometheus/procfs v0.10.1 // indirect + github.com/prometheus/procfs v0.11.0 // indirect github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/tigrisdata/tigris-client-go v1.0.0 // indirect github.com/xdg-go/pbkdf2 v1.0.0 // indirect diff --git a/integration/go.sum b/integration/go.sum index 207b7788245e..967a43ea7b2b 100644 --- a/integration/go.sum +++ b/integration/go.sum @@ -9,8 +9,8 @@ github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tS github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= -github.com/SAP/go-hdb v1.3.6 h1:8j4FLYwp6s6JV/bjfs7P15rBqKBofGHKX0KLCV9BmiI= -github.com/SAP/go-hdb v1.3.6/go.mod h1:oHuCjMonQNRgjWLPaHkDZPlJhjBUMciFB6mDNQaPN1s= +github.com/SAP/go-hdb v1.3.8 h1:o1HyjliL9SrrGLN+P+yI4K0vJFhn/SWiXuaQhak/TgY= +github.com/SAP/go-hdb v1.3.8/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= @@ -107,8 +107,8 @@ github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/ github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f h1:ahoGnXfh4wiCisojvzq1PzgxzFwJEUHMI26pUY6oluk= github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f/go.mod h1:m9tCxmy1PSUQa5o0aL4rQTowmJD1BK2Zc7dgnK/IrXc= -github.com/jackc/pgx/v5 v5.4.0 h1:BSr+GCm4N6QcgIwv0DyTFHK9ugfEFF9DzSbbzxOiXU0= -github.com/jackc/pgx/v5 v5.4.0/go.mod h1:q6iHT8uDNXWiFNOlRqJzBTaSH3+2xCXkokxHZC5qWFY= +github.com/jackc/pgx/v5 v5.4.1 h1:oKfB/FhuVtit1bBM3zNRRsZ925ZkMN3HXL+LgLUM9lE= +github.com/jackc/pgx/v5 v5.4.1/go.mod h1:q6iHT8uDNXWiFNOlRqJzBTaSH3+2xCXkokxHZC5qWFY= github.com/jackc/puddle/v2 v2.2.0 h1:RdcDk92EJBuBS55nQMMYFXTxwstHug4jkhT5pq8VxPk= github.com/jackc/puddle/v2 v2.2.0/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jaswdr/faker v1.18.0 h1:sJ8HQLxvNRH+Ond1pTLR01BAxMN0iuYe+6aD30H0cRE= @@ -155,8 +155,8 @@ github.com/prometheus/client_model v0.4.0 h1:5lQXD3cAg1OXBf4Wq03gTrXHeaV0TQvGfUo github.com/prometheus/client_model v0.4.0/go.mod h1:oMQmHW1/JoDwqLtg57MGgP/Fb1CJEYF2imWWhWtMkYU= github.com/prometheus/common v0.44.0 h1:+5BrQJwiBB9xsMygAB3TNvpQKOwlkc25LbISbrdOOfY= github.com/prometheus/common v0.44.0/go.mod h1:ofAIvZbQ1e/nugmZGz4/qCb9Ap1VoSTIO7x0VV9VvuY= -github.com/prometheus/procfs v0.10.1 h1:kYK1Va/YMlutzCGazswoHKo//tZVlFpKYh+PymziUAg= -github.com/prometheus/procfs v0.10.1/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM= +github.com/prometheus/procfs v0.11.0 h1:5EAgkfkMl659uZPbe9AS2N68a7Cc1TJbPEuGzFuRbyk= +github.com/prometheus/procfs v0.11.0/go.mod h1:nwNm2aOCAYw8uTR/9bWRREkZFxAUcWzPHWJq+XBB/FM= github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= diff --git a/tools/go.mod b/tools/go.mod index 1c92eb3a2f23..90f7a707031c 100644 --- a/tools/go.mod +++ b/tools/go.mod @@ -8,30 +8,32 @@ require ( github.com/google/go-github/v41 v41.0.0 github.com/goreleaser/nfpm/v2 v2.30.1 github.com/quasilyte/go-consistent v0.0.0-20220429160651-4e46040fbc82 - github.com/reviewdog/reviewdog v0.14.1 + github.com/reviewdog/reviewdog v0.14.2 github.com/stretchr/testify v1.8.4 golang.org/x/oauth2 v0.9.0 golang.org/x/perf v0.0.0-20230427221525-d343f6398b76 - golang.org/x/tools v0.9.3 + golang.org/x/tools v0.10.0 golang.org/x/vuln v0.1.0 mvdan.cc/gofumpt v0.5.0 ) require ( - cloud.google.com/go v0.100.2 // indirect - cloud.google.com/go/compute v1.6.0 // indirect - cloud.google.com/go/datastore v1.6.0 // indirect + cloud.google.com/go v0.110.0 // indirect + cloud.google.com/go/compute v1.18.0 // indirect + cloud.google.com/go/compute/metadata v0.2.3 // indirect + cloud.google.com/go/datastore v1.11.0 // indirect github.com/AlekSi/pointer v1.2.0 // indirect github.com/Masterminds/goutils v1.1.1 // indirect github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.2.1 // indirect github.com/Masterminds/sprig v2.22.0+incompatible // indirect github.com/Microsoft/go-winio v0.5.1 // indirect - github.com/ProtonMail/go-crypto v0.0.0-20210512092938-c05353c2d58c // indirect + github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 // indirect github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794 // indirect github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb // indirect - github.com/bradleyfalzon/ghinstallation/v2 v2.0.4 // indirect + github.com/bradleyfalzon/ghinstallation/v2 v2.5.0 // indirect github.com/cavaliergopher/cpio v1.0.1 // indirect + github.com/cloudflare/circl v1.3.3 // indirect github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/emirpasic/gods v1.12.0 // indirect @@ -46,18 +48,19 @@ require ( github.com/go-toolsmith/pkgload v1.0.0 // indirect github.com/go-toolsmith/typep v1.0.0 // indirect github.com/gobwas/glob v0.2.3 // indirect - github.com/golang-jwt/jwt/v4 v4.0.0 // indirect - github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect - github.com/golang/protobuf v1.5.2 // indirect + github.com/golang-jwt/jwt/v4 v4.5.0 // indirect + github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect + github.com/golang/protobuf v1.5.3 // indirect github.com/google/go-cmp v0.5.9 // indirect - github.com/google/go-github/v39 v39.2.0 // indirect + github.com/google/go-github/v53 v53.1.0 // indirect github.com/google/go-querystring v1.1.0 // indirect github.com/google/uuid v1.3.0 // indirect - github.com/googleapis/gax-go/v2 v2.3.0 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect + github.com/googleapis/gax-go/v2 v2.7.1 // indirect github.com/goreleaser/chglog v0.4.2 // indirect github.com/goreleaser/fileglob v1.3.0 // indirect - github.com/hashicorp/go-cleanhttp v0.5.1 // indirect - github.com/hashicorp/go-retryablehttp v0.6.8 // indirect + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect + github.com/hashicorp/go-retryablehttp v0.7.2 // indirect github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01 // indirect github.com/huandu/xstrings v1.3.2 // indirect github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 // indirect @@ -85,7 +88,7 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/radovskyb/watcher v1.0.7 // indirect github.com/reva2/bitbucket-insights-api v1.0.0 // indirect - github.com/reviewdog/errorformat v0.0.0-20220309155058-b075c45b6d9a // indirect + github.com/reviewdog/errorformat v0.0.0-20230613135350-0fa2b5b7c727 // indirect github.com/reviewdog/go-bitbucket v0.0.0-20201024094602-708c3f6a7de0 // indirect github.com/russross/blackfriday/v2 v2.1.0 // indirect github.com/sajari/fuzzy v1.0.0 // indirect @@ -93,30 +96,29 @@ require ( github.com/spf13/cobra v1.7.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/ulikunitz/xz v0.5.11 // indirect - github.com/vvakame/sdlog v0.0.0-20200409072131-7c0d359efddc // indirect - github.com/xanzy/go-gitlab v0.63.0 // indirect + github.com/vvakame/sdlog v1.1.2 // indirect + github.com/xanzy/go-gitlab v0.85.0 // indirect github.com/xanzy/ssh-agent v0.3.1 // indirect github.com/yuin/goldmark v1.4.13 // indirect gitlab.com/digitalxero/go-conventional-commit v1.0.7 // indirect - go.opencensus.io v0.23.0 // indirect - golang.org/x/build v0.0.0-20200616162219-07bebbe343e9 // indirect + go.opencensus.io v0.24.0 // indirect + golang.org/x/build v0.0.0-20230616225607-2f704e01b36f // indirect golang.org/x/crypto v0.10.0 // indirect - golang.org/x/exp v0.0.0-20230212135524-a684f29349b6 // indirect - golang.org/x/mod v0.10.0 // indirect + golang.org/x/exp v0.0.0-20230510235704-dd950f8aeaea // indirect + golang.org/x/mod v0.11.0 // indirect golang.org/x/net v0.11.0 // indirect - golang.org/x/sync v0.2.0 // indirect + golang.org/x/sync v0.3.0 // indirect golang.org/x/sys v0.9.0 // indirect golang.org/x/term v0.9.0 // indirect golang.org/x/text v0.10.0 // indirect - golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect - golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect - google.golang.org/api v0.74.0 // indirect + golang.org/x/time v0.3.0 // indirect + golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect + google.golang.org/api v0.114.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9 // indirect - google.golang.org/grpc v1.45.0 // indirect - google.golang.org/protobuf v1.28.0 // indirect + google.golang.org/genproto v0.0.0-20230320184635-7606e756e683 // indirect + google.golang.org/grpc v1.53.0 // indirect + google.golang.org/protobuf v1.30.0 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect - gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect mvdan.cc/sh/v3 v3.6.0 // indirect ) diff --git a/tools/go.sum b/tools/go.sum index 44b08e66351d..9d59c87f344e 100644 --- a/tools/go.sum +++ b/tools/go.sum @@ -14,36 +14,24 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs= cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc= cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY= -cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI= -cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk= -cloud.google.com/go v0.78.0/go.mod h1:QjdrLG0uq+YwhjoVOLsS1t7TW8fs36kLs4XO5R5ECHg= -cloud.google.com/go v0.79.0/go.mod h1:3bzgcEeQlzbuEAYu4mrWhKqWjmpprinYgKJLgKHnbb8= -cloud.google.com/go v0.81.0/go.mod h1:mk/AM35KwGk/Nm2YSeZbxXdrNK3KZOYHmLkOqC2V6E0= -cloud.google.com/go v0.83.0/go.mod h1:Z7MJUsANfY0pYPdw0lbnivPx4/vhy/e2FEkSkF7vAVY= -cloud.google.com/go v0.84.0/go.mod h1:RazrYuxIK6Kb7YrzzhPoLmCVzl7Sup4NrbKPg8KHSUM= -cloud.google.com/go v0.87.0/go.mod h1:TpDYlFy7vuLzZMMZ+B6iRiELaY7z/gJPaqbMx6mlWcY= -cloud.google.com/go v0.90.0/go.mod h1:kRX0mNRHe0e2rC6oNakvwQqzyDmg57xJ+SZU1eT2aDQ= -cloud.google.com/go v0.93.3/go.mod h1:8utlLll2EF5XMAV15woO4lSbWQlk8rer9aLOfLh7+YI= -cloud.google.com/go v0.94.1/go.mod h1:qAlAugsXlC+JWO+Bke5vCtc9ONxjQT3drlTTnAplMW4= -cloud.google.com/go v0.97.0/go.mod h1:GF7l59pYBVlXQIBLx3a761cZ41F9bBH3JUlihCt2Udc= -cloud.google.com/go v0.99.0/go.mod h1:w0Xx2nLzqWJPuozYQX+hFfCSI8WioryfRDzkoI/Y2ZA= -cloud.google.com/go v0.100.2 h1:t9Iw5QH5v4XtlEQaCtUY7x6sCABps8sW0acw7e2WQ6Y= -cloud.google.com/go v0.100.2/go.mod h1:4Xra9TjzAeYHrl5+oeLlzbM2k3mjVhZh4UqTZ//w99A= +cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys= +cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg= cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= -cloud.google.com/go/compute v0.1.0/go.mod h1:GAesmwr110a34z04OlxYkATPBEfVhkymfTBXtfbBFow= -cloud.google.com/go/compute v1.3.0/go.mod h1:cCZiE1NHEtai4wiufUhW8I8S1JKkAnhnQJWM7YD99wM= -cloud.google.com/go/compute v1.5.0/go.mod h1:9SMHyhJlzhlkJqrPAc839t2BZFTSk6Jdj6mkzQJeu0M= -cloud.google.com/go/compute v1.6.0 h1:XdQIN5mdPTSBVwSIVDuY5e8ZzVAccsHvD3qTEz4zIps= -cloud.google.com/go/compute v1.6.0/go.mod h1:T29tfhtVbq1wvAPo0E3+7vhgmkOYeXjhFvz/FMzPu0s= +cloud.google.com/go/compute v1.18.0 h1:FEigFqoDbys2cvFkZ9Fjq4gnHBP55anJ0yQyau2f9oY= +cloud.google.com/go/compute v1.18.0/go.mod h1:1X7yHxec2Ga+Ss6jPyjxRxpu2uu7PLgsOVXvgU0yacs= +cloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k= +cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= +cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= -cloud.google.com/go/datastore v1.6.0 h1:wZaHIqu1tebvGRYhVgcfNX6jN2q638OGO23JyJckxuI= -cloud.google.com/go/datastore v1.6.0/go.mod h1:q3ZJj1GMQRdU0OCv5XXpCqfLqHHZnI5zcumkvuYDmHI= +cloud.google.com/go/datastore v1.11.0 h1:iF6I/HaLs3Ado8uRKMvZRvF/ZLkWaWE9i8AiHzbC774= +cloud.google.com/go/datastore v1.11.0/go.mod h1:TvGxBIHCS50u8jzG+AW/ppf87v1of8nwzFNgEZU1D3c= +cloud.google.com/go/longrunning v0.4.1 h1:v+yFJOfKC3yZdY6ZUI933pIYdhyhV8S3NpWrXWmg7jM= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= @@ -75,10 +63,8 @@ github.com/Masterminds/sprig v2.22.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuN github.com/Microsoft/go-winio v0.5.0/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84= github.com/Microsoft/go-winio v0.5.1 h1:aPJp2QD7OOrhO5tQXqQoGSJc+DjDtWTGLOmNyAm6FgY= github.com/Microsoft/go-winio v0.5.1/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84= -github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c= -github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/ProtonMail/go-crypto v0.0.0-20210512092938-c05353c2d58c h1:bNpaLLv2Y4kslsdkdCwAYu8Bak1aGVtxwi8Z/wy4Yuo= -github.com/ProtonMail/go-crypto v0.0.0-20210512092938-c05353c2d58c/go.mod h1:z4/9nQmJSSwwds7ejkxaJwO37dru3geImFUdJlaLzQo= +github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 h1:wPbRQzjjwFc0ih8puEVAOFGELsn1zoIIYdxvML7mDxA= +github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8/go.mod h1:I0gYDMZ6Z5GRU7l58bNFSkPTFN6Yl12dsUlAZ8xy98g= github.com/ProtonMail/go-mime v0.0.0-20190923161245-9b5a4261663a h1:W6RrgN/sTxg1msqzFFb+G80MFmpjMw61IU+slm+wln4= github.com/ProtonMail/gopenpgp/v2 v2.2.2 h1:u2m7xt+CZWj88qK1UUNBoXeJCFJwJCZ/Ff4ymGoxEXs= github.com/aclements/go-gg v0.0.0-20170118225347-6dbb4e4fefb0/go.mod h1:55qNq4vcpkIuHowELi5C8e+1yUHtoLoOUR9QU5j7Tes= @@ -88,41 +74,33 @@ github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3 github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw= github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs= github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs= -github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA= github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c= -github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= +github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= -github.com/aws/aws-sdk-go v1.30.15/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0= github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb h1:m935MPodAbYS46DG4pJSv7WO+VECIWUQ7OJYSoTrMh4= github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb/go.mod h1:PkYb9DJNAwrSvRx5DYA+gUcOIgTGVMNkfSCbZM8cWpI= github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= github.com/boombuler/barcode v1.0.1/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= -github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g= -github.com/bradleyfalzon/ghinstallation/v2 v2.0.4 h1:tXKVfhE7FcSkhkv0UwkLvPDeZ4kz6OXd0PKPlFqf81M= -github.com/bradleyfalzon/ghinstallation/v2 v2.0.4/go.mod h1:B40qPqJxWE0jDZgOR1JmaMy+4AY1eBP+IByOvqyAKp0= +github.com/bradleyfalzon/ghinstallation/v2 v2.5.0 h1:yaYcGQ7yEIGbsJfW/9z7v1sLiZg/5rSNNXwmMct5XaE= +github.com/bradleyfalzon/ghinstallation/v2 v2.5.0/go.mod h1:amcvPQMrRkWNdueWOjPytGL25xQGzox7425qMgzo+Vo= +github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0= +github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0= github.com/caarlos0/go-rpmutils v0.2.1-0.20211112020245-2cd62ff89b11 h1:IRrDwVlWQr6kS1U8/EtyA1+EHcc4yl8pndcqXWrEamg= github.com/caarlos0/testfs v0.4.4 h1:3PHvzHi5Lt+g332CiShwS8ogTgS3HjrmzZxCm6JCDr8= github.com/caarlos0/testfs v0.4.4/go.mod h1:bRN55zgG4XCUVVHZCeU+/Tz1Q6AxEJOEJTliBy+1DMk= github.com/cavaliergopher/cpio v1.0.1 h1:KQFSeKmZhv0cr+kawA3a0xTQCU4QxXF1vhU7P7av2KM= github.com/cavaliergopher/cpio v1.0.1/go.mod h1:pBdaqQjnvXxdS/6CvNDwIANIFSP0xRKI16PX4xejRQc= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= -github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= -github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I= +github.com/cloudflare/circl v1.3.3 h1:fE/Qz0QdIGqeWfnwq0RE0R7MI51s0M2E4Ga9kq5AEMs= +github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA= github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= -github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= -github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= -github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI= -github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= -github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w= github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= @@ -135,11 +113,6 @@ github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3 github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= -github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po= -github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= -github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= -github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ= -github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs= github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw= @@ -147,10 +120,8 @@ github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k= github.com/fogleman/gg v1.3.0/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k= github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY= -github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= -github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= -github.com/gliderlabs/ssh v0.2.2 h1:6zsha5zo/TWhRhwqCD3+EarCAgZ2yN28ipRnGPnwkI0= github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= +github.com/gliderlabs/ssh v0.3.3 h1:mBQ8NiOgDkINJrZtoizkC3nDNYgSaWtxyem6S2XHBtA= github.com/go-fonts/dejavu v0.1.0/go.mod h1:4Wt4I4OU2Nq9asgDCteaAaWZOV24E+0/Pwo0gppep4g= github.com/go-fonts/latin-modern v0.2.0/go.mod h1:rQVLdDMK+mK1xscDwsqM5J8U2jrRa3T0ecnM9pNujks= github.com/go-fonts/liberation v0.1.1/go.mod h1:K6qoJYypsmfVjWg8KOVDQhLc8UDgIK2HYqyqAO9z7GY= @@ -172,7 +143,6 @@ github.com/go-latex/latex v0.0.0-20210118124228-b3d85cf34e07/go.mod h1:CO1AlKB2C github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81/go.mod h1:SX0U8uGpxhq9o2S/CELCSUxEWWAuoCUcVCQWv7G2OCk= github.com/go-pdf/fpdf v0.5.0/go.mod h1:HzcnA+A23uwogo0tp9yU+l3V+KXhiESpt1PMayhOh5M= github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w= -github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43I= github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE= github.com/go-task/task/v3 v3.26.0 h1:NfYktTh/XXJUFtUFdle+BZFtvGlmhk4Q7YfoeOBLPlE= @@ -191,14 +161,15 @@ github.com/go-toolsmith/typep v1.0.0 h1:zKymWyA1TRYvqYrYDrfEMZULyrhcnGY3x7LDKU2X github.com/go-toolsmith/typep v1.0.0/go.mod h1:JSQCQMUPdRlMZFswiq3TGpNp1GMktqkR2Ns5AIQkATU= github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= -github.com/golang-jwt/jwt/v4 v4.0.0 h1:RAqyYixv1p7uEnocuy8P1nru5wprCh/MH2BIlW5z5/o= -github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg= +github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= +github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY= github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= @@ -206,8 +177,6 @@ github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4= -github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8= -github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= @@ -223,10 +192,9 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx4u74HPM= -github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac/go.mod h1:P32wAyui1PQ58Oce/KYkOqQv8cVw1zAapXOl+dRFGbc= github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82/go.mod h1:PxC8OnwL11+aosOB5+iEPoV3picfs8tUpkVd0pDo+Kg= github.com/gonum/internal v0.0.0-20181124074243-f884aa714029/go.mod h1:Pu4dmpkhSyOzRwuXkOgAvijx4o+4YMUJJo9OvPYMkks= @@ -244,24 +212,20 @@ github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/ github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE= +github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= -github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ= -github.com/google/go-github/v39 v39.2.0 h1:rNNM311XtPOz5rDdsJXAp2o8F67X9FnROXTvto3aSnQ= -github.com/google/go-github/v39 v39.2.0/go.mod h1:C1s8C5aCC9L+JXIYpJM5GYytdX52vC1bLvHEF1IhBrE= github.com/google/go-github/v41 v41.0.0 h1:HseJrM2JFf2vfiZJ8anY2hqBjdfY1Vlj/K27ueww4gg= github.com/google/go-github/v41 v41.0.0/go.mod h1:XgmCA5H323A9rtgExdTcnDkcqp6S30AVACCBDOonIxg= -github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck= +github.com/google/go-github/v53 v53.0.0/go.mod h1:XhFRObz+m/l+UCm9b7KSIC3lT3NWSXGt7mOsAWEloao= +github.com/google/go-github/v53 v53.1.0 h1:mKJnR9lzZwD1fvbp27aK1i6rxyAbycWsXlN+r9JKPqM= +github.com/google/go-github/v53 v53.1.0/go.mod h1:XhFRObz+m/l+UCm9b7KSIC3lT3NWSXGt7mOsAWEloao= github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8= github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU= github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= -github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= -github.com/google/martian/v3 v3.2.1/go.mod h1:oBOf6HBosgwRXnUGWUB05QECsc6uvmMiJ3+6W4l/CUk= github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= @@ -269,13 +233,6 @@ github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hf github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= -github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/safehtml v0.0.2/go.mod h1:L4KWwDsUJdECRAEpZoBn3O64bQaywRscowZjJAzjHnU= @@ -283,14 +240,13 @@ github.com/google/subcommands v1.0.1/go.mod h1:ZjhPrFU+Olkh9WazFPsl27BQ4UPiG37m3 github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.2.3 h1:yk9/cqRKtT9wXZSsRH9aurXEpJX+U6FLtpYTdC3R06k= +github.com/googleapis/enterprise-certificate-proxy v0.2.3/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k= github.com/googleapis/gax-go v0.0.0-20161107002406-da06d194a00e/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= -github.com/googleapis/gax-go/v2 v2.1.0/go.mod h1:Q3nei7sK6ybPYH7twZdmQpAd1MKb7pfu6SK+H1/DsU0= -github.com/googleapis/gax-go/v2 v2.1.1/go.mod h1:hddJymUZASv3XPyGkUpKj8pPO47Rmb0eJc8R6ouapiM= -github.com/googleapis/gax-go/v2 v2.2.0/go.mod h1:as02EH8zWkzwUoLbBaFeQ+arQaj/OthfcblKl4IGNaM= -github.com/googleapis/gax-go/v2 v2.3.0 h1:nRJtk3y8Fm770D42QV6T90ZnvFZyk7agSo3Q+Z9p3WI= -github.com/googleapis/gax-go/v2 v2.3.0/go.mod h1:b8LNqSzNabLiUpXKkY7HAR5jr6bIT99EXz9pXxye9YM= +github.com/googleapis/gax-go/v2 v2.7.1 h1:gF4c0zjUP2H/s/hEGyLA3I0fA2ZWjzYiONAD6cvPr8A= +github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38/qKbhSAKP6QI= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8= github.com/goreleaser/chglog v0.4.2 h1:afmbT1d7lX/q+GF8wv3a1Dofs2j/Y9YkiCpGemWR6mI= github.com/goreleaser/chglog v0.4.2/go.mod h1:u/F03un4hMCQrp65qSWCkkC6T+G7YLKZ+AM2mITE47s= @@ -298,14 +254,12 @@ github.com/goreleaser/fileglob v1.3.0 h1:/X6J7U8lbDpQtBvGcwwPS6OpzkNVlVEsFUVRx9+ github.com/goreleaser/fileglob v1.3.0/go.mod h1:Jx6BoXv3mbYkEzwm9THo7xbr5egkAraxkGorbJb4RxU= github.com/goreleaser/nfpm/v2 v2.30.1 h1:mn3nrLRvCRW/SO86z2IBTctU6BZSXKkyRR8Zkpw344Y= github.com/goreleaser/nfpm/v2 v2.30.1/go.mod h1:2zdXNdSziz4veeXBVIcLE5Y8oiycm6BOSfflz2UhWGk= -github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= -github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw= -github.com/hashicorp/go-cleanhttp v0.5.1 h1:dH3aiDG9Jvb5r5+bYHsikaOUIpcM0xvgMXVoDkXMzJM= -github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= +github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= +github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= -github.com/hashicorp/go-retryablehttp v0.6.8 h1:92lWxgpa+fF3FozM4B3UZtHZMJX8T5XT+TFdCxsPyWs= -github.com/hashicorp/go-retryablehttp v0.6.8/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY= +github.com/hashicorp/go-retryablehttp v0.7.2 h1:AcYqCvkpalPnPF2pn0KamgwamS42TqUDDYFRKq/RAd0= +github.com/hashicorp/go-retryablehttp v0.7.2/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01 h1:HiJF8Mek+I7PY0Bm+SuhkwaAZSZP83sw6rrTMrgZ0io= @@ -317,7 +271,6 @@ github.com/huandu/xstrings v1.3.2/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 h1:i462o439ZjprVSFSZLZxcsoAe592sZB1rci2Z8j4wdk= github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0/go.mod h1:N0Wam8K1arqPXNWjMo21EXnBPOPp36vB07FNRdD2geA= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= -github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/imdario/mergo v0.3.9/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4= github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY= @@ -327,9 +280,7 @@ github.com/invopop/jsonschema v0.7.0 h1:2vgQcBz1n256N+FpX3Jq7Y17AjYt46Ig3zIWyy77 github.com/invopop/jsonschema v0.7.0/go.mod h1:O9uiLokuu0+MGFlyiaqtWxwqJm41/+8Nj0lD7A36YH0= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= -github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU= github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= -github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik= github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= @@ -349,7 +300,6 @@ github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= @@ -399,13 +349,12 @@ github.com/radovskyb/watcher v1.0.7 h1:AYePLih6dpmS32vlHfhCeli8127LzkIgwJGcwwe8t github.com/radovskyb/watcher v1.0.7/go.mod h1:78okwvY5wPdzcb1UYnip1pvrZNIVEIh/Cm+ZuvsUYIg= github.com/reva2/bitbucket-insights-api v1.0.0 h1:lpQ/Q7OmnG04w/EM77piOwZBxP41PeTlbytXxVrnplA= github.com/reva2/bitbucket-insights-api v1.0.0/go.mod h1:pLs+ki3MKUntrPryxaGIvpRLiEtBhwfJ/uvxQIMfqHU= -github.com/reviewdog/errorformat v0.0.0-20220309155058-b075c45b6d9a h1:HIL+jTKsWmNT5WoTNwHQ0jUNJpFOmgeHLOsHMZInrF8= -github.com/reviewdog/errorformat v0.0.0-20220309155058-b075c45b6d9a/go.mod h1:AqhrP0G7F9YRROF10JQwdd4cNO8bdm6bY6KzcOc3Cp8= +github.com/reviewdog/errorformat v0.0.0-20230613135350-0fa2b5b7c727 h1:IMel3dOWPESKUzIoobTkhIJSWoXDv53Gkq+yynKkK/A= +github.com/reviewdog/errorformat v0.0.0-20230613135350-0fa2b5b7c727/go.mod h1:AqhrP0G7F9YRROF10JQwdd4cNO8bdm6bY6KzcOc3Cp8= github.com/reviewdog/go-bitbucket v0.0.0-20201024094602-708c3f6a7de0 h1:XZ60Bp2UqwaJ6fDQExoFVrgs4nIzwBCy9ct6GCj9hH8= github.com/reviewdog/go-bitbucket v0.0.0-20201024094602-708c3f6a7de0/go.mod h1:5JbWAMFyq9hbISZawRyIe7QTcLaptvCIvmZnYo+1VvA= -github.com/reviewdog/reviewdog v0.14.1 h1:+B2340ddsSC+tbSSTxb5oC24YPMLzQAlt4kVAmNS/3o= -github.com/reviewdog/reviewdog v0.14.1/go.mod h1:408Nr2GPWc7Qozz0X1RZ/3b7EVQIPJ6XF6BssRJ8ZKY= -github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ= +github.com/reviewdog/reviewdog v0.14.2 h1:0az+EeJZA+cqrQkpP2VOBU+xpcEcCGVuzNWp9QUqlKI= +github.com/reviewdog/reviewdog v0.14.2/go.mod h1:M45lrCOcy2LIqsi1Oj+K5xe38Qs2UCQC7MnLwkZt5yA= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk= @@ -421,29 +370,29 @@ github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= github.com/smartystreets/goconvey v1.7.2 h1:9RBaZCeXEQ3UselpuwUQHltGVXvdwm6cv1hgR6gDIPg= -github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= -github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.3.1-0.20190311161405-34c6fa2dc709/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= -github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8= github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= -github.com/vvakame/sdlog v0.0.0-20200409072131-7c0d359efddc h1:El7LEavRpa49dYFE9ezO8aQxQn5E7u7eQkFsaXsoQAY= -github.com/vvakame/sdlog v0.0.0-20200409072131-7c0d359efddc/go.mod h1:MmhrKtbECoUJTctfak+MnOFoJ9XQqYZ7chcwV9O7v3I= -github.com/xanzy/go-gitlab v0.63.0 h1:a9fXpKWykUS6dowapFej/2Wjf4aOAEFC1q2ZIcz4IpI= -github.com/xanzy/go-gitlab v0.63.0/go.mod h1:F0QEXwmqiBUxCgJm8fE9S+1veX4XC9Z4cfaAbqwk4YM= +github.com/vvakame/sdlog v1.1.2 h1:hWV5ESsYMn/vubf1uFWdkMJomrQucAU+9gc/S3A/JE8= +github.com/vvakame/sdlog v1.1.2/go.mod h1:jQWcS5V7V6xNIpNUEdLRwWlub0SYicmkE9c6N6FgJ44= +github.com/xanzy/go-gitlab v0.85.0 h1:E/wjnsd/mM5kV6O9y5+i6zxjx+wfAwa97sgcT1ETNwk= +github.com/xanzy/go-gitlab v0.85.0/go.mod h1:5ryv+MnpZStBH8I/77HuQBsMbBGANtVpLWC15qOjWAw= github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4= github.com/xanzy/ssh-agent v0.3.1 h1:AmzO1SSWxw73zxFZPRwaMN1MohDw8UyHnmuxyceTEGo= github.com/xanzy/ssh-agent v0.3.1/go.mod h1:QIE4lCeL7nkC25x+yA3LBIYfwCc1TFziCtG7cBAac6w= @@ -451,8 +400,6 @@ github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofm github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.13 h1:fVcFKWvrslecOb/tg+Cc05dkeYx540o0FuFt3nUVDoE= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= @@ -463,13 +410,10 @@ go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= -go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= -go.opencensus.io v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M= -go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= -go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI= -go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE= -golang.org/x/build v0.0.0-20200616162219-07bebbe343e9 h1:SgmspiKqqI4Du0T87bPBEezUSzVOKhKDgconpLrfyuc= -golang.org/x/build v0.0.0-20200616162219-07bebbe343e9/go.mod h1:ia5pRNoJUuxRhXkmwkySu4YBTbXHSKig2ie6daQXihg= +go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= +go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= +golang.org/x/build v0.0.0-20230616225607-2f704e01b36f h1:+DJe+d0u+/q3cXpk/n2r4D7MmyECuvojksgwIkifUkw= +golang.org/x/build v0.0.0-20230616225607-2f704e01b36f/go.mod h1:YAvpYk9SQ69z5Nt7IfnPJCr4/1XKsyE7gI/x9nKRqc4= golang.org/x/crypto v0.0.0-20190219172222-a4c6cb3142f2/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -477,10 +421,11 @@ golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= +golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM= golang.org/x/crypto v0.10.0/go.mod h1:o4eNf7Ede1fv+hwOwZsTHl9EsPFO6q6ZvYR8vYfY45I= golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -497,8 +442,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0 golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= -golang.org/x/exp v0.0.0-20230212135524-a684f29349b6 h1:Ic9KukPQ7PegFzHckNiMTQXGgEszA7mY2Fn4ZMtnMbw= -golang.org/x/exp v0.0.0-20230212135524-a684f29349b6/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= +golang.org/x/exp v0.0.0-20230510235704-dd950f8aeaea h1:vLCWI/yYrdEHyN2JzIzPO3aaQJHQdp89IZBA/+azVC4= +golang.org/x/exp v0.0.0-20230510235704-dd950f8aeaea/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= golang.org/x/exp/typeparams v0.0.0-20221208152030-732eee02a75a h1:Jw5wfR+h9mnIYH+OtGT2im5wV1YGGDora5vTv/aa5bE= golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= @@ -521,8 +466,6 @@ golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHl golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= -golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= -golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= @@ -531,12 +474,11 @@ golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY= -golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= -golang.org/x/mod v0.10.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU= +golang.org/x/mod v0.11.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -563,19 +505,14 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= -golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc= -golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= -golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= -golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= -golang.org/x/net v0.0.0-20220325170049-de3da57026de/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU= golang.org/x/net v0.11.0/go.mod h1:2L/ixqYpgIVXmeoSA/4Lu7BzTG4KIyPIryS4IsOd1oQ= golang.org/x/oauth2 v0.0.0-20170207211851-4464e7848382/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -585,21 +522,10 @@ golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= -golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= +golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE= golang.org/x/oauth2 v0.9.0 h1:BPpt2kU7oMRq3kCHAA1tbSEshXRw1LpG2ztgDwrzuAs= golang.org/x/oauth2 v0.9.0/go.mod h1:qYgFZaFiu6Wg24azG8bdV52QJXJGbZzIIsRCdVKzbLw= -golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw= golang.org/x/perf v0.0.0-20230427221525-d343f6398b76 h1:cPGZx8Liyx5Pq/yX80/6WMKe2yidT0xvVCQBOGa8WHU= golang.org/x/perf v0.0.0-20230427221525-d343f6398b76/go.mod h1:UBKtEnL8aqnd+0JHqZ+2qoMDwtuy6cYhhKNoHLBiTQc= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -610,11 +536,11 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI= -golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E= +golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190221075227-b4e8571b14e0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -642,42 +568,31 @@ golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210304124612-50617c2ba197/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210908233432-aa78b53d3365/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211210111614-af8b64212486/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s= golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= +golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.9.0 h1:GRRCnKYhdQrD8kfRAdQ6Zcw1P0OcELxGLKJvtjVMZ28= golang.org/x/term v0.9.0/go.mod h1:M6DEAAIenWoTxdKrOltXcmDY3rSplQUkrvaDU5FcQyo= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -685,16 +600,20 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58= golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= +golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190110163146-51295c7ec13a/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -732,7 +651,6 @@ golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapK golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= -golang.org/x/tools v0.0.0-20200406213809-066fd1390ee0/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= @@ -740,27 +658,19 @@ golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roY golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= -golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE= -golang.org/x/tools v0.0.0-20201110124207-079ba7bd75cd/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20201201161351-ac6f37ff4c2a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= -golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= -golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.10/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E= -golang.org/x/tools v0.9.3 h1:Gn1I8+64MsuTb/HpH+LmQtNas23LhUVr3rYZ0eKuaMM= -golang.org/x/tools v0.9.3/go.mod h1:owI94Op576fPu3cIGQeHs3joujW/2Oc6MtlxbF5dfNc= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/tools v0.10.0 h1:tvDr/iQoUqNdohiYm0LmmKcBk+q86lb9EprIUFhHHGg= +golang.org/x/tools v0.10.0/go.mod h1:UJwyiVBsOA2uwvK/e5OY3GTpDUJriEd+/YlqAwLPmyM= golang.org/x/vuln v0.1.0 h1:9GRdj6wAIkDrsMevuolY+SXERPjQPp2P1ysYA0jpZe0= golang.org/x/vuln v0.1.0/go.mod h1:/YuzZYjGbwB8y19CisAppfyw3uTZnuCz3r+qgx/QRzU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= +golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= gonum.org/v1/gonum v0.0.0-20180816165407-929014505bf4/go.mod h1:Y+Yx5eoAFn32cQvJDxZx5Dpnq+c3wtXuadVZAcxbbBo= gonum.org/v1/gonum v0.8.2/go.mod h1:oe/vMfY3deqTw+1EZJhuvEW2iwGF1bW9wwu7XCu0+v0= gonum.org/v1/gonum v0.9.3/go.mod h1:TZumC3NeyVQskjXqmyWt4S3bINhy7B4eYwW69EbyX+0= @@ -785,26 +695,8 @@ google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0M google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM= google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc= -google.golang.org/api v0.35.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg= -google.golang.org/api v0.36.0/go.mod h1:+z5ficQTmoYpPn8LCUNVpK5I7hwkpjbcgqA7I34qYtE= -google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjRCQ8= -google.golang.org/api v0.41.0/go.mod h1:RkxM5lITDfTzmyKFPt+wGrCJbVfniCr2ool8kTBzRTU= -google.golang.org/api v0.43.0/go.mod h1:nQsDGjRXMo4lvh5hP0TKqF244gqhGcr/YSIykhUk/94= -google.golang.org/api v0.47.0/go.mod h1:Wbvgpq1HddcWVtzsVLyfLp8lDg6AA241LmgIL59tHXo= -google.golang.org/api v0.48.0/go.mod h1:71Pr1vy+TAZRPkPs/xlCf5SsU8WjuAWv1Pfjbtukyy4= -google.golang.org/api v0.50.0/go.mod h1:4bNT5pAuq5ji4SRZm+5QIkjny9JAyVD/3gaSihNefaw= -google.golang.org/api v0.51.0/go.mod h1:t4HdrdoNgyN5cbEfm7Lum0lcLDLiise1F8qDKX00sOU= -google.golang.org/api v0.54.0/go.mod h1:7C4bFFOvVDGXjfDTAsgGwDgAxRDeQ4X8NvUedIt6z3k= -google.golang.org/api v0.55.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= -google.golang.org/api v0.56.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= -google.golang.org/api v0.57.0/go.mod h1:dVPlbZyBo2/OjBpmvNdpn2GRm6rPy75jyU7bmhdrMgI= -google.golang.org/api v0.61.0/go.mod h1:xQRti5UdCmoCEqFxcz93fTl338AVqDgyaDRuOZ3hg9I= -google.golang.org/api v0.63.0/go.mod h1:gs4ij2ffTRXwuzzgJl/56BdwJaA194ijkfn++9tDuPo= -google.golang.org/api v0.67.0/go.mod h1:ShHKP8E60yPsKNw/w8w+VYaj9H6buA5UqDp8dhbQZ6g= -google.golang.org/api v0.70.0/go.mod h1:Bs4ZM2HGifEvXwd50TtW70ovgJffJYw2oRCOFU/SkfA= -google.golang.org/api v0.71.0/go.mod h1:4PyU6e6JogV1f9eA4voyrTY2batOLdgZ5qZ5HOCc4j8= -google.golang.org/api v0.74.0 h1:ExR2D+5TYIrMphWgs5JCgwRhEDlPDXXrLwHHMgPHTXE= -google.golang.org/api v0.74.0/go.mod h1:ZpfMZOVRMywNyvJFeqL9HRWBgAuRfSjJFpe9QtRRyDs= +google.golang.org/api v0.114.0 h1:1xQPji6cO2E2vLiI+C/XiFAnsn1WV3mjaEwGLhi3grE= +google.golang.org/api v0.114.0/go.mod h1:ifYI2ZsFK6/uGddGfAD5BMxlnkBqCmqHSDUVi45N5Yg= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -836,53 +728,14 @@ google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfG google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= -google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA= google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201109203340-2640f1f9cdfb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210303154014-9728d6b83eeb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210310155132-4ce2db91004e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210319143718-93e7006c17a6/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210402141018-6c239bbf2bb1/go.mod h1:9lPAdzaEmUacj36I+k7YKbEc5CXzPIeORRgDAUOu28A= -google.golang.org/genproto v0.0.0-20210513213006-bf773b8c8384/go.mod h1:P3QM42oQyzQSnHPnZ/vqoCdDmzH28fzWByN9asMeM8A= -google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= -google.golang.org/genproto v0.0.0-20210604141403-392c879c8b08/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= -google.golang.org/genproto v0.0.0-20210608205507-b6d2f5bf0d7d/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= -google.golang.org/genproto v0.0.0-20210624195500-8bfb893ecb84/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24= -google.golang.org/genproto v0.0.0-20210713002101-d411969a0d9a/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k= -google.golang.org/genproto v0.0.0-20210716133855-ce7ef5c701ea/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k= -google.golang.org/genproto v0.0.0-20210728212813-7823e685a01f/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= -google.golang.org/genproto v0.0.0-20210805201207-89edb61ffb67/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= -google.golang.org/genproto v0.0.0-20210813162853-db860fec028c/go.mod h1:cFeNkxwySK631ADgubI+/XFU/xp8FD5KIVV4rj8UC5w= -google.golang.org/genproto v0.0.0-20210821163610-241b8fcbd6c8/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= -google.golang.org/genproto v0.0.0-20210828152312-66f60bf46e71/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= -google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= -google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= -google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= -google.golang.org/genproto v0.0.0-20210924002016-3dee208752a0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20211206160659-862468c7d6e0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20211221195035-429b39de9b1c/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20220126215142-9970aeb2e350/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20220207164111-0872dc986b00/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20220218161850-94dd64e39d7c/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= -google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= -google.golang.org/genproto v0.0.0-20220304144024-325a89244dc8/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= -google.golang.org/genproto v0.0.0-20220310185008-1973136f34c6/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= -google.golang.org/genproto v0.0.0-20220324131243-acbaeb5b85eb/go.mod h1:hAL49I2IFola2sVEjAn7MEwsja0xp51I0tlGAf9hz4E= -google.golang.org/genproto v0.0.0-20220407144326-9054f6ed7bac/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= -google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9 h1:XGQ6tc+EnM35IAazg4y6AHmUg4oK8NXsXaILte1vRlk= -google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20230320184635-7606e756e683 h1:khxVcsk/FhnzxMKOyD+TDGwjbEOpcPuIpmafPGFmhMA= +google.golang.org/genproto v0.0.0-20230320184635-7606e756e683/go.mod h1:NWraEVixdDnqcqQ30jipen1STv2r/n24Wb7twVTGR4s= google.golang.org/grpc v0.0.0-20170208002647-2a6bf6142e96/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= @@ -896,24 +749,9 @@ google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKa google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk= google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= -google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= -google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0= google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= -google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8= -google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= -google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= -google.golang.org/grpc v1.36.1/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= -google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= -google.golang.org/grpc v1.37.1/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= -google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= -google.golang.org/grpc v1.39.0/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= -google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= -google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= -google.golang.org/grpc v1.40.1/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= -google.golang.org/grpc v1.44.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= -google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M= -google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ= -google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw= +google.golang.org/grpc v1.53.0 h1:LAv2ds7cmFV/XTS3XG1NneeENYrXGmorPxsBbptIjNc= +google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= @@ -926,27 +764,22 @@ google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGj google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw= google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= +google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU= gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= -gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME= gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= -gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From c68c93681c600bd7f62e2a87e9d3b43bc5820061 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 15:16:30 +0400 Subject: [PATCH 09/46] Fix fuzzing corpus collection (#2879) --- .github/workflows/go.yml | 17 +++++++++++++++-- cmd/fuzztool/fuzztool.go | 3 ++- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 62825511caeb..cd7775f02b3e 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -379,20 +379,33 @@ jobs: # `secrets` are not supported in `if`, so we have to do this trick with environment variable: # https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability # https://github.com/actions/runner/issues/520 - - name: Push corpus - if: ${{ always() && env.KEY != '' }} + + - name: Push corpus if possible + if: always() && env.KEY != '' env: KEY: ${{ secrets.FUZZ_CORPUS_DEPLOY_KEY }} run: | git config user.name ferretdb-bot git config user.email ferretdb-bot@ferretdb.io git add --all . + git diff-index --patch git diff-index --quiet HEAD || git commit --message 'Update corpus' git fetch git rebase origin/main git push working-directory: fuzz-corpus + - name: Upload corpus if push is not possible + if: failure() && env.KEY == '' + env: + KEY: ${{ secrets.FUZZ_CORPUS_DEPLOY_KEY }} + uses: actions/upload-artifact@v3 + with: + name: fuzz-corpus + path: fuzz-corpus.zip + retention-days: 1 + if-no-files-found: error + - name: Collect Linux logs if: failure() run: | diff --git a/cmd/fuzztool/fuzztool.go b/cmd/fuzztool/fuzztool.go index 5ff5b68b5861..67d2e9f9beb1 100644 --- a/cmd/fuzztool/fuzztool.go +++ b/cmd/fuzztool/fuzztool.go @@ -16,6 +16,7 @@ package main import ( "bytes" + "encoding/hex" "io" "io/fs" "os" @@ -71,7 +72,7 @@ func collectFiles(root string, logger *zap.SugaredLogger) (map[string]struct{}, } // skip other files - if len(info.Name()) != 64 { + if _, err = hex.DecodeString(info.Name()); err != nil { return nil } From 24683b652ada4c0799f0246a56f56fcc15f871fd Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Wed, 21 Jun 2023 21:33:59 +0900 Subject: [PATCH 10/46] Update supported commands (#2876) --- website/docs/reference/supported-commands.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/website/docs/reference/supported-commands.md b/website/docs/reference/supported-commands.md index a555eeb40efb..0ba79b1ddcfe 100644 --- a/website/docs/reference/supported-commands.md +++ b/website/docs/reference/supported-commands.md @@ -304,7 +304,7 @@ Related [issue](https://github.com/FerretDB/FerretDB/issues/1917). | Stage | Status | Comments | | -------------------- | ------ | --------------------------------------------------------- | -| `$addFields` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1413) | +| `$addFields` | ⚠️ | [Issue](https://github.com/FerretDB/FerretDB/issues/1413) | | `$bucket` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1414) | | `$bucketAuto` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1414) | | `$changeStream` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1415) | @@ -336,13 +336,13 @@ Related [issue](https://github.com/FerretDB/FerretDB/issues/1917). | `$sample` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1435) | | `$search` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1436) | | `$searchMeta` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1436) | -| `$set` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1413) | +| `$set` | ⚠️ | [Issue](https://github.com/FerretDB/FerretDB/issues/1413) | | `$setWindowFields` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1437) | | `$skip` | ✅️ | | | `$sort` | ✅️ | | | `$sortByCount` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1440) | | `$unionWith` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1441) | -| `$unset` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1432) | +| `$unset` | ✅️ | | | `$unwind` | ✅️ | | ### Aggregation pipeline operators @@ -492,7 +492,8 @@ Related [issue](https://github.com/FerretDB/FerretDB/issues/1917). | `$substrCP` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1463) | | `$subtract` (arithmetic) | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1453) | | `$subtract` (date) | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1460) | -| `$sum` | ✅️ | | +| `$sum` (accumulator) | ✅️ | | +| `$sum` (operator) | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/2680) | | `$switch` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1457) | | `$tan` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1465) | | `$tanh` | ❌ | [Issue](https://github.com/FerretDB/FerretDB/issues/1465) | From 93b80a4de2d034ca7b469571d2614bc024baaf3c Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 16:55:01 +0400 Subject: [PATCH 11/46] Add basic tests for iterators (#2880) --- integration/commands_administration_test.go | 2 + integration/create_test.go | 8 +++ integration/renamecollection_test.go | 2 + internal/handlers/pg/pgdb/indexes_test.go | 2 + internal/types/array_iterator_test.go | 5 ++ internal/types/document_iterator_test.go | 5 ++ .../iterator/testiterator/testiterator.go | 45 ++++++++++++++ internal/util/iterator/testiterator_test.go | 50 ++++++++++++++++ internal/util/teststress/stress.go | 58 +++++++++++++++++++ 9 files changed, 177 insertions(+) create mode 100644 internal/util/iterator/testiterator/testiterator.go create mode 100644 internal/util/iterator/testiterator_test.go create mode 100644 internal/util/teststress/stress.go diff --git a/integration/commands_administration_test.go b/integration/commands_administration_test.go index c8a6df7863c4..ba3be669cdc2 100644 --- a/integration/commands_administration_test.go +++ b/integration/commands_administration_test.go @@ -1071,6 +1071,8 @@ func TestCommandsAdministrationServerStatusFreeMonitoring(t *testing.T) { } func TestCommandsAdministrationServerStatusStress(t *testing.T) { + // TODO rewrite using teststress.Stress + setup.SkipForTigrisWithReason(t, "https://github.com/FerretDB/FerretDB/issues/1507") ctx, collection := setup.Setup(t) // no providers there, we will create collections concurrently diff --git a/integration/create_test.go b/integration/create_test.go index 28d384501e8b..50f39c1220fc 100644 --- a/integration/create_test.go +++ b/integration/create_test.go @@ -31,6 +31,8 @@ import ( ) func TestCreateStress(t *testing.T) { + // TODO rewrite using teststress.Stress + ctx, collection := setup.Setup(t) // no providers there, we will create collections concurrently db := collection.Database() @@ -109,6 +111,8 @@ func TestCreateStress(t *testing.T) { } func TestCreateOnInsertStressSameCollection(t *testing.T) { + // TODO rewrite using teststress.Stress + setup.SkipForTigrisWithReason(t, "https://github.com/FerretDB/FerretDB/issues/1341") ctx, collection := setup.Setup(t) // do not toLower() db name as it may contain uppercase letters @@ -148,6 +152,8 @@ func TestCreateOnInsertStressSameCollection(t *testing.T) { } func TestCreateOnInsertStressDiffCollection(t *testing.T) { + // TODO rewrite using teststress.Stress + ctx, collection := setup.Setup(t) // do not toLower() db name as it may contain uppercase letters db := collection.Database().Client().Database(t.Name()) @@ -187,6 +193,8 @@ func TestCreateOnInsertStressDiffCollection(t *testing.T) { } func TestCreateStressSameCollection(t *testing.T) { + // TODO rewrite using teststress.Stress + ctx, collection := setup.Setup(t) // no providers there, we will create collection from the test db := collection.Database() diff --git a/integration/renamecollection_test.go b/integration/renamecollection_test.go index 4af5315c61f4..e66cbab0bbcb 100644 --- a/integration/renamecollection_test.go +++ b/integration/renamecollection_test.go @@ -28,6 +28,8 @@ import ( ) func TestRenameCollectionStress(t *testing.T) { + // TODO rewrite using teststress.Stress + setup.SkipForTigrisWithReason(t, "Command renameCollection is not supported for Tigris") ctx, collection := setup.Setup(t) // no providers there, we will create collections for this test diff --git a/internal/handlers/pg/pgdb/indexes_test.go b/internal/handlers/pg/pgdb/indexes_test.go index a0007fce1450..21b7a86c8263 100644 --- a/internal/handlers/pg/pgdb/indexes_test.go +++ b/internal/handlers/pg/pgdb/indexes_test.go @@ -340,6 +340,8 @@ func TestDropIndexes(t *testing.T) { } func TestDropIndexesStress(t *testing.T) { + // TODO rewrite using teststress.Stress + ctx := testutil.Ctx(t) pool := getPool(ctx, t) diff --git a/internal/types/array_iterator_test.go b/internal/types/array_iterator_test.go index 723f83b94a87..b35544eaa45f 100644 --- a/internal/types/array_iterator_test.go +++ b/internal/types/array_iterator_test.go @@ -21,12 +21,17 @@ import ( "github.com/stretchr/testify/require" "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/iterator/testiterator" "github.com/FerretDB/FerretDB/internal/util/must" ) func TestArrayIterator(t *testing.T) { t.Parallel() + testiterator.TestIterator(t, func() iterator.Interface[int, any] { + return must.NotFail(NewArray(int32(1), int32(2))).Iterator() + }) + t.Run("Normal", func(t *testing.T) { t.Parallel() diff --git a/internal/types/document_iterator_test.go b/internal/types/document_iterator_test.go index cf6f77c44147..2425a5c09de3 100644 --- a/internal/types/document_iterator_test.go +++ b/internal/types/document_iterator_test.go @@ -21,12 +21,17 @@ import ( "github.com/stretchr/testify/require" "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/iterator/testiterator" "github.com/FerretDB/FerretDB/internal/util/must" ) func TestDocumentIterator(t *testing.T) { t.Parallel() + testiterator.TestIterator(t, func() iterator.Interface[string, any] { + return must.NotFail(NewDocument("foo", "bar", "baz", "qux")).Iterator() + }) + t.Run("Normal", func(t *testing.T) { t.Parallel() diff --git a/internal/util/iterator/testiterator/testiterator.go b/internal/util/iterator/testiterator/testiterator.go new file mode 100644 index 000000000000..856f67941bd5 --- /dev/null +++ b/internal/util/iterator/testiterator/testiterator.go @@ -0,0 +1,45 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package testiterator provides a helper for checking iterator implementations. +// +// It is in a separate package to avoid import cycles. +package testiterator + +import ( + "testing" + + "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/teststress" +) + +// TestIterator checks that the iterator implementation is correct. +func TestIterator[K, V any](t *testing.T, newIter func() iterator.Interface[K, V]) { + t.Helper() + + // TODO more tests https://github.com/FerretDB/FerretDB/issues/2867 + + t.Run("Close", func(t *testing.T) { + t.Parallel() + + iter := newIter() + + teststress.Stress(t, func(ready chan<- struct{}, start <-chan struct{}) { + ready <- struct{}{} + <-start + + iter.Close() + }) + }) +} diff --git a/internal/util/iterator/testiterator_test.go b/internal/util/iterator/testiterator_test.go new file mode 100644 index 000000000000..aaac47d9c0b6 --- /dev/null +++ b/internal/util/iterator/testiterator_test.go @@ -0,0 +1,50 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package iterator_test // to avoid import cycle + +import ( + "testing" + + "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/iterator/testiterator" +) + +func TestTestIteratorForFunc(t *testing.T) { + t.Parallel() + + testiterator.TestIterator(t, func() iterator.Interface[struct{}, int] { + var i int + var k struct{} + + f := func() (struct{}, int, error) { + i++ + if i > 3 { + return k, 0, iterator.ErrIteratorDone + } + + return k, i, nil + } + + return iterator.ForFunc(f) + }) +} + +func TestTestIteratorForSlice(t *testing.T) { + t.Parallel() + + testiterator.TestIterator(t, func() iterator.Interface[int, int] { + return iterator.ForSlice([]int{1, 2, 3}) + }) +} diff --git a/internal/util/teststress/stress.go b/internal/util/teststress/stress.go new file mode 100644 index 000000000000..972c95ef0daa --- /dev/null +++ b/internal/util/teststress/stress.go @@ -0,0 +1,58 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package teststress provides a helper for stress testing. +// +// It is in a separate package to avoid import cycles. +package teststress + +import ( + "runtime" + "sync" + "testing" +) + +// Stress runs function f in multiple goroutines. +// +// Function f should do a needed setup, send a message to ready channel when it is ready to start, +// wait for start channel to be closed, and then do the actual work. +func Stress(tb testing.TB, f func(ready chan<- struct{}, start <-chan struct{})) { + tb.Helper() + + n := runtime.GOMAXPROCS(-1) * 10 + + // do a bit more work to reduce a chance that one goroutine would finish + // before the other one is still being created + var wg sync.WaitGroup + readyCh := make(chan struct{}, n) + startCh := make(chan struct{}) + + for i := 0; i < n; i++ { + wg.Add(1) + + go func() { + defer wg.Done() + + f(readyCh, startCh) + }() + } + + for i := 0; i < n; i++ { + <-readyCh + } + + close(startCh) + + wg.Wait() +} From 643b387bafaf5f8d79f3ceb631cff0b274acf820 Mon Sep 17 00:00:00 2001 From: Lorant Polya Date: Wed, 21 Jun 2023 06:20:40 -0700 Subject: [PATCH 12/46] Implement basic `insert` support for SAP HANA (#2732) Co-authored-by: Lorant Polya Co-authored-by: Alexey Palazhchenko --- .github/workflows/go-trust.yml | 20 ++- .golangci.yml | 1 + internal/handlers/hana/hanadb/collections.go | 53 +++++++- internal/handlers/hana/hanadb/databases.go | 119 +++++++++++++++++- internal/handlers/hana/hanadb/insert.go | 57 +++++++++ internal/handlers/hana/hanadb/query.go | 21 ++++ internal/handlers/hana/msg_create.go | 9 +- internal/handlers/hana/msg_drop.go | 7 +- internal/handlers/hana/msg_dropdatabase.go | 6 +- internal/handlers/hana/msg_insert.go | 124 ++++++++++++++++++- internal/handlers/hana/msg_listdatabases.go | 97 ++++++++++++++- 11 files changed, 490 insertions(+), 24 deletions(-) create mode 100644 internal/handlers/hana/hanadb/insert.go create mode 100644 internal/handlers/hana/hanadb/query.go diff --git a/.github/workflows/go-trust.yml b/.github/workflows/go-trust.yml index 5dc8c72ebc12..1b21ca25de5a 100644 --- a/.github/workflows/go-trust.yml +++ b/.github/workflows/go-trust.yml @@ -32,10 +32,16 @@ env: jobs: integration: - name: Integration ${{ matrix.name }} + name: Integration ${{ matrix.name }} ${{ matrix.shard_index }}/${{ matrix.shard_total }} runs-on: group: paid - timeout-minutes: 30 + timeout-minutes: 20 + + # Do not run this job in parallel for any PR change or branch/tag push + # to save some resources. + concurrency: + group: ${{ github.workflow }}-${{ matrix.name }}-${{ matrix.shard_index }}-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true if: > github.event_name != 'pull_request_target' || @@ -48,7 +54,9 @@ jobs: fail-fast: false matrix: include: - - { name: "Hana", task: "hana" } + - { name: "Hana", task: "hana", shard_index: 1, shard_total: 3 } + - { name: "Hana", task: "hana", shard_index: 2, shard_total: 3 } + - { name: "Hana", task: "hana", shard_index: 3, shard_total: 3 } steps: - name: Checkout code @@ -90,8 +98,8 @@ jobs: - name: Wait for and setup environment run: bin/task env-setup - - name: Run ${{ matrix.task }} tests - run: bin/task test-integration-${{ matrix.task }} + - name: Run ${{ matrix.name }} tests (${{ matrix.shard_index }}/${{ matrix.shard_total }}) + run: bin/task test-integration-${{ matrix.task }} SHARD_INDEX=${{ matrix.shard_index }} SHARD_TOTAL=${{ matrix.shard_total }} env: GOFLAGS: ${{ runner.debug == '1' && '-v' || '' }} FERRETDB_HANA_URL: ${{ secrets.FERRETDB_HANA_URL }} @@ -108,7 +116,7 @@ jobs: with: token: 22159d7c-856d-4fe9-8fdb-5d9ecff35514 files: ./integration/integration-${{ matrix.task }}.txt - flags: integration,${{ matrix.task }} + flags: integration,${{ matrix.task }},shard-${{ matrix.shard_index }} fail_ci_if_error: true verbose: true diff --git a/.golangci.yml b/.golangci.yml index 6f882caee867..e6ef30dd403b 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -41,6 +41,7 @@ linters-settings: - $all - "!**/internal/backends/sqlite/*.go" - "!**/internal/handlers/pg/pgdb/*.go" + - "!**/internal/handlers/hana/hanadb/*.go" deny: - pkg: github.com/FerretDB/FerretDB/internal/handlers/sjson pgdb: diff --git a/internal/handlers/hana/hanadb/collections.go b/internal/handlers/hana/hanadb/collections.go index c2ca1dc6ddb9..03147e9bcc3d 100644 --- a/internal/handlers/hana/hanadb/collections.go +++ b/internal/handlers/hana/hanadb/collections.go @@ -16,27 +16,68 @@ package hanadb import ( "context" + "errors" "fmt" + + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" ) // CreateCollection creates a new SAP HANA JSON Document Store collection. // // It returns ErrAlreadyExist if collection already exist. -func (hanaPool *Pool) CreateCollection(ctx context.Context, db, collection string) error { - sql := fmt.Sprintf("CREATE COLLECTION %q.%q", db, collection) +func (hanaPool *Pool) CreateCollection(ctx context.Context, qp *QueryParams) error { + sql := fmt.Sprintf("CREATE COLLECTION %q.%q", qp.DB, qp.Collection) _, err := hanaPool.ExecContext(ctx, sql) return getHanaErrorIfExists(err) } -// DropCollection drops collection +// CreateCollectionIfNotExists creates a new SAP HANA JSON Document Store collection. +// +// Returns nil if collection already exist. +func (hanaPool *Pool) CreateCollectionIfNotExists(ctx context.Context, qp *QueryParams) error { + err := hanaPool.CreateCollection(ctx, qp) + + switch { + case errors.Is(err, ErrCollectionAlreadyExist): + return nil + default: + return err + } +} + +// DropCollection drops collection. // -// It returns ErrTableNotExist is collection does not exist. -func (hanaPool *Pool) DropCollection(ctx context.Context, db, collection string) error { - sql := fmt.Sprintf("DROP COLLECTION %q.%q", db, collection) +// Returns ErrTableNotExist is collection does not exist. +func (hanaPool *Pool) DropCollection(ctx context.Context, qp *QueryParams) error { + sql := fmt.Sprintf("DROP COLLECTION %q.%q", qp.DB, qp.Collection) _, err := hanaPool.ExecContext(ctx, sql) return getHanaErrorIfExists(err) } + +// CollectionSize calculates the size of the given collection. +// +// Returns 0 if schema or collection doesn't exist, otherwise returns its size. +func (hanaPool *Pool) CollectionSize(ctx context.Context, qp *QueryParams) (int64, error) { + sqlStmt := "SELECT TABLE_SIZE FROM M_TABLES WHERE SCHEMA_NAME = $1 AND TABLE_NAME = $2 AND TABLE_TYPE = 'COLLECTION'" + + var size any + if err := hanaPool.QueryRowContext(ctx, sqlStmt, qp.DB, qp.Collection).Scan(&size); err != nil { + return 0, lazyerrors.Error(err) + } + + var collectionSize int64 + switch size := size.(type) { + case int64: + collectionSize = size + case nil: + collectionSize = 0 + default: + return 0, lazyerrors.Errorf("Got wrong type for tableSize. Got: %T", collectionSize) + } + + return collectionSize, nil +} diff --git a/internal/handlers/hana/hanadb/databases.go b/internal/handlers/hana/hanadb/databases.go index 5ac55dd53e4e..80cf538d407c 100644 --- a/internal/handlers/hana/hanadb/databases.go +++ b/internal/handlers/hana/hanadb/databases.go @@ -16,25 +16,132 @@ package hanadb import ( "context" + "errors" "fmt" + + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" ) // CreateSchema creates a schema in SAP HANA JSON Document Store. -func (hanaPool *Pool) CreateSchema(ctx context.Context, db string) error { - sqlStmt := fmt.Sprintf("CREATE SCHEMA %q", db) +// +// Returns ErrSchemaAlreadyExist if schema already exists. +func (hanaPool *Pool) CreateSchema(ctx context.Context, qp *QueryParams) error { + sqlStmt := fmt.Sprintf("CREATE SCHEMA %q", qp.DB) _, err := hanaPool.ExecContext(ctx, sqlStmt) return getHanaErrorIfExists(err) } -// DropSchema drops database +// CreateSchemaIfNotExists creates a schema in SAP HANA JSON Document Store. +// +// Returns nil if the schema already exists. +func (hanaPool *Pool) CreateSchemaIfNotExists(ctx context.Context, qp *QueryParams) error { + err := hanaPool.CreateSchema(ctx, qp) + + switch { + case errors.Is(err, ErrSchemaAlreadyExist): + return nil + default: + return err + } +} + +// DropSchema drops database. // -// It returns ErrSchemaNotExist if schema does not exist. -func (hanaPool *Pool) DropSchema(ctx context.Context, db string) error { - sql := fmt.Sprintf("DROP SCHEMA %q CASCADE", db) +// Returns ErrSchemaNotExist if schema does not exist. +func (hanaPool *Pool) DropSchema(ctx context.Context, qp *QueryParams) error { + sql := fmt.Sprintf("DROP SCHEMA %q CASCADE", qp.DB) _, err := hanaPool.ExecContext(ctx, sql) return getHanaErrorIfExists(err) } + +// ListSchemas lists all schemas that aren't related to Hana SYS schemas and SYS owner. +func (hanaPool *Pool) ListSchemas(ctx context.Context) ([]string, error) { + const excludeSYS = "%SYS%" + sqlStmt := "SELECT SCHEMA_NAME FROM SCHEMAS WHERE SCHEMA_NAME NOT LIKE $1 AND SCHEMA_OWNER NOT LIKE $2" + + rows, err := hanaPool.QueryContext(ctx, sqlStmt, excludeSYS, excludeSYS) + if err != nil { + return nil, lazyerrors.Error(err) + } + defer rows.Close() + + res := make([]string, 0, 2) + + for rows.Next() { + var name string + if err = rows.Scan(&name); err != nil { + return nil, lazyerrors.Error(err) + } + + res = append(res, name) + } + + if err = rows.Err(); err != nil { + return nil, lazyerrors.Error(err) + } + + return res, nil +} + +// ListCollections lists all collections under a given schema. +// +// Returns an empty array if schema doesn't exist. +func (hanaPool *Pool) ListCollections(ctx context.Context, qp *QueryParams) ([]string, error) { + sqlStmt := "SELECT TABLE_NAME FROM M_TABLES WHERE SCHEMA_NAME = $1 AND TABLE_TYPE = 'COLLECTION'" + + rows, err := hanaPool.QueryContext(ctx, sqlStmt, qp.DB) + if err != nil { + return nil, lazyerrors.Error(err) + } + defer rows.Close() + + res := make([]string, 0, 2) + + for rows.Next() { + var name string + if err = rows.Scan(&name); err != nil { + return nil, lazyerrors.Error(err) + } + + res = append(res, name) + } + + if err = rows.Err(); err != nil { + return nil, lazyerrors.Error(err) + } + + return res, nil +} + +// SchemaSize calculates the total size of collections under schema. +// +// Returns 0 if schema doesn't exist, otherwise returns its size. +func (hanaPool *Pool) SchemaSize(ctx context.Context, qp *QueryParams) (int64, error) { + collections, err := hanaPool.ListCollections(ctx, qp) + if err != nil { + return 0, lazyerrors.Error(err) + } + + qpCopy := QueryParams{ + DB: qp.DB, + } + + var totalSize int64 + + for _, collection := range collections { + qpCopy.Collection = collection + + size, err := hanaPool.CollectionSize(ctx, &qpCopy) + if err != nil { + return 0, lazyerrors.Error(err) + } + + totalSize += size + } + + return totalSize, nil +} diff --git a/internal/handlers/hana/hanadb/insert.go b/internal/handlers/hana/hanadb/insert.go new file mode 100644 index 000000000000..28ec4aba0136 --- /dev/null +++ b/internal/handlers/hana/hanadb/insert.go @@ -0,0 +1,57 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package hanadb + +import ( + "context" + "fmt" + + "github.com/FerretDB/FerretDB/internal/handlers/sjson" + "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/must" +) + +// InsertOne inserts a document into a collection in SAP HANA JSON Document Store. +func (hanaPool *Pool) InsertOne(ctx context.Context, qp *QueryParams, doc *types.Document) error { + err := hanaPool.CreateSchemaIfNotExists(ctx, qp) + + switch { + case err == nil: + // Success case + default: + return err + } + + err = hanaPool.CreateCollectionIfNotExists(ctx, qp) + + switch { + case err == nil: + // Success case + default: + return err + } + + return hanaPool.insert(ctx, qp, doc) +} + +// insert inserts a document into a collection in SAP HANA JSON Document Store. +func (hanaPool *Pool) insert(ctx context.Context, qp *QueryParams, doc *types.Document) error { + sqlStmt := fmt.Sprintf("insert into %q.%q values($1)", qp.DB, qp.Collection) + + // sjson.MarshalSingleValue can be used because the Hana insert json format is just a json string + _, err := hanaPool.ExecContext(ctx, sqlStmt, must.NotFail(sjson.MarshalSingleValue(doc))) + + return getHanaErrorIfExists(err) +} diff --git a/internal/handlers/hana/hanadb/query.go b/internal/handlers/hana/hanadb/query.go new file mode 100644 index 000000000000..ea968fa59449 --- /dev/null +++ b/internal/handlers/hana/hanadb/query.go @@ -0,0 +1,21 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package hanadb + +// QueryParams represents options/parameters used for SQL query/statement. +type QueryParams struct { + DB string + Collection string +} diff --git a/internal/handlers/hana/msg_create.go b/internal/handlers/hana/msg_create.go index 7af30795871a..90b44378a239 100644 --- a/internal/handlers/hana/msg_create.go +++ b/internal/handlers/hana/msg_create.go @@ -87,7 +87,12 @@ func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, return nil, err } - err = dbPool.CreateSchema(ctx, db) + qp := hanadb.QueryParams{ + DB: db, + Collection: collection, + } + + err = dbPool.CreateSchema(ctx, &qp) switch { case err == nil: @@ -101,7 +106,7 @@ func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, return nil, lazyerrors.Error(err) } - err = dbPool.CreateCollection(ctx, db, collection) + err = dbPool.CreateCollection(ctx, &qp) switch { case err == nil: diff --git a/internal/handlers/hana/msg_drop.go b/internal/handlers/hana/msg_drop.go index 1922d9a0300f..98e20115afe2 100644 --- a/internal/handlers/hana/msg_drop.go +++ b/internal/handlers/hana/msg_drop.go @@ -53,7 +53,12 @@ func (h *Handler) MsgDrop(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } - err = dbPool.DropCollection(ctx, db, collection) + qp := hanadb.QueryParams{ + DB: db, + Collection: collection, + } + + err = dbPool.DropCollection(ctx, &qp) switch { case err == nil: diff --git a/internal/handlers/hana/msg_dropdatabase.go b/internal/handlers/hana/msg_dropdatabase.go index 3bf43bcd628a..10afb7a86242 100644 --- a/internal/handlers/hana/msg_dropdatabase.go +++ b/internal/handlers/hana/msg_dropdatabase.go @@ -45,9 +45,13 @@ func (h *Handler) MsgDropDatabase(ctx context.Context, msg *wire.OpMsg) (*wire.O return nil, err } + qp := hanadb.QueryParams{ + DB: db, + } + res := must.NotFail(types.NewDocument()) - err = dbPool.DropSchema(ctx, db) + err = dbPool.DropSchema(ctx, &qp) switch { case err == nil: diff --git a/internal/handlers/hana/msg_insert.go b/internal/handlers/hana/msg_insert.go index 6c59d8b77e27..4a610fe1f792 100644 --- a/internal/handlers/hana/msg_insert.go +++ b/internal/handlers/hana/msg_insert.go @@ -16,12 +16,134 @@ package hana import ( "context" + "errors" + "fmt" + "github.com/FerretDB/FerretDB/internal/handlers/common" + "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" + "github.com/FerretDB/FerretDB/internal/handlers/hana/hanadb" + "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" "github.com/FerretDB/FerretDB/internal/util/must" "github.com/FerretDB/FerretDB/internal/wire" ) // MsgInsert implements HandlerInterface. func (h *Handler) MsgInsert(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return nil, notImplemented(must.NotFail(msg.Document()).Command()) + dbPool, err := h.DBPool(ctx) + if err != nil { + return nil, lazyerrors.Error(err) + } + + document, err := msg.Document() + if err != nil { + return nil, lazyerrors.Error(err) + } + + ignoredFields := []string{ + "writeConcern", + } + common.Ignored(document, h.L, ignoredFields...) + + params, err := common.GetInsertParams(document, h.L) + if err != nil { + return nil, err + } + + qp := hanadb.QueryParams{ + DB: params.DB, + Collection: params.Collection, + } + + inserted, insErrors := insertMany(ctx, dbPool, &qp, params.Docs, params.Ordered) + + replyDoc := must.NotFail(types.NewDocument( + "n", inserted, + "ok", float64(1), + )) + + if insErrors.Len() > 0 { + replyDoc = insErrors.Document() + } + + var reply wire.OpMsg + must.NoError(reply.SetSections(wire.OpMsgSection{ + Documents: []*types.Document{replyDoc}, + })) + + return &reply, nil +} + +func insertMany(ctx context.Context, dbPool *hanadb.Pool, qp *hanadb.QueryParams, docs *types.Array, ordered bool) (int32, *commonerrors.WriteErrors) { //nolint:lll // argument list is too long + var inserted int32 + var insErrors commonerrors.WriteErrors + + // TODO: Bulk Insert + // Attempt to insert all the documents in the same request to make insert faster. + /*if err := dbPool.InsertManyDocuments(ctx, qp docs); err == nil { + return int32(docs.Len()), &insErrors + }*/ + + // If the transaction failed, attempt to insert each document separately. + for i := 0; i < docs.Len(); i++ { + doc := must.NotFail(docs.Get(i)) + + err := insertOne(ctx, dbPool, qp, doc.(*types.Document)) + + var we *commonerrors.WriteErrors + + switch { + case err == nil: + inserted++ + continue + case errors.As(err, &we): + insErrors.Merge(we, int32(i)) + default: + insErrors.Append(err, int32(i)) + } + + if ordered { + return inserted, &insErrors + } + } + + return inserted, &insErrors +} + +// insertOne checks if database and collection exist, create them if needed and attempts to insertDocument the given doc. +func insertOne(ctx context.Context, dbPool *hanadb.Pool, qp *hanadb.QueryParams, doc *types.Document) error { + toInsert := doc + + if !toInsert.Has("_id") { + // Make a copy so that original document could be sent to the proxy as it is. + toInsert = doc.DeepCopy() + + toInsert.Set("_id", types.NewObjectID()) + } + + err := dbPool.InsertOne(ctx, qp, toInsert) + + switch { + case err == nil: + return nil + case errors.Is(err, hanadb.ErrInvalidCollectionName), errors.Is(err, hanadb.ErrInvalidDatabaseName): + msg := fmt.Sprintf("Invalid namespace: %s.%s", qp.DB, qp.Collection) + return commonerrors.NewCommandErrorMsg(commonerrors.ErrInvalidNamespace, msg) + + // TODO: set up some sort of metadata table to keep track of '_ids' so we can track duplicates + /*case errors.Is(err, hanzdb.ErrUniqueViolation): + // TODO Extend message for non-_id unique indexes in https://github.com/FerretDB/FerretDB/issues/2045 + idMasrshaled := must.NotFail(json.Marshal(must.NotFail(d.Get("_id")))) + + return commonerrors.NewWriteErrorMsg( + commonerrors.ErrDuplicateKey, + fmt.Sprintf( + `E11000 duplicate key error collection: %s.%s index: _id_ dup key: { _id: %s }`, + qp.DB, qp.Collection, idMasrshaled, + ), + ) + */ + default: + return lazyerrors.Error(err) + } } diff --git a/internal/handlers/hana/msg_listdatabases.go b/internal/handlers/hana/msg_listdatabases.go index 5cb1a2139521..facd557f1426 100644 --- a/internal/handlers/hana/msg_listdatabases.go +++ b/internal/handlers/hana/msg_listdatabases.go @@ -17,11 +17,106 @@ package hana import ( "context" + "github.com/FerretDB/FerretDB/internal/handlers/common" + "github.com/FerretDB/FerretDB/internal/handlers/commonparams" + "github.com/FerretDB/FerretDB/internal/handlers/hana/hanadb" + "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" "github.com/FerretDB/FerretDB/internal/util/must" "github.com/FerretDB/FerretDB/internal/wire" ) // MsgListDatabases implements HandlerInterface. func (h *Handler) MsgListDatabases(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return nil, notImplemented(must.NotFail(msg.Document()).Command()) + dbPool, err := h.DBPool(ctx) + if err != nil { + return nil, lazyerrors.Error(err) + } + + document, err := msg.Document() + if err != nil { + return nil, lazyerrors.Error(err) + } + + var filter *types.Document + if filter, err = common.GetOptionalParam(document, "filter", filter); err != nil { + return nil, err + } + + common.Ignored(document, h.L, "comment", "authorizedDatabases") + + databaseNames, err := dbPool.ListSchemas(ctx) + if err != nil { + return nil, err + } + + var nameOnly bool + if v, _ := document.Get("nameOnly"); v != nil { + nameOnly, err = commonparams.GetBoolOptionalParam("nameOnly", v) + if err != nil { + return nil, err + } + } + + var totalSize int64 + databases := types.MakeArray(len(databaseNames)) + qp := hanadb.QueryParams{} + + for _, databaseName := range databaseNames { + qp.DB = databaseName + + size, err := dbPool.SchemaSize(ctx, &qp) + if err != nil { + return nil, lazyerrors.Error(err) + } + + totalSize += size + + d := must.NotFail(types.NewDocument( + "name", databaseName, + "sizeOnDisk", size, + "empty", size == 0, + )) + + matches, err := common.FilterDocument(d, filter) + if err != nil { + return nil, err + } + + if !matches { + continue + } + + if nameOnly { + d = must.NotFail(types.NewDocument( + "name", databaseName, + )) + } + + databases.Append(d) + } + + if nameOnly { + var reply wire.OpMsg + must.NoError(reply.SetSections(wire.OpMsgSection{ + Documents: []*types.Document{must.NotFail(types.NewDocument( + "databases", databases, + "ok", float64(1), + ))}, + })) + + return &reply, nil + } + + var reply wire.OpMsg + must.NoError(reply.SetSections(wire.OpMsgSection{ + Documents: []*types.Document{must.NotFail(types.NewDocument( + "databases", databases, + "totalSize", totalSize, + "totalSizeMb", totalSize/1024/1024, + "ok", float64(1), + ))}, + })) + + return &reply, nil } From f4bb3f395038fc53851b8b68909502307e15ddb7 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 17:50:17 +0400 Subject: [PATCH 13/46] Update contributing docs (#2828) --- CONTRIBUTING.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 23879c713b51..c4ec59dfae43 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,12 +44,13 @@ Finally, you will also need [git-lfs](https://git-lfs.github.com) installed and Fork the [FerretDB repository on GitHub](https://github.com/FerretDB/FerretDB/fork). To have all the tags in the repository and what they point to, copy all branches by removing checkmark for `copy the main branch only` before forking. -After forking FerretDB on GitHub, you can clone the repository: +After forking FerretDB on GitHub, you can clone the repository and add upstream repository as a remote: ```sh git clone git@github.com:/FerretDB.git cd FerretDB git remote add upstream https://github.com/FerretDB/FerretDB.git +git fetch --all --tags ``` To run development commands, you should first install the [`task`](https://taskfile.dev/) tool. @@ -220,6 +221,18 @@ Some of our idiosyncrasies: It may seem random, but it is only pseudo-random and follows BSON spec: 2. We generally pass and return `struct`s by pointers. There are some exceptions like `types.Path` that has value semantics, but when in doubt – use pointers. +3. Code comments: + - All top-level declarations, even unexported, should have documentation comments. + - In documentation comments do not describe the name in terms of the name itself (`// Registry is a registry of …`). + Use other words instead; often, they could add additional information and make reading more pleasant (`// Registry stores …`). + - In code comments, in general, do not describe _what_ the code does: it should be clear from the code itself + (and when it doesn't and the code is tricky, simplify it instead). + Instead, describe _why_ the code does that if it is not clear from the surrounding context, names, etc. + There is no need to add comments just because there are no comments if everything is already clear without them. + - For code comments, write either + sentence fragments (do not start it with a capital letter, do not end it with a dot, use the simplified grammar) for short notes + or full sentences (do start them with capital letters, do end them with dots, do check their grammar) when a longer (2+ sentences) + explanation is needed (and the code could not be simplified). #### Integration tests conventions From 258bad366aa43df04d4b7b67b9af0d8560ba5d15 Mon Sep 17 00:00:00 2001 From: Alexander Tobi Fashakin Date: Wed, 21 Jun 2023 15:40:55 +0100 Subject: [PATCH 14/46] Add blog post "FerretDB Demo: Launch and Test a Database in Minutes" (#2851) --- ...retdb-demo-launch-test-database-minutes.md | 177 ++++++++++++++++++ .../blog/ferretdb-demo-creating-database.png | 3 + .../static/img/blog/ferretdb-demo-page.png | 3 + .../blog/launch-ferretdb-demo-database.jpg | 3 + 4 files changed, 186 insertions(+) create mode 100644 website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md create mode 100644 website/static/img/blog/ferretdb-demo-creating-database.png create mode 100644 website/static/img/blog/ferretdb-demo-page.png create mode 100644 website/static/img/blog/launch-ferretdb-demo-database.jpg diff --git a/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md b/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md new file mode 100644 index 000000000000..55cf73188e3e --- /dev/null +++ b/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md @@ -0,0 +1,177 @@ +--- +slug: ferretdb-demo-launch-test-database-minutes +title: 'FerretDB Demo: Launch and Test a Database in Minutes' +authors: [alex] +description: > + Want to find out how FerretDB really works? Let's guide you through our demo to learn more. +image: /img/blog/ferretdb-meteor.jpg +keywords: [launch ferretdb database, test ferretdb, ferretdb demo] +tags: [tutorial, demo, cloud] +--- + +![Launch and test a ferretdb database in minutes](/img/blog/launch-ferretdb-demo-database.jpg) + +Want to know what a [truly open-source MongoDB alternative](https://blog.ferretdb.io/mongodb-compatibility-whats-really-important/) looks like? +No worries! +We've got you covered. + + + +We've set up the [FerretDB demo page](https://try.ferretdb.io/) so you can experiment and see what FerretDB looks like in action: creating your own database, performing CRUD operations, running indexes, or aggregation pipeline commands, basically all the commands you're already used to. + +If you're new to FerretDB and wondering what it's all about, [FerretDB](https://www.ferretdb.io/) is an open-source database committed to restoring MongoDB workloads to their open-source origins while enabling developers to use the same syntax and commands. +Basically, we [convert MongoDB wire protocols to SQL with PostgreSQL](https://blog.ferretdb.io/pjson-how-to-store-bson-in-jsonb/) as the backend engine. + +Now that we know what FerretDB is all about, let's jump into the demo and try it out. + +## Prerequisites + +Before proceeding, please make sure you have: + +- `mongosh`. + Please [install mongosh](https://www.mongodb.com/docs/mongodb-shell/install/) if you don't currently have it installed. + +## How to launch your own FerretDB demo database + +In this blog post, we'll be showcasing the FerretDB demo page and what you can achieve with it. + +The FerretDB demo is quick to set up, and you can get started through the demo site. + +- [Demo site](https://try.ferretdb.io/) + +Once you are on the site, you can click the "launch a database for free" button to create your own FerretDB database at Civo.com - a cloud marketplace and service platform. + +![Homescreen of demo page](/img/blog/ferretdb-demo-page.png) + +This process involves getting the necessary credentials for the setup from Civo and then installing the latest FerretDB version on the Kubernetes cluster for you. +This whole setup takes about 2-4 minutes to complete; once the database is ready, you'll get a connection URI with your authentication credentials. + +![Database creation process](/img/blog/ferretdb-demo-creating-database.png) + +Note that all FerretDB databases created in this demo are deployed on a managed Kubernetes cluster provided by Civo, and are only available for 2 hours after creation. + +Remember that the FerretDB instance is only available for 2 hours, so if you would like to go all the way and test out FerretDB for a longer period of time, you can install FerretDB through the Civo Marketplace or through any of our installation guides. + +## Connecting to your FerretDB instance + +Now that you have your FerretDB connection URI, you can connect to your FerretDB instance. + +Copy the FerretDB connection URI; it typically follows this format: + +```sh +mongodb://username:password@host:port/database?authMechanism=PLAIN +``` + +Connect to FerretDB through `mongosh` with the connection URI: + +```sh +mongosh 'mongodb://username:password@host:port/database?authMechanism=PLAIN' +``` + +And that's about it! +You have your FerretDB database running. + +## Run basic operations on your FerretDB database + +Now that you're connected to FerretDB, let's explore some basic database operations: + +**Inserting data:** Let's try inserting 4 documents into a database collection `demo`: + +```js +db.demo.insertMany([ + { + name: 'Chinedu Eze', + age: 20, + email: 'chinedu.eze@example.com', + major: 'Computer Science', + sports: ['Basketball', 'Running'] + }, + { + name: 'Maria Rodriguez', + age: 21, + email: 'maria.rodriguez@example.com', + major: 'Business Administration', + sports: ['Yoga', 'Swimming'] + }, + { + name: 'Kelly Li', + age: 19, + email: 'kelly.li@example.com', + major: 'Engineering', + sports: ['Soccer', 'Cycling'] + }, + { + name: 'Sara Nguyen', + age: 22, + email: 'sara.nguyen@example.com', + major: 'Biology', + sports: ['Tennis', 'Dancing'] + } +]) +``` + +**Delete data:** Let's remove a document from the database using a `name` field as the specified query. + +```js +db.demo.deleteOne({ name: 'Sara Nguyen' }) +``` + +**Update data:** Let's update the documents in the database with `age` less than or equal to `21`. + +```js +db.demo.updateMany({ age: { $lt: 20 } }, { $addToSet: { sports: 'aerobics' } }) +``` + +To see the current state of your database, use the `find` command, which should give you a good view of all the prior changes made to the collection: + +```js +db.demo.find() +``` + +Output: + +```json5 +[ + { + _id: ObjectId("648bc0619df6027209a65b40"), + name: 'Chinedu Eze', + age: 20, + email: 'chinedu.eze@example.com', + major: 'Computer Science', + sports: [ 'Basketball', 'Running' ] + }, + { + _id: ObjectId("648bc0619df6027209a65b41"), + name: 'Maria Rodriguez', + age: 21, + email: 'maria.rodriguez@example.com', + major: 'Business Administration', + sports: [ 'Yoga', 'Swimming' ] + }, + { + _id: ObjectId("648bc0619df6027209a65b42"), + name: 'Kelly Li', + age: 19, + email: 'kelly.li@example.com', + major: 'Engineering', + sports: [ 'Soccer', 'Cycling' ] + } +] +``` + +**Indexing:** In FerretDB, you can also create indexes; let's set an index on the `email` field the same way you would on MongoDB, like this: + +```js +db.demo.createIndex({ email: 1 }) +``` + +You can try several combinations of commands; here is a list of the currently [supported commands on FerretDB](https://docs.ferretdb.io/reference/supported-commands/). +You can also check out this blog post on some of the [basic CRUD commands you can run on FerretDB](https://blog.ferretdb.io/mongodb-crud-operations-with-ferretdb/). + +Feel free to experiment and explore all the capabilities of FerretDB. + +## Conclusion + +Powered by Civo's seamless setup process, you can have a FerretDB instance up and running in minutes, but note that the database cluster is only up for two hours before it's removed, so you may need to create another one, or install [FerretDB through the Civo Marketplace](https://www.civo.com/marketplace/FerretDB) or through the [FerretDB installation guide](https://docs.ferretdb.io/quickstart-guide/). + +If you have any questions, feedback, or requests for FerretDB, please feel free to reach out to us on our [community channels](https://docs.ferretdb.io/#community). diff --git a/website/static/img/blog/ferretdb-demo-creating-database.png b/website/static/img/blog/ferretdb-demo-creating-database.png new file mode 100644 index 000000000000..80917667f6aa --- /dev/null +++ b/website/static/img/blog/ferretdb-demo-creating-database.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:46ce5852e6d5afee47820fca39a16e915c61239626e330ea7c23c5fbf4095e85 +size 328374 diff --git a/website/static/img/blog/ferretdb-demo-page.png b/website/static/img/blog/ferretdb-demo-page.png new file mode 100644 index 000000000000..2fd34ab4b3e0 --- /dev/null +++ b/website/static/img/blog/ferretdb-demo-page.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:81b12d41d343d011f92d95682b447250a0ea5d828f8437a38d64b80c6d043103 +size 374166 diff --git a/website/static/img/blog/launch-ferretdb-demo-database.jpg b/website/static/img/blog/launch-ferretdb-demo-database.jpg new file mode 100644 index 000000000000..ef55ab9ccf6a --- /dev/null +++ b/website/static/img/blog/launch-ferretdb-demo-database.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:803a2be5d9ccda7c86423b94378615fe3b6c4193ec44c7230d0ef45a1b093e86 +size 51056 From 0c6f3a5d32c6fde306cbfeff59ecefb4a56dc05a Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 21 Jun 2023 18:54:28 +0400 Subject: [PATCH 15/46] Fix blog post's social preview --- .../2023-06-21-ferretdb-demo-launch-test-database-minutes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md b/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md index 55cf73188e3e..43eaf5b8b924 100644 --- a/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md +++ b/website/blog/2023-06-21-ferretdb-demo-launch-test-database-minutes.md @@ -4,7 +4,7 @@ title: 'FerretDB Demo: Launch and Test a Database in Minutes' authors: [alex] description: > Want to find out how FerretDB really works? Let's guide you through our demo to learn more. -image: /img/blog/ferretdb-meteor.jpg +image: /img/blog/launch-ferretdb-demo-database.jpg keywords: [launch ferretdb database, test ferretdb, ferretdb demo] tags: [tutorial, demo, cloud] --- From ced475653afdf066dad956f035bbc52d594c73cf Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Thu, 22 Jun 2023 18:15:39 +0400 Subject: [PATCH 16/46] Improve `wire` and `sjson` fuzzing (#2883) --- Taskfile.yml | 18 ++-- cmd/fuzztool/fuzztool.go | 5 +- internal/handlers/sjson/sjson_test.go | 46 ++++++++++ internal/util/testutil/fuzz.go | 46 ---------- internal/wire/bits.go | 3 + internal/wire/msg_body.go | 9 +- internal/wire/op_msg.go | 2 +- internal/wire/op_msg_test.go | 6 +- internal/wire/record.go | 122 ++++++++++++++++++++++++++ internal/wire/records_test.go | 115 ------------------------ internal/wire/wire_test.go | 47 ++++++---- 11 files changed, 219 insertions(+), 200 deletions(-) delete mode 100644 internal/util/testutil/fuzz.go create mode 100644 internal/wire/record.go delete mode 100644 internal/wire/records_test.go diff --git a/Taskfile.yml b/Taskfile.yml index 80897dbd5024..e3b560a5648c 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -279,20 +279,18 @@ tasks: cmds: - go test -count=0 ./... - # Those commands should still run tests (i.e., should not have -run=XXX flags) - # to fill seed corpus for fuzz tests that use WriteSeedCorpusFile (e.g., FuzzHandler). fuzz: desc: "Fuzz for about 2 minutes (with default FUZZ_TIME)" cmds: - go test -list='Fuzz.*' ./... - - go test -fuzz=FuzzArray -fuzztime={{.FUZZ_TIME}} ./internal/bson/ - - go test -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/bson/ - - go test -fuzz=FuzzArray -fuzztime={{.FUZZ_TIME}} ./internal/handlers/sjson/ - - go test -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/handlers/sjson/ - - go test -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/handlers/tigris/tjson/ - - go test -fuzz=FuzzMsg -fuzztime={{.FUZZ_TIME}} ./internal/wire/ - - go test -fuzz=FuzzQuery -fuzztime={{.FUZZ_TIME}} ./internal/wire/ - - go test -fuzz=FuzzReply -fuzztime={{.FUZZ_TIME}} ./internal/wire/ + - go test -run=XXX -fuzz=FuzzArray -fuzztime={{.FUZZ_TIME}} ./internal/bson/ + - go test -run=XXX -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/bson/ + - go test -run=XXX -fuzz=FuzzArray -fuzztime={{.FUZZ_TIME}} ./internal/handlers/sjson/ + - go test -run=XXX -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/handlers/sjson/ + - go test -run=XXX -fuzz=FuzzDocument -fuzztime={{.FUZZ_TIME}} ./internal/handlers/tigris/tjson/ + - go test -run=XXX -fuzz=FuzzMsg -fuzztime={{.FUZZ_TIME}} ./internal/wire/ + - go test -run=XXX -fuzz=FuzzQuery -fuzztime={{.FUZZ_TIME}} ./internal/wire/ + - go test -run=XXX -fuzz=FuzzReply -fuzztime={{.FUZZ_TIME}} ./internal/wire/ fuzz-corpus: desc: "Sync seed and generated fuzz corpora with FUZZ_CORPUS" diff --git a/cmd/fuzztool/fuzztool.go b/cmd/fuzztool/fuzztool.go index 67d2e9f9beb1..edf1db8f213b 100644 --- a/cmd/fuzztool/fuzztool.go +++ b/cmd/fuzztool/fuzztool.go @@ -17,6 +17,7 @@ package main import ( "bytes" "encoding/hex" + "errors" "io" "io/fs" "os" @@ -43,7 +44,7 @@ func generatedCorpus() (string, error) { path := filepath.Join(string(bytes.TrimSpace(b)), "fuzz", "github.com", "FerretDB", "FerretDB") if _, err = os.Stat(path); err != nil { - if os.IsNotExist(err) { + if errors.Is(err, fs.ErrNotExist) { err = os.MkdirAll(path, 0o777) } @@ -129,7 +130,7 @@ func copyFile(src, dst string) error { dir := filepath.Dir(dst) _, err = os.Stat(dir) - if os.IsNotExist(err) { + if errors.Is(err, fs.ErrNotExist) { err = os.MkdirAll(dir, 0o777) } diff --git a/internal/handlers/sjson/sjson_test.go b/internal/handlers/sjson/sjson_test.go index 1ff34da899c8..8263ef4cfd02 100644 --- a/internal/handlers/sjson/sjson_test.go +++ b/internal/handlers/sjson/sjson_test.go @@ -20,6 +20,7 @@ import ( "errors" "fmt" "math" + "path/filepath" "testing" "github.com/stretchr/testify/assert" @@ -27,6 +28,7 @@ import ( "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/must" + "github.com/FerretDB/FerretDB/internal/wire" ) type testCase struct { @@ -159,6 +161,50 @@ func fuzzJSON(f *testing.F, testCases []testCase, newFunc func() sjsontype) { } } + // load recorded documents only if we are fuzzing documents + if _, ok := newFunc().(*documentType); ok && !testing.Short() { + records, err := wire.LoadRecords(filepath.Join("..", "..", "..", "tmp", "records"), 1000) + require.NoError(f, err) + + var n int + + for _, rec := range records { + var docs []*types.Document + + switch b := rec.Body.(type) { + case *wire.OpMsg: + doc, err := b.Document() + require.NoError(f, err) + docs = append(docs, doc) + + case *wire.OpQuery: + if doc := b.Query; doc != nil { + docs = append(docs, doc) + } + + if doc := b.ReturnFieldsSelector; doc != nil { + docs = append(docs, doc) + } + + case *wire.OpReply: + docs = append(docs, b.Documents...) + } + + for _, doc := range docs { + j, err := MarshalSingleValue(doc) + require.NoError(f, err) + + sch, err := marshalSchemaForDoc(doc) + require.NoError(f, err) + + f.Add(string(j), string(sch)) + n++ + } + } + + f.Logf("%d recorded documents were added to the seed corpus", n) + } + f.Fuzz(func(t *testing.T, j, jsch string) { t.Parallel() diff --git a/internal/util/testutil/fuzz.go b/internal/util/testutil/fuzz.go deleted file mode 100644 index b11a510f3c4c..000000000000 --- a/internal/util/testutil/fuzz.go +++ /dev/null @@ -1,46 +0,0 @@ -// Copyright 2021 FerretDB Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package testutil - -import ( - "bytes" - "crypto/sha256" - "fmt" - "os" - "path/filepath" - "testing" - - "github.com/stretchr/testify/require" -) - -// WriteSeedCorpusFile adds given data to the fuzzing seed corpus for given fuzz function. -// -// It can be an alternative to using f.Add. -func WriteSeedCorpusFile(tb testing.TB, funcName string, b []byte) { - tb.Helper() - - var buf bytes.Buffer - buf.WriteString("go test fuzz v1\n") - _, err := fmt.Fprintf(&buf, "[]byte(%q)\n", b) - require.NoError(tb, err) - - dir := filepath.Join("testdata", "fuzz", funcName) - err = os.MkdirAll(dir, 0o777) - require.NoError(tb, err) - - filename := filepath.Join(dir, fmt.Sprintf("test-%x", sha256.Sum256(b))) - err = os.WriteFile(filename, buf.Bytes(), 0o666) - require.NoError(tb, err) -} diff --git a/internal/wire/bits.go b/internal/wire/bits.go index 1aa48f357240..8872e4baca9a 100644 --- a/internal/wire/bits.go +++ b/internal/wire/bits.go @@ -20,6 +20,9 @@ type flagBit uint32 type flags uint32 +// flagsSize represents flags size in bytes. +const flagsSize = 4 + func (flags flags) strings(bitStringer func(flagBit) string) []string { res := make([]string, 0, 2) for shift := 0; shift < 32; shift++ { diff --git a/internal/wire/msg_body.go b/internal/wire/msg_body.go index 148cd1abe1e7..04313bc3b053 100644 --- a/internal/wire/msg_body.go +++ b/internal/wire/msg_body.go @@ -42,9 +42,6 @@ type MsgBody interface { // indicating that connection was closed by the client. var ErrZeroRead = errors.New("zero bytes read") -// kFlagBitSize represents the size of the flag bits field. -const kFlagBitSize = 4 - // ReadMessage reads from reader and returns wire header and body. // // Error is (possibly wrapped) ErrZeroRead if zero bytes was read. @@ -144,7 +141,7 @@ func getChecksum(data []byte) (uint32, error) { // ensure that the length of the body is at least the size of a flagbit // and a crc32 checksum n := len(data) - if n < crc32.Size+kFlagBitSize { + if n < crc32.Size+flagsSize { return 0, lazyerrors.New("Invalid message size for an OpMsg containing a checksum") } @@ -158,11 +155,11 @@ func getChecksum(data []byte) (uint32, error) { // // TODO The callers of checksum validation should be closer to OP_MSG handling: https://github.com/FerretDB/FerretDB/issues/2690 func validateChecksum(header *MsgHeader, body []byte) error { - if len(body) < kFlagBitSize { + if len(body) < flagsSize { return lazyerrors.New("Message contains illegal flags value") } - flagBit := OpMsgFlags(binary.LittleEndian.Uint32(body[:kFlagBitSize])) + flagBit := OpMsgFlags(binary.LittleEndian.Uint32(body[:flagsSize])) if !flagBit.FlagSet(OpMsgChecksumPresent) { return nil } diff --git a/internal/wire/op_msg.go b/internal/wire/op_msg.go index 3954f6d8b7ed..97214922597c 100644 --- a/internal/wire/op_msg.go +++ b/internal/wire/op_msg.go @@ -40,8 +40,8 @@ type OpMsgSection struct { type OpMsg struct { FlagBits OpMsgFlags - checksum uint32 sections []OpMsgSection + checksum uint32 } // SetSections of the OpMsg. diff --git a/internal/wire/op_msg_test.go b/internal/wire/op_msg_test.go index 38b704b06813..d2d2f59c7abb 100644 --- a/internal/wire/op_msg_test.go +++ b/internal/wire/op_msg_test.go @@ -292,7 +292,6 @@ var msgTestCases = []testCase{{ }, msgBody: &OpMsg{ FlagBits: OpMsgFlags(OpMsgChecksumPresent), - checksum: 1737537506, sections: []OpMsgSection{{ Kind: 1, Identifier: "documents", @@ -307,6 +306,7 @@ var msgTestCases = []testCase{{ "$db", "test", ))}, }}, + checksum: 1737537506, }, command: "insert", }, { @@ -364,7 +364,6 @@ var msgTestCases = []testCase{{ }, msgBody: &OpMsg{ FlagBits: OpMsgFlags(OpMsgChecksumPresent), - checksum: 2932997361, sections: []OpMsgSection{{ Kind: 1, Identifier: "updates", @@ -387,6 +386,7 @@ var msgTestCases = []testCase{{ "$db", "test", ))}, }}, + checksum: 2932997361, }, command: "update", }, { @@ -429,7 +429,6 @@ var msgTestCases = []testCase{{ }, msgBody: &OpMsg{ FlagBits: OpMsgFlags(OpMsgChecksumPresent), - checksum: 1737537506, sections: []OpMsgSection{{ Kind: 1, Identifier: "documents", @@ -444,6 +443,7 @@ var msgTestCases = []testCase{{ "$db", "test", ))}, }}, + checksum: 1737537506, }, err: "OP_MSG checksum does not match contents.", }} diff --git a/internal/wire/record.go b/internal/wire/record.go new file mode 100644 index 000000000000..b2d8da10cfe2 --- /dev/null +++ b/internal/wire/record.go @@ -0,0 +1,122 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package wire + +import ( + "bufio" + "errors" + "io/fs" + "math/rand" + "os" + "path/filepath" + + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" +) + +// Record represents a single recorded wire protocol message, loaded from a .bin file. +type Record struct { + Header *MsgHeader + Body MsgBody + HeaderB []byte + BodyB []byte +} + +// LoadRecords finds all .bin files recursively, selects up to the limit at random (or all if limit <= 0), and parses them. +func LoadRecords(dir string, limit int) ([]Record, error) { + var files []string + err := filepath.WalkDir(dir, func(path string, entry fs.DirEntry, err error) error { + if err != nil { + return lazyerrors.Error(err) + } + + if filepath.Ext(entry.Name()) == ".bin" { + files = append(files, path) + } + + return nil + }) + + switch { + case errors.Is(err, fs.ErrNotExist): + return nil, nil + case err != nil: + return nil, lazyerrors.Error(err) + } + + if limit > 0 && len(files) > limit { + f := make([]string, limit) + for fI, filesI := range rand.Perm(len(files))[:limit] { + f[fI] = files[filesI] + } + files = f + } + + var res []Record + + for _, file := range files { + r, err := loadRecordFile(file) + if err != nil { + return nil, lazyerrors.Errorf("%s: %w", file, err) + } + + res = append(res, r...) + } + + return res, nil +} + +// loadRecordFile parses a single .bin file. +func loadRecordFile(file string) ([]Record, error) { + f, err := os.Open(file) + if err != nil { + return nil, lazyerrors.Error(err) + } + + defer f.Close() //nolint:errcheck // we are only reading it + + r := bufio.NewReader(f) + + var res []Record + + for { + header, body, err := ReadMessage(r) + if errors.Is(err, ErrZeroRead) { + break + } + + if err != nil { + return nil, lazyerrors.Error(err) + } + + headerB, err := header.MarshalBinary() + if err != nil { + return nil, lazyerrors.Error(err) + } + + bodyB, err := body.MarshalBinary() + if err != nil { + return nil, lazyerrors.Error(err) + } + + res = append(res, Record{ + Header: header, + Body: body, + HeaderB: headerB, + BodyB: bodyB, + }) + } + + return res, nil +} diff --git a/internal/wire/records_test.go b/internal/wire/records_test.go deleted file mode 100644 index 37291ae0d7dc..000000000000 --- a/internal/wire/records_test.go +++ /dev/null @@ -1,115 +0,0 @@ -// Copyright 2021 FerretDB Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package wire - -import ( - "bufio" - "encoding/binary" - "errors" - "io/fs" - "math/rand" - "os" - "path/filepath" - - "github.com/FerretDB/FerretDB/internal/util/lazyerrors" -) - -// loadRecords gets recursively all .bin files from the recordsPath directory, -// parses their content to wire Msgs and returns them as an array of testCase -// structs with headerB and bodyB fields set. -// If no records are found, it returns nil and no error. -func loadRecords(recordsPath string) ([]testCase, error) { - // Load recursively every file path with ".bin" extension from recordsPath directory - var recordFiles []string - - err := filepath.WalkDir(recordsPath, func(path string, entry fs.DirEntry, err error) error { - if err != nil { - return err - } - - if filepath.Ext(entry.Name()) == ".bin" { - recordFiles = append(recordFiles, path) - } - - return nil - }) - - switch { - case os.IsNotExist(err): - return nil, nil - case err != nil: - return nil, err - } - - // Select random 1000 number of files from an array of files - if len(recordFiles) > 1000 { - idx := rand.Perm(len(recordFiles)) - idx = idx[:1000] - - sel := []string{} - for _, i := range idx { - sel = append(sel, recordFiles[i]) - } - recordFiles = sel - } - - var resMsgs []testCase - - // Read every record file, parse their content to wire messages - // and store them in the testCase struct - for _, path := range recordFiles { - f, err := os.Open(path) - if err != nil { - return nil, lazyerrors.Errorf("%s: %w", path, err) - } - - defer f.Close() - - r := bufio.NewReader(f) - - for { - header, body, err := ReadMessage(r) - if errors.Is(err, ErrZeroRead) { - break - } - if err != nil { - return nil, lazyerrors.Errorf("%s: %w", path, err) - } - - headBytes, err := header.MarshalBinary() - if err != nil { - return nil, lazyerrors.Errorf("%s: %w", path, err) - } - - bodyBytes, err := body.MarshalBinary() - if err != nil { - return nil, lazyerrors.Errorf("%s: %w", path, err) - } - - // unset flagBit (if present) - flag := binary.LittleEndian.Uint32(bodyBytes[:kFlagBitSize]) - flag &^= 0x01 // flips the LSB of the flagbit - - binary.LittleEndian.PutUint32(bodyBytes[:kFlagBitSize], flag) - - resMsgs = append(resMsgs, testCase{ - headerB: headBytes, - bodyB: bodyBytes, - }) - } - } - - return resMsgs, nil -} diff --git a/internal/wire/wire_test.go b/internal/wire/wire_test.go index 5bac08729c02..5a6fbbf6e086 100644 --- a/internal/wire/wire_test.go +++ b/internal/wire/wire_test.go @@ -50,6 +50,27 @@ type testCase struct { err string // unwrapped } +// setExpectedB checks and sets expectedB fields from headerB and bodyB. +func (tc *testCase) setExpectedB(tb testing.TB) { + tb.Helper() + + if (len(tc.headerB) == 0) != (len(tc.bodyB) == 0) { + tb.Fatalf("header dump and body dump are not in sync") + } + + if (len(tc.headerB) == 0) == (len(tc.expectedB) == 0) { + tb.Fatalf("header/body dumps and expectedB are not in sync") + } + + if len(tc.expectedB) == 0 { + tc.expectedB = make([]byte, 0, len(tc.headerB)+len(tc.bodyB)) + tc.expectedB = append(tc.expectedB, tc.headerB...) + tc.expectedB = append(tc.expectedB, tc.bodyB...) + tc.headerB = nil + tc.bodyB = nil + } +} + func testMessages(t *testing.T, testCases []testCase) { for _, tc := range testCases { tc := tc @@ -58,19 +79,7 @@ func testMessages(t *testing.T, testCases []testCase) { require.NotEmpty(t, tc.name, "name should not be empty") - if (len(tc.headerB) == 0) != (len(tc.bodyB) == 0) { - t.Fatalf("header dump and body dump are not in sync") - } - if (len(tc.headerB) == 0) == (len(tc.expectedB) == 0) { - t.Fatalf("header/body dumps and expectedB are not in sync") - } - - if len(tc.expectedB) == 0 { - expectedB := make([]byte, 0, len(tc.headerB)+len(tc.bodyB)) - expectedB = append(expectedB, tc.headerB...) - expectedB = append(expectedB, tc.bodyB...) - tc.expectedB = expectedB - } + tc.setExpectedB(t) t.Run("ReadMessage", func(t *testing.T) { t.Parallel() @@ -128,18 +137,22 @@ func testMessages(t *testing.T, testCases []testCase) { func fuzzMessages(f *testing.F, testCases []testCase) { for _, tc := range testCases { + tc.setExpectedB(f) f.Add(tc.expectedB) } if !testing.Short() { - records, err := loadRecords(filepath.Join("..", "..", "tmp", "records")) + records, err := LoadRecords(filepath.Join("..", "..", "tmp", "records"), 1000) require.NoError(f, err) - f.Logf("%d recorded messages were added to the seed corpus", len(records)) - for _, rec := range records { - f.Add(rec.bodyB) + b := make([]byte, 0, len(rec.HeaderB)+len(rec.BodyB)) + b = append(b, rec.HeaderB...) + b = append(b, rec.BodyB...) + f.Add(b) } + + f.Logf("%d recorded messages were added to the seed corpus", len(records)) } f.Fuzz(func(t *testing.T, b []byte) { From 72df2bc39deab67015993dff30687514e49170e2 Mon Sep 17 00:00:00 2001 From: Patryk Kwiatek Date: Fri, 23 Jun 2023 04:55:46 +0200 Subject: [PATCH 17/46] Add operators support for all aggregation stages (#2850) --- .../aggregate_documents_compat_test.go | 174 +++++++++++++++++- .../handlers/common/add_fields_iterator.go | 72 ++++++++ .../aggregations/operators/operators.go | 66 +++---- .../operators/operators_errors.go | 9 - .../common/aggregations/operators/type.go | 10 +- .../stages/projection/projection.go | 41 ++++- .../common/aggregations/stages/validate.go | 11 +- internal/handlers/commonerrors/error.go | 4 + .../handlers/commonerrors/errorcode_string.go | 48 ++--- 9 files changed, 337 insertions(+), 98 deletions(-) diff --git a/integration/aggregate_documents_compat_test.go b/integration/aggregate_documents_compat_test.go index b817c7bf0c8b..d36cf7e673ae 100644 --- a/integration/aggregate_documents_compat_test.go +++ b/integration/aggregate_documents_compat_test.go @@ -1597,6 +1597,40 @@ func TestAggregateCompatProject(t *testing.T) { }, resultType: emptyResult, }, + "EmptyDocument": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{{"foo", bson.D{}}}}}, + }, + resultType: emptyResult, + }, + "Document": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{{"foo", bson.D{{"v", "foo"}}}}}}, + }, + }, + "IDDocument": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{{"_id", bson.D{{"v", "foo"}}}}}}, + }, + }, + "IDType": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{{"_id", bson.D{{"$type", "$v"}}}}}}, + }, + }, + "DocumentAndValue": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{ + {"foo", bson.D{{"v", "foo"}}}, + {"v", 1}, + }}}, + }, + }, "Type": { pipeline: bson.A{ bson.D{{"$sort", bson.D{{"_id", -1}}}}, @@ -1618,20 +1652,20 @@ func TestAggregateCompatProject(t *testing.T) { "TypeRecursive": { pipeline: bson.A{ bson.D{{"$sort", bson.D{{"_id", -1}}}}, - bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"$type", bson.D{{"$type", "$v"}}}}}}}}}}, + bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"$type", "$v"}}}}}}}}, }, }, "TypeRecursiveNonExistent": { pipeline: bson.A{ bson.D{{"$sort", bson.D{{"_id", -1}}}}, - bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"$type", bson.D{{"$non-existent", "$v"}}}}}}}}}}, + bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"$non-existent", "$v"}}}}}}}}, }, resultType: emptyResult, }, "TypeRecursiveInvalid": { pipeline: bson.A{ bson.D{{"$sort", bson.D{{"_id", -1}}}}, - bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"$type", bson.D{{"v", "$v"}}}}}}}}}}, + bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"v", "$v"}}}}}}}}, }, }, "TypeRecursiveArrayInvalid": { @@ -1666,6 +1700,12 @@ func TestAggregateCompatProject(t *testing.T) { bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{{"foo", "bar"}}}}}}}}, }, }, + "TypeEmpty": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$project", bson.D{{"type", bson.D{{"$type", bson.D{}}}}}}}, + }, + }, "TypeArraySingleItem": { pipeline: bson.A{ bson.D{{"$sort", bson.D{{"_id", -1}}}}, @@ -1806,6 +1846,134 @@ func TestAggregateCompatAddFields(t *testing.T) { resultType: emptyResult, skip: "https://github.com/FerretDB/FerretDB/issues/1413", }, + + "InvalidOperator": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"value", bson.D{{"$invalid-operator", "foo"}}}}}}, + }, + resultType: emptyResult, + }, + "Type": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "$v"}}}}}}, + }, + }, + "TypeNonExistent": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "$foo"}}}}}}, + }, + }, + "TypeDotNotation": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "$v.foo"}}}}}}, + }, + }, + "TypeRecursive": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.D{{"$type", "$v"}}}}}}}}, + }, + }, + "TypeRecursiveNonExistent": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.D{{"$non-existent", "$v"}}}}}}}}, + }, + resultType: emptyResult, + }, + "TypeRecursiveInvalid": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.D{{"v", "$v"}}}}}}}}, + }, + }, + + "TypeInt": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", int32(42)}}}}}}, + }, + }, + "TypeLong": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", int64(42)}}}}}}, + }, + }, + "TypeString": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "42"}}}}}}, + }, + }, + "TypeDocument": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.D{{"foo", "bar"}}}}}}}}, + }, + }, + "TypeEmpty": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.D{}}}}}}}, + }, + }, + "MultipleOperators": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "foo"}, {"$operator", "foo"}}}}}}, + }, + resultType: emptyResult, + }, + "MultipleOperatorFirst": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", "foo"}, {"not-operator", "foo"}}}}}}, + }, + resultType: emptyResult, + }, + "MultipleOperatorLast": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"not-operator", "foo"}, {"$type", "foo"}}}}}}, + }, + resultType: emptyResult, + }, + "TypeArraySingleItem": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.A{int32(42)}}}}}}}, + }, + }, + "TypeArray": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.A{"foo", "bar"}}}}}}}, + }, + resultType: emptyResult, + }, + "TypeNestedArray": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", bson.A{bson.A{"foo", "bar"}}}}}}}}, + }, + }, + "TypeObjectID": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", primitive.NewObjectID()}}}}}}, + }, + }, + "TypeBool": { + pipeline: bson.A{ + bson.D{{"$sort", bson.D{{"_id", -1}}}}, + bson.D{{"$addFields", bson.D{{"type", bson.D{{"$type", true}}}}}}, + }, + }, } testAggregateStagesCompat(t, testCases) diff --git a/internal/handlers/common/add_fields_iterator.go b/internal/handlers/common/add_fields_iterator.go index 2c4ab941cd5a..aa133ee417f7 100644 --- a/internal/handlers/common/add_fields_iterator.go +++ b/internal/handlers/common/add_fields_iterator.go @@ -15,6 +15,10 @@ package common import ( + "errors" + + "github.com/FerretDB/FerretDB/internal/handlers/common/aggregations/operators" + "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/iterator" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" @@ -54,6 +58,24 @@ func (iter *addFieldsIterator) Next() (struct{}, *types.Document, error) { for _, key := range iter.newField.Keys() { val := must.NotFail(iter.newField.Get(key)) + + switch v := val.(type) { + case *types.Document: + if !operators.IsOperator(v) { + break + } + + op, err := operators.NewOperator(v) + if err = processAddFieldsError(err); err != nil { + return unused, nil, err + } + + val, err = op.Process(doc) + if err = processAddFieldsError(err); err != nil { + return unused, nil, err + } + } + doc.Set(key, val) } @@ -65,6 +87,56 @@ func (iter *addFieldsIterator) Close() { iter.iter.Close() } +// processAddFieldsError takes internal error related to operator evaluation and +// returns proper CommandError that can be returned by $addFields aggregation stage. +func processAddFieldsError(err error) error { + if err == nil { + return nil + } + + var opErr operators.OperatorError + + if !errors.As(err, &opErr) { + return err + } + + switch opErr.Code() { + case operators.ErrTooManyFields: + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrAddFieldsExpressionWrongAmountOfArgs, + "Invalid $addFields :: caused by :: FieldPath field names may not start with '$'."+ + " Consider using $getField or $setField.", + "$addFields (stage)", + ) + case operators.ErrNotImplemented: + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrNotImplemented, + "Invalid $addFields :: caused by :: "+opErr.Error(), + "$addFields (stage)", + ) + case operators.ErrArgsInvalidLen: + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrOperatorWrongLenOfArgs, + "Invalid $addFields :: caused by :: "+opErr.Error(), + "$addFields (stage)", + ) + case operators.ErrInvalidExpression: + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidPipelineOperator, + "Invalid $addFields :: caused by :: "+opErr.Error(), + "$addFields (stage)", + ) + case operators.ErrInvalidNestedExpression: + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidPipelineOperator, + "Invalid $addFields :: caused by :: "+opErr.Error(), + "$addFields (stage)", + ) + default: + return lazyerrors.Error(err) + } +} + // check interfaces var ( _ types.DocumentsIterator = (*addFieldsIterator)(nil) diff --git a/internal/handlers/common/aggregations/operators/operators.go b/internal/handlers/common/aggregations/operators/operators.go index 667e4e03380f..498363576582 100644 --- a/internal/handlers/common/aggregations/operators/operators.go +++ b/internal/handlers/common/aggregations/operators/operators.go @@ -45,60 +45,50 @@ type Operator interface { Process(in *types.Document) (any, error) } -// NewOperator returns operator from provided document. -// The document should look like: `{<$operator>: }`. -func NewOperator(doc any) (Operator, error) { - operatorDoc, ok := doc.(*types.Document) - if !ok { - return nil, newOperatorError( - ErrWrongType, - "Invalid type of operator field (expected document)", - ) - } - - if operatorDoc.Len() == 0 { - return nil, newOperatorError( - ErrEmptyField, - "The operator field is empty (expected document)", - ) - } - - iter := operatorDoc.Iterator() +// IsOperator returns true if provided document should be +// treated as operator document. +func IsOperator(doc *types.Document) bool { + iter := doc.Iterator() defer iter.Close() - var operatorExists bool - for { - k, _, err := iter.Next() - if errors.Is(err, iterator.ErrIteratorDone) { - break - } - + key, _, err := iter.Next() if err != nil { - return nil, lazyerrors.Error(err) + if errors.Is(err, iterator.ErrIteratorDone) { + break + } + + return false } - if strings.HasPrefix(k, "$") { - operatorExists = true - break + if strings.HasPrefix(key, "$") { + return true } } - switch { - case !operatorExists: - return nil, newOperatorError( - ErrNoOperator, - "No operator in document", + return false +} + +// NewOperator returns operator from provided document. +// The document should look like: `{<$operator>: }`. +// +// Before calling NewOperator on document it's recommended to validate +// document before by using IsOperator on it. +func NewOperator(doc *types.Document) (Operator, error) { + if doc.Len() == 0 { + return nil, lazyerrors.New( + "The operator field is empty", ) + } - case operatorDoc.Len() > 1: + if doc.Len() > 1 { return nil, newOperatorError( ErrTooManyFields, "The operator field specifies more than one operator", ) } - operator := operatorDoc.Command() + operator := doc.Command() newOperator, supported := Operators[operator] _, unsupported := unsupportedOperators[operator] @@ -107,7 +97,7 @@ func NewOperator(doc any) (Operator, error) { case supported && unsupported: panic(fmt.Sprintf("operator %q is in both `operators` and `unsupportedOperators`", operator)) case supported && !unsupported: - return newOperator(operatorDoc) + return newOperator(doc) case !supported && unsupported: return nil, newOperatorError( ErrNotImplemented, diff --git a/internal/handlers/common/aggregations/operators/operators_errors.go b/internal/handlers/common/aggregations/operators/operators_errors.go index 7dafcedc59ca..3f28d769aacf 100644 --- a/internal/handlers/common/aggregations/operators/operators_errors.go +++ b/internal/handlers/common/aggregations/operators/operators_errors.go @@ -24,12 +24,6 @@ const ( // ErrArgsInvalidLen indicates that operator have invalid amount of arguments. ErrArgsInvalidLen - // ErrWrongType indicates that operator field is not a document. - ErrWrongType - - // ErrEmptyField indicates that operator field is empty. - ErrEmptyField - // ErrTooManyFields indicates that operator field specifes more than one operators. ErrTooManyFields @@ -41,9 +35,6 @@ const ( // ErrInvalidNestedExpression indicates that operator inside the target operator does not exist. ErrInvalidNestedExpression - - // ErrNoOperator indicates that given document does not contain any operator. - ErrNoOperator ) // newOperatorError returns new OperatorError. diff --git a/internal/handlers/common/aggregations/operators/type.go b/internal/handlers/common/aggregations/operators/type.go index 91c149cec79a..a8da847b6017 100644 --- a/internal/handlers/common/aggregations/operators/type.go +++ b/internal/handlers/common/aggregations/operators/type.go @@ -55,6 +55,11 @@ func (t *typeOp) Process(doc *types.Document) (any, error) { switch param := typeParam.(type) { case *types.Document: + if !IsOperator(param) { + res = param + break + } + operator, err := NewOperator(param) if err != nil { var opErr OperatorError @@ -62,11 +67,6 @@ func (t *typeOp) Process(doc *types.Document) (any, error) { return nil, lazyerrors.Error(err) } - if opErr.Code() == ErrNoOperator { - res = param - continue - } - if opErr.Code() == ErrInvalidExpression { opErr.code = ErrInvalidNestedExpression } diff --git a/internal/handlers/common/aggregations/stages/projection/projection.go b/internal/handlers/common/aggregations/stages/projection/projection.go index 3a7fb13a0511..78a25e0a2d35 100644 --- a/internal/handlers/common/aggregations/stages/projection/projection.go +++ b/internal/handlers/common/aggregations/stages/projection/projection.go @@ -141,6 +141,22 @@ func ValidateProjection(projection *types.Document) (*types.Document, bool, erro switch value := value.(type) { case *types.Document: + if !operators.IsOperator(value) { + if value.Len() == 0 { + return nil, false, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrEmptySubProject, + "Invalid $project :: caused by :: An empty sub-projection is not a valid value."+ + " Found empty object at path", + "$project (stage)", + ) + } + + validated.Set(key, value) + result = true + + break + } + op, err := operators.NewOperator(value) if err = processOperatorError(err); err != nil { return nil, false, err @@ -232,16 +248,24 @@ func ProjectDocument(doc, projection *types.Document, inclusion bool) (*types.Do var op operators.Operator var value any + if !operators.IsOperator(idValue) { + projected.Set("_id", idValue) + set = true + + break + } + op, err = operators.NewOperator(idValue) if err != nil { return nil, processOperatorError(err) } - value, err = op.Process(projected) + value, err = op.Process(doc) if err != nil { return nil, err } + set = true projected.Set("_id", value) case *types.Array, string, types.Binary, types.ObjectID, @@ -249,6 +273,7 @@ func ProjectDocument(doc, projection *types.Document, inclusion bool) (*types.Do projected.Set("_id", idValue) set = true + case bool: set = idValue @@ -312,6 +337,11 @@ func projectDocumentWithoutID(doc *types.Document, projection *types.Document, i var op operators.Operator var v any + if !operators.IsOperator(value) { + projected.Set(key, value) + break + } + op, err = operators.NewOperator(value) if err != nil { return nil, processOperatorError(err) @@ -605,13 +635,6 @@ func processOperatorError(err error) error { } switch opErr.Code() { - case operators.ErrEmptyField: - return commonerrors.NewCommandErrorMsgWithArgument( - commonerrors.ErrEmptySubProject, - "Invalid $project :: caused by :: An empty sub-projection is not a valid value."+ - " Found empty object at path", - "$project (stage)", - ) case operators.ErrTooManyFields: return commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrFieldPathInvalidName, @@ -643,8 +666,6 @@ func processOperatorError(err error) error { "Invalid $project :: caused by :: "+opErr.Error(), "$project (stage)", ) - case operators.ErrNoOperator, operators.ErrWrongType: - fallthrough default: return lazyerrors.Error(err) } diff --git a/internal/handlers/common/aggregations/stages/validate.go b/internal/handlers/common/aggregations/stages/validate.go index d1a3558f81bd..06dc1dcf4847 100644 --- a/internal/handlers/common/aggregations/stages/validate.go +++ b/internal/handlers/common/aggregations/stages/validate.go @@ -35,7 +35,7 @@ func validateExpression(stage string, doc *types.Document) error { defer iter.Close() for { - k, v, err := iter.Next() + _, v, err := iter.Next() if errors.Is(err, iterator.ErrIteratorDone) { return nil } @@ -44,15 +44,6 @@ func validateExpression(stage string, doc *types.Document) error { return lazyerrors.Error(err) } - if strings.HasPrefix(k, "$") { - // TODO: https://github.com/FerretDB/FerretDB/issues/2165 - return commonerrors.NewCommandErrorMsgWithArgument( - commonerrors.ErrNotImplemented, - fmt.Sprintf("%s operator is not implemented for %s key expression yet", k, stage), - fmt.Sprintf("%s (stage)", stage), - ) - } - switch value := v.(type) { case *types.Document: if err := validateExpression(stage, value); err != nil { diff --git a/internal/handlers/commonerrors/error.go b/internal/handlers/commonerrors/error.go index d619aff1fa50..fb9bc3534b46 100644 --- a/internal/handlers/commonerrors/error.go +++ b/internal/handlers/commonerrors/error.go @@ -231,6 +231,10 @@ const ( // ErrStageCountBadValue indicates that $count stage contains invalid value. ErrStageCountBadValue = ErrorCode(40160) // Location40160 + // ErrAddFieldsExpressionWrongAmountOfArgs indicates that $addFields stage expression contain invalid + // amount of arguments. + ErrAddFieldsExpressionWrongAmountOfArgs = ErrorCode(40181) // Location40181 + // ErrStageGroupUnaryOperator indicates that $sum is a unary operator. ErrStageGroupUnaryOperator = ErrorCode(40237) // Location40237 diff --git a/internal/handlers/commonerrors/errorcode_string.go b/internal/handlers/commonerrors/errorcode_string.go index a54647fcf5d7..7522b5da5b16 100644 --- a/internal/handlers/commonerrors/errorcode_string.go +++ b/internal/handlers/commonerrors/errorcode_string.go @@ -74,6 +74,7 @@ func _() { _ = x[ErrStageCountNonEmptyString-40157] _ = x[ErrStageCountBadPrefix-40158] _ = x[ErrStageCountBadValue-40160] + _ = x[ErrAddFieldsExpressionWrongAmountOfArgs-40181] _ = x[ErrStageGroupUnaryOperator-40237] _ = x[ErrStageGroupMultipleAccumulator-40238] _ = x[ErrStageGroupInvalidAccumulator-40234] @@ -98,7 +99,7 @@ func _() { _ = x[ErrStageCollStatsInvalidArg-5447000] } -const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseUnauthorizedTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" +const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseUnauthorizedTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40181Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" var _ErrorCode_map = map[ErrorCode]string{ 0: _ErrorCode_name[0:5], @@ -166,28 +167,29 @@ var _ErrorCode_map = map[ErrorCode]string{ 40157: _ErrorCode_name[897:910], 40158: _ErrorCode_name[910:923], 40160: _ErrorCode_name[923:936], - 40234: _ErrorCode_name[936:949], - 40237: _ErrorCode_name[949:962], - 40238: _ErrorCode_name[962:975], - 40272: _ErrorCode_name[975:988], - 40323: _ErrorCode_name[988:1001], - 40352: _ErrorCode_name[1001:1014], - 40353: _ErrorCode_name[1014:1027], - 40414: _ErrorCode_name[1027:1040], - 40415: _ErrorCode_name[1040:1053], - 50840: _ErrorCode_name[1053:1066], - 51024: _ErrorCode_name[1066:1079], - 51075: _ErrorCode_name[1079:1092], - 51091: _ErrorCode_name[1092:1105], - 51108: _ErrorCode_name[1105:1118], - 51246: _ErrorCode_name[1118:1131], - 51247: _ErrorCode_name[1131:1144], - 51270: _ErrorCode_name[1144:1157], - 51272: _ErrorCode_name[1157:1170], - 4822819: _ErrorCode_name[1170:1185], - 5107200: _ErrorCode_name[1185:1200], - 5107201: _ErrorCode_name[1200:1215], - 5447000: _ErrorCode_name[1215:1230], + 40181: _ErrorCode_name[936:949], + 40234: _ErrorCode_name[949:962], + 40237: _ErrorCode_name[962:975], + 40238: _ErrorCode_name[975:988], + 40272: _ErrorCode_name[988:1001], + 40323: _ErrorCode_name[1001:1014], + 40352: _ErrorCode_name[1014:1027], + 40353: _ErrorCode_name[1027:1040], + 40414: _ErrorCode_name[1040:1053], + 40415: _ErrorCode_name[1053:1066], + 50840: _ErrorCode_name[1066:1079], + 51024: _ErrorCode_name[1079:1092], + 51075: _ErrorCode_name[1092:1105], + 51091: _ErrorCode_name[1105:1118], + 51108: _ErrorCode_name[1118:1131], + 51246: _ErrorCode_name[1131:1144], + 51247: _ErrorCode_name[1144:1157], + 51270: _ErrorCode_name[1157:1170], + 51272: _ErrorCode_name[1170:1183], + 4822819: _ErrorCode_name[1183:1198], + 5107200: _ErrorCode_name[1198:1213], + 5107201: _ErrorCode_name[1213:1228], + 5447000: _ErrorCode_name[1228:1243], } func (i ErrorCode) String() string { From 00aeea87da254b76297c64f2c0d8ecf0275ce268 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Fri, 23 Jun 2023 12:17:54 +0400 Subject: [PATCH 18/46] Unskip test that passes now (#2885) All TLS parameters are now a part of URI. Closes #1759. --- integration/commands_diagnostic_test.go | 2 -- internal/handlers/common/aggregations/stages/group.go | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/integration/commands_diagnostic_test.go b/integration/commands_diagnostic_test.go index 672351dbb785..7a1a2085b24c 100644 --- a/integration/commands_diagnostic_test.go +++ b/integration/commands_diagnostic_test.go @@ -346,8 +346,6 @@ func TestCommandsDiagnosticValidateError(t *testing.T) { } func TestCommandsDiagnosticWhatsMyURI(t *testing.T) { - t.Skip("https://github.com/FerretDB/FerretDB/issues/1759") - t.Parallel() s := setup.SetupWithOpts(t, nil) diff --git a/internal/handlers/common/aggregations/stages/group.go b/internal/handlers/common/aggregations/stages/group.go index 2fc5c92adfa5..d12bdb316542 100644 --- a/internal/handlers/common/aggregations/stages/group.go +++ b/internal/handlers/common/aggregations/stages/group.go @@ -117,6 +117,7 @@ func newGroup(stage *types.Document) (aggregations.Stage, error) { // Process implements Stage interface. func (g *group) Process(ctx context.Context, iter types.DocumentsIterator, closer *iterator.MultiCloser) (types.DocumentsIterator, error) { //nolint:lll // for readability + // TODO https://github.com/FerretDB/FerretDB/issues/2863 docs, err := iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) if err != nil { return nil, lazyerrors.Error(err) From bbed64014dd515d8bfbf057e97722a0068f7e544 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Fri, 23 Jun 2023 16:12:06 +0400 Subject: [PATCH 19/46] Tweak contributing guidelines (#2886) --- CONTRIBUTING.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c4ec59dfae43..5d83a6b0d0ca 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -63,14 +63,15 @@ You can also [install `task` globally](https://taskfile.dev/#/installation), but that might lead to the version skew. With `task` installed, -you should install development tools with `task init` +you can see all available tasks by running `task -l` in the root of the repository +(not in the `tools` directory). + +After that, you should install development tools with `task init` and download required Docker images with `task env-pull`. If something does not work correctly, you can reset the environment with `task env-reset`. -You can see all available `task` tasks with `task -l`. - ### Building a production release binary To build a production release binary, run `task build-release`. From a1ff494a25195749c53a03f206b5a7c5f424267b Mon Sep 17 00:00:00 2001 From: Matt Cornillon Date: Fri, 23 Jun 2023 15:18:10 +0200 Subject: [PATCH 20/46] Fix Github link for Dance repository (#2887) --- website/docs/contributing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/contributing/index.md b/website/docs/contributing/index.md index 586627d02dad..621e93eab421 100644 --- a/website/docs/contributing/index.md +++ b/website/docs/contributing/index.md @@ -38,7 +38,7 @@ More information on contributing to the documentation can be found [here](https: ## Contributing to dance -- To start contributing to the [dance repository](https://github.com/FerretDB/github-actions) for integration testing, follow the guidelines in the [CONTRIBUTING.md](https://github.com/FerretDB/dance/blob/main/CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](https://github.com/FerretDB/dance/blob/main/CODE_OF_CONDUCT.md) files. +- To start contributing to the [dance repository](https://github.com/FerretDB/dance) for integration testing, follow the guidelines in the [CONTRIBUTING.md](https://github.com/FerretDB/dance/blob/main/CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](https://github.com/FerretDB/dance/blob/main/CODE_OF_CONDUCT.md) files. - Looking for something to work on? Check out the [open issues](https://github.com/FerretDB/dance/issues) for this repository. From 82f54cda4968fc641f43ceef998c4cb0ede238af Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Fri, 23 Jun 2023 22:41:01 +0900 Subject: [PATCH 21/46] Support cursors for aggregation pipelines (#2861) Closes #1892. --- integration/aggregate_documents_test.go | 252 ++++++++++++++++++++++++ integration/query_test.go | 91 +++++---- internal/handlers/common/getmore.go | 7 +- internal/handlers/pg/msg_aggregate.go | 61 +++++- internal/handlers/pg/msg_find.go | 4 - 5 files changed, 362 insertions(+), 53 deletions(-) diff --git a/integration/aggregate_documents_test.go b/integration/aggregate_documents_test.go index 2a3edb96c066..84afdacfa52c 100644 --- a/integration/aggregate_documents_test.go +++ b/integration/aggregate_documents_test.go @@ -17,10 +17,13 @@ package integration import ( "testing" + "github.com/AlekSi/pointer" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" "go.mongodb.org/mongo-driver/mongo" + "go.mongodb.org/mongo-driver/mongo/options" "github.com/FerretDB/FerretDB/integration/setup" ) @@ -659,3 +662,252 @@ func TestAggregateUnsetErrors(t *testing.T) { }) } } + +func TestAggregateCommandCursor(t *testing.T) { + t.Parallel() + ctx, collection := setup.Setup(t) + + // the number of documents is set above the default batchSize of 101 + // for testing unset batchSize returning default batchSize + docs := generateDocuments(0, 110) + _, err := collection.InsertMany(ctx, docs) + require.NoError(t, err) + + for name, tc := range map[string]struct { //nolint:vet // used for testing only + pipeline any // optional, defaults to bson.A{} + cursor any // optional, nil to leave cursor unset + + firstBatch primitive.A // optional, expected firstBatch + err *mongo.CommandError // optional, expected error from MongoDB + altMessage string // optional, alternative error message for FerretDB, ignored if empty + skip string // optional, skip test with a specified reason + }{ + "Int": { + cursor: bson.D{{"batchSize", 1}}, + firstBatch: docs[:1], + }, + "Long": { + cursor: bson.D{{"batchSize", int64(2)}}, + firstBatch: docs[:2], + }, + "LongZero": { + cursor: bson.D{{"batchSize", int64(0)}}, + firstBatch: bson.A{}, + }, + "LongNegative": { + cursor: bson.D{{"batchSize", int64(-1)}}, + err: &mongo.CommandError{ + Code: 51024, + Name: "Location51024", + Message: "BSON field 'batchSize' value must be >= 0, actual value '-1'", + }, + altMessage: "BSON field 'batchSize' value must be >= 0, actual value '-1'", + }, + "DoubleZero": { + cursor: bson.D{{"batchSize", float64(0)}}, + firstBatch: bson.A{}, + }, + "DoubleNegative": { + cursor: bson.D{{"batchSize", -1.1}}, + err: &mongo.CommandError{ + Code: 51024, + Name: "Location51024", + Message: "BSON field 'batchSize' value must be >= 0, actual value '-1'", + }, + }, + "DoubleFloor": { + cursor: bson.D{{"batchSize", 1.9}}, + firstBatch: docs[:1], + }, + "Bool": { + cursor: bson.D{{"batchSize", true}}, + firstBatch: docs[:1], + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "BSON field 'cursor.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double']", + }, + altMessage: "BSON field 'aggregate.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double]'", + }, + "Unset": { + cursor: nil, + firstBatch: docs[:101], + err: &mongo.CommandError{ + Code: 9, + Name: "FailedToParse", + Message: "The 'cursor' option is required, except for aggregate with the explain argument", + }, + }, + "Empty": { + cursor: bson.D{}, + firstBatch: docs[:101], + }, + "String": { + cursor: "invalid", + firstBatch: docs[:101], + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "cursor field must be missing or an object", + }, + altMessage: "BSON field 'cursor' is the wrong type 'string', expected type 'object'", + }, + "LargeBatchSize": { + cursor: bson.D{{"batchSize", 102}}, + firstBatch: docs[:102], + }, + "LargeBatchSizeMatch": { + pipeline: bson.A{ + bson.D{{"$match", bson.D{{"_id", bson.D{{"$in", bson.A{0, 1, 2, 3, 4, 5}}}}}}}, + }, + cursor: bson.D{{"batchSize", 102}}, + firstBatch: docs[:6], + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Parallel() + + var pipeline any = bson.A{} + if tc.pipeline != nil { + pipeline = tc.pipeline + } + + var rest bson.D + if tc.cursor != nil { + rest = append(rest, bson.E{Key: "cursor", Value: tc.cursor}) + } + + command := append( + bson.D{ + {"aggregate", collection.Name()}, + {"pipeline", pipeline}, + }, + rest..., + ) + + var res bson.D + err := collection.Database().RunCommand(ctx, command).Decode(&res) + if tc.err != nil { + assert.Nil(t, res) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) + + return + } + + require.NoError(t, err) + + v, ok := res.Map()["cursor"] + require.True(t, ok) + + cursor, ok := v.(bson.D) + require.True(t, ok) + + // do not check the value of cursor id, FerretDB has a different id + cursorID := cursor.Map()["id"] + assert.NotNil(t, cursorID) + + firstBatch, ok := cursor.Map()["firstBatch"] + require.True(t, ok) + require.Equal(t, tc.firstBatch, firstBatch) + }) + } +} + +func TestAggregateBatchSize(t *testing.T) { + t.Parallel() + ctx, collection := setup.Setup(t) + + // The test cases call `aggregate`, then may implicitly call `getMore` upon `cursor.Next()`. + // The batchSize set by `aggregate` is used also by `getMore` unless + // `aggregate` has default batchSize or 0 batchSize, then `getMore` has unlimited batchSize. + // To test that, the number of documents is set to more than the double of default batchSize 101. + docs := generateDocuments(0, 220) + _, err := collection.InsertMany(ctx, docs) + require.NoError(t, err) + + t.Run("SetBatchSize", func(t *testing.T) { + t.Parallel() + + cursor, err := collection.Aggregate(ctx, bson.D{}, &options.AggregateOptions{BatchSize: pointer.ToInt32(2)}) + require.NoError(t, err) + + defer cursor.Close(ctx) + + require.Equal(t, 2, cursor.RemainingBatchLength(), "expected 2 documents in first batch") + + for i := 2; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in first batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } + + // batchSize of 2 is applied to second batch which is obtained by implicit call to `getMore` + for i := 2; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in second batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } + + cursor.SetBatchSize(5) + + for i := 5; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in third batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } + + // get rest of documents from the cursor to ensure cursor is exhausted + var res bson.D + err = cursor.All(ctx, &res) + require.NoError(t, err) + + ok := cursor.Next(ctx) + require.False(t, ok, "cursor exhausted, not expecting next document") + }) + + t.Run("ZeroBatchSize", func(t *testing.T) { + t.Parallel() + + cursor, err := collection.Aggregate(ctx, bson.D{}, &options.AggregateOptions{BatchSize: pointer.ToInt32(0)}) + require.NoError(t, err) + + defer cursor.Close(ctx) + + require.Equal(t, 0, cursor.RemainingBatchLength()) + + // next batch obtain from implicit call to `getMore` has the rest of the documents, not 0 batchSize + // TODO: 16MB batchSize limit https://github.com/FerretDB/FerretDB/issues/2824 + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document") + require.Equal(t, 219, cursor.RemainingBatchLength()) + }) + + t.Run("DefaultBatchSize", func(t *testing.T) { + t.Parallel() + + // unset batchSize uses default batchSize 101 for the first batch + cursor, err := collection.Aggregate(ctx, bson.D{}) + require.NoError(t, err) + + defer cursor.Close(ctx) + + require.Equal(t, 101, cursor.RemainingBatchLength()) + + for i := 101; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } + + // next batch obtain from implicit call to `getMore` has the rest of the documents, not default batchSize + // TODO: 16MB batchSize limit https://github.com/FerretDB/FerretDB/issues/2824 + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document") + require.Equal(t, 118, cursor.RemainingBatchLength()) + }) +} diff --git a/integration/query_test.go b/integration/query_test.go index af115d1968a0..6cae94c6d811 100644 --- a/integration/query_test.go +++ b/integration/query_test.go @@ -619,7 +619,8 @@ func TestQueryCommandBatchSize(t *testing.T) { t.Parallel() ctx, collection := setup.Setup(t) - // the number of documents is set to slightly above the default batchSize of 101 + // the number of documents is set above the default batchSize of 101 + // for testing unset batchSize returning default batchSize docs := generateDocuments(0, 110) _, err := collection.InsertMany(ctx, docs) require.NoError(t, err) @@ -839,7 +840,10 @@ func TestQueryBatchSize(t *testing.T) { t.Parallel() ctx, collection := setup.Setup(t) - // the number of documents is set to much bigger than the default batchSize of 101 + // The test cases call `find`, then may implicitly call `getMore` upon `cursor.Next()`. + // The batchSize set by `find` is used also by `getMore` unless + // `find` has default batchSize or 0 batchSize, then `getMore` has unlimited batchSize. + // To test that, the number of documents is set to more than the double of default batchSize 101. docs := generateDocuments(0, 220) _, err := collection.InsertMany(ctx, docs) require.NoError(t, err) @@ -847,78 +851,82 @@ func TestQueryBatchSize(t *testing.T) { t.Run("SetBatchSize", func(t *testing.T) { t.Parallel() - // set BatchSize to 2 cursor, err := collection.Find(ctx, bson.D{}, &options.FindOptions{BatchSize: pointer.ToInt32(2)}) require.NoError(t, err) defer cursor.Close(ctx) - // firstBatch has remaining 2 documents - require.Equal(t, 2, cursor.RemainingBatchLength()) + require.Equal(t, 2, cursor.RemainingBatchLength(), "expected 2 documents in first batch") - // get first document from firstBatch - ok := cursor.Next(ctx) - require.True(t, ok, "expected to have next document") - require.Equal(t, 1, cursor.RemainingBatchLength()) - - // get second document from firstBatch - ok = cursor.Next(ctx) - require.True(t, ok, "expected to have next document") - require.Equal(t, 0, cursor.RemainingBatchLength()) - - // get first document from secondBatch - ok = cursor.Next(ctx) - require.True(t, ok, "expected to have next document") - require.Equal(t, 1, cursor.RemainingBatchLength()) + for i := 2; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in first batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } - // get second document from secondBatch - ok = cursor.Next(ctx) - require.True(t, ok, "expected to have next document") - require.Equal(t, 0, cursor.RemainingBatchLength()) + // batchSize of 2 is applied to second batch which is obtained by implicit call to `getMore` + for i := 2; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in second batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } - // increase batchSize cursor.SetBatchSize(5) - // get first document from thirdBatch - ok = cursor.Next(ctx) - require.True(t, ok, "expected to have next document") - require.Equal(t, 4, cursor.RemainingBatchLength()) + for i := 5; i > 0; i-- { + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document in third batch") + require.Equal(t, i-1, cursor.RemainingBatchLength()) + } - // get rest of documents from the cursor + // get rest of documents from the cursor to ensure cursor is exhausted var res bson.D err = cursor.All(ctx, &res) require.NoError(t, err) - // cursor is exhausted - ok = cursor.Next(ctx) + ok := cursor.Next(ctx) require.False(t, ok, "cursor exhausted, not expecting next document") }) t.Run("DefaultBatchSize", func(t *testing.T) { t.Parallel() - // leave batchSize unset, firstBatch uses default batchSize 101 + // unset batchSize uses default batchSize 101 for the first batch cursor, err := collection.Find(ctx, bson.D{}) require.NoError(t, err) defer cursor.Close(ctx) - // firstBatch has remaining 101 documents require.Equal(t, 101, cursor.RemainingBatchLength()) - // get 101 documents from firstBatch - for i := 0; i < 101; i++ { + for i := 101; i > 0; i-- { ok := cursor.Next(ctx) require.True(t, ok, "expected to have next document") + require.Equal(t, i-1, cursor.RemainingBatchLength()) } + // next batch obtain from implicit call to `getMore` has the rest of the documents, not default batchSize + // TODO: 16MB batchSize limit https://github.com/FerretDB/FerretDB/issues/2824 + ok := cursor.Next(ctx) + require.True(t, ok, "expected to have next document") + require.Equal(t, 118, cursor.RemainingBatchLength()) + }) + + t.Run("ZeroBatchSize", func(t *testing.T) { + t.Parallel() + + cursor, err := collection.Find(ctx, bson.D{}, &options.FindOptions{BatchSize: pointer.ToInt32(0)}) + require.NoError(t, err) + + defer cursor.Close(ctx) + require.Equal(t, 0, cursor.RemainingBatchLength()) - // secondBatch has the rest of the documents, not only 109 documents + // next batch obtain from implicit call to `getMore` has the rest of the documents, not 0 batchSize // TODO: 16MB batchSize limit https://github.com/FerretDB/FerretDB/issues/2824 ok := cursor.Next(ctx) require.True(t, ok, "expected to have next document") - require.Equal(t, 118, cursor.RemainingBatchLength()) + require.Equal(t, 219, cursor.RemainingBatchLength()) }) t.Run("NegativeLimit", func(t *testing.T) { @@ -933,15 +941,13 @@ func TestQueryBatchSize(t *testing.T) { defer cursor.Close(ctx) - // firstBatch has remaining 1 document - require.Equal(t, 1, cursor.RemainingBatchLength()) + require.Equal(t, 1, cursor.RemainingBatchLength(), "expected 1 document in first batch") - // firstBatch contains single document ok := cursor.Next(ctx) require.True(t, ok, "expected to have next document") require.Equal(t, 0, cursor.RemainingBatchLength()) - // there is no remaining batch, cursor is exhausted + // there is no remaining batch due to negative limit ok = cursor.Next(ctx) require.False(t, ok, "cursor exhausted, not expecting next document") require.Equal(t, 0, cursor.RemainingBatchLength()) @@ -952,7 +958,8 @@ func TestQueryCommandGetMore(t *testing.T) { t.Parallel() ctx, collection := setup.Setup(t) - // the number of documents is set to slightly above the default batchSize of 101 + // the number of documents is set above the default batchSize of 101 + // for testing unset batchSize returning default batchSize docs := generateDocuments(0, 110) _, err := collection.InsertMany(ctx, docs) require.NoError(t, err) diff --git a/internal/handlers/common/getmore.go b/internal/handlers/common/getmore.go index a04fa431714e..1e0eb65825ce 100644 --- a/internal/handlers/common/getmore.go +++ b/internal/handlers/common/getmore.go @@ -42,8 +42,8 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* } // TODO: Use ExtractParam https://github.com/FerretDB/FerretDB/issues/2859 - v, err := document.Get("collection") - if err != nil { + v, _ := document.Get("collection") + if v == nil { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrMissingField, "BSON field 'getMore.collection' is missing but a required field", @@ -110,7 +110,8 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* if cursor.DB != db || cursor.Collection != collection { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrUnauthorized, - fmt.Sprintf("Requested getMore on namespace '%s.%s', but cursor belongs to a different namespace %s.%s", + fmt.Sprintf( + "Requested getMore on namespace '%s.%s', but cursor belongs to a different namespace %s.%s", db, collection, cursor.DB, diff --git a/internal/handlers/pg/msg_aggregate.go b/internal/handlers/pg/msg_aggregate.go index 90ad62bf6b9c..9b46e8977a56 100644 --- a/internal/handlers/pg/msg_aggregate.go +++ b/internal/handlers/pg/msg_aggregate.go @@ -22,10 +22,13 @@ import ( "github.com/jackc/pgx/v5" + "github.com/FerretDB/FerretDB/internal/clientconn/conninfo" + "github.com/FerretDB/FerretDB/internal/clientconn/cursor" "github.com/FerretDB/FerretDB/internal/handlers/common" "github.com/FerretDB/FerretDB/internal/handlers/common/aggregations" "github.com/FerretDB/FerretDB/internal/handlers/common/aggregations/stages" "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" + "github.com/FerretDB/FerretDB/internal/handlers/commonparams" "github.com/FerretDB/FerretDB/internal/handlers/pg/pgdb" "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/iterator" @@ -46,8 +49,7 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs return nil, lazyerrors.Error(err) } - // TODO https://github.com/FerretDB/FerretDB/issues/1892 - common.Ignored(document, h.L, "cursor", "lsid") + common.Ignored(document, h.L, "lsid") if err = common.Unimplemented(document, "explain", "collation", "let"); err != nil { return nil, err @@ -130,6 +132,38 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs } } + // validate cursor after validating pipeline stages to keep compatibility + v, _ := document.Get("cursor") + if v == nil { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrFailedToParse, + "The 'cursor' option is required, except for aggregate with the explain argument", + document.Command(), + ) + } + + cursorDoc, ok := v.(*types.Document) + if !ok { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrTypeMismatch, + fmt.Sprintf( + `BSON field 'cursor' is the wrong type '%s', expected type 'object'`, + commonparams.AliasFromType(v), + ), + document.Command(), + ) + } + + v, _ = cursorDoc.Get("batchSize") + if v == nil { + v = int32(101) + } + + batchSize, err := commonparams.GetValidatedNumberParamWithMinValue(document.Command(), "batchSize", v, 0) + if err != nil { + return nil, err + } + var resDocs []*types.Document // At this point we have a list of stages to apply to the documents or stats. @@ -166,7 +200,26 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs return nil, err } - // TODO https://github.com/FerretDB/FerretDB/issues/1892 + var cursorID int64 + + if h.EnableCursors && int64(len(resDocs)) > batchSize { + // Cursor is not created when resDocs is less than batchSize, it all fits in the firstBatch. + iter := iterator.Values(iterator.ForSlice(resDocs)) + c := cursor.New(&cursor.NewParams{ + Iter: iter, + DB: db, + Collection: collection, + BatchSize: int32(batchSize), + }) + username, _ := conninfo.Get(ctx).Auth() + cursorID = h.registry.StoreCursor(username, c) + resDocs, err = iterator.ConsumeValuesN(iter, int(batchSize)) + + if err != nil { + return nil, lazyerrors.Error(err) + } + } + firstBatch := types.MakeArray(len(resDocs)) for _, doc := range resDocs { firstBatch.Append(doc) @@ -177,7 +230,7 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs Documents: []*types.Document{must.NotFail(types.NewDocument( "cursor", must.NotFail(types.NewDocument( "firstBatch", firstBatch, - "id", int64(0), + "id", cursorID, "ns", db+"."+collection, )), "ok", float64(1), diff --git a/internal/handlers/pg/msg_find.go b/internal/handlers/pg/msg_find.go index e8a874b33870..0112352c4a8a 100644 --- a/internal/handlers/pg/msg_find.go +++ b/internal/handlers/pg/msg_find.go @@ -80,10 +80,6 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er var resDocs []*types.Document err = dbPool.InTransaction(ctx, func(tx pgx.Tx) error { - if params.BatchSize == 0 { - return nil - } - var iter types.DocumentsIterator var queryRes pgdb.QueryResults From d2219f2d883917e8a92c550129f6eddcbfd858fc Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Fri, 23 Jun 2023 19:28:56 +0400 Subject: [PATCH 22/46] Add handler's metrics registration (#2895) --- internal/clientconn/conn.go | 16 ---------------- internal/clientconn/listener.go | 2 ++ internal/handlers/hana/hana.go | 11 +++++++++++ internal/handlers/handlers.go | 4 ++++ internal/handlers/pg/pg.go | 11 +++++++++++ internal/handlers/sqlite/sqlite.go | 11 +++++++++++ internal/handlers/tigris/tigris.go | 11 +++++++++++ 7 files changed, 50 insertions(+), 16 deletions(-) diff --git a/internal/clientconn/conn.go b/internal/clientconn/conn.go index 5ff8e98d5b62..0a00c6ba8314 100644 --- a/internal/clientconn/conn.go +++ b/internal/clientconn/conn.go @@ -31,7 +31,6 @@ import ( "time" "github.com/pmezard/go-difflib/difflib" - "github.com/prometheus/client_golang/prometheus" "go.uber.org/zap" "go.uber.org/zap/zapcore" @@ -591,18 +590,3 @@ func (c *conn) logResponse(who string, resHeader *wire.MsgHeader, resBody wire.M return level } - -// Describe implements prometheus.Collector. -func (c *conn) Describe(ch chan<- *prometheus.Desc) { - c.m.Describe(ch) -} - -// Collect implements prometheus.Collector. -func (c *conn) Collect(ch chan<- prometheus.Metric) { - c.m.Collect(ch) -} - -// check interfaces -var ( - _ prometheus.Collector = (*conn)(nil) -) diff --git a/internal/clientconn/listener.go b/internal/clientconn/listener.go index 8a119888629d..6e09b4f3fa36 100644 --- a/internal/clientconn/listener.go +++ b/internal/clientconn/listener.go @@ -339,11 +339,13 @@ func (l *Listener) TLSAddr() net.Addr { // Describe implements prometheus.Collector. func (l *Listener) Describe(ch chan<- *prometheus.Desc) { l.Metrics.Describe(ch) + l.Handler.Describe(ch) } // Collect implements prometheus.Collector. func (l *Listener) Collect(ch chan<- prometheus.Metric) { l.Metrics.Collect(ch) + l.Handler.Collect(ch) } // check interfaces diff --git a/internal/handlers/hana/hana.go b/internal/handlers/hana/hana.go index 6467924f780e..b6b48ecccbef 100644 --- a/internal/handlers/hana/hana.go +++ b/internal/handlers/hana/hana.go @@ -19,6 +19,7 @@ import ( "context" "sync" + "github.com/prometheus/client_golang/prometheus" "go.uber.org/zap" "github.com/FerretDB/FerretDB/internal/clientconn/connmetrics" @@ -110,6 +111,16 @@ func (h *Handler) DBPool(ctx context.Context) (*hanadb.Pool, error) { return p, nil } +// Describe implements handlers.Interface. +func (h *Handler) Describe(ch chan<- *prometheus.Desc) { + // TODO +} + +// Collect implements handlers.Interface. +func (h *Handler) Collect(ch chan<- prometheus.Metric) { + // TODO +} + // check interfaces var ( _ handlers.Interface = (*Handler)(nil) diff --git a/internal/handlers/handlers.go b/internal/handlers/handlers.go index f4a052aba467..268c4b9b579a 100644 --- a/internal/handlers/handlers.go +++ b/internal/handlers/handlers.go @@ -18,6 +18,8 @@ package handlers import ( "context" + "github.com/prometheus/client_golang/prometheus" + "github.com/FerretDB/FerretDB/internal/wire" ) @@ -35,6 +37,8 @@ type Interface interface { // Close gracefully shutdowns handler. Close() + prometheus.Collector + // CmdQuery queries collections for documents. // Used by deprecated OP_QUERY message during connection handshake with an old client. CmdQuery(ctx context.Context, query *wire.OpQuery) (*wire.OpReply, error) diff --git a/internal/handlers/pg/pg.go b/internal/handlers/pg/pg.go index 494decd9d46a..09c51d32fd55 100644 --- a/internal/handlers/pg/pg.go +++ b/internal/handlers/pg/pg.go @@ -21,6 +21,7 @@ import ( "sync" "time" + "github.com/prometheus/client_golang/prometheus" "go.uber.org/zap" "github.com/FerretDB/FerretDB/internal/clientconn/conninfo" @@ -144,6 +145,16 @@ func (h *Handler) DBPool(ctx context.Context) (*pgdb.Pool, error) { return p, nil } +// Describe implements handlers.Interface. +func (h *Handler) Describe(ch chan<- *prometheus.Desc) { + // TODO +} + +// Collect implements handlers.Interface. +func (h *Handler) Collect(ch chan<- prometheus.Metric) { + // TODO +} + // check interfaces var ( _ handlers.Interface = (*Handler)(nil) diff --git a/internal/handlers/sqlite/sqlite.go b/internal/handlers/sqlite/sqlite.go index 67bfceb87c2c..89b98b4f8a4d 100644 --- a/internal/handlers/sqlite/sqlite.go +++ b/internal/handlers/sqlite/sqlite.go @@ -18,6 +18,7 @@ package sqlite import ( + "github.com/prometheus/client_golang/prometheus" "go.uber.org/zap" "github.com/FerretDB/FerretDB/internal/backends" @@ -73,6 +74,16 @@ func New(opts *NewOpts) (handlers.Interface, error) { // Close implements handlers.Interface. func (h *Handler) Close() {} +// Describe implements handlers.Interface. +func (h *Handler) Describe(ch chan<- *prometheus.Desc) { + // TODO +} + +// Collect implements handlers.Interface. +func (h *Handler) Collect(ch chan<- prometheus.Metric) { + // TODO +} + // check interfaces var ( _ handlers.Interface = (*Handler)(nil) diff --git a/internal/handlers/tigris/tigris.go b/internal/handlers/tigris/tigris.go index 08cd3a6ba549..285a8af09d0c 100644 --- a/internal/handlers/tigris/tigris.go +++ b/internal/handlers/tigris/tigris.go @@ -20,6 +20,7 @@ import ( "sync" "time" + "github.com/prometheus/client_golang/prometheus" "github.com/tigrisdata/tigris-client-go/config" "go.uber.org/zap" @@ -147,6 +148,16 @@ func (h *Handler) DBPool(ctx context.Context) (*tigrisdb.TigrisDB, error) { return p, nil } +// Describe implements handlers.Interface. +func (h *Handler) Describe(ch chan<- *prometheus.Desc) { + // TODO +} + +// Collect implements handlers.Interface. +func (h *Handler) Collect(ch chan<- prometheus.Metric) { + // TODO +} + // check interfaces var ( _ handlers.Interface = (*Handler)(nil) From 362758a1e3192f326098c45264db73f26fc48893 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Fri, 23 Jun 2023 21:40:34 +0400 Subject: [PATCH 23/46] Clean-up some code and comments (#2904) --- cmd/envtool/envtool.go | 2 +- ferretdb/ferretdb.go | 2 +- integration/setup/client.go | 4 ++-- integration/setup/listener.go | 6 +++--- integration/setup/setup.go | 3 ++- integration/setup/setup_compat.go | 3 ++- internal/clientconn/conn.go | 11 ++++++++--- internal/clientconn/listener.go | 8 ++++---- internal/handlers/common/count_iterator.go | 2 ++ internal/handlers/commoncommands/msg_listcommands.go | 2 ++ internal/util/iterator/func.go | 1 - internal/util/iterator/slice.go | 1 - internal/util/iterator/values.go | 1 - internal/util/iterator/with_close.go | 1 - internal/util/telemetry/reporter.go | 2 +- 15 files changed, 28 insertions(+), 21 deletions(-) diff --git a/cmd/envtool/envtool.go b/cmd/envtool/envtool.go index 33ec5d4f1e09..4f3d6af8be23 100644 --- a/cmd/envtool/envtool.go +++ b/cmd/envtool/envtool.go @@ -57,7 +57,7 @@ var ( // versionFile contains version information with leading v. const versionFile = "build/version/version.txt" -// waitForPort waits for the given port to be available until ctx is done. +// waitForPort waits for the given port to be available until ctx is canceled. func waitForPort(ctx context.Context, logger *zap.SugaredLogger, port uint16) error { addr := fmt.Sprintf("127.0.0.1:%d", port) logger.Infof("Waiting for %s to be up...", addr) diff --git a/ferretdb/ferretdb.go b/ferretdb/ferretdb.go index 89c194b12238..95bc0016c863 100644 --- a/ferretdb/ferretdb.go +++ b/ferretdb/ferretdb.go @@ -157,7 +157,7 @@ func New(config *Config) (*FerretDB, error) { }, nil } -// Run runs FerretDB until ctx is done. +// Run runs FerretDB until ctx is canceled. // // When this method returns, listener and all connections are closed. func (f *FerretDB) Run(ctx context.Context) error { diff --git a/integration/setup/client.go b/integration/setup/client.go index a2d6a78a7132..91db78793b08 100644 --- a/integration/setup/client.go +++ b/integration/setup/client.go @@ -18,7 +18,6 @@ import ( "context" "net/url" "path/filepath" - "runtime/trace" "testing" "github.com/stretchr/testify/require" @@ -29,6 +28,7 @@ import ( "go.opentelemetry.io/otel" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" + "github.com/FerretDB/FerretDB/internal/util/observability" ) // mongoDBURIOpts represents mongoDBURI's options. @@ -109,7 +109,7 @@ func setupClient(tb testing.TB, ctx context.Context, uri string) *mongo.Client { ctx, span := otel.Tracer("").Start(ctx, "setupClient") defer span.End() - defer trace.StartRegion(ctx, "setupClient").End() + defer observability.FuncCall(ctx)() client, err := makeClient(ctx, uri) if err != nil { diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 169fbcfcc650..9041ed489c6e 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -19,7 +19,6 @@ import ( "errors" "os" "path/filepath" - "runtime/trace" "strings" "sync/atomic" "testing" @@ -32,6 +31,7 @@ import ( "github.com/FerretDB/FerretDB/internal/clientconn" "github.com/FerretDB/FerretDB/internal/clientconn/connmetrics" "github.com/FerretDB/FerretDB/internal/handlers/registry" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/state" ) @@ -63,7 +63,7 @@ func unixSocketPath(tb testing.TB) string { return f.Name() } -// setupListener starts in-process FerretDB server that runs until ctx is done. +// setupListener starts in-process FerretDB server that runs until ctx is canceled. // It returns client and MongoDB URI of that listener. func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mongo.Client, string) { tb.Helper() @@ -71,7 +71,7 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mon _, span := otel.Tracer("").Start(ctx, "setupListener") defer span.End() - defer trace.StartRegion(ctx, "setupListener").End() + defer observability.FuncCall(ctx)() require.Empty(tb, *targetURLF, "-target-url must be empty for in-process FerretDB") diff --git a/integration/setup/setup.go b/integration/setup/setup.go index 4bc3d30dad32..42f97821e5c2 100644 --- a/integration/setup/setup.go +++ b/integration/setup/setup.go @@ -34,6 +34,7 @@ import ( "github.com/FerretDB/FerretDB/integration/shareddata" "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/testutil" ) @@ -173,7 +174,7 @@ func setupCollection(tb testing.TB, ctx context.Context, client *mongo.Client, o ctx, span := otel.Tracer("").Start(ctx, "setupCollection") defer span.End() - defer trace.StartRegion(ctx, "setupCollection").End() + defer observability.FuncCall(ctx)() var ownDatabase bool databaseName := opts.DatabaseName diff --git a/integration/setup/setup_compat.go b/integration/setup/setup_compat.go index a647f617cbb7..f6edfaccc867 100644 --- a/integration/setup/setup_compat.go +++ b/integration/setup/setup_compat.go @@ -29,6 +29,7 @@ import ( "go.uber.org/zap" "github.com/FerretDB/FerretDB/integration/shareddata" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/testutil" ) @@ -132,7 +133,7 @@ func setupCompatCollections(tb testing.TB, ctx context.Context, client *mongo.Cl ctx, span := otel.Tracer("").Start(ctx, "setupCompatCollections") defer span.End() - defer trace.StartRegion(ctx, "setupCompatCollections").End() + defer observability.FuncCall(ctx)() database := client.Database(opts.databaseName) diff --git a/internal/clientconn/conn.go b/internal/clientconn/conn.go index 0a00c6ba8314..0cbfd2cab02a 100644 --- a/internal/clientconn/conn.go +++ b/internal/clientconn/conn.go @@ -26,7 +26,6 @@ import ( "net" "os" "path/filepath" - "runtime/trace" "sync/atomic" "time" @@ -43,6 +42,7 @@ import ( "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" "github.com/FerretDB/FerretDB/internal/util/must" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/wire" ) @@ -121,7 +121,7 @@ func newConn(opts *newConnOpts) (*conn, error) { }, nil } -// run runs the client connection until ctx is done, client disconnects, +// run runs the client connection until ctx is canceled, client disconnects, // or fatal error or panic is encountered. // // Returned error is always non-nil. @@ -386,6 +386,8 @@ func (c *conn) run(ctx context.Context) (err error) { // route sends request to a handler's command based on the op code provided in the request header. // +// The passed context is canceled when the client disconnects. +// // Handlers to which it routes, should not panic on bad input, but may do so in "impossible" cases. // They also should not use recover(). That allows us to use fuzzing. // @@ -544,11 +546,14 @@ func (c *conn) route(ctx context.Context, reqHeader *wire.MsgHeader, reqBody wir return } +// handleOpMsg processes OP_MSG request. +// +// The passed context is canceled when the client disconnects. func (c *conn) handleOpMsg(ctx context.Context, msg *wire.OpMsg, command string) (*wire.OpMsg, error) { if cmd, ok := commoncommands.Commands[command]; ok { if cmd.Handler != nil { // TODO move it to route, closer to Prometheus metrics - defer trace.StartRegion(ctx, command).End() + defer observability.FuncCall(ctx)() return cmd.Handler(c.h, ctx, msg) } diff --git a/internal/clientconn/listener.go b/internal/clientconn/listener.go index 6e09b4f3fa36..8ae3603cf55a 100644 --- a/internal/clientconn/listener.go +++ b/internal/clientconn/listener.go @@ -78,7 +78,7 @@ func NewListener(opts *NewListenerOpts) *Listener { } } -// Run runs the listener until ctx is done or some unrecoverable error occurs. +// Run runs the listener until ctx is canceled or some unrecoverable error occurs. // // When this method returns, listener and all connections are closed. func (l *Listener) Run(ctx context.Context) error { @@ -241,15 +241,15 @@ func setupTLSListener(opts *setupTLSListenerOpts) (net.Listener, error) { return listener, nil } -// acceptLoop runs listener's connection accepting loop. +// acceptLoop runs listener's connection accepting loop until context is canceled. func acceptLoop(ctx context.Context, listener net.Listener, wg *sync.WaitGroup, l *Listener, logger *zap.Logger) { var retry int64 for { netConn, err := listener.Accept() if err != nil { // Run closed listener on context cancellation - if ctx.Err() != nil { - break + if context.Cause(ctx) != nil { + return } l.Metrics.Accepts.WithLabelValues("1").Inc() diff --git a/internal/handlers/common/count_iterator.go b/internal/handlers/common/count_iterator.go index c277ae7fc8dd..fdae8416180c 100644 --- a/internal/handlers/common/count_iterator.go +++ b/internal/handlers/common/count_iterator.go @@ -31,6 +31,8 @@ import ( // If input iterator contains no document, it returns ErrIteratorDone. // // Close method closes the underlying iterator. +// +// Deprecated: remove this function, use iterator.ConsumeCount instead. func CountIterator(iter types.DocumentsIterator, closer *iterator.MultiCloser, field string) types.DocumentsIterator { res := &countIterator{ iter: iter, diff --git a/internal/handlers/commoncommands/msg_listcommands.go b/internal/handlers/commoncommands/msg_listcommands.go index 62f29213e7f4..ca0ed1147dd8 100644 --- a/internal/handlers/commoncommands/msg_listcommands.go +++ b/internal/handlers/commoncommands/msg_listcommands.go @@ -33,6 +33,8 @@ type command struct { Help string // Handler processes this command. + // + // The passed context is canceled when the client disconnects. Handler func(handlers.Interface, context.Context, *wire.OpMsg) (*wire.OpMsg, error) } diff --git a/internal/util/iterator/func.go b/internal/util/iterator/func.go index 3f2b0b301fef..de6017cdfe08 100644 --- a/internal/util/iterator/func.go +++ b/internal/util/iterator/func.go @@ -77,6 +77,5 @@ func (iter *funcIterator[K, V]) Close() { // check interfaces var ( _ Interface[any, any] = (*funcIterator[any, any])(nil) - _ NextFunc[any, any] = (*funcIterator[any, any])(nil).Next _ Closer = (*funcIterator[any, any])(nil) ) diff --git a/internal/util/iterator/slice.go b/internal/util/iterator/slice.go index 32dbb3c9c1c7..a389c3f2a6a2 100644 --- a/internal/util/iterator/slice.go +++ b/internal/util/iterator/slice.go @@ -71,6 +71,5 @@ func (iter *sliceIterator[V]) Close() { // check interfaces var ( _ Interface[int, any] = (*sliceIterator[any])(nil) - _ NextFunc[int, any] = (*sliceIterator[any])(nil).Next _ Closer = (*sliceIterator[any])(nil) ) diff --git a/internal/util/iterator/values.go b/internal/util/iterator/values.go index c2e44dd59c87..13e95248c3ba 100644 --- a/internal/util/iterator/values.go +++ b/internal/util/iterator/values.go @@ -117,6 +117,5 @@ func (iter *valuesIterator[K, V]) Close() { // check interfaces var ( _ Interface[struct{}, any] = (*valuesIterator[any, any])(nil) - _ NextFunc[struct{}, any] = (*valuesIterator[any, any])(nil).Next _ Closer = (*valuesIterator[any, any])(nil) ) diff --git a/internal/util/iterator/with_close.go b/internal/util/iterator/with_close.go index 812393cf151d..6050184bb12a 100644 --- a/internal/util/iterator/with_close.go +++ b/internal/util/iterator/with_close.go @@ -44,6 +44,5 @@ func (iter *withCloseIterator[K, V]) Close() { // check interfaces var ( _ Interface[any, any] = (*withCloseIterator[any, any])(nil) - _ NextFunc[any, any] = (*withCloseIterator[any, any])(nil).Next _ Closer = (*withCloseIterator[any, any])(nil) ) diff --git a/internal/util/telemetry/reporter.go b/internal/util/telemetry/reporter.go index 4d158db725ba..92fd12a762bf 100644 --- a/internal/util/telemetry/reporter.go +++ b/internal/util/telemetry/reporter.go @@ -117,7 +117,7 @@ func (r *Reporter) Run(ctx context.Context) { r.firstReportDelay(ctx, ch) - for ctx.Err() == nil { + for context.Cause(ctx) == nil { r.report(ctx) ctxutil.Sleep(ctx, r.ReportInterval) From eb6170cb3fb51f0e007b0a8ed5436a3474dbc5d4 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Mon, 26 Jun 2023 14:29:04 +0400 Subject: [PATCH 24/46] Fix cancelation signals propagation (#2908) --- cmd/ferretdb/main.go | 1 - ferretdb/ferretdb.go | 4 +-- integration/setup/listener.go | 3 +- integration/setup/setup.go | 6 ++-- internal/clientconn/listener.go | 12 +++++--- internal/handlers/handlers.go | 1 + internal/util/resource/resource.go | 4 ++- internal/util/testutil/testutil.go | 47 ++++++++++++++++++------------ 8 files changed, 45 insertions(+), 33 deletions(-) diff --git a/cmd/ferretdb/main.go b/cmd/ferretdb/main.go index 3fcf934d81ed..98e2c6f3d781 100644 --- a/cmd/ferretdb/main.go +++ b/cmd/ferretdb/main.go @@ -378,7 +378,6 @@ func run() { if err != nil { logger.Fatal(err.Error()) } - defer h.Close() l := clientconn.NewListener(&clientconn.NewListenerOpts{ TCP: cli.Listen.Addr, diff --git a/ferretdb/ferretdb.go b/ferretdb/ferretdb.go index 95bc0016c863..e3faee3589c4 100644 --- a/ferretdb/ferretdb.go +++ b/ferretdb/ferretdb.go @@ -159,10 +159,8 @@ func New(config *Config) (*FerretDB, error) { // Run runs FerretDB until ctx is canceled. // -// When this method returns, listener and all connections are closed. +// When this method returns, listener and all connections, as well as handler are closed. func (f *FerretDB) Run(ctx context.Context) error { - defer f.l.Handler.Close() - err := f.l.Run(ctx) if errors.Is(err, context.Canceled) { err = nil diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 9041ed489c6e..6600113202bc 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -192,10 +192,9 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mon } }() - // ensure that all listener's logs are written before test ends + // ensure that all listener's and handler's logs are written before test ends tb.Cleanup(func() { <-runDone - h.Close() }) var clientOpts mongoDBURIOpts diff --git a/integration/setup/setup.go b/integration/setup/setup.go index 42f97821e5c2..7a97ce591e36 100644 --- a/integration/setup/setup.go +++ b/integration/setup/setup.go @@ -137,16 +137,16 @@ func SetupWithOpts(tb testing.TB, opts *SetupOpts) *SetupResult { var uri string if *targetURLF == "" { - client, uri = setupListener(tb, ctx, logger) + client, uri = setupListener(tb, setupCtx, logger) } else { - client = setupClient(tb, ctx, *targetURLF) + client = setupClient(tb, setupCtx, *targetURLF) uri = *targetURLF } // register cleanup function after setupListener registers its own to preserve full logs tb.Cleanup(cancel) - collection := setupCollection(tb, ctx, client, opts) + collection := setupCollection(tb, setupCtx, client, opts) level.SetLevel(*logLevelF) diff --git a/internal/clientconn/listener.go b/internal/clientconn/listener.go index 8ae3603cf55a..6eb674c41c02 100644 --- a/internal/clientconn/listener.go +++ b/internal/clientconn/listener.go @@ -80,8 +80,10 @@ func NewListener(opts *NewListenerOpts) *Listener { // Run runs the listener until ctx is canceled or some unrecoverable error occurs. // -// When this method returns, listener and all connections are closed. +// When this method returns, listener and all connections, as well as handler are closed. func (l *Listener) Run(ctx context.Context) error { + defer l.Handler.Close() + logger := l.Logger.Named("listener") if l.TCP != "" { @@ -122,8 +124,12 @@ func (l *Listener) Run(ctx context.Context) error { logger.Sugar().Infof("Listening on TLS %s ...", l.TLSAddr()) } - // close listeners on context cancellation to exit from listenLoop + var wg sync.WaitGroup + + wg.Add(1) go func() { + defer wg.Done() + <-ctx.Done() if l.tcpListener != nil { @@ -139,8 +145,6 @@ func (l *Listener) Run(ctx context.Context) error { } }() - var wg sync.WaitGroup - if l.TCP != "" { wg.Add(1) diff --git a/internal/handlers/handlers.go b/internal/handlers/handlers.go index 268c4b9b579a..ecc7e20bf191 100644 --- a/internal/handlers/handlers.go +++ b/internal/handlers/handlers.go @@ -35,6 +35,7 @@ import ( // Please keep methods documentation in sync with commands help text in the handlers/common package. type Interface interface { // Close gracefully shutdowns handler. + // It should be called after listener closes all client connections and stops listening. Close() prometheus.Collector diff --git a/internal/util/resource/resource.go b/internal/util/resource/resource.go index 7d4b215a0f58..ea14b2a40407 100644 --- a/internal/util/resource/resource.go +++ b/internal/util/resource/resource.go @@ -86,7 +86,9 @@ func Track(obj any, token *Token) { msg += "\nObject created by " + string(stack) } - panic(msg) + // TODO + _ = msg + // panic(msg) }) } diff --git a/internal/util/testutil/testutil.go b/internal/util/testutil/testutil.go index d682ee8fdf63..b58947f40df5 100644 --- a/internal/util/testutil/testutil.go +++ b/internal/util/testutil/testutil.go @@ -26,35 +26,44 @@ import ( ) // Ctx returns test context. +// It is canceled when test is finished or interrupted. func Ctx(tb testing.TB) context.Context { tb.Helper() - ctx, span := otel.Tracer("").Start(context.Background(), tb.Name()) - tb.Cleanup(func() { - span.End() - }) + signalsCtx, signalsCancel := notifyTestsTermination(context.Background()) - ctx, task := trace.NewTask(ctx, tb.Name()) - tb.Cleanup(task.End) + testDone := make(chan struct{}) - ctx, stop := notifyTestsTermination(ctx) + tb.Cleanup(func() { + close(testDone) + }) go func() { - <-ctx.Done() - - tb.Log("Stopping...") - stop() + select { + case <-testDone: + signalsCancel() + return - // There is a weird interaction between terminal's process group/session signal handling, - // Task's signal handling, - // and this attempt to handle signals gracefully. - // It may cause tests to continue running in the background - // while terminal shows command-line prompt already. - // - // Panic to surely stop tests. - panic("Stopping everything") + case <-signalsCtx.Done(): + // There is a weird interaction between terminal's process group/session signal handling, + // Task's signal handling, + // and this attempt to handle signals gracefully. + // It may cause tests to continue running in the background + // while terminal shows command-line prompt already. + // + // Panic to surely stop tests. + panic("Stopping everything") + } }() + ctx, span := otel.Tracer("").Start(signalsCtx, tb.Name()) + tb.Cleanup(func() { + span.End() + }) + + ctx, task := trace.NewTask(ctx, tb.Name()) + tb.Cleanup(task.End) + return ctx } From 49d338c5cdc071a63381e61586ef8c95f1c85f58 Mon Sep 17 00:00:00 2001 From: Patryk Kwiatek Date: Mon, 26 Jun 2023 20:35:36 +0200 Subject: [PATCH 25/46] Fix collection name starting with dot validation (#2912) --- integration/basic_test.go | 8 ++++++++ internal/handlers/pg/msg_create.go | 4 ++++ internal/handlers/pg/pgdb/collections.go | 8 +++++++- internal/handlers/pg/pgdb/pgdb.go | 3 +++ 4 files changed, 22 insertions(+), 1 deletion(-) diff --git a/integration/basic_test.go b/integration/basic_test.go index c2310dc10087..7906d97f4dc0 100644 --- a/integration/basic_test.go +++ b/integration/basic_test.go @@ -379,6 +379,14 @@ func TestCollectionName(t *testing.T) { }, altMessage: "Invalid collection name: 'TestCollectionName.\x00'", }, + "DotSurround": { + collection: ".collection..", + err: &mongo.CommandError{ + Name: "InvalidNamespace", + Code: 73, + Message: "Collection names cannot start with '.': .collection..", + }, + }, "Dot": { collection: "collection.name", }, diff --git a/internal/handlers/pg/msg_create.go b/internal/handlers/pg/msg_create.go index 4b75a10d0d70..a0106fa529b3 100644 --- a/internal/handlers/pg/msg_create.go +++ b/internal/handlers/pg/msg_create.go @@ -105,6 +105,10 @@ func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, msg := fmt.Sprintf("Invalid collection name: '%s.%s'", db, collection) return commonerrors.NewCommandErrorMsg(commonerrors.ErrInvalidNamespace, msg) + case errors.Is(err, pgdb.ErrCollectionStartsWithDot): + msg := fmt.Sprintf("Collection names cannot start with '.': %s", collection) + return commonerrors.NewCommandErrorMsg(commonerrors.ErrInvalidNamespace, msg) + default: return lazyerrors.Error(err) } diff --git a/internal/handlers/pg/pgdb/collections.go b/internal/handlers/pg/pgdb/collections.go index d639e0df2c0d..dce2d369e6b3 100644 --- a/internal/handlers/pg/pgdb/collections.go +++ b/internal/handlers/pg/pgdb/collections.go @@ -35,7 +35,8 @@ import ( // validateCollectionNameRe validates collection names. // Empty collection name, names with `$` and `\x00` are not allowed. -var validateCollectionNameRe = regexp.MustCompile("^[^$\x00]{1,235}$") +// Collection names that start with `.` are also not allowed. +var validateCollectionNameRe = regexp.MustCompile("^[^.$\x00][^$\x00]{0,234}$") // Collections returns a sorted list of FerretDB collection names. // @@ -108,9 +109,14 @@ func CollectionExists(ctx context.Context, tx pgx.Tx, db, collection string) (bo // It returns possibly wrapped error: // - ErrInvalidDatabaseName - if the given database name doesn't conform to restrictions. // - ErrInvalidCollectionName - if the given collection name doesn't conform to restrictions. +// - ErrCollectionStartsWithDot - if the given collection name starts with dot. // - ErrAlreadyExist - if a FerretDB collection with the given name already exists. // - *transactionConflictError - if a PostgreSQL conflict occurs (the caller could retry the transaction). func CreateCollection(ctx context.Context, tx pgx.Tx, db, collection string) error { + if strings.HasPrefix(collection, ".") { + return ErrCollectionStartsWithDot + } + if !validateCollectionNameRe.MatchString(collection) || strings.HasPrefix(collection, reservedPrefix) || !utf8.ValidString(collection) { diff --git a/internal/handlers/pg/pgdb/pgdb.go b/internal/handlers/pg/pgdb/pgdb.go index e2e69a2e3e07..e2f6c318b6e6 100644 --- a/internal/handlers/pg/pgdb/pgdb.go +++ b/internal/handlers/pg/pgdb/pgdb.go @@ -45,6 +45,9 @@ var ( // ErrInvalidCollectionName indicates that a collection didn't pass name checks. ErrInvalidCollectionName = fmt.Errorf("invalid FerretDB collection name") + // ErrCollectionStartsWithDot indicates that collection name starts with dot, but it shouldn't. + ErrCollectionStartsWithDot = fmt.Errorf("Collection names cannot start with '.'") + // ErrInvalidDatabaseName indicates that a database didn't pass name checks. ErrInvalidDatabaseName = fmt.Errorf("invalid FerretDB database name") From 9355acc6525c6369b2c36a663591da78dfc65233 Mon Sep 17 00:00:00 2001 From: Alexander Tobi Fashakin Date: Tue, 27 Jun 2023 06:34:01 +0100 Subject: [PATCH 26/46] Add blog post on "How to Configure FerretDB to work on Percona Distribution for PostgreSQL" --- ...db-work-percona-distribution-postgresql.md | 302 ++++++++++++++++++ .../img/blog/displaying-studio3t-data.png | 3 + website/static/img/blog/percona-ferretdb.png | 3 + website/static/img/blog/postgresql.png | 3 + 4 files changed, 311 insertions(+) create mode 100644 website/blog/2023-06-26-configure-ferretdb-work-percona-distribution-postgresql.md create mode 100644 website/static/img/blog/displaying-studio3t-data.png create mode 100644 website/static/img/blog/percona-ferretdb.png create mode 100644 website/static/img/blog/postgresql.png diff --git a/website/blog/2023-06-26-configure-ferretdb-work-percona-distribution-postgresql.md b/website/blog/2023-06-26-configure-ferretdb-work-percona-distribution-postgresql.md new file mode 100644 index 000000000000..95bdbda74176 --- /dev/null +++ b/website/blog/2023-06-26-configure-ferretdb-work-percona-distribution-postgresql.md @@ -0,0 +1,302 @@ +--- +slug: configure-ferretdb-work-percona-distribution-postgresql +title: 'How to Configure FerretDB to Work on Percona Distribution for PostgreSQL' +authors: [alex] +description: > + In this article, we’ll guide you through the advantages of using FerretDB and how you can configure it to work natively on Percona Distribution for PostgreSQL. +image: /img/blog/percona-ferretdb.png +keywords: + [enterprise postgresql, run mongodb workload on postgresql, postgresql tools] +tags: [tutorial, postgresql tools] +--- + +![How to Configure FerretDB to Work on Percona Distribution for PostgreSQL](/img/blog/percona-ferretdb.png) + +Imagine being able to leverage the flexibility and simplicity of the MongoDB query language through FerretDB together with the robust enterprise PostgreSQL tools and services provided by Percona. + + + +That's exactly what you get when you configure FerretDB for your Percona Distribution for PostgreSQL! + +In this article, we'll guide you through the advantages of using FerretDB and how you can configure it to work natively on Percona Distribution for PostgreSQL and explore how FerretDB's stored data appear in PostgreSQL. + +Before we go into the technical details, let's find out what these two solutions are all about. + +## What is Percona Distribution for PostgreSQL? + +[Percona Distribution for PostgreSQL](https://www.percona.com/software/postgresql-distribution) is a software package provided by Percona, a world-class company that provides open source database software, support, and services. + +Basically, this is a distribution of the PostgreSQL database management system (DBMS) that includes enhanced features to make it more manageable, scalable, and performant. + +PostgreSQL database, widely used by millions of developers across the globe, has had a significant impact on open-source development. + +![PostgreSQL Database](/img/blog/postgresql.png) +_source: [PostgreSQL website](https://www.postgresql.org/)_ + +Many software companies have built their applications on top of the database, extended its functionality, and provided robust infrastructures and services for it. +Percona is one of those companies, and have contributed immensely to the ecosystem with their array of PostgreSQL tools, extensions, and services. + +With the PostgreSQL Distribution provided by Percona, users get an enhanced version of the PostgreSQL DBMS itself, containing optimized features for better query performance, storage engine, and monitoring capabilities. + +Notable solutions in the Percona Distribution for PostgreSQL include pgAudit, pgBackRest, Patroni, pgRepack, among others. + +## Introducing FerretDB + +[FerretDB](https://www.ferretdb.io/) is open source document database that acts as a MongoDB alternative for users looking for open-source solutions with the same query language and commands, ease of use, and flexibility. + +Using PostgreSQL as the backend, FerretDB converts the wire protocols of MongoDB to SQL, enabling you to manage MongoDB workloads with PostgreSQL. + +This means you can take advantage of many MongoDB tools as well while leveraging all the operational and management features for your PostgreSQL DBMS. +Pretty neat, right? +We think so too. + +FerretDB works natively with Percona Distribution for PostgreSQL, meaning you can take advantage of all its exceptional PostgreSQL features right out of the box, such as backup, monitoring, and more. + +## Setting up environment and packages + +### Prerequisites + +We'll be using the Debian package for both FerretDB and Percona Distribution for PostgreSQL, so having a Unix-like operating system such as Ubuntu is important. + +You also need to remove any preexisting installation of PostgreSQL to avoid conflicts. +To remove PostgreSQL, use the following command: + +```sh +sudo apt-get --purge remove postgresql +``` + +Also, you'll need to download the [Studio 3T linux version](https://studio3t.com/knowledge-base/articles/how-to-install-studio-3t-on-linux/); we'll be using the MongoDB GUI tool to showcase how FerretDB works with Percona Distribution for PostgreSQL. + +### Installation + +We need to install the Debian packages for both software. + +#### Installing Debian Package of Percona Distribution for PostgreSQL + +Start by downloading the Debian package for Percona release packages. + +```sh +wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb +``` + +Then you can go ahead to install the package using `dpkg`: + +```sh +sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb +``` + +Once that's done, you should refresh your local cache: + +```sh +sudo apt update +``` + +See [here](https://docs.percona.com/postgresql/15/installing.html) for the installation guide on all the PostgreSQL solutions provided by Percona. + +The next step is to install the latest version of Percona Distribution for PostgreSQL from the repositories provided by Percona. + +```sh +sudo percona-release setup ppg-15 +sudo apt install percona-ppg-server-15 +``` + +At this point, you should have the Percona Distribution for PostgreSQL running on your system. +Run this command just to be sure: + +```sh +sudo systemctl status postgresql.service +``` + +#### FerretDB Debian Download + +Now let's download the Debian package for FerretDB. +We are going to download the latest version which is FerretDB v1.4.0. +You can find the [latest version on the releases here](https://github.com/FerretDB/FerretDB/releases): + +```sh +wget https://github.com/FerretDB/FerretDB/releases/download/v1.4.0/ferretdb.deb +``` + +Once downloaded, install FerretDB using this command: + +```sh +sudo dpkg -i ferretdb.deb +``` + +Please check that the installation works fine by running `ferretdb --version`. + +## Configuring Percona Distribution for PostgreSQL to work with FerretDB + +The FerretDB installation does not include PostgreSQL or any other backend so you'll need to have that separately and since you have Percona Distribution for PostgreSQL installed, then we can use that. + +And since we're already running Percona Distribution for PostgreSQL, we can just connect to it. + +To connect to the Percona Distribution for PostgreSQL using appropriate FerretDB flags, run the command: + +```sh +ferretdb --handler=pg --postgresql-url=postgres://127.0.0.1:5432/ferretdb --listen-addr=127.0.0.1:27017 +``` + +Great! +We've connected to the Percona Distribution for PostgreSQL and we're ready to start using FerretDB. + +Note that FerretDB provides a list of flags for configuring your database – see them [here](https://docs.ferretdb.io/configuration/flags/). + +From a new terminal, let's access the PostgreSQL command line: + +```sh +sudo su - postgres -c psql +``` + +In the `psql` terminal, we are going to create a database to hold all data that'll be passed through the MongoDB URI. + +```sql +CREATE DATABASE ferretdb; +``` + +Create a new user with username and password privileges: + +```sql +CREATE USER username WITH PASSWORD 'password'; +``` + +Be sure to set the database, username, and password to your preferred options. + +Grant all privileges on the `ferretdb` database to the new user: + +```sql +GRANT ALL PRIVILEGES ON DATABASE ferretdb TO username; +``` + +Now, you should be able to connect to FerretDB using the following MongoDB URI format on any MongoDB tools or GUI. + +## Experimenting with FerretDB through Studio 3T + +To showcase the connection of FerretDB and Percona Distribution, we'll need to use `mongosh` or any MongoDB GUI tool. +For the purpose of this guide, we'll be using Studio 3T – a MongoDB GUI tool. + +Once Studio 3T is installed, launch the application and add a connection using the right MongoDB URI. + +```text +mongodb://username:password@127.0.0.1/ferretdb?authMechanism=PLAIN +``` + +After setting up the connection, navigate and switch the context to the `ferretdb` database. +Using the Intellishell, let's insert some documents through the shell on Studio 3T. + +```js +db.test.insertMany([ + { + name: 'John Doe', + email: 'johndoe@example.com', + age: 30, + gender: 'Male', + interests: ['reading', 'hiking', 'photography'], + address: { + street: '123 Main St', + city: 'Anytown', + state: 'CA', + zip: '12345' + } + }, + { + name: 'Jane Smith', + email: 'janesmith@example.com', + age: 25, + gender: 'Female', + interests: ['painting', 'travel', 'yoga'], + address: { + street: '456 Elm St', + city: 'Othertown', + state: 'NY', + zip: '67890' + } + }, + { + name: 'Bob Johnson', + email: 'bjohnson@example.com', + age: 40, + gender: 'Male', + interests: ['cooking', 'gardening', 'fishing'], + address: { + street: '789 Oak St', + city: 'Somewhere', + state: 'TX', + zip: '23456' + } + }, + { + name: 'Samantha Lee', + email: 'slee@example.com', + age: 28, + gender: 'Female', + interests: ['music', 'dancing', 'skiing'], + address: { + street: '321 Pine St', + city: 'Anotherplace', + state: 'FL', + zip: '34567' + } + } +]) +``` + +![Displaying data through Studio 3T](/img/blog/displaying-studio3t-data.png) + +What we want to do now is to explore and view the data we just inserted through FerretDB on the Percona Distribution for PostgreSQL. + +In your terminal, we will open a new `psql` terminal that takes us directly to the `ferretdb` database we created earlier, and contains our data. + +```text +~$ sudo su - postgres +postgres@Alexander-ubuntu:~$ psql ferretdb +psql (15.2 - Percona Distribution, server 13.10 - Percona Distribution (Ubuntu 2:13.10-1.focal)) +Type "help" for help. +``` + +`ferretdb` database should now be in context. +Let's proceed by setting `SET search_path TO ferretdb;` and then displaying the tables (this is akin to the collection we created earlier) in the database using `\dt`. + +```text +ferretdb=# set search_path to ferretdb; +SET +ferretdb=# \dt + List of relations + Schema | Name | Type | Owner +----------+-----------------------------+-------+---------- + ferretdb | _ferretdb_database_metadata | table | username + ferretdb | test_afd071e5 | table | username +(2 rows) +``` + +We can see two tables: one containing the metadata for the database and the other containing the `test` collection we created on Studio 3T. +Let's view the table content in PostgreSQL: + +```text +ferretdb=# table test_afd071e5; +``` + +```text + _jsonb + +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + {"$s": {"p": {"_id": {"t": "objectId"}, "age": {"t": "int"}, "name": {"t": "string"}, "email": {"t": "string"}, "gender": {"t": "string"}, "address": {"t": "object", "$s": {"p": {"zip": {"t": "string"}, "city": {"t": "string"}, "state": {"t": "string"}, "street": {"t": "string"}}, "$k": ["street", "city", "state", "zip"]}}, "interests": {"i": [{"t": "string"}, {"t": "string"}, {"t": "string"}], "t": "array"}}, "$k": ["_id", "name", "email", "age", "gender", "interests", "address"]}, "_id": "64955a05cc7e30485cdeeee3", "age": 30, "name": "John Doe", "email": "johndoe@example.com", "gender": "Male", "address": {"zip": "12345", "city": "Anytown", "state": "CA", "street": "123 Main St"}, "interests": ["reading", "hiking", "photography"]} + {"$s": {"p": {"_id": {"t": "objectId"}, "age": {"t": "int"}, "name": {"t": "string"}, "email": {"t": "string"}, "gender": {"t": "string"}, "address": {"t": "object", "$s": {"p": {"zip": {"t": "string"}, "city": {"t": "string"}, "state": {"t": "string"}, "street": {"t": "string"}}, "$k": ["street", "city", "state", "zip"]}}, "interests": {"i": [{"t": "string"}, {"t": "string"}, {"t": "string"}], "t": "array"}}, "$k": ["_id", "name", "email", "age", "gender", "interests", "address"]}, "_id": "64955a05cc7e30485cdeeee4", "age": 25, "name": "Jane Smith", "email": "janesmith@example.com", "gender": "Female", "address": {"zip": "67890", "city": "Othertown", "state": "NY", "street": "456 Elm St"}, "interests": ["painting", "travel", "yoga"]} + {"$s": {"p": {"_id": {"t": "objectId"}, "age": {"t": "int"}, "name": {"t": "string"}, "email": {"t": "string"}, "gender": {"t": "string"}, "address": {"t": "object", "$s": {"p": {"zip": {"t": "string"}, "city": {"t": "string"}, "state": {"t": "string"}, "street": {"t": "string"}}, "$k": ["street", "city", "state", "zip"]}}, "interests": {"i": [{"t": "string"}, {"t": "string"}, {"t": "string"}], "t": "array"}}, "$k": ["_id", "name", "email", "age", "gender", "interests", "address"]}, "_id": "64955a05cc7e30485cdeeee5", "age": 40, "name": "Bob Johnson", "email": "bjohnson@example.com", "gender": "Male", "address": {"zip": "23456", "city": "Somewhere", "state": "TX", "street": "789 Oak St"}, "interests": ["cooking", "gardening", "fishing"]} + {"$s": {"p": {"_id": {"t": "objectId"}, "age": {"t": "int"}, "name": {"t": "string"}, "email": {"t": "string"}, "gender": {"t": "string"}, "address": {"t": "object", "$s": {"p": {"zip": {"t": "string"}, "city": {"t": "string"}, "state": {"t": "string"}, "street": {"t": "string"}}, "$k": ["street", "city", "state", "zip"]}}, "interests": {"i": [{"t": "string"}, {"t": "string"}, {"t": "string"}], "t": "array"}}, "$k": ["_id", "name", "email", "age", "gender", "interests", "address"]}, "_id": "64955a05cc7e30485cdeeee6", "age": 28, "name": "Samantha Lee", "email": "slee@example.com", "gender": "Female", "address": {"zip": "34567", "city": "Anotherplace", "state": "FL", "street": "321 Pine St"}, "interests": ["music", "dancing", "skiing"]} +(4 rows) +``` + +Brilliant! +Just by using FerretDB, we've been able to store data using MongoDB commands and query language and have it displayed in Percona Distribution for PostgreSQL. + +## Round-up + +As we've showcased in this guide, FerretDB gives you the chance to run and manage MongoDB production workloads in PostgreSQL; by having Percona Distribution for PostgreSQL as your backend, you can tap into all the other open source enterprise tools, extensions, and services provided by Percona – and they all work seamlessly together! + +For enterprise users already familiar with PostgreSQL management, this is a great way to also leverage your existing knowledge and skills, for backup, monitoring, security, replication, and so on. + +FerretDB is an open source project, and as such we welcome all contributions and feedback that can help us improve it. +So please do check our [GitHub repository](https://github.com/FerretDB/FerretDB/) and feel free to leave a comment on [any of our channels](https://docs.ferretdb.io/#community). + +And if you would like to learn more on how to get started with FerretDB, please check out [our installation guide](https://docs.ferretdb.io/quickstart-guide/). diff --git a/website/static/img/blog/displaying-studio3t-data.png b/website/static/img/blog/displaying-studio3t-data.png new file mode 100644 index 000000000000..c8593d34644f --- /dev/null +++ b/website/static/img/blog/displaying-studio3t-data.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0fd3d77be3d71b972f08f4a70a87484cc18eb7b4f68ee397432a503a079e36ff +size 222290 diff --git a/website/static/img/blog/percona-ferretdb.png b/website/static/img/blog/percona-ferretdb.png new file mode 100644 index 000000000000..5d40ddb3435d --- /dev/null +++ b/website/static/img/blog/percona-ferretdb.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:885df81fbedb43f1aa06dc6aa88056cb108daad7ae01badc1d07dfffba469716 +size 124302 diff --git a/website/static/img/blog/postgresql.png b/website/static/img/blog/postgresql.png new file mode 100644 index 000000000000..b4d88baf6bf5 --- /dev/null +++ b/website/static/img/blog/postgresql.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b7c82ae9b0346bf45d9f4f17a11b0511007bcac41af2451e3d65fbfc5d45f0e5 +size 1440761 From 4768f4706f7c21db28de987a4fbd8c9fa6213d2e Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Tue, 27 Jun 2023 10:36:20 +0400 Subject: [PATCH 27/46] Enable cursor support for PostgreSQL and SQLite (#2864) --- README.md | 1 - Taskfile.yml | 16 +- cmd/ferretdb/main.go | 2 - integration/setup/listener.go | 1 - integration/setup/setup.go | 1 - internal/backends/collection.go | 4 + internal/backends/sqlite/backend.go | 3 +- internal/backends/sqlite/metadata/pool/db.go | 6 +- internal/backends/sqlite/query_iterator.go | 44 ++--- internal/clientconn/cursor/cursor.go | 79 +++++++-- internal/clientconn/cursor/registry.go | 176 +++++++++++++++---- internal/handlers/common/count.go | 1 + internal/handlers/common/find.go | 1 + internal/handlers/common/getmore.go | 15 +- internal/handlers/common/insert.go | 1 + internal/handlers/pg/msg_aggregate.go | 41 +++-- internal/handlers/pg/msg_find.go | 54 +++--- internal/handlers/pg/msg_getmore.go | 2 +- internal/handlers/pg/pg.go | 15 +- internal/handlers/registry/pg.go | 1 - internal/handlers/registry/registry.go | 1 - internal/handlers/registry/sqlite.go | 2 +- internal/handlers/registry/tigris.go | 1 - internal/handlers/sjson/sjson_test.go | 4 + internal/handlers/sqlite/msg_drop.go | 12 ++ internal/handlers/sqlite/msg_find.go | 72 ++++---- internal/handlers/sqlite/msg_getmore.go | 4 +- internal/handlers/sqlite/sqlite.go | 18 +- internal/handlers/tigris/tigris.go | 5 +- internal/util/iterator/closer.go | 13 ++ internal/util/resource/resource.go | 4 +- internal/util/testutil/testutil.go | 1 - 32 files changed, 418 insertions(+), 183 deletions(-) diff --git a/README.md b/README.md index 2f18aa99852d..7b36689fd138 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,6 @@ [![Go Reference](https://pkg.go.dev/badge/github.com/FerretDB/FerretDB/ferretdb.svg)](https://pkg.go.dev/github.com/FerretDB/FerretDB/ferretdb) [![Go](https://github.com/FerretDB/FerretDB/actions/workflows/go.yml/badge.svg?branch=main)](https://github.com/FerretDB/FerretDB/actions/workflows/go.yml) -[![Integration](https://github.com/FerretDB/FerretDB/actions/workflows/integration.yml/badge.svg?branch=main)](https://github.com/FerretDB/FerretDB/actions/workflows/integration.yml) [![codecov](https://codecov.io/gh/FerretDB/FerretDB/branch/main/graph/badge.svg?token=JZ56XFT3DM)](https://codecov.io/gh/FerretDB/FerretDB) [![Security](https://github.com/FerretDB/FerretDB/actions/workflows/security.yml/badge.svg?branch=main)](https://github.com/FerretDB/FerretDB/actions/workflows/security.yml) diff --git a/Taskfile.yml b/Taskfile.yml index e3b560a5648c..2ac822676847 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -199,7 +199,6 @@ tasks: -target-tls -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb -compat-url='mongodb://username:password@127.0.0.1:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem' - -enable-cursors vars: SHARD_RUN: sh: go run -C .. ./cmd/envtool tests shard --index={{.SHARD_INDEX}} --total={{.SHARD_TOTAL}} @@ -401,6 +400,21 @@ tasks: --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb --test-records-dir=tmp/records + run-sqlite-proxy: + desc: "Run FerretDB with `sqlite` handler in diff-proxy mode" + deps: [build-host] + cmds: + - go run ./cmd/envtool shell mkdir tmp/sqlite + - > + bin/ferretdb{{exeExt}} -test.coverprofile=cover.txt -- + --listen-addr=:27017 + --proxy-addr=127.0.0.1:47017 + --mode=diff-proxy + --handler=sqlite + --sqlite-uri=tmp/sqlite + --test-records-dir=tmp/records + --test-disable-filter-pushdown + lint: desc: "Run linters" cmds: diff --git a/cmd/ferretdb/main.go b/cmd/ferretdb/main.go index 98e2c6f3d781..ca9d3a716749 100644 --- a/cmd/ferretdb/main.go +++ b/cmd/ferretdb/main.go @@ -81,7 +81,6 @@ var cli struct { RecordsDir string `default:"" help:"Experimental: directory for record files."` DisableFilterPushdown bool `default:"false" help:"Experimental: disable filter pushdown."` EnableSortPushdown bool `default:"false" help:"Experimental: enable sort pushdown."` - EnableCursors bool `default:"false" help:"Experimental: enable cursors."` //nolint:lll // for readability Telemetry struct { @@ -372,7 +371,6 @@ func run() { TestOpts: registry.TestOpts{ DisableFilterPushdown: cli.Test.DisableFilterPushdown, EnableSortPushdown: cli.Test.EnableSortPushdown, - EnableCursors: cli.Test.EnableCursors, }, }) if err != nil { diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 6600113202bc..2456440660ef 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -134,7 +134,6 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mon TestOpts: registry.TestOpts{ DisableFilterPushdown: *disableFilterPushdownF, EnableSortPushdown: *enableSortPushdownF, - EnableCursors: *enableCursorsF, }, } h, err := registry.NewHandler(handler, handlerOpts) diff --git a/integration/setup/setup.go b/integration/setup/setup.go index 7a97ce591e36..6dc3ad076b75 100644 --- a/integration/setup/setup.go +++ b/integration/setup/setup.go @@ -61,7 +61,6 @@ var ( disableFilterPushdownF = flag.Bool("disable-filter-pushdown", false, "disable filter pushdown") enableSortPushdownF = flag.Bool("enable-sort-pushdown", false, "enable sort pushdown") - enableCursorsF = flag.Bool("enable-cursors", false, "enable cursors") ) // Other globals. diff --git a/internal/backends/collection.go b/internal/backends/collection.go index 0bc00a96daf4..957261b98c31 100644 --- a/internal/backends/collection.go +++ b/internal/backends/collection.go @@ -67,6 +67,10 @@ type QueryResult struct { } // Query executes a query against the collection. +// +// The passed context should be used for canceling the initial query. +// It also can be used to close the returned iterator and free underlying resources, +// but doing so is not necessary - the handler will do that anyway. func (cc *collectionContract) Query(ctx context.Context, params *QueryParams) (res *QueryResult, err error) { defer observability.FuncCall(ctx)() defer checkError(err) diff --git a/internal/backends/sqlite/backend.go b/internal/backends/sqlite/backend.go index 4602277a8ca9..25138800b3ed 100644 --- a/internal/backends/sqlite/backend.go +++ b/internal/backends/sqlite/backend.go @@ -35,6 +35,7 @@ type backend struct { // NewBackendParams represents the parameters of NewBackend function. type NewBackendParams struct { Dir string + L *zap.Logger } // NewBackend creates a new SQLite backend. @@ -48,7 +49,7 @@ func NewBackend(params *NewBackendParams) (backends.Backend, error) { return nil, lazyerrors.Errorf("%q should be an existing directory", params.Dir) } - r, err := metadata.NewRegistry(params.Dir, zap.L().Named("sqlite").Named("metadata")) + r, err := metadata.NewRegistry(params.Dir, params.L.Named("metadata")) if err != nil { return nil, lazyerrors.Error(err) } diff --git a/internal/backends/sqlite/metadata/pool/db.go b/internal/backends/sqlite/metadata/pool/db.go index fbd0a408e853..12cce2b97d8a 100644 --- a/internal/backends/sqlite/metadata/pool/db.go +++ b/internal/backends/sqlite/metadata/pool/db.go @@ -36,11 +36,13 @@ func openDB(uri string) (*db, error) { return nil, lazyerrors.Error(err) } + // TODO https://github.com/FerretDB/FerretDB/issues/2909 + // TODO https://github.com/FerretDB/FerretDB/issues/2755 sqlDB.SetConnMaxIdleTime(0) sqlDB.SetConnMaxLifetime(0) - sqlDB.SetMaxIdleConns(1) - sqlDB.SetMaxOpenConns(1) + // sqlDB.SetMaxIdleConns(5) + // sqlDB.SetMaxOpenConns(5) if err = sqlDB.Ping(); err != nil { _ = sqlDB.Close() diff --git a/internal/backends/sqlite/query_iterator.go b/internal/backends/sqlite/query_iterator.go index 022068ecb447..3a87e1ffd15b 100644 --- a/internal/backends/sqlite/query_iterator.go +++ b/internal/backends/sqlite/query_iterator.go @@ -23,26 +23,29 @@ import ( "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/iterator" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/resource" ) // queryIterator implements iterator.Interface to fetch documents from the database. -// -//nolint:vet // for readability type queryIterator struct { - ctx context.Context - - m sync.Mutex - rows *sql.Rows + // the order of fields is weird to make the struct smaller due to alignment + ctx context.Context + rows *sql.Rows // protected by m token *resource.Token + m sync.Mutex } // newQueryIterator returns a new queryIterator for the given *sql.Rows. // // Iterator's Close method closes rows. +// They are also closed by the Next method on any error, including context cancellation, +// to make sure that the database connection is released as early as possible. +// In that case, the iterator's Close method should still be called. // // Nil rows are possible and return already done iterator. +// It still should be Close'd. func newQueryIterator(ctx context.Context, rows *sql.Rows) types.DocumentsIterator { iter := &queryIterator{ ctx: ctx, @@ -55,15 +58,9 @@ func newQueryIterator(ctx context.Context, rows *sql.Rows) types.DocumentsIterat } // Next implements iterator.Interface. -// -// Errors (possibly wrapped) are: -// - iterator.ErrIteratorDone; -// - context.Canceled; -// - context.DeadlineExceeded; -// - something else. -// -// Otherwise, the next document is returned. func (iter *queryIterator) Next() (struct{}, *types.Document, error) { + defer observability.FuncCall(iter.ctx)() + iter.m.Lock() defer iter.m.Unlock() @@ -75,28 +72,31 @@ func (iter *queryIterator) Next() (struct{}, *types.Document, error) { } if err := context.Cause(iter.ctx); err != nil { + iter.close() return unused, nil, lazyerrors.Error(err) } if !iter.rows.Next() { - if err := iter.rows.Err(); err != nil { - return unused, nil, lazyerrors.Error(err) - } + err := iter.rows.Err() - // to avoid context cancellation changing the next `Next()` error - // from `iterator.ErrIteratorDone` to `context.Canceled` iter.close() - return unused, nil, iterator.ErrIteratorDone + if err == nil { + err = iterator.ErrIteratorDone + } + + return unused, nil, lazyerrors.Error(err) } var b []byte if err := iter.rows.Scan(&b); err != nil { + iter.close() return unused, nil, lazyerrors.Error(err) } doc, err := sjson.Unmarshal(b) if err != nil { + iter.close() return unused, nil, lazyerrors.Error(err) } @@ -105,6 +105,8 @@ func (iter *queryIterator) Next() (struct{}, *types.Document, error) { // Close implements iterator.Interface. func (iter *queryIterator) Close() { + defer observability.FuncCall(iter.ctx)() + iter.m.Lock() defer iter.m.Unlock() @@ -115,6 +117,8 @@ func (iter *queryIterator) Close() { // // This should be called only when the caller already holds the mutex. func (iter *queryIterator) close() { + defer observability.FuncCall(iter.ctx)() + if iter.rows != nil { iter.rows.Close() iter.rows = nil diff --git a/internal/clientconn/cursor/cursor.go b/internal/clientconn/cursor/cursor.go index b946c3b35b5f..666e1787fc8d 100644 --- a/internal/clientconn/cursor/cursor.go +++ b/internal/clientconn/cursor/cursor.go @@ -16,38 +16,81 @@ package cursor import ( + "sync" + "time" + "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/resource" ) -// Cursor allows clients to iterate over a result set. -type cursor struct { - *NewParams +// The implementation of the cursor and registry is quite complicated and entangled. +// That's because there are many cases when cursor / iterator / underlying database connection +// must be closed to free resources, including when no handler and backend code is running; +// for example, when the client disconnects between `getMore` commands. +// At the same time, we want to shift complexity away from the handler and from backend implementations +// because they are already quite complex. +// The current design enables ease of use at the expense of the implementation complexity. - token *resource.Token -} +// Cursor allows clients to iterate over a result set. +// +// It implements types.DocumentsIterator interface by wrapping another iterator with documents +// with additional metadata and registration in the registry. +// +// Closing the cursor removes it from the registry. +type Cursor struct { + // the order of fields is weird to make the struct smaller due to alignment -// NewParams contains the parameters for creating a new cursor. -type NewParams struct { - Iter types.DocumentsIterator + created time.Time + iter types.DocumentsIterator + r *Registry + token *resource.Token + closed chan struct{} DB string Collection string - BatchSize int32 + Username string + ID int64 + closeOnce sync.Once } -// New creates a new cursor. -func New(params *NewParams) *cursor { - c := &cursor{ - NewParams: params, - token: resource.NewToken(), +// newCursor creates a new cursor. +func newCursor(id int64, db, collection, username string, iter types.DocumentsIterator, r *Registry) *Cursor { + c := &Cursor{ + ID: id, + DB: db, + Collection: collection, + Username: username, + iter: iter, + r: r, + created: time.Now(), + closed: make(chan struct{}), + token: resource.NewToken(), } + resource.Track(c, c.token) return c } -// Close closes the cursor. -func (c *cursor) Close() { - c.Iter.Close() - resource.Untrack(c, c.token) +// Next implements types.DocumentsIterator interface. +func (c *Cursor) Next() (struct{}, *types.Document, error) { + return c.iter.Next() } + +// Close implements types.DocumentsIterator interface. +func (c *Cursor) Close() { + c.closeOnce.Do(func() { + c.iter.Close() + c.iter = nil + + c.r.delete(c) + + close(c.closed) + + resource.Untrack(c, c.token) + }) +} + +// check interfaces +var ( + _ types.DocumentsIterator = (*Cursor)(nil) +) diff --git a/internal/clientconn/cursor/registry.go b/internal/clientconn/cursor/registry.go index b0f39c0d5590..aefa09614580 100644 --- a/internal/clientconn/cursor/registry.go +++ b/internal/clientconn/cursor/registry.go @@ -15,13 +15,26 @@ package cursor import ( + "context" "math/rand" "sync" "sync/atomic" + "time" + "github.com/prometheus/client_golang/prometheus" + "go.uber.org/zap" + "golang.org/x/exp/maps" + + "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/debugbuild" ) +// Parts of Prometheus metric names. +const ( + namespace = "ferretdb" + subsystem = "cursors" +) + // Global last cursor ID. var lastCursorID atomic.Uint32 @@ -34,72 +47,163 @@ func init() { // Registry stores cursors. // -// TODO add cleanup -// TODO add metrics +// TODO better cleanup (?), more metrics https://github.com/FerretDB/FerretDB/issues/2862 // //nolint:vet // for readability type Registry struct { rw sync.RWMutex - m map[string]map[int64]*cursor // username -> ID -> cursor + m map[int64]*Cursor + + l *zap.Logger + wg sync.WaitGroup + + created *prometheus.CounterVec + duration *prometheus.HistogramVec } // NewRegistry creates a new Registry. -func NewRegistry() *Registry { +func NewRegistry(l *zap.Logger) *Registry { return &Registry{ - m: map[string]map[int64]*cursor{}, + m: map[int64]*Cursor{}, + l: l, + created: prometheus.NewCounterVec( + prometheus.CounterOpts{ + Namespace: namespace, + Subsystem: subsystem, + Name: "created_total", + Help: "Total number of cursors created.", + }, + []string{"db", "collection", "username"}, + ), + duration: prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: namespace, + Subsystem: subsystem, + Name: "duration_seconds", + Help: "Cursors lifetime in seconds.", + Buckets: []float64{ + 1 * time.Millisecond.Seconds(), + 5 * time.Millisecond.Seconds(), + 10 * time.Millisecond.Seconds(), + 25 * time.Millisecond.Seconds(), + 50 * time.Millisecond.Seconds(), + 100 * time.Millisecond.Seconds(), + 250 * time.Millisecond.Seconds(), + 500 * time.Millisecond.Seconds(), + 1000 * time.Millisecond.Seconds(), + 2500 * time.Millisecond.Seconds(), + 5000 * time.Millisecond.Seconds(), + 10000 * time.Millisecond.Seconds(), + }, + }, + []string{"db", "collection", "username"}, + ), } } -// Cursor returns stored cursor by username and ID, or nil. -func (r *Registry) Cursor(username string, id int64) *cursor { - r.rw.RLock() - defer r.rw.RUnlock() +// Close waits for all cursors to be closed. +func (r *Registry) Close() { + // we mainly do that for tests; see https://github.com/uber-go/zap/issues/687 - if u := r.m[username]; u == nil { - return nil - } + r.wg.Wait() +} - return r.m[username][id] +// NewParams represent parameters for NewCursor. +type NewParams struct { + Iter types.DocumentsIterator + DB string + Collection string + Username string } -// StoreCursor stores cursor and return its ID. -func (r *Registry) StoreCursor(username string, c *cursor) int64 { +// NewCursor creates and stores a new cursor. +// +// The cursor will be closed automatically when a given context is canceled, +// even if the cursor is not being used at that time. +func (r *Registry) NewCursor(ctx context.Context, params *NewParams) *Cursor { r.rw.Lock() defer r.rw.Unlock() - if u := r.m[username]; u == nil { - r.m[username] = map[int64]*cursor{} - } - // use global, sequential, positive, short cursor IDs to make debugging easier var id int64 - for id == 0 || r.m[username][id] != nil { + for id == 0 || r.m[id] != nil { id = int64(lastCursorID.Add(1)) } - r.m[username][id] = c + r.l.Debug( + "Creating", + zap.Int64("id", id), + zap.String("db", params.DB), + zap.String("collection", params.Collection), + ) + + r.created.WithLabelValues(params.DB, params.Collection, params.Username).Inc() + + c := newCursor(id, params.DB, params.Collection, params.Username, params.Iter, r) + r.m[id] = c - return id + r.wg.Add(1) + + go func() { + defer r.wg.Done() + + select { + case <-ctx.Done(): + c.Close() + case <-c.closed: + } + }() + + return c } -// DeleteCursor closes and deletes cursor. -func (r *Registry) DeleteCursor(username string, id int64) { +// Get returns stored cursor by ID, or nil. +func (r *Registry) Get(id int64) *Cursor { + r.rw.RLock() + defer r.rw.RUnlock() + + return r.m[id] +} + +// All returns a shallow copy of all stored cursors. +func (r *Registry) All() []*Cursor { + r.rw.RLock() + defer r.rw.RUnlock() + + return maps.Values(r.m) +} + +// This method should be called only from cursor.Close(). +func (r *Registry) delete(c *Cursor) { r.rw.Lock() defer r.rw.Unlock() - if u := r.m[username]; u == nil { - return - } + d := time.Since(c.created) + r.l.Debug( + "Deleting", + zap.Int("total", len(r.m)), + zap.Int64("id", c.ID), + zap.Duration("duration", d), + ) - c := r.m[username][id] - if c == nil { - return - } + r.duration.WithLabelValues(c.DB, c.Collection, c.Username).Observe(d.Seconds()) - c.Close() - delete(r.m[username], id) + delete(r.m, c.ID) +} - if len(r.m[username]) == 0 { - delete(r.m, username) - } +// Describe implements prometheus.Collector. +func (r *Registry) Describe(ch chan<- *prometheus.Desc) { + r.created.Describe(ch) + r.duration.Describe(ch) } + +// Collect implements prometheus.Collector. +func (r *Registry) Collect(ch chan<- prometheus.Metric) { + r.created.Collect(ch) + r.duration.Collect(ch) +} + +// check interfaces +var ( + _ prometheus.Collector = (*Registry)(nil) +) diff --git a/internal/handlers/common/count.go b/internal/handlers/common/count.go index 925ffa115677..9653bbff8a92 100644 --- a/internal/handlers/common/count.go +++ b/internal/handlers/common/count.go @@ -35,6 +35,7 @@ type CountParams struct { Hint any `ferretdb:"hint,ignored"` ReadConcern *types.Document `ferretdb:"readConcern,ignored"` Comment string `ferretdb:"comment,ignored"` + LSID any `ferretdb:"lsid,ignored"` } // GetCountParams returns the parameters for the count command. diff --git a/internal/handlers/common/find.go b/internal/handlers/common/find.go index adcbeab52656..6ae93a290b49 100644 --- a/internal/handlers/common/find.go +++ b/internal/handlers/common/find.go @@ -48,6 +48,7 @@ type FindParams struct { Max *types.Document `ferretdb:"max,ignored"` Min *types.Document `ferretdb:"min,ignored"` Hint any `ferretdb:"hint,ignored"` + LSID any `ferretdb:"lsid,ignored"` ReturnKey bool `ferretdb:"returnKey,unimplemented-non-default"` ShowRecordId bool `ferretdb:"showRecordId,unimplemented-non-default"` diff --git a/internal/handlers/common/getmore.go b/internal/handlers/common/getmore.go index 1e0eb65825ce..d2e8c99925c1 100644 --- a/internal/handlers/common/getmore.go +++ b/internal/handlers/common/getmore.go @@ -80,13 +80,14 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* ) } - // TODO maxTimeMS, comment + // TODO maxTimeMS https://github.com/FerretDB/FerretDB/issues/1808 + // TODO comment username, _ := conninfo.Get(ctx).Auth() // TODO: Use ExtractParam https://github.com/FerretDB/FerretDB/issues/2859 - cursor := registry.Cursor(username, cursorID) - if cursor == nil { + cursor := registry.Get(cursorID) + if cursor == nil || cursor.Username != username { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrCursorNotFound, fmt.Sprintf("cursor id %d not found", cursorID), @@ -121,7 +122,7 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* ) } - resDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor.Iter), int(batchSize)) + resDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor), int(batchSize)) if err != nil { return nil, lazyerrors.Error(err) } @@ -131,6 +132,12 @@ func GetMore(ctx context.Context, msg *wire.OpMsg, registry *cursor.Registry) (* nextBatch.Append(doc) } + if nextBatch.Len() < int(batchSize) { + // Cursor ID 0 lets the client know that there are no more results. + // Cursor is already closed and removed from the registry by this point. + cursorID = 0 + } + var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ Documents: []*types.Document{must.NotFail(types.NewDocument( diff --git a/internal/handlers/common/insert.go b/internal/handlers/common/insert.go index 9f17b1c3bb7b..7109d77fd1cf 100644 --- a/internal/handlers/common/insert.go +++ b/internal/handlers/common/insert.go @@ -31,6 +31,7 @@ type InsertParams struct { WriteConcern any `ferretdb:"writeConcern,ignored"` BypassDocumentValidation bool `ferretdb:"bypassDocumentValidation,ignored"` Comment string `ferretdb:"comment,ignored"` + LSID any `ferretdb:"lsid,ignored"` } // GetInsertParams returns the parameters for an insert command. diff --git a/internal/handlers/pg/msg_aggregate.go b/internal/handlers/pg/msg_aggregate.go index 9b46e8977a56..613effc18aaa 100644 --- a/internal/handlers/pg/msg_aggregate.go +++ b/internal/handlers/pg/msg_aggregate.go @@ -84,6 +84,8 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs ) } + username, _ := conninfo.Get(ctx).Auth() + pipeline, err := common.GetRequiredParam[*types.Array](document, "pipeline") if err != nil { return nil, commonerrors.NewCommandErrorMsgWithArgument( @@ -186,6 +188,7 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs qp.Sort = sort } + // TODO https://github.com/FerretDB/FerretDB/issues/1733 resDocs, err = processStagesDocuments(ctx, &stagesDocumentsParams{dbPool, &qp, stagesDocuments}) } else { // stats stages are provided - fetch stats from the DB and apply stages to them @@ -200,31 +203,33 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs return nil, err } - var cursorID int64 + cursor := h.cursors.NewCursor(ctx, &cursor.NewParams{ + Iter: iterator.Values(iterator.ForSlice(resDocs)), + DB: db, + Collection: collection, + Username: username, + }) - if h.EnableCursors && int64(len(resDocs)) > batchSize { - // Cursor is not created when resDocs is less than batchSize, it all fits in the firstBatch. - iter := iterator.Values(iterator.ForSlice(resDocs)) - c := cursor.New(&cursor.NewParams{ - Iter: iter, - DB: db, - Collection: collection, - BatchSize: int32(batchSize), - }) - username, _ := conninfo.Get(ctx).Auth() - cursorID = h.registry.StoreCursor(username, c) - resDocs, err = iterator.ConsumeValuesN(iter, int(batchSize)) + cursorID := cursor.ID - if err != nil { - return nil, lazyerrors.Error(err) - } + firstBatchDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor), int(batchSize)) + if err != nil { + cursor.Close() + return nil, lazyerrors.Error(err) } - firstBatch := types.MakeArray(len(resDocs)) - for _, doc := range resDocs { + firstBatch := types.MakeArray(len(firstBatchDocs)) + for _, doc := range firstBatchDocs { firstBatch.Append(doc) } + if firstBatch.Len() < int(batchSize) { + // let the client know that there are no more results + cursorID = 0 + + cursor.Close() + } + var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ Documents: []*types.Document{must.NotFail(types.NewDocument( diff --git a/internal/handlers/pg/msg_find.go b/internal/handlers/pg/msg_find.go index 0112352c4a8a..58ee8b713b13 100644 --- a/internal/handlers/pg/msg_find.go +++ b/internal/handlers/pg/msg_find.go @@ -50,11 +50,13 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } - if params.MaxTimeMS != 0 { - ctxWithTimeout, cancel := context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) - defer cancel() + username, _ := conninfo.Get(ctx).Auth() - ctx = ctxWithTimeout + cancel := func() {} + if params.MaxTimeMS != 0 { + // It is not clear if maxTimeMS affects only find, or both find and getMore (as the current code does). + // TODO https://github.com/FerretDB/FerretDB/issues/1808 + ctx, cancel = context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) } qp := &pgdb.QueryParams{ @@ -66,6 +68,7 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er // get comment from query, e.g. db.collection.find({$comment: "test"}) if params.Filter != nil { if qp.Comment, err = common.GetOptionalParam(params.Filter, "$comment", qp.Comment); err != nil { + cancel() return nil, err } } @@ -118,42 +121,47 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return lazyerrors.Error(err) } + // TODO https://github.com/FerretDB/FerretDB/issues/1733 resDocs, err = iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) return err }) if err != nil { + cancel() return nil, lazyerrors.Error(err) } - var cursorID int64 + iter := iterator.Values(iterator.ForSlice(resDocs)) + closer := iterator.NewMultiCloser(iter, iterator.CloserFunc(cancel)) - if h.EnableCursors && !params.SingleBatch && int64(len(resDocs)) > params.BatchSize { - // Cursor is not created for singleBatch, and when resDocs is less than batchSize, - // all response fits in the firstBatch. - iter := iterator.Values(iterator.ForSlice(resDocs)) - c := cursor.New(&cursor.NewParams{ - Iter: iter, - DB: params.DB, - Collection: params.Collection, - BatchSize: int32(params.BatchSize), - }) + cursor := h.cursors.NewCursor(ctx, &cursor.NewParams{ + Iter: iterator.WithClose(iter, closer.Close), + DB: params.DB, + Collection: params.Collection, + Username: username, + }) - username, _ := conninfo.Get(ctx).Auth() - cursorID = h.registry.StoreCursor(username, c) + cursorID := cursor.ID - resDocs, err = iterator.ConsumeValuesN(iter, int(params.BatchSize)) - if err != nil { - return nil, lazyerrors.Error(err) - } + firstBatchDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor), int(params.BatchSize)) + if err != nil { + cursor.Close() + return nil, lazyerrors.Error(err) } - firstBatch := types.MakeArray(len(resDocs)) - for _, doc := range resDocs { + firstBatch := types.MakeArray(len(firstBatchDocs)) + for _, doc := range firstBatchDocs { firstBatch.Append(doc) } + if params.SingleBatch || firstBatch.Len() < int(params.BatchSize) { + // let the client know that there are no more results + cursorID = 0 + + cursor.Close() + } + var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ Documents: []*types.Document{must.NotFail(types.NewDocument( diff --git a/internal/handlers/pg/msg_getmore.go b/internal/handlers/pg/msg_getmore.go index b6aed32ca5c0..e8a0a4b84c78 100644 --- a/internal/handlers/pg/msg_getmore.go +++ b/internal/handlers/pg/msg_getmore.go @@ -23,5 +23,5 @@ import ( // MsgGetMore implements handlers.Interface. func (h *Handler) MsgGetMore(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return common.GetMore(ctx, msg, h.registry) + return common.GetMore(ctx, msg, h.cursors) } diff --git a/internal/handlers/pg/pg.go b/internal/handlers/pg/pg.go index 09c51d32fd55..0c21060b1a3e 100644 --- a/internal/handlers/pg/pg.go +++ b/internal/handlers/pg/pg.go @@ -37,8 +37,8 @@ import ( type Handler struct { *NewOpts - url url.URL - registry *cursor.Registry + url url.URL + cursors *cursor.Registry // accessed by DBPool(ctx) rw sync.RWMutex @@ -56,7 +56,6 @@ type NewOpts struct { // test options DisableFilterPushdown bool EnableSortPushdown bool - EnableCursors bool } // New returns a new handler. @@ -71,10 +70,10 @@ func New(opts *NewOpts) (handlers.Interface, error) { } h := &Handler{ - NewOpts: opts, - url: *u, - registry: cursor.NewRegistry(), - pools: make(map[string]*pgdb.Pool, 1), + NewOpts: opts, + url: *u, + cursors: cursor.NewRegistry(opts.L.Named("cursors")), + pools: make(map[string]*pgdb.Pool, 1), } return h, nil @@ -89,6 +88,8 @@ func (h *Handler) Close() { p.Close() delete(h.pools, k) } + + h.cursors.Close() } // DBPool returns database connection pool for the given client connection. diff --git a/internal/handlers/registry/pg.go b/internal/handlers/registry/pg.go index 6d8d6d0811fc..e71f8fab2d52 100644 --- a/internal/handlers/registry/pg.go +++ b/internal/handlers/registry/pg.go @@ -31,7 +31,6 @@ func init() { DisableFilterPushdown: opts.DisableFilterPushdown, EnableSortPushdown: opts.EnableSortPushdown, - EnableCursors: opts.EnableCursors, } return pg.New(handlerOpts) diff --git a/internal/handlers/registry/registry.go b/internal/handlers/registry/registry.go index 9382d4497ef4..a5a570781f9f 100644 --- a/internal/handlers/registry/registry.go +++ b/internal/handlers/registry/registry.go @@ -62,7 +62,6 @@ type NewHandlerOpts struct { type TestOpts struct { DisableFilterPushdown bool EnableSortPushdown bool - EnableCursors bool } // NewHandler constructs a new handler. diff --git a/internal/handlers/registry/sqlite.go b/internal/handlers/registry/sqlite.go index 62a7e906cfb4..246a05115237 100644 --- a/internal/handlers/registry/sqlite.go +++ b/internal/handlers/registry/sqlite.go @@ -27,7 +27,7 @@ func init() { handlerOpts := &sqlite.NewOpts{ Dir: opts.SQLiteURI, - L: opts.Logger, + L: opts.Logger.Named("sqlite"), Metrics: opts.Metrics, StateProvider: opts.StateProvider, diff --git a/internal/handlers/registry/tigris.go b/internal/handlers/registry/tigris.go index 0e93a6394f26..7da8c4de12e3 100644 --- a/internal/handlers/registry/tigris.go +++ b/internal/handlers/registry/tigris.go @@ -38,7 +38,6 @@ func init() { StateProvider: opts.StateProvider, DisableFilterPushdown: opts.DisableFilterPushdown, - EnableCursors: opts.EnableCursors, } return tigris.New(handlerOpts) diff --git a/internal/handlers/sjson/sjson_test.go b/internal/handlers/sjson/sjson_test.go index 8263ef4cfd02..66c97d566703 100644 --- a/internal/handlers/sjson/sjson_test.go +++ b/internal/handlers/sjson/sjson_test.go @@ -191,6 +191,10 @@ func fuzzJSON(f *testing.F, testCases []testCase, newFunc func() sjsontype) { } for _, doc := range docs { + if doc.ValidateData() != nil { + continue + } + j, err := MarshalSingleValue(doc) require.NoError(f, err) diff --git a/internal/handlers/sqlite/msg_drop.go b/internal/handlers/sqlite/msg_drop.go index a6e68e8340dd..7b9b00611265 100644 --- a/internal/handlers/sqlite/msg_drop.go +++ b/internal/handlers/sqlite/msg_drop.go @@ -47,6 +47,18 @@ func (h *Handler) MsgDrop(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } + // This is mainly needed for SQLite and our cursor tests to avoid SQLITE_BUSY errors, + // but fine for other backends too. + // + // There is a race condition: another client could create a new cursor for that collection + // after we closed all of them, but before we drop the collection itself. + // In that case, we expect the client to retry the operation. + for _, c := range h.cursors.All() { + if c.DB == dbName && c.Collection == collectionName { + c.Close() + } + } + db := h.b.Database(dbName) defer db.Close() diff --git a/internal/handlers/sqlite/msg_find.go b/internal/handlers/sqlite/msg_find.go index a593b8f12680..233b383d96a6 100644 --- a/internal/handlers/sqlite/msg_find.go +++ b/internal/handlers/sqlite/msg_find.go @@ -19,6 +19,8 @@ import ( "errors" "time" + "github.com/FerretDB/FerretDB/internal/clientconn/conninfo" + "github.com/FerretDB/FerretDB/internal/clientconn/cursor" "github.com/FerretDB/FerretDB/internal/handlers/common" "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" "github.com/FerretDB/FerretDB/internal/types" @@ -40,46 +42,36 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } - if params.MaxTimeMS != 0 { - ctxWithTimeout, cancel := context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) - defer cancel() - - ctx = ctxWithTimeout - } - - if params.BatchSize == 0 { - var reply wire.OpMsg - must.NoError(reply.SetSections(wire.OpMsgSection{ - Documents: []*types.Document{must.NotFail(types.NewDocument( - "cursor", must.NotFail(types.NewDocument( - "firstBatch", types.MakeArray(0), - "id", int64(0), - "ns", params.DB+"."+params.Collection, - )), - "ok", float64(1), - ))}, - })) - - return &reply, nil - } + username, _ := conninfo.Get(ctx).Auth() db := h.b.Database(params.DB) defer db.Close() + cancel := func() {} + if params.MaxTimeMS != 0 { + // It is not clear if maxTimeMS affects only find, or both find and getMore (as the current code does). + // TODO https://github.com/FerretDB/FerretDB/issues/1808 + ctx, cancel = context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) + } + queryRes, err := db.Collection(params.Collection).Query(ctx, nil) if err != nil { + cancel() return nil, lazyerrors.Error(err) } - iter := queryRes.Iter + // closer accumulates all things that should be closed / canceled + closer := iterator.NewMultiCloser(queryRes.Iter) - closer := iterator.NewMultiCloser(iter) - defer closer.Close() + // to free maxTimeMS's context resources when they are not needed + closer.Add(iterator.CloserFunc(cancel)) - iter = common.FilterIterator(iter, closer, params.Filter) + iter := common.FilterIterator(queryRes.Iter, closer, params.Filter) iter, err = common.SortIterator(iter, closer, params.Sort) if err != nil { + closer.Close() + var pathErr *types.DocumentPathError if errors.As(err, &pathErr) && pathErr.Code() == types.ErrDocumentPathEmptyKey { return nil, commonerrors.NewCommandErrorMsgWithArgument( @@ -98,21 +90,39 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er iter, err = common.ProjectionIterator(iter, closer, params.Projection, params.Filter) if err != nil { + closer.Close() return nil, lazyerrors.Error(err) } - res, err := iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) + // Combine iterators chain and closer into a cursor to pass around. + // The context will be canceled when client disconnects or after maxTimeMS. + cursor := h.cursors.NewCursor(ctx, &cursor.NewParams{ + Iter: iterator.WithClose(iterator.Interface[struct{}, *types.Document](iter), closer.Close), + DB: params.DB, + Collection: params.Collection, + Username: username, + }) + + cursorID := cursor.ID + + firstBatchDocs, err := iterator.ConsumeValuesN(iterator.Interface[struct{}, *types.Document](cursor), int(params.BatchSize)) if err != nil { + cursor.Close() return nil, lazyerrors.Error(err) } - var cursorID int64 - - firstBatch := types.MakeArray(len(res)) - for _, doc := range res { + firstBatch := types.MakeArray(len(firstBatchDocs)) + for _, doc := range firstBatchDocs { firstBatch.Append(doc) } + if params.SingleBatch || firstBatch.Len() < int(params.BatchSize) { + // let the client know that there are no more results + cursorID = 0 + + cursor.Close() + } + var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ Documents: []*types.Document{must.NotFail(types.NewDocument( diff --git a/internal/handlers/sqlite/msg_getmore.go b/internal/handlers/sqlite/msg_getmore.go index 10507019c446..f9c3b6f07f40 100644 --- a/internal/handlers/sqlite/msg_getmore.go +++ b/internal/handlers/sqlite/msg_getmore.go @@ -17,11 +17,11 @@ package sqlite import ( "context" - "github.com/FerretDB/FerretDB/internal/util/must" + "github.com/FerretDB/FerretDB/internal/handlers/common" "github.com/FerretDB/FerretDB/internal/wire" ) // MsgGetMore implements handlers.Interface. func (h *Handler) MsgGetMore(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return nil, notImplemented(must.NotFail(msg.Document()).Command()) + return common.GetMore(ctx, msg, h.cursors) } diff --git a/internal/handlers/sqlite/sqlite.go b/internal/handlers/sqlite/sqlite.go index 89b98b4f8a4d..b01390ede4a4 100644 --- a/internal/handlers/sqlite/sqlite.go +++ b/internal/handlers/sqlite/sqlite.go @@ -24,6 +24,7 @@ import ( "github.com/FerretDB/FerretDB/internal/backends" "github.com/FerretDB/FerretDB/internal/backends/sqlite" "github.com/FerretDB/FerretDB/internal/clientconn/connmetrics" + "github.com/FerretDB/FerretDB/internal/clientconn/cursor" "github.com/FerretDB/FerretDB/internal/handlers" "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" "github.com/FerretDB/FerretDB/internal/util/state" @@ -40,7 +41,10 @@ func notImplemented(command string) error { // Handler implements handlers.Interface. type Handler struct { *NewOpts + b backends.Backend + + cursors *cursor.Registry } // NewOpts represents handler configuration. @@ -53,6 +57,7 @@ type NewOpts struct { Metrics *connmetrics.ConnMetrics StateProvider *state.Provider + // test options DisableFilterPushdown bool } @@ -60,28 +65,33 @@ type NewOpts struct { func New(opts *NewOpts) (handlers.Interface, error) { b, err := sqlite.NewBackend(&sqlite.NewBackendParams{ Dir: opts.Dir, + L: opts.L, }) if err != nil { return nil, err } return &Handler{ - NewOpts: opts, b: b, + NewOpts: opts, + cursors: cursor.NewRegistry(opts.L.Named("cursors")), }, nil } // Close implements handlers.Interface. -func (h *Handler) Close() {} +func (h *Handler) Close() { + h.cursors.Close() + h.b.Close() +} // Describe implements handlers.Interface. func (h *Handler) Describe(ch chan<- *prometheus.Desc) { - // TODO + h.cursors.Describe(ch) } // Collect implements handlers.Interface. func (h *Handler) Collect(ch chan<- prometheus.Metric) { - // TODO + h.cursors.Collect(ch) } // check interfaces diff --git a/internal/handlers/tigris/tigris.go b/internal/handlers/tigris/tigris.go index 285a8af09d0c..2d8ff602a80c 100644 --- a/internal/handlers/tigris/tigris.go +++ b/internal/handlers/tigris/tigris.go @@ -56,7 +56,6 @@ type NewOpts struct { // test options DisableFilterPushdown bool - EnableCursors bool } // AuthParams represents authentication parameters. @@ -74,7 +73,7 @@ func New(opts *NewOpts) (handlers.Interface, error) { return &Handler{ NewOpts: opts, - registry: cursor.NewRegistry(), + registry: cursor.NewRegistry(opts.L.Named("cursors")), pools: make(map[AuthParams]*tigrisdb.TigrisDB, 1), }, nil } @@ -88,6 +87,8 @@ func (h *Handler) Close() { p.Driver.Close() delete(h.pools, k) } + + h.registry.Close() } // DBPool returns database connection pool for the given client connection. diff --git a/internal/util/iterator/closer.go b/internal/util/iterator/closer.go index 90017516cd86..819c8b3b53be 100644 --- a/internal/util/iterator/closer.go +++ b/internal/util/iterator/closer.go @@ -21,6 +21,14 @@ type Closer interface { Close() } +// CloserFunc converts a function (such as context.CancelFunc) to a Closer. +type CloserFunc func() + +// Close implements Closer. +func (cf CloserFunc) Close() { + cf() +} + // MultiCloser is a helper for closing multiple closers. type MultiCloser struct { token *resource.Token @@ -60,3 +68,8 @@ func (mc *MultiCloser) Close() { resource.Untrack(mc, mc.token) } + +// check interfaces +var ( + _ Closer = CloserFunc(nil) +) diff --git a/internal/util/resource/resource.go b/internal/util/resource/resource.go index ea14b2a40407..7d4b215a0f58 100644 --- a/internal/util/resource/resource.go +++ b/internal/util/resource/resource.go @@ -86,9 +86,7 @@ func Track(obj any, token *Token) { msg += "\nObject created by " + string(stack) } - // TODO - _ = msg - // panic(msg) + panic(msg) }) } diff --git a/internal/util/testutil/testutil.go b/internal/util/testutil/testutil.go index b58947f40df5..866978d4340a 100644 --- a/internal/util/testutil/testutil.go +++ b/internal/util/testutil/testutil.go @@ -42,7 +42,6 @@ func Ctx(tb testing.TB) context.Context { select { case <-testDone: signalsCancel() - return case <-signalsCtx.Done(): // There is a weird interaction between terminal's process group/session signal handling, From a15bb132baafa31806ab9aaf3d0b6bbf424736f9 Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Tue, 27 Jun 2023 16:33:01 +0900 Subject: [PATCH 28/46] Improve validation for `createIndexes` and `dropIndexes` (#2884) Closes #2311. --- integration/indexes_compat_test.go | 58 ++-- integration/indexes_test.go | 260 ++++++++++++++++-- internal/handlers/commonerrors/error.go | 6 + .../handlers/commonerrors/errorcode_string.go | 142 +++++----- internal/handlers/pg/msg_createindexes.go | 196 +++++++++++-- internal/handlers/pg/msg_dropindexes.go | 8 + 6 files changed, 538 insertions(+), 132 deletions(-) diff --git a/integration/indexes_compat_test.go b/integration/indexes_compat_test.go index 03cd3a2951b9..f6b9a10daa04 100644 --- a/integration/indexes_compat_test.go +++ b/integration/indexes_compat_test.go @@ -104,7 +104,6 @@ func TestCreateIndexesCompat(t *testing.T) { Keys: bson.D{{"_id", 1}}, // this index is already created by default }, }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "DescendingID": { models: []mongo.IndexModel{ @@ -173,7 +172,18 @@ func TestCreateIndexesCompat(t *testing.T) { {Keys: bson.D{{"v", 1}}}, }, resultType: emptyResult, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", + skip: "https://github.com/FerretDB/FerretDB/issues/2910", + // the error for existing and non-existing collection are different, + // below is the error for existing collection. + // + // &mongo.CommandError{ + // Code: 96, + // Name: "OperationFailed", + // Message: `Index build failed: 7a1c4cc3-8ac6-44d3-92e0-57853e6bc837: Collection ` + + // `TestCreateIndexesCommandInvalidSpec-SameIndex.TestCreateIndexesCommandInvalidSpec-SameIndex ` + + // `( 020f17e0-7847-45f2-8397-c631c5e9bdaf ) :: caused by :: Cannot build two identical indexes. ` + + // `Try again without duplicate indexes.`, + // }, }, "MultiWithInvalid": { models: []mongo.IndexModel{ @@ -336,21 +346,18 @@ func TestCreateIndexesCommandCompat(t *testing.T) { key: bson.D{{"v", -1}}, indexName: "custom-name", resultType: emptyResult, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "IndexNameNotSet": { collectionName: "test", key: bson.D{{"v", -1}}, indexName: nil, resultType: emptyResult, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "EmptyIndexName": { collectionName: "test", key: bson.D{{"v", -1}}, indexName: "", resultType: emptyResult, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "NonStringIndexName": { collectionName: "test", @@ -362,7 +369,6 @@ func TestCreateIndexesCommandCompat(t *testing.T) { collectionName: "test", key: bson.D{{"_id", 1}, {"v", 1}}, indexName: "_id_", // the same name as the default index - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "InvalidKey": { collectionName: "test", @@ -377,14 +383,12 @@ func TestCreateIndexesCommandCompat(t *testing.T) { "KeyNotSet": { collectionName: "test", resultType: emptyResult, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, "UniqueFalse": { collectionName: "unique_false", key: bson.D{{"v", 1}}, indexName: "unique_false", unique: false, - skip: "https://github.com/FerretDB/FerretDB/issues/2845", }, "UniqueTypeDocument": { collectionName: "test", @@ -451,27 +455,31 @@ func TestCreateIndexesCommandCompat(t *testing.T) { assert.Equal(t, compatRes, targetRes) - targetErr = targetCollection.Database().RunCommand( - ctx, bson.D{{"listIndexes", tc.collectionName}}, - ).Decode(&targetRes) + targetCursor, targetErr := targetCollection.Indexes().List(ctx) + compatCursor, compatErr := compatCollection.Indexes().List(ctx) - compatErr = compatCollection.Database().RunCommand( - ctx, bson.D{{"listIndexes", tc.collectionName}}, - ).Decode(&targetRes) + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } - require.Nil(t, targetRes) - require.Nil(t, compatRes) + require.NoError(t, targetErr) + require.NoError(t, compatErr) - if targetErr != nil { - t.Logf("Target error: %v", targetErr) - t.Logf("Compat error: %v", compatErr) + targetListRes := FetchAll(t, ctx, targetCursor) + compatListRes := FetchAll(t, ctx, compatCursor) - // error messages are intentionally not compared - AssertMatchesCommandError(t, compatErr, targetErr) + assert.Equal(t, compatListRes, targetListRes) - return - } - require.NoError(t, compatErr, "compat error; target returned no error") + targetSpec, targetErr := targetCollection.Indexes().ListSpecifications(ctx) + compatSpec, compatErr := compatCollection.Indexes().ListSpecifications(ctx) + + require.NoError(t, compatErr) + require.NoError(t, targetErr) + + assert.Equal(t, compatSpec, targetSpec) }) } } @@ -806,7 +814,7 @@ func TestDropIndexesCommandCompat(t *testing.T) { } } -func TestCreateIndexesUniqueCompat(t *testing.T) { +func TestCreateIndexesCompatUnique(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes creation is not supported for Tigris") t.Parallel() diff --git a/integration/indexes_test.go b/integration/indexes_test.go index d674f29e9836..a4b2e242a05c 100644 --- a/integration/indexes_test.go +++ b/integration/indexes_test.go @@ -26,7 +26,7 @@ import ( "github.com/FerretDB/FerretDB/integration/shareddata" ) -func TestIndexesDropCommandErrors(t *testing.T) { +func TestDropIndexesCommandErrors(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") t.Parallel() @@ -81,7 +81,11 @@ func TestIndexesDropCommandErrors(t *testing.T) { }, "InvalidDocumentIndex": { toDrop: bson.D{{"invalid", "invalid"}}, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", + err: &mongo.CommandError{ + Code: 27, + Name: "IndexNotFound", + Message: "can't find index with key: { invalid: \"invalid\" }", + }, }, "NonExistentKey": { toDrop: bson.D{{"non-existent", 1}}, @@ -173,16 +177,19 @@ func TestIndexesDropCommandErrors(t *testing.T) { } } -func TestCreateIndexesInvalidSpec(t *testing.T) { +func TestCreateIndexesCommandInvalidSpec(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") t.Parallel() for name, tc := range map[string]struct { - indexes any - err *mongo.CommandError - altMessage string - skip string + indexes any // optional + missingIndexes bool // optional, if set indexes must be nil + noProvider bool // if set, no provider is added. + + err *mongo.CommandError // required, expected error from MongoDB + altMessage string // optional, alternative error message for FerretDB, ignored if empty + skip string // optional, skip test with a specified reason }{ "EmptyIndexes": { indexes: bson.A{}, @@ -192,6 +199,14 @@ func TestCreateIndexesInvalidSpec(t *testing.T) { Message: "Must specify at least one index to create", }, }, + "MissingIndexes": { + missingIndexes: true, + err: &mongo.CommandError{ + Code: 40414, + Name: "Location40414", + Message: "BSON field 'createIndexes.indexes' is missing but a required field", + }, + }, "NilIndexes": { indexes: nil, err: &mongo.CommandError{ @@ -199,16 +214,30 @@ func TestCreateIndexesInvalidSpec(t *testing.T) { Name: "Location10065", Message: "invalid parameter: expected an object (indexes)", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, - "InvalidType": { + "InvalidTypeObject": { + indexes: bson.D{}, + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "BSON field 'createIndexes.indexes' is the wrong type 'object', expected type 'array'", + }, + }, + "InvalidTypeInt": { indexes: 42, err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", Message: "BSON field 'createIndexes.indexes' is the wrong type 'int', expected type 'array'", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", + }, + "InvalidTypeArrayString": { + indexes: bson.A{"invalid"}, + err: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "BSON field 'createIndexes.indexes.0' is the wrong type 'string', expected type 'object'", + }, }, "IDIndex": { indexes: bson.A{ @@ -225,6 +254,106 @@ func TestCreateIndexesInvalidSpec(t *testing.T) { ` Specification: { key: { _id: 1 }, name: "_id_", unique: true, v: 2 }`, }, }, + "MissingName": { + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", 1}}}, + }, + }, + err: &mongo.CommandError{ + Code: 9, + Name: "FailedToParse", + Message: `Error in specification { key: { v: 1 } } :: caused by :: ` + + `The 'name' field is a required property of an index specification`, + }, + }, + "EmptyName": { + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", -1}}}, + {"name", ""}, + }, + }, + err: &mongo.CommandError{ + Code: 67, + Name: "CannotCreateIndex", + Message: `Error in specification { key: { v: -1 }, name: "", v: 2 } :: caused by :: index name cannot be empty`, + }, + altMessage: `Error in specification { key: { v: -1 }, name: "" } :: caused by :: index name cannot be empty`, + }, + "MissingKey": { + indexes: bson.A{ + bson.D{}, + }, + err: &mongo.CommandError{ + Code: 9, + Name: "FailedToParse", + Message: `Error in specification {} :: caused by :: The 'key' field is a required property of an index specification`, + }, + }, + "IdenticalIndex": { + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", 1}}}, + {"name", "v_1"}, + }, + bson.D{ + {"key", bson.D{{"v", 1}}}, + {"name", "v_1"}, + }, + }, + noProvider: true, + err: &mongo.CommandError{ + Code: 68, + Name: "IndexAlreadyExists", + Message: `Identical index already exists: v_1`, + }, + }, + "SameName": { + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"foo", -1}}}, + {"name", "index-name"}, + }, + bson.D{ + {"key", bson.D{{"bar", -1}}}, + {"name", "index-name"}, + }, + }, + noProvider: true, + err: &mongo.CommandError{ + Code: 86, + Name: "IndexKeySpecsConflict", + Message: "An existing index has the same name as the requested index. " + + "When index names are not specified, they are auto generated and can " + + "cause conflicts. Please refer to our documentation. " + + "Requested index: { v: 2, key: { bar: -1 }, name: \"index-name\" }, " + + "existing index: { v: 2, key: { foo: -1 }, name: \"index-name\" }", + }, + altMessage: "An existing index has the same name as the requested index. " + + "When index names are not specified, they are auto generated and can " + + "cause conflicts. Please refer to our documentation. " + + "Requested index: { key: { bar: -1 }, name: \"index-name\" }, " + + "existing index: { key: { foo: -1 }, name: \"index-name\" }", + }, + "SameIndex": { + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", -1}}}, + {"name", "foo"}, + }, + bson.D{ + {"key", bson.D{{"v", -1}}}, + {"name", "bar"}, + }, + }, + noProvider: true, + err: &mongo.CommandError{ + Code: 85, + Name: "IndexOptionsConflict", + Message: "Index already exists with a different name: foo", + }, + }, "UniqueTypeDocument": { indexes: bson.A{ bson.D{ @@ -251,11 +380,109 @@ func TestCreateIndexesInvalidSpec(t *testing.T) { t.Parallel() + if tc.missingIndexes { + require.Nil(t, tc.indexes, "indexes must be nil if missingIndexes is true") + } + + var providers []shareddata.Provider + if !tc.noProvider { + // one provider is enough to check for errors + providers = append(providers, shareddata.ArrayDocuments) + } + + ctx, collection := setup.Setup(t, providers...) + + var rest bson.D + + if !tc.missingIndexes { + rest = append(rest, bson.E{Key: "indexes", Value: tc.indexes}) + } + + command := append(bson.D{ + {"createIndexes", collection.Name()}, + }, + rest..., + ) + + var res bson.D + err := collection.Database().RunCommand(ctx, command).Decode(&res) + + require.Nil(t, res) + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) + }) + } +} + +func TestCreateIndexesCommandInvalidCollection(t *testing.T) { + setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") + + t.Parallel() + + for name, tc := range map[string]struct { + collectionName any + indexes any + err *mongo.CommandError + altMessage string + skip string + }{ + "InvalidTypeCollection": { + collectionName: 42, + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", 1}}}, + {"name", "v_1"}, + }, + }, + err: &mongo.CommandError{ + Code: 2, + Name: "BadValue", + Message: "collection name has invalid type int", + }, + altMessage: "required parameter \"createIndexes\" has type int32 (expected string)", + }, + "NilCollection": { + collectionName: nil, + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", 1}}}, + {"name", "v_1"}, + }, + }, + err: &mongo.CommandError{ + Code: 2, + Name: "BadValue", + Message: "collection name has invalid type null", + }, + altMessage: "required parameter \"createIndexes\" has type types.NullType (expected string)", + }, + "EmptyCollection": { + collectionName: "", + indexes: bson.A{ + bson.D{ + {"key", bson.D{{"v", 1}}}, + {"name", "v_1"}, + }, + }, + err: &mongo.CommandError{ + Code: 73, + Name: "InvalidNamespace", + Message: "Invalid namespace specified 'TestCreateIndexesCommandInvalidCollection-EmptyCollection.'", + }, + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Parallel() + provider := shareddata.ArrayDocuments // one provider is enough to check for errors ctx, collection := setup.Setup(t, provider) command := bson.D{ - {"createIndexes", collection.Name()}, + {"createIndexes", tc.collectionName}, {"indexes", tc.indexes}, } @@ -268,7 +495,7 @@ func TestCreateIndexesInvalidSpec(t *testing.T) { } } -func TestDropIndexesInvalidCollection(t *testing.T) { +func TestDropIndexesCommandInvalidCollection(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") t.Parallel() @@ -286,7 +513,7 @@ func TestDropIndexesInvalidCollection(t *testing.T) { err: &mongo.CommandError{ Code: 26, Name: "NamespaceNotFound", - Message: "ns not found TestDropIndexesInvalidCollection-NonExistentCollection.non-existent", + Message: "ns not found TestDropIndexesCommandInvalidCollection-NonExistentCollection.non-existent", }, }, "InvalidTypeCollection": { @@ -297,7 +524,7 @@ func TestDropIndexesInvalidCollection(t *testing.T) { Name: "BadValue", Message: "collection name has invalid type int", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", + altMessage: "required parameter \"dropIndexes\" has type int32 (expected string)", }, "NilCollection": { collectionName: nil, @@ -307,7 +534,7 @@ func TestDropIndexesInvalidCollection(t *testing.T) { Name: "BadValue", Message: "collection name has invalid type null", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", + altMessage: "required parameter \"dropIndexes\" has type types.NullType (expected string)", }, "EmptyCollection": { collectionName: "", @@ -315,9 +542,8 @@ func TestDropIndexesInvalidCollection(t *testing.T) { err: &mongo.CommandError{ Code: 73, Name: "InvalidNamespace", - Message: "Invalid namespace specified 'TestIndexesDropInvalidCollection-EmptyCollection.'", + Message: "Invalid namespace specified 'TestDropIndexesCommandInvalidCollection-EmptyCollection.'", }, - skip: "https://github.com/FerretDB/FerretDB/issues/2311", }, } { name, tc := name, tc diff --git a/internal/handlers/commonerrors/error.go b/internal/handlers/commonerrors/error.go index fb9bc3534b46..91297f05824b 100644 --- a/internal/handlers/commonerrors/error.go +++ b/internal/handlers/commonerrors/error.go @@ -89,6 +89,9 @@ const ( // ErrCannotCreateIndex indicates that index creation process failed because some data are not valid. ErrCannotCreateIndex = ErrorCode(67) // CannotCreateIndex + // ErrIndexAlreadyExists indicates that identical index already exists. + ErrIndexAlreadyExists = ErrorCode(68) // IndexAlreadyExists + // ErrInvalidOptions indicates that _id index cannot be deleted. ErrInvalidOptions = ErrorCode(72) // InvalidOptions @@ -116,6 +119,9 @@ const ( // ErrNotImplemented indicates that a flag or command is not implemented. ErrNotImplemented = ErrorCode(238) // NotImplemented + // ErrIndexesWrongType indicates that indexes parameter has wrong type. + ErrIndexesWrongType = ErrorCode(10065) // Location10065 + // ErrDuplicateKeyInsert indicates duplicate key violation on inserting document. ErrDuplicateKeyInsert = ErrorCode(11000) // Location11000 diff --git a/internal/handlers/commonerrors/errorcode_string.go b/internal/handlers/commonerrors/errorcode_string.go index 7522b5da5b16..ea547a4c7dee 100644 --- a/internal/handlers/commonerrors/errorcode_string.go +++ b/internal/handlers/commonerrors/errorcode_string.go @@ -28,6 +28,7 @@ func _() { _ = x[ErrCommandNotFound-59] _ = x[ErrImmutableField-66] _ = x[ErrCannotCreateIndex-67] + _ = x[ErrIndexAlreadyExists-68] _ = x[ErrInvalidOptions-72] _ = x[ErrInvalidNamespace-73] _ = x[ErrIndexOptionsConflict-85] @@ -37,6 +38,7 @@ func _() { _ = x[ErrInvalidIndexSpecificationOption-197] _ = x[ErrInvalidPipelineOperator-168] _ = x[ErrNotImplemented-238] + _ = x[ErrIndexesWrongType-10065] _ = x[ErrDuplicateKeyInsert-11000] _ = x[ErrSetBadExpression-40272] _ = x[ErrStageGroupInvalidFields-15947] @@ -99,7 +101,7 @@ func _() { _ = x[ErrStageCollStatsInvalidArg-5447000] } -const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseUnauthorizedTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40181Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" +const _ErrorCode_name = "UnsetInternalErrorBadValueFailedToParseUnauthorizedTypeMismatchAuthenticationFailedIllegalOperationNamespaceNotFoundIndexNotFoundPathNotViableConflictingUpdateOperatorsCursorNotFoundNamespaceExistsDollarPrefixedFieldNameInvalidIDEmptyFieldNameCommandNotFoundImmutableFieldCannotCreateIndexIndexAlreadyExistsInvalidOptionsInvalidNamespaceIndexOptionsConflictIndexKeySpecsConflictOperationFailedDocumentValidationFailureInvalidPipelineOperatorInvalidIndexSpecificationOptionNotImplementedLocation10065Location11000Location15947Location15948Location15955Location15958Location15959Location15969Location15973Location15974Location15975Location15976Location15981Location15998Location16020Location16410Location16872Location17276Location28667Location28724Location28812Location28818Location31002Location31119Location31120Location31249Location31250Location31253Location31254Location31324Location31325Location31394Location31395Location40156Location40157Location40158Location40160Location40181Location40234Location40237Location40238Location40272Location40323Location40352Location40353Location40414Location40415Location50840Location51024Location51075Location51091Location51108Location51246Location51247Location51270Location51272Location4822819Location5107200Location5107201Location5447000" var _ErrorCode_map = map[ErrorCode]string{ 0: _ErrorCode_name[0:5], @@ -122,74 +124,76 @@ var _ErrorCode_map = map[ErrorCode]string{ 59: _ErrorCode_name[243:258], 66: _ErrorCode_name[258:272], 67: _ErrorCode_name[272:289], - 72: _ErrorCode_name[289:303], - 73: _ErrorCode_name[303:319], - 85: _ErrorCode_name[319:339], - 86: _ErrorCode_name[339:360], - 96: _ErrorCode_name[360:375], - 121: _ErrorCode_name[375:400], - 168: _ErrorCode_name[400:423], - 197: _ErrorCode_name[423:454], - 238: _ErrorCode_name[454:468], - 11000: _ErrorCode_name[468:481], - 15947: _ErrorCode_name[481:494], - 15948: _ErrorCode_name[494:507], - 15955: _ErrorCode_name[507:520], - 15958: _ErrorCode_name[520:533], - 15959: _ErrorCode_name[533:546], - 15969: _ErrorCode_name[546:559], - 15973: _ErrorCode_name[559:572], - 15974: _ErrorCode_name[572:585], - 15975: _ErrorCode_name[585:598], - 15976: _ErrorCode_name[598:611], - 15981: _ErrorCode_name[611:624], - 15998: _ErrorCode_name[624:637], - 16020: _ErrorCode_name[637:650], - 16410: _ErrorCode_name[650:663], - 16872: _ErrorCode_name[663:676], - 17276: _ErrorCode_name[676:689], - 28667: _ErrorCode_name[689:702], - 28724: _ErrorCode_name[702:715], - 28812: _ErrorCode_name[715:728], - 28818: _ErrorCode_name[728:741], - 31002: _ErrorCode_name[741:754], - 31119: _ErrorCode_name[754:767], - 31120: _ErrorCode_name[767:780], - 31249: _ErrorCode_name[780:793], - 31250: _ErrorCode_name[793:806], - 31253: _ErrorCode_name[806:819], - 31254: _ErrorCode_name[819:832], - 31324: _ErrorCode_name[832:845], - 31325: _ErrorCode_name[845:858], - 31394: _ErrorCode_name[858:871], - 31395: _ErrorCode_name[871:884], - 40156: _ErrorCode_name[884:897], - 40157: _ErrorCode_name[897:910], - 40158: _ErrorCode_name[910:923], - 40160: _ErrorCode_name[923:936], - 40181: _ErrorCode_name[936:949], - 40234: _ErrorCode_name[949:962], - 40237: _ErrorCode_name[962:975], - 40238: _ErrorCode_name[975:988], - 40272: _ErrorCode_name[988:1001], - 40323: _ErrorCode_name[1001:1014], - 40352: _ErrorCode_name[1014:1027], - 40353: _ErrorCode_name[1027:1040], - 40414: _ErrorCode_name[1040:1053], - 40415: _ErrorCode_name[1053:1066], - 50840: _ErrorCode_name[1066:1079], - 51024: _ErrorCode_name[1079:1092], - 51075: _ErrorCode_name[1092:1105], - 51091: _ErrorCode_name[1105:1118], - 51108: _ErrorCode_name[1118:1131], - 51246: _ErrorCode_name[1131:1144], - 51247: _ErrorCode_name[1144:1157], - 51270: _ErrorCode_name[1157:1170], - 51272: _ErrorCode_name[1170:1183], - 4822819: _ErrorCode_name[1183:1198], - 5107200: _ErrorCode_name[1198:1213], - 5107201: _ErrorCode_name[1213:1228], - 5447000: _ErrorCode_name[1228:1243], + 68: _ErrorCode_name[289:307], + 72: _ErrorCode_name[307:321], + 73: _ErrorCode_name[321:337], + 85: _ErrorCode_name[337:357], + 86: _ErrorCode_name[357:378], + 96: _ErrorCode_name[378:393], + 121: _ErrorCode_name[393:418], + 168: _ErrorCode_name[418:441], + 197: _ErrorCode_name[441:472], + 238: _ErrorCode_name[472:486], + 10065: _ErrorCode_name[486:499], + 11000: _ErrorCode_name[499:512], + 15947: _ErrorCode_name[512:525], + 15948: _ErrorCode_name[525:538], + 15955: _ErrorCode_name[538:551], + 15958: _ErrorCode_name[551:564], + 15959: _ErrorCode_name[564:577], + 15969: _ErrorCode_name[577:590], + 15973: _ErrorCode_name[590:603], + 15974: _ErrorCode_name[603:616], + 15975: _ErrorCode_name[616:629], + 15976: _ErrorCode_name[629:642], + 15981: _ErrorCode_name[642:655], + 15998: _ErrorCode_name[655:668], + 16020: _ErrorCode_name[668:681], + 16410: _ErrorCode_name[681:694], + 16872: _ErrorCode_name[694:707], + 17276: _ErrorCode_name[707:720], + 28667: _ErrorCode_name[720:733], + 28724: _ErrorCode_name[733:746], + 28812: _ErrorCode_name[746:759], + 28818: _ErrorCode_name[759:772], + 31002: _ErrorCode_name[772:785], + 31119: _ErrorCode_name[785:798], + 31120: _ErrorCode_name[798:811], + 31249: _ErrorCode_name[811:824], + 31250: _ErrorCode_name[824:837], + 31253: _ErrorCode_name[837:850], + 31254: _ErrorCode_name[850:863], + 31324: _ErrorCode_name[863:876], + 31325: _ErrorCode_name[876:889], + 31394: _ErrorCode_name[889:902], + 31395: _ErrorCode_name[902:915], + 40156: _ErrorCode_name[915:928], + 40157: _ErrorCode_name[928:941], + 40158: _ErrorCode_name[941:954], + 40160: _ErrorCode_name[954:967], + 40181: _ErrorCode_name[967:980], + 40234: _ErrorCode_name[980:993], + 40237: _ErrorCode_name[993:1006], + 40238: _ErrorCode_name[1006:1019], + 40272: _ErrorCode_name[1019:1032], + 40323: _ErrorCode_name[1032:1045], + 40352: _ErrorCode_name[1045:1058], + 40353: _ErrorCode_name[1058:1071], + 40414: _ErrorCode_name[1071:1084], + 40415: _ErrorCode_name[1084:1097], + 50840: _ErrorCode_name[1097:1110], + 51024: _ErrorCode_name[1110:1123], + 51075: _ErrorCode_name[1123:1136], + 51091: _ErrorCode_name[1136:1149], + 51108: _ErrorCode_name[1149:1162], + 51246: _ErrorCode_name[1162:1175], + 51247: _ErrorCode_name[1175:1188], + 51270: _ErrorCode_name[1188:1201], + 51272: _ErrorCode_name[1201:1214], + 4822819: _ErrorCode_name[1214:1229], + 5107200: _ErrorCode_name[1229:1244], + 5107201: _ErrorCode_name[1244:1259], + 5447000: _ErrorCode_name[1259:1274], } func (i ErrorCode) String() string { diff --git a/internal/handlers/pg/msg_createindexes.go b/internal/handlers/pg/msg_createindexes.go index d29e34484fe2..c45dc311a2a8 100644 --- a/internal/handlers/pg/msg_createindexes.go +++ b/internal/handlers/pg/msg_createindexes.go @@ -59,9 +59,41 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. return nil, err } - idxArr, err := common.GetRequiredParam[*types.Array](document, "indexes") - if err != nil { - return nil, err + if collection == "" { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidNamespace, + fmt.Sprintf("Invalid namespace specified '%s.'", db), + command, + ) + } + + v, _ := document.Get("indexes") + if v == nil { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrMissingField, + "BSON field 'createIndexes.indexes' is missing but a required field", + document.Command(), + ) + } + + idxArr, ok := v.(*types.Array) + if !ok { + if _, ok = v.(types.NullType); ok { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrIndexesWrongType, + "invalid parameter: expected an object (indexes)", + document.Command(), + ) + } + + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrTypeMismatch, + fmt.Sprintf( + "BSON field 'createIndexes.indexes' is the wrong type '%s', expected type 'array'", + commonparams.AliasFromType(v), + ), + document.Command(), + ) } if idxArr.Len() == 0 { @@ -75,16 +107,41 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. iter := idxArr.Iterator() defer iter.Close() + indexes := map[*types.Document]*pgdb.Index{} + + var isUniqueSpecified bool + var numIndexesBefore, numIndexesAfter int32 err = dbPool.InTransactionRetry(ctx, func(tx pgx.Tx) error { + var indexesBefore []pgdb.Index + indexesBefore, err = pgdb.Indexes(ctx, tx, db, collection) + if err == nil { + numIndexesBefore = int32(len(indexesBefore)) + } + if errors.Is(err, pgdb.ErrTableNotExist) { + numIndexesBefore = 1 + err = nil + } + + if err != nil { + return lazyerrors.Error(err) + } + for { - var val any - _, val, err = iter.Next() + var key, val any + key, val, err = iter.Next() switch { case err == nil: // do nothing case errors.Is(err, iterator.ErrIteratorDone): - // iterator is done, no more indexes to create + var indexesAfter []pgdb.Index + indexesAfter, err = pgdb.Indexes(ctx, tx, db, collection) + if err != nil { + return lazyerrors.Error(err) + } + + numIndexesAfter = int32(len(indexesAfter)) + return nil default: return lazyerrors.Error(err) @@ -92,8 +149,15 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. indexDoc, ok := val.(*types.Document) if !ok { - // TODO Add better validation and return proper error: https://github.com/FerretDB/FerretDB/issues/2311 - return lazyerrors.Errorf("expected index document, got %T", val) + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrTypeMismatch, + fmt.Sprintf( + "BSON field 'createIndexes.indexes.%d' is the wrong type '%s', expected type 'object'", + key, + commonparams.AliasFromType(val), + ), + document.Command(), + ) } var index *pgdb.Index @@ -102,9 +166,63 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. return err } - if err = pgdb.CreateIndexIfNotExists(ctx, tx, db, collection, index); err != nil { + if index.Name == "" { + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrCannotCreateIndex, + fmt.Sprintf( + "Error in specification %s :: caused by :: index name cannot be empty", + types.FormatAnyValue(indexDoc), + ), + document.Command(), + ) + } + + for doc, existing := range indexes { + if existing.Key.Equal(index.Key) && existing.Name == index.Name { + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrIndexAlreadyExists, + fmt.Sprintf("Identical index already exists: %s", existing.Name), + document.Command(), + ) + } + + if existing.Key.Equal(index.Key) { + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrIndexOptionsConflict, + fmt.Sprintf("Index already exists with a different name: %s", existing.Name), + document.Command(), + ) + } + + if existing.Name == index.Name { + return commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrIndexKeySpecsConflict, + fmt.Sprintf("An existing index has the same name as the requested index. "+ + "When index names are not specified, they are auto generated and can "+ + "cause conflicts. Please refer to our documentation. "+ + "Requested index: %s, "+ + "existing index: %s", + types.FormatAnyValue(indexDoc), + types.FormatAnyValue(doc), + ), + document.Command(), + ) + } + } + + indexes[indexDoc] = index + + err = pgdb.CreateIndexIfNotExists(ctx, tx, db, collection, index) + if errors.Is(err, pgdb.ErrIndexKeyAlreadyExist) && index.Name == "_id_1" { + // ascending _id index is created by default + return nil + } + + if err != nil { return err } + + isUniqueSpecified = index.Unique != nil } }) @@ -119,7 +237,7 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. ) case errors.Is(err, pgdb.ErrIndexNameAlreadyExist): return nil, commonerrors.NewCommandErrorMsgWithArgument( - commonerrors.ErrIndexKeySpecsConflict, + commonerrors.ErrBadValue, "One of the specified indexes already exists with a different key", document.Command(), ) @@ -134,11 +252,19 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. return nil, lazyerrors.Error(err) } + res := new(types.Document) + + if isUniqueSpecified { + res.Set("numIndexesBefore", numIndexesBefore) + res.Set("numIndexesAfter", numIndexesAfter) + res.Set("createdCollectionAutomatically", true) + } + + res.Set("ok", float64(1)) + var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ - Documents: []*types.Document{must.NotFail(types.NewDocument( - "ok", float64(1), - ))}, + Documents: []*types.Document{res}, })) return &reply, nil @@ -151,6 +277,7 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { iter := indexDoc.Iterator() defer iter.Close() + var hasValue bool for { opt, _, err := iter.Next() @@ -158,11 +285,24 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { case err == nil: // do nothing case errors.Is(err, iterator.ErrIteratorDone): + if !hasValue { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrFailedToParse, + fmt.Sprintf( + "Error in specification {} :: caused by :: "+ + "The 'key' field is a required property of an index specification", + ), + "createIndexes", + ) + } + return &index, nil default: return nil, lazyerrors.Error(err) } + hasValue = true + // Process required param "key" var keyDoc *types.Document @@ -204,9 +344,23 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { return nil, err } - // Process required param "name" - index.Name, err = common.GetRequiredParam[string](indexDoc, "name") - if err != nil { + v, _ := indexDoc.Get("name") + if v == nil { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrFailedToParse, + fmt.Sprintf( + "Error in specification { key: %s } :: caused by :: "+ + "The 'name' field is a required property of an index specification", + types.FormatAnyValue(keyDoc), + ), + "createIndexes", + ) + } + + var ok bool + index.Name, ok = v.(string) + + if !ok { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrTypeMismatch, "'name' option must be specified as a string", @@ -219,9 +373,9 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { // already processed, do nothing case "unique": - uniqueVal := must.NotFail(indexDoc.Get("unique")) + v := must.NotFail(indexDoc.Get("unique")) - _, ok := uniqueVal.(bool) + _, ok := v.(bool) if !ok { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrTypeMismatch, @@ -230,7 +384,7 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { ":: caused by :: "+ "The field 'unique' has value unique: %[3]s, which is not convertible to bool", types.FormatAnyValue(must.NotFail(indexDoc.Get("key"))), - index.Name, types.FormatAnyValue(uniqueVal), + index.Name, types.FormatAnyValue(v), ), "createIndexes", ) @@ -309,8 +463,8 @@ func processIndexKey(keyDoc *types.Document) (pgdb.IndexKey, error) { if orderParam, err = commonparams.GetWholeNumberParam(order); err != nil { return nil, commonerrors.NewCommandErrorMsgWithArgument( - commonerrors.ErrNotImplemented, - fmt.Sprintf("Index key value %q is not implemented yet", order), + commonerrors.ErrIndexNotFound, + fmt.Sprintf("can't find index with key: { %s: \"%s\" }", field, order), "createIndexes", ) } diff --git a/internal/handlers/pg/msg_dropindexes.go b/internal/handlers/pg/msg_dropindexes.go index b5b50c6827cc..bd344ddfd31b 100644 --- a/internal/handlers/pg/msg_dropindexes.go +++ b/internal/handlers/pg/msg_dropindexes.go @@ -58,6 +58,14 @@ func (h *Handler) MsgDropIndexes(ctx context.Context, msg *wire.OpMsg) (*wire.Op return nil, err } + if collection == "" { + return nil, commonerrors.NewCommandErrorMsgWithArgument( + commonerrors.ErrInvalidNamespace, + fmt.Sprintf("Invalid namespace specified '%s.'", db), + command, + ) + } + var nIndexesWas int32 var responseMsg string From 5b1e6b61547e7e644e3c023694baa2adff21afde Mon Sep 17 00:00:00 2001 From: Alexander Tobi Fashakin Date: Tue, 27 Jun 2023 10:00:18 +0100 Subject: [PATCH 29/46] Update incorrect blog post image (#2920) --- website/static/img/blog/percona-ferretdb.png | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/static/img/blog/percona-ferretdb.png b/website/static/img/blog/percona-ferretdb.png index 5d40ddb3435d..7b2be3935f1d 100644 --- a/website/static/img/blog/percona-ferretdb.png +++ b/website/static/img/blog/percona-ferretdb.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:885df81fbedb43f1aa06dc6aa88056cb108daad7ae01badc1d07dfffba469716 -size 124302 +oid sha256:be17587a9d1a6f9fab306f0f7f9916bd352dfc49857ddb618524014366fea39a +size 126198 From 483f927aded5fd6a5ae06da6c36404536238163d Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 28 Jun 2023 12:50:38 +0400 Subject: [PATCH 30/46] Bump deps, add permissions monitoring (#2930) --- .github/workflows/clean.yml | 5 +- .github/workflows/conform-pr.yml | 3 + .github/workflows/debug.yml | 5 +- .github/workflows/docs.yml | 6 ++ .github/workflows/format.yml | 5 +- .github/workflows/go-trust.yml | 13 +-- .github/workflows/go.yml | 55 +++++++++-- .github/workflows/js.yml | 5 +- .github/workflows/packages.yml | 3 + .github/workflows/permissions.yml | 30 ++++++ .github/workflows/security.yml | 5 +- .github/workflows/website-preview.yml | 6 ++ build/deps/ferretdb-wrangler.Dockerfile | 2 +- go.mod | 10 +- go.sum | 20 ++-- integration/go.mod | 15 ++- integration/go.sum | 44 +++++---- tools/go.mod | 33 ++++--- tools/go.sum | 118 ++++++++++++------------ 19 files changed, 247 insertions(+), 136 deletions(-) create mode 100644 .github/workflows/permissions.yml diff --git a/.github/workflows/clean.yml b/.github/workflows/clean.yml index 0025a4fd09f5..11c799962dcf 100644 --- a/.github/workflows/clean.yml +++ b/.github/workflows/clean.yml @@ -6,12 +6,15 @@ on: jobs: scan-old-package: - runs-on: ubuntu-latest + runs-on: ubuntu-22.04 # disable for now if: false steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - uses: actions/checkout@v3 - name: Set up Go uses: actions/setup-go@v4 diff --git a/.github/workflows/conform-pr.yml b/.github/workflows/conform-pr.yml index 36cc768db23d..df82a16fb20b 100644 --- a/.github/workflows/conform-pr.yml +++ b/.github/workflows/conform-pr.yml @@ -74,6 +74,9 @@ jobs: # Do not add a source code checkout step because we don't check the `trust` label. # See the warning at the top of the file. + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Setup Go uses: FerretDB/github-actions/setup-go@main with: diff --git a/.github/workflows/debug.yml b/.github/workflows/debug.yml index f822e5a0fe07..3a43e4e95f62 100644 --- a/.github/workflows/debug.yml +++ b/.github/workflows/debug.yml @@ -22,7 +22,7 @@ on: # Do not run this workflow in parallel for any PR change. concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: false + cancel-in-progress: true jobs: contexts: @@ -34,6 +34,9 @@ jobs: if: false steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 498c1766c57b..863e8156eb79 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -36,6 +36,9 @@ jobs: if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -113,6 +116,9 @@ jobs: url: ${{ steps.deploy.outputs.page_url }} steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Deploy to GitHub Pages id: deploy uses: actions/deploy-pages@v2 diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index ce38f146ae70..a7c1bc0f604e 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -17,7 +17,7 @@ on: # Do not run this workflow in parallel for any PR change. concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: false + cancel-in-progress: true env: GOPATH: /home/runner/go @@ -35,6 +35,9 @@ jobs: if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/go-trust.yml b/.github/workflows/go-trust.yml index 1b21ca25de5a..bef199ea6ed6 100644 --- a/.github/workflows/go-trust.yml +++ b/.github/workflows/go-trust.yml @@ -17,12 +17,6 @@ on: schedule: - cron: "12 2 * * *" # after integration workflow to reuse cache -# Do not run this workflow in parallel for any PR change or branch/tag push -# to save some resources. -concurrency: - group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: true - env: GOPATH: /home/runner/go GOCACHE: /home/runner/go/cache @@ -37,10 +31,10 @@ jobs: group: paid timeout-minutes: 20 - # Do not run this job in parallel for any PR change or branch/tag push + # Do not run this job in parallel for any PR change or branch push # to save some resources. concurrency: - group: ${{ github.workflow }}-${{ matrix.name }}-${{ matrix.shard_index }}-${{ github.head_ref || github.ref_name }} + group: ${{ github.workflow }}-integration-${{ matrix.name }}-${{ matrix.shard_index }}-${{ github.head_ref || github.ref_name }} cancel-in-progress: true if: > @@ -59,6 +53,9 @@ jobs: - { name: "Hana", task: "hana", shard_index: 3, shard_total: 3 } steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index cd7775f02b3e..e55c4272ea90 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -14,11 +14,6 @@ on: schedule: - cron: "12 0 * * *" -# Do not run this workflow in parallel for any PR change. -concurrency: - group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: false - env: GOPATH: /home/runner/go GOCACHE: /home/runner/go/cache @@ -32,9 +27,17 @@ jobs: runs-on: ubuntu-22.04 timeout-minutes: 10 + # Do not run this job in parallel for any PR change or branch push. + concurrency: + group: ${{ github.workflow }}-short-test-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true + if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -73,9 +76,17 @@ jobs: runs-on: ubuntu-22.04 timeout-minutes: 10 + # Do not run this job in parallel for any PR change or branch push. + concurrency: + group: ${{ github.workflow }}-test-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true + if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -150,10 +161,10 @@ jobs: group: paid timeout-minutes: 20 - # Do not run this job in parallel for any PR change or branch/tag push + # Do not run this job in parallel for any PR change or branch push # to save some resources. concurrency: - group: ${{ github.workflow }}-${{ matrix.name }}-${{ matrix.shard_index }}-${{ github.head_ref || github.ref_name }} + group: ${{ github.workflow }}-integration-${{ matrix.name }}-${{ matrix.shard_index }}-${{ github.head_ref || github.ref_name }} cancel-in-progress: true if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') @@ -171,6 +182,9 @@ jobs: - { name: "SQLite", task: "sqlite", shard_index: 3, shard_total: 3 } steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -242,6 +256,9 @@ jobs: if: always() && (github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready')) steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Submit coveralls uses: coverallsapp/github-action@v2 with: @@ -252,9 +269,17 @@ jobs: runs-on: ubuntu-22.04 timeout-minutes: 10 + # Do not run this job in parallel for any PR change or branch push. + concurrency: + group: ${{ github.workflow }}-env-data-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true + if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -289,9 +314,17 @@ jobs: runs-on: ubuntu-22.04 timeout-minutes: 10 + # Do not run this job in parallel for any PR change or branch push. + concurrency: + group: ${{ github.workflow }}-golangci-lint-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true + if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -311,9 +344,17 @@ jobs: runs-on: ubuntu-22.04 timeout-minutes: 10 + # Do not run this job in parallel for any PR change or branch push. + concurrency: + group: ${{ github.workflow }}-fuzz-short-${{ github.head_ref || github.ref_name }} + cancel-in-progress: true + if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index e45dc721c9cc..25e4d85725e6 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -19,7 +19,7 @@ on: # Do not run this workflow in parallel for any PR change. concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: false + cancel-in-progress: true env: GOPATH: /home/runner/go @@ -37,6 +37,9 @@ jobs: if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/packages.yml b/.github/workflows/packages.yml index a91716cd8f73..723d67ba1126 100644 --- a/.github/workflows/packages.yml +++ b/.github/workflows/packages.yml @@ -60,6 +60,9 @@ jobs: packages: write steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code if: github.event_name != 'pull_request_target' uses: actions/checkout@v3 diff --git a/.github/workflows/permissions.yml b/.github/workflows/permissions.yml new file mode 100644 index 000000000000..8f82b139e8c8 --- /dev/null +++ b/.github/workflows/permissions.yml @@ -0,0 +1,30 @@ +--- +name: Permissions +on: + workflow_dispatch: + inputs: + name: + description: "The name of the workflow file to analyze" + required: true + type: string + count: + description: "How many last runs to analyze" + required: false + type: number + default: 10 + +permissions: + actions: read + +jobs: + advisor: + name: Advisor + runs-on: ubuntu-22.04 + timeout-minutes: 10 + + steps: + - name: Analyze permissions + uses: GitHubSecurityLab/actions-permissions/advisor@v1 + with: + name: ${{ inputs.name }} + count: ${{ inputs.count }} diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index 56b4dc3698c7..c109bb487623 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -17,7 +17,7 @@ on: # Do not run this workflow in parallel for any PR change. concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} - cancel-in-progress: false + cancel-in-progress: true env: GOPATH: /home/runner/go @@ -35,6 +35,9 @@ jobs: if: github.event_name != 'pull_request' || !contains(github.event.pull_request.labels.*.name, 'not ready') steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/.github/workflows/website-preview.yml b/.github/workflows/website-preview.yml index f2e5b239977f..656d3a4d451b 100644 --- a/.github/workflows/website-preview.yml +++ b/.github/workflows/website-preview.yml @@ -39,6 +39,9 @@ jobs: url: ${{ steps.extract.outputs.extracted_url }} steps: + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: @@ -104,6 +107,9 @@ jobs: # Just build them again instead. # That also allows us to pass branch name, commit hash, etc from git to wrangler. + - name: Setup permissions monitoring + uses: GitHubSecurityLab/actions-permissions/monitor@v1 + - name: Checkout code uses: actions/checkout@v3 with: diff --git a/build/deps/ferretdb-wrangler.Dockerfile b/build/deps/ferretdb-wrangler.Dockerfile index 58f21620ca45..acb7f6f3daa7 100644 --- a/build/deps/ferretdb-wrangler.Dockerfile +++ b/build/deps/ferretdb-wrangler.Dockerfile @@ -1 +1 @@ -FROM ghcr.io/ferretdb/ferretdb-wrangler:3.1.0-1 +FROM ghcr.io/ferretdb/ferretdb-wrangler:3.1.1-1 diff --git a/go.mod b/go.mod index a7ab24d69d2a..db6d3054c691 100644 --- a/go.mod +++ b/go.mod @@ -4,8 +4,8 @@ go 1.20 require ( github.com/AlekSi/pointer v1.2.0 - github.com/SAP/go-hdb v1.3.8 - github.com/alecthomas/kong v0.7.1 + github.com/SAP/go-hdb v1.3.9 + github.com/alecthomas/kong v0.8.0 github.com/google/uuid v1.3.0 github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f @@ -20,8 +20,8 @@ require ( go.opentelemetry.io/otel/trace v1.16.0 go.uber.org/zap v1.24.0 golang.org/x/crypto v0.10.0 // indirect; always use @latest - golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 - golang.org/x/net v0.11.0 + golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df + golang.org/x/net v0.10.0 golang.org/x/sys v0.9.0 modernc.org/sqlite v1.23.1 ) @@ -57,7 +57,7 @@ require ( go.opentelemetry.io/otel/metric v1.16.0 // indirect go.uber.org/atomic v1.10.0 // indirect go.uber.org/multierr v1.6.0 // indirect - golang.org/x/mod v0.8.0 // indirect + golang.org/x/mod v0.11.0 // indirect golang.org/x/oauth2 v0.8.0 // indirect golang.org/x/sync v0.2.0 // indirect golang.org/x/text v0.10.0 // indirect diff --git a/go.sum b/go.sum index 3a6791a68f0c..e0d7094cdc98 100644 --- a/go.sum +++ b/go.sum @@ -9,11 +9,11 @@ github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tS github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= -github.com/SAP/go-hdb v1.3.8 h1:o1HyjliL9SrrGLN+P+yI4K0vJFhn/SWiXuaQhak/TgY= -github.com/SAP/go-hdb v1.3.8/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= +github.com/SAP/go-hdb v1.3.9 h1:vaOwlwJJ6j5MBuizV3p/rtr5YDXiBIfvs73ohCGtdto= +github.com/SAP/go-hdb v1.3.9/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= github.com/alecthomas/assert/v2 v2.1.0 h1:tbredtNcQnoSd3QBhQWI7QZ3XHOVkw1Moklp2ojoH/0= -github.com/alecthomas/kong v0.7.1 h1:azoTh0IOfwlAX3qN9sHWTxACE2oV8Bg2gAwBsMwDQY4= -github.com/alecthomas/kong v0.7.1/go.mod h1:n1iCIO2xS46oE8ZfYCNDqdR0b0wZNrXAIAqro/2132U= +github.com/alecthomas/kong v0.8.0 h1:ryDCzutfIqJPnNn0omnrgHLbAggDQM2VWHikE1xqK7s= +github.com/alecthomas/kong v0.8.0/go.mod h1:n1iCIO2xS46oE8ZfYCNDqdR0b0wZNrXAIAqro/2132U= github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= @@ -201,8 +201,8 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM= golang.org/x/crypto v0.10.0/go.mod h1:o4eNf7Ede1fv+hwOwZsTHl9EsPFO6q6ZvYR8vYfY45I= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 h1:k/i9J1pBpvlfR+9QsetwPyERsqu1GIbi967PQMq3Ivc= -golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df h1:UA2aFVmmsIlefxMk29Dp2juaUSth8Pyn3Tq5Y5mJGME= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -210,8 +210,8 @@ golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHl golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= -golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU= +golang.org/x/mod v0.11.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -225,8 +225,8 @@ golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81R golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU= -golang.org/x/net v0.11.0/go.mod h1:2L/ixqYpgIVXmeoSA/4Lu7BzTG4KIyPIryS4IsOd1oQ= +golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= diff --git a/integration/go.mod b/integration/go.mod index 472c709e8aaf..6ce9518b60a7 100644 --- a/integration/go.mod +++ b/integration/go.mod @@ -10,19 +10,19 @@ require ( github.com/jaswdr/faker v1.18.0 github.com/prometheus/client_golang v1.16.0 github.com/stretchr/testify v1.8.4 - go.mongodb.org/mongo-driver v1.11.7 + go.mongodb.org/mongo-driver v1.12.0 go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo v0.42.0 go.opentelemetry.io/otel v1.16.0 go.opentelemetry.io/otel/exporters/jaeger v1.16.0 go.opentelemetry.io/otel/sdk v1.16.0 go.uber.org/zap v1.24.0 - golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 + golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df ) require ( cloud.google.com/go/compute v1.19.1 // indirect cloud.google.com/go/compute/metadata v0.2.3 // indirect - github.com/SAP/go-hdb v1.3.8 // indirect + github.com/SAP/go-hdb v1.3.9 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/benbjohnson/clock v1.1.0 // indirect github.com/beorn7/perks v1.0.1 // indirect @@ -53,7 +53,6 @@ require ( github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe // indirect - github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/prometheus/client_model v0.4.0 // indirect github.com/prometheus/common v0.44.0 // indirect @@ -61,16 +60,16 @@ require ( github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/tigrisdata/tigris-client-go v1.0.0 // indirect github.com/xdg-go/pbkdf2 v1.0.0 // indirect - github.com/xdg-go/scram v1.1.1 // indirect - github.com/xdg-go/stringprep v1.0.3 // indirect + github.com/xdg-go/scram v1.1.2 // indirect + github.com/xdg-go/stringprep v1.0.4 // indirect github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect go.opentelemetry.io/otel/metric v1.16.0 // indirect go.opentelemetry.io/otel/trace v1.16.0 // indirect go.uber.org/atomic v1.10.0 // indirect go.uber.org/multierr v1.6.0 // indirect golang.org/x/crypto v0.10.0 // indirect - golang.org/x/mod v0.8.0 // indirect - golang.org/x/net v0.11.0 // indirect + golang.org/x/mod v0.11.0 // indirect + golang.org/x/net v0.10.0 // indirect golang.org/x/oauth2 v0.8.0 // indirect golang.org/x/sync v0.2.0 // indirect golang.org/x/sys v0.9.0 // indirect diff --git a/integration/go.sum b/integration/go.sum index 967a43ea7b2b..fb929a45a086 100644 --- a/integration/go.sum +++ b/integration/go.sum @@ -9,8 +9,8 @@ github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tS github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= -github.com/SAP/go-hdb v1.3.8 h1:o1HyjliL9SrrGLN+P+yI4K0vJFhn/SWiXuaQhak/TgY= -github.com/SAP/go-hdb v1.3.8/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= +github.com/SAP/go-hdb v1.3.9 h1:vaOwlwJJ6j5MBuizV3p/rtr5YDXiBIfvs73ohCGtdto= +github.com/SAP/go-hdb v1.3.9/go.mod h1:rGEDo4STw4LOWnmd+pJpqXr+7QEoy32bREK6hMn/Emk= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= @@ -145,7 +145,6 @@ github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJ github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_golang v1.16.0 h1:yk/hx9hDbrGHovbci4BY+pRMfSuuat626eFsHb7tmT8= @@ -173,20 +172,17 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= -github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4= -github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tigrisdata/tigris-client-go v1.0.0 h1:07Qw8Tm0qL15WiadP0hp4iBiRzfNSJ+GH4/ozO0nNs0= github.com/tigrisdata/tigris-client-go v1.0.0/go.mod h1:2n6TQUdoTbzuTtakHT/ZNuK5X+I/i57BqqCcYAzG7y4= github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c= github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI= -github.com/xdg-go/scram v1.1.1 h1:VOMT+81stJgXW3CpHyqHN3AXDYIMsx56mEFrB37Mb/E= -github.com/xdg-go/scram v1.1.1/go.mod h1:RaEWvsqvNKKvBPvcKeFjrG2cJqOkHTiyTpzz23ni57g= -github.com/xdg-go/stringprep v1.0.3 h1:kdwGpVNwPFtjs98xCGkHjQtGKh86rDcRZN17QEMCOIs= -github.com/xdg-go/stringprep v1.0.3/go.mod h1:W3f5j4i+9rC0kuIEJL0ky1VpHXQU3ocBgklLGvcBnW8= +github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY= +github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4= +github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8= +github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ= github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQluxsYJ78Id3Y= @@ -195,8 +191,9 @@ github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7Jul github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= -go.mongodb.org/mongo-driver v1.11.7 h1:LIwYxASDLGUg/8wOhgOOZhX8tQa/9tgZPgzZoVqJvcs= -go.mongodb.org/mongo-driver v1.11.7/go.mod h1:G9TgswdsWjX4tmDA5zfs2+6AEPpYJwqblyjsfuh8oXY= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= +go.mongodb.org/mongo-driver v1.12.0 h1:aPx33jmn/rQuJXPQLZQ8NtfPQG8CaqgLThFtqRb0PiE= +go.mongodb.org/mongo-driver v1.12.0/go.mod h1:AZkxhPnFJUoH7kZlFkVKucV20K387miPfm7oimrSmK0= go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo v0.42.0 h1:PL1iPuCLd14uZf2CZmN3mEGF9KurGs9IBt6UvO4owJk= go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo v0.42.0/go.mod h1:r8zTHTSZ9+o69VyAtF9ZaFJPDJdOSG950GEV6uiA99U= go.opentelemetry.io/otel v1.16.0 h1:Z7GVAX/UkAXPKsy94IU+i6thsQS4nb7LviLpnaNeW8s= @@ -223,12 +220,13 @@ go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM= golang.org/x/crypto v0.10.0/go.mod h1:o4eNf7Ede1fv+hwOwZsTHl9EsPFO6q6ZvYR8vYfY45I= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 h1:k/i9J1pBpvlfR+9QsetwPyERsqu1GIbi967PQMq3Ivc= -golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df h1:UA2aFVmmsIlefxMk29Dp2juaUSth8Pyn3Tq5Y5mJGME= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= @@ -236,8 +234,9 @@ golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHl golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= -golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU= +golang.org/x/mod v0.11.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -249,11 +248,13 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU= -golang.org/x/net v0.11.0/go.mod h1:2L/ixqYpgIVXmeoSA/4Lu7BzTG4KIyPIryS4IsOd1oQ= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8= @@ -265,6 +266,7 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI= golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -279,16 +281,21 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s= golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58= golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -301,6 +308,7 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/tools/go.mod b/tools/go.mod index 90f7a707031c..871cb32bfc28 100644 --- a/tools/go.mod +++ b/tools/go.mod @@ -6,7 +6,7 @@ require ( github.com/BurntSushi/go-sumtype v0.0.0-20221020234012-480526a59796 github.com/go-task/task/v3 v3.26.0 github.com/google/go-github/v41 v41.0.0 - github.com/goreleaser/nfpm/v2 v2.30.1 + github.com/goreleaser/nfpm/v2 v2.31.0 github.com/quasilyte/go-consistent v0.0.0-20220429160651-4e46040fbc82 github.com/reviewdog/reviewdog v0.14.2 github.com/stretchr/testify v1.8.4 @@ -24,23 +24,23 @@ require ( cloud.google.com/go/datastore v1.11.0 // indirect github.com/AlekSi/pointer v1.2.0 // indirect github.com/Masterminds/goutils v1.1.1 // indirect - github.com/Masterminds/semver v1.5.0 // indirect github.com/Masterminds/semver/v3 v3.2.1 // indirect - github.com/Masterminds/sprig v2.22.0+incompatible // indirect - github.com/Microsoft/go-winio v0.5.1 // indirect - github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 // indirect + github.com/Masterminds/sprig/v3 v3.2.3 // indirect + github.com/Microsoft/go-winio v0.5.2 // indirect + github.com/ProtonMail/go-crypto v0.0.0-20230626094100-7e9e0395ebec // indirect github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794 // indirect + github.com/acomagu/bufpipe v1.0.4 // indirect github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb // indirect github.com/bradleyfalzon/ghinstallation/v2 v2.5.0 // indirect github.com/cavaliergopher/cpio v1.0.1 // indirect github.com/cloudflare/circl v1.3.3 // indirect github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect github.com/davecgh/go-spew v1.1.1 // indirect - github.com/emirpasic/gods v1.12.0 // indirect + github.com/emirpasic/gods v1.18.1 // indirect github.com/fatih/color v1.15.0 // indirect - github.com/go-git/gcfg v1.5.0 // indirect - github.com/go-git/go-billy/v5 v5.3.1 // indirect - github.com/go-git/go-git/v5 v5.2.0 // indirect + github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect + github.com/go-git/go-billy/v5 v5.4.1 // indirect + github.com/go-git/go-git/v5 v5.7.0 // indirect github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect github.com/go-toolsmith/astcast v1.0.0 // indirect github.com/go-toolsmith/astequal v1.0.0 // indirect @@ -57,34 +57,34 @@ require ( github.com/google/uuid v1.3.0 // indirect github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect github.com/googleapis/gax-go/v2 v2.7.1 // indirect - github.com/goreleaser/chglog v0.4.2 // indirect + github.com/goreleaser/chglog v0.5.0 // indirect github.com/goreleaser/fileglob v1.3.0 // indirect github.com/hashicorp/go-cleanhttp v0.5.2 // indirect github.com/hashicorp/go-retryablehttp v0.7.2 // indirect github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01 // indirect - github.com/huandu/xstrings v1.3.2 // indirect + github.com/huandu/xstrings v1.3.3 // indirect github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 // indirect github.com/imdario/mergo v0.3.16 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/invopop/jsonschema v0.7.0 // indirect github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect github.com/joho/godotenv v1.5.1 // indirect - github.com/kevinburke/ssh_config v1.1.0 // indirect + github.com/kevinburke/ssh_config v1.2.0 // indirect github.com/kisielk/gotool v1.0.0 // indirect - github.com/klauspost/compress v1.16.5 // indirect + github.com/klauspost/compress v1.16.6 // indirect github.com/klauspost/pgzip v1.2.6 // indirect github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.17 // indirect github.com/mattn/go-shellwords v1.0.12 // indirect github.com/mattn/go-zglob v0.0.4 // indirect github.com/mitchellh/copystructure v1.2.0 // indirect - github.com/mitchellh/go-homedir v1.1.0 // indirect github.com/mitchellh/hashstructure/v2 v2.0.2 // indirect github.com/mitchellh/reflectwalk v1.0.2 // indirect github.com/muesli/mango v0.1.0 // indirect github.com/muesli/mango-cobra v1.2.0 // indirect github.com/muesli/mango-pflag v0.1.0 // indirect github.com/muesli/roff v0.1.0 // indirect + github.com/pjbgf/sha1cd v0.3.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/radovskyb/watcher v1.0.7 // indirect github.com/reva2/bitbucket-insights-api v1.0.0 // indirect @@ -93,12 +93,15 @@ require ( github.com/russross/blackfriday/v2 v2.1.0 // indirect github.com/sajari/fuzzy v1.0.0 // indirect github.com/sergi/go-diff v1.2.0 // indirect + github.com/shopspring/decimal v1.2.0 // indirect + github.com/skeema/knownhosts v1.1.1 // indirect + github.com/spf13/cast v1.5.1 // indirect github.com/spf13/cobra v1.7.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/ulikunitz/xz v0.5.11 // indirect github.com/vvakame/sdlog v1.1.2 // indirect github.com/xanzy/go-gitlab v0.85.0 // indirect - github.com/xanzy/ssh-agent v0.3.1 // indirect + github.com/xanzy/ssh-agent v0.3.3 // indirect github.com/yuin/goldmark v1.4.13 // indirect gitlab.com/digitalxero/go-conventional-commit v1.0.7 // indirect go.opencensus.io v0.24.0 // indirect diff --git a/tools/go.sum b/tools/go.sum index 9d59c87f344e..ad4e52d4e6ba 100644 --- a/tools/go.sum +++ b/tools/go.sum @@ -54,30 +54,27 @@ github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ= github.com/GoogleCloudPlatform/cloudsql-proxy v0.0.0-20190129172621-c8b1d7a94ddf/go.mod h1:aJ4qN3TfrelA6NZ6AXsXRfmEVaYin3EDbSPJrKS8OXo= github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI= github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU= -github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww= -github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y= +github.com/Masterminds/semver/v3 v3.2.0/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ= github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0= github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ= -github.com/Masterminds/sprig v2.22.0+incompatible h1:z4yfnGrZ7netVz+0EDJ0Wi+5VZCSYp4Z0m2dk6cEM60= -github.com/Masterminds/sprig v2.22.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuNhlNS5hqE0NB0E6fgfo2Br3o= -github.com/Microsoft/go-winio v0.5.0/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84= -github.com/Microsoft/go-winio v0.5.1 h1:aPJp2QD7OOrhO5tQXqQoGSJc+DjDtWTGLOmNyAm6FgY= -github.com/Microsoft/go-winio v0.5.1/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84= -github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 h1:wPbRQzjjwFc0ih8puEVAOFGELsn1zoIIYdxvML7mDxA= +github.com/Masterminds/sprig/v3 v3.2.3 h1:eL2fZNezLomi0uOLqjQoN6BfsDD+fyLtgbJMAj9n6YA= +github.com/Masterminds/sprig/v3 v3.2.3/go.mod h1:rXcFaZ2zZbLRJv/xSysmlgIM1u11eBaRMhvYXJNkGuM= +github.com/Microsoft/go-winio v0.5.2 h1:a9IhgEQBCUEk6QCdml9CiJGhAws+YwffDHEMp1VMrpA= +github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY= github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8/go.mod h1:I0gYDMZ6Z5GRU7l58bNFSkPTFN6Yl12dsUlAZ8xy98g= -github.com/ProtonMail/go-mime v0.0.0-20190923161245-9b5a4261663a h1:W6RrgN/sTxg1msqzFFb+G80MFmpjMw61IU+slm+wln4= -github.com/ProtonMail/gopenpgp/v2 v2.2.2 h1:u2m7xt+CZWj88qK1UUNBoXeJCFJwJCZ/Ff4ymGoxEXs= +github.com/ProtonMail/go-crypto v0.0.0-20230626094100-7e9e0395ebec h1:vV3RryLxt42+ZIVOFbYJCH1jsZNTNmj2NYru5zfx+4E= +github.com/ProtonMail/go-crypto v0.0.0-20230626094100-7e9e0395ebec/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0= +github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f h1:tCbYj7/299ekTTXpdwKYF8eBlsYsDVoggDAuAjoK66k= +github.com/ProtonMail/gopenpgp/v2 v2.7.1 h1:Awsg7MPc2gD3I7IFac2qE3Gdls0lZW8SzrFZ3k1oz0s= github.com/aclements/go-gg v0.0.0-20170118225347-6dbb4e4fefb0/go.mod h1:55qNq4vcpkIuHowELi5C8e+1yUHtoLoOUR9QU5j7Tes= github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794 h1:xlwdaKcTNVW4PtpQb8aKA4Pjy0CdJHEqvFbAnvR5m2g= github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794/go.mod h1:7e+I0LQFUI9AXWxOfsQROs9xPhoJtbsyWcjJqDd4KPY= +github.com/acomagu/bufpipe v1.0.4 h1:e3H4WUzM3npvo5uv95QuJM3cQspFNtFBzvJ2oNjKIDQ= +github.com/acomagu/bufpipe v1.0.4/go.mod h1:mxdxdup/WdsKVreO5GpW4+M/1CE2sMG4jeGJ2sYmHc4= github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw= github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw= -github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs= -github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs= -github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c= github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= -github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb h1:m935MPodAbYS46DG4pJSv7WO+VECIWUQ7OJYSoTrMh4= github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb/go.mod h1:PkYb9DJNAwrSvRx5DYA+gUcOIgTGVMNkfSCbZM8cWpI= github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= @@ -108,34 +105,31 @@ github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg= -github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o= +github.com/elazarl/goproxy v0.0.0-20221015165544-a0805db90819 h1:RIB4cRk+lBqKK3Oy0r2gRX4ui7tuhiZq2SuTtTCi0/0= +github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc= +github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ= github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs= github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw= -github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc= github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k= github.com/fogleman/gg v1.3.0/go.mod h1:R/bRT+9gY/C5z7JzPU0zXsXHKM4/ayA+zqcVNZzPa1k= github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY= -github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= -github.com/gliderlabs/ssh v0.3.3 h1:mBQ8NiOgDkINJrZtoizkC3nDNYgSaWtxyem6S2XHBtA= +github.com/gliderlabs/ssh v0.3.5 h1:OcaySEmAQJgyYcArR+gGGTHCyE7nvhEMTlYY+Dp8CpY= github.com/go-fonts/dejavu v0.1.0/go.mod h1:4Wt4I4OU2Nq9asgDCteaAaWZOV24E+0/Pwo0gppep4g= github.com/go-fonts/latin-modern v0.2.0/go.mod h1:rQVLdDMK+mK1xscDwsqM5J8U2jrRa3T0ecnM9pNujks= github.com/go-fonts/liberation v0.1.1/go.mod h1:K6qoJYypsmfVjWg8KOVDQhLc8UDgIK2HYqyqAO9z7GY= github.com/go-fonts/liberation v0.2.0/go.mod h1:K6qoJYypsmfVjWg8KOVDQhLc8UDgIK2HYqyqAO9z7GY= github.com/go-fonts/stix v0.1.0/go.mod h1:w/c1f0ldAUlJmLBvlbkvVXLAD+tAMqobIIQpmnUIzUY= -github.com/go-git/gcfg v1.5.0 h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4= -github.com/go-git/gcfg v1.5.0/go.mod h1:5m20vg6GwYabIxaOonVkTdrILxQMpEShl1xiMF4ua+E= -github.com/go-git/go-billy/v5 v5.0.0/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0= -github.com/go-git/go-billy/v5 v5.3.1 h1:CPiOUAzKtMRvolEKw+bG1PLRpT7D3LIs3/3ey4Aiu34= -github.com/go-git/go-billy/v5 v5.3.1/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0= -github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12 h1:PbKy9zOy4aAKrJ5pibIRpVO2BXnK1Tlcg+caKI7Ox5M= -github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12/go.mod h1:m+ICp2rF3jDhFgEZ/8yziagdT1C+ZpZcrJjappBCDSw= -github.com/go-git/go-git/v5 v5.2.0 h1:YPBLG/3UK1we1ohRkncLjaXWLW+HKp5QNM/jTli2JgI= -github.com/go-git/go-git/v5 v5.2.0/go.mod h1:kh02eMX+wdqqxgNMEyq8YgwlIOsDOa9homkUq1PoTMs= +github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI= +github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic= +github.com/go-git/go-billy/v5 v5.4.1 h1:Uwp5tDRkPr+l/TnbHOQzp+tmJfLceOlbVucgpTz8ix4= +github.com/go-git/go-billy/v5 v5.4.1/go.mod h1:vjbugF6Fz7JIflbVpl1hJsGjSHNltrSw45YK/ukIvQg= +github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20230305113008-0c11038e723f h1:Pz0DHeFij3XFhoBRGUDPzSJ+w2UcK5/0JvF8DRI58r8= +github.com/go-git/go-git/v5 v5.7.0 h1:t9AudWVLmqzlo+4bqdf7GY+46SUuRsx59SboFxkq2aE= +github.com/go-git/go-git/v5 v5.7.0/go.mod h1:coJHKEOk5kUClpsNlXrUvPrDxY3w3gjHvhcZd8Fodw8= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= @@ -237,6 +231,7 @@ github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/safehtml v0.0.2/go.mod h1:L4KWwDsUJdECRAEpZoBn3O64bQaywRscowZjJAzjHnU= github.com/google/subcommands v1.0.1/go.mod h1:ZjhPrFU+Olkh9WazFPsl27BQ4UPiG37m3yTrtFlrHVk= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= @@ -247,13 +242,13 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.7.1 h1:gF4c0zjUP2H/s/hEGyLA3I0fA2ZWjzYiONAD6cvPr8A= github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38/qKbhSAKP6QI= -github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8= -github.com/goreleaser/chglog v0.4.2 h1:afmbT1d7lX/q+GF8wv3a1Dofs2j/Y9YkiCpGemWR6mI= -github.com/goreleaser/chglog v0.4.2/go.mod h1:u/F03un4hMCQrp65qSWCkkC6T+G7YLKZ+AM2mITE47s= +github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g= +github.com/goreleaser/chglog v0.5.0 h1:Sk6BMIpx8+vpAf8KyPit34OgWui8c7nKTMHhYx88jJ4= +github.com/goreleaser/chglog v0.5.0/go.mod h1:Ri46M3lrMuv76FHszs3vtABR8J8k1w9JHYAzxeeOl28= github.com/goreleaser/fileglob v1.3.0 h1:/X6J7U8lbDpQtBvGcwwPS6OpzkNVlVEsFUVRx9+k+7I= github.com/goreleaser/fileglob v1.3.0/go.mod h1:Jx6BoXv3mbYkEzwm9THo7xbr5egkAraxkGorbJb4RxU= -github.com/goreleaser/nfpm/v2 v2.30.1 h1:mn3nrLRvCRW/SO86z2IBTctU6BZSXKkyRR8Zkpw344Y= -github.com/goreleaser/nfpm/v2 v2.30.1/go.mod h1:2zdXNdSziz4veeXBVIcLE5Y8oiycm6BOSfflz2UhWGk= +github.com/goreleaser/nfpm/v2 v2.31.0 h1:cb8QSZ7tPnUlWPEdYcWwNWXiRvmVPznJ6LYiOIdOJ6Y= +github.com/goreleaser/nfpm/v2 v2.31.0/go.mod h1:qlMQCbOTapyqRss16vAPwK/WAjWKdt0gY3vh4wipm8I= github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= @@ -266,12 +261,12 @@ github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01 h1:H github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01/go.mod h1:1DWDZmeYf0LX30zscWb7K9rUMeirNeBMd5Dum+seUhc= github.com/haya14busa/go-checkstyle v0.0.0-20170303121022-5e9d09f51fa1/go.mod h1:RsN5RGgVYeXpcXNtWyztD5VIe7VNSEqpJvF2iEH7QvI= github.com/haya14busa/go-sarif v0.0.0-20210102043135-e2c5fed2fa3d/go.mod h1:1Hkn3JseGMB/hv1ywzkapVQDWV3bFgp6POZobZmR/5g= -github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw= -github.com/huandu/xstrings v1.3.2/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE= +github.com/huandu/xstrings v1.3.3 h1:/Gcsuc1x8JVbJ9/rlye4xZnVAbEkGauT8lbebqcQws4= +github.com/huandu/xstrings v1.3.3/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE= github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 h1:i462o439ZjprVSFSZLZxcsoAe592sZB1rci2Z8j4wdk= github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0/go.mod h1:N0Wam8K1arqPXNWjMo21EXnBPOPp36vB07FNRdD2geA= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= -github.com/imdario/mergo v0.3.9/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= +github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4= github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= @@ -280,7 +275,6 @@ github.com/invopop/jsonschema v0.7.0 h1:2vgQcBz1n256N+FpX3Jq7Y17AjYt46Ig3zIWyy77 github.com/invopop/jsonschema v0.7.0/go.mod h1:O9uiLokuu0+MGFlyiaqtWxwqJm41/+8Nj0lD7A36YH0= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= -github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= @@ -288,22 +282,23 @@ github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/X github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= github.com/jung-kurt/gofpdf v1.0.0/go.mod h1:7Id9E/uU8ce6rXgefFLlgrJj/GYY22cpxn+r32jIOes= github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5/go.mod h1:7Id9E/uU8ce6rXgefFLlgrJj/GYY22cpxn+r32jIOes= -github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM= -github.com/kevinburke/ssh_config v1.1.0 h1:pH/t1WS9NzT8go394IqZeJTMHVm6Cr6ZJ6AQ+mdNo/o= -github.com/kevinburke/ssh_config v1.1.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM= +github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4= +github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM= github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= -github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.16.6 h1:91SKEy4K37vkp255cJ8QesJhjyRO0hn9i9G0GoUwLsk= +github.com/klauspost/compress v1.16.6/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU= github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/matryer/is v1.2.0/go.mod h1:2fLPjFQM9rhQ15aVEtbuwhJinnOqrmgXPNdZsdwlWXA= github.com/matryer/is v1.4.0 h1:sosSmIWwkYITGrxZ25ULNDeKiMNzFSr4V/eqBQP0PeE= github.com/matryer/is v1.4.0/go.mod h1:8I/i5uYgLzgsgEloJE1U6xx5HkBQpAZvepWuujKwMRU= github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= @@ -316,12 +311,12 @@ github.com/mattn/go-shellwords v1.0.12/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lL github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI= github.com/mattn/go-zglob v0.0.4 h1:LQi2iOm0/fGgu80AioIJ/1j9w9Oh+9DZ39J4VAGzHQM= github.com/mattn/go-zglob v0.0.4/go.mod h1:MxxjyoXXnMxfIpxTK2GAkw1w8glPsQILx3N5wrKakiY= +github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw= github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= -github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= -github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4= github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE= +github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= github.com/muesli/mango v0.1.0 h1:DZQK45d2gGbql1arsYA4vfg4d7I9Hfx5rX/GCmzsAvI= @@ -332,11 +327,12 @@ github.com/muesli/mango-pflag v0.1.0 h1:UADqbYgpUyRoBja3g6LUL+3LErjpsOwaC9ywvBWe github.com/muesli/mango-pflag v0.1.0/go.mod h1:YEQomTxaCUp8PrbhFh10UfbhbQrM/xJ4i2PB8VTLLW0= github.com/muesli/roff v0.1.0 h1:YD0lalCotmYuF5HhZliKWlIx7IEhiXeSfq7hNjFqGF8= github.com/muesli/roff v0.1.0/go.mod h1:pjAHQM9hdUUwm/krAfrLGgJkXJ+YuhtsfZ42kieB2Ig= -github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs= github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2dXMnm1mY= github.com/phpdave11/gofpdi v1.0.12/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI= github.com/phpdave11/gofpdi v1.0.13/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI= +github.com/pjbgf/sha1cd v0.3.0 h1:4D5XXmUUBUl/xQ6IjCkEAbqXskkq/4O7LmGn0AqMDs4= +github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFzPPsI= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -363,13 +359,18 @@ github.com/ruudk/golang-pdf417 v0.0.0-20181029194003-1af4ab5afa58/go.mod h1:6lfF github.com/ruudk/golang-pdf417 v0.0.0-20201230142125-a7e3863a1245/go.mod h1:pQAZKsJ8yyVxGRWYNEm9oFB8ieLgKFnamEyDmSA0BRk= github.com/sajari/fuzzy v1.0.0 h1:+FmwVvJErsd0d0hAPlj4CxqxUtQY/fOoY0DwX4ykpRY= github.com/sajari/fuzzy v1.0.0/go.mod h1:OjYR6KxoWOe9+dOlXeiCJd4dIbED4Oo8wpS89o0pwOo= -github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ= github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= -github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM= +github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ= +github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= -github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= -github.com/smartystreets/goconvey v1.7.2 h1:9RBaZCeXEQ3UselpuwUQHltGVXvdwm6cv1hgR6gDIPg= +github.com/skeema/knownhosts v1.1.1 h1:MTk78x9FPgDFVFkDLTrsnnfCJl7g1C/nnKvePgrIngE= +github.com/skeema/knownhosts v1.1.1/go.mod h1:g4fPeYpque7P0xefxtGzV81ihjC8sX2IqpAoNkjxbMo= +github.com/smartystreets/assertions v1.13.1 h1:Ef7KhSmjZcK6AVf9YbJdvPYG9avaF0ZxudX+ThRdWfU= +github.com/smartystreets/goconvey v1.8.0 h1:Oi49ha/2MURE0WexF052Z0m+BNSGirfjg5RL+JXWq3w= +github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cast v1.5.1 h1:R+kOtfhWQE6TVQzY+4D7wJLBgkdVasCEFxSUBYBYIlA= +github.com/spf13/cast v1.5.1/go.mod h1:b9PdjNptOpzXr7Rq1q9gJML/2cdGQAo69NKzQ10KN48= github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= @@ -393,9 +394,8 @@ github.com/vvakame/sdlog v1.1.2 h1:hWV5ESsYMn/vubf1uFWdkMJomrQucAU+9gc/S3A/JE8= github.com/vvakame/sdlog v1.1.2/go.mod h1:jQWcS5V7V6xNIpNUEdLRwWlub0SYicmkE9c6N6FgJ44= github.com/xanzy/go-gitlab v0.85.0 h1:E/wjnsd/mM5kV6O9y5+i6zxjx+wfAwa97sgcT1ETNwk= github.com/xanzy/go-gitlab v0.85.0/go.mod h1:5ryv+MnpZStBH8I/77HuQBsMbBGANtVpLWC15qOjWAw= -github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4= -github.com/xanzy/ssh-agent v0.3.1 h1:AmzO1SSWxw73zxFZPRwaMN1MohDw8UyHnmuxyceTEGo= -github.com/xanzy/ssh-agent v0.3.1/go.mod h1:QIE4lCeL7nkC25x+yA3LBIYfwCc1TFziCtG7cBAac6w= +github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM= +github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw= github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -414,16 +414,15 @@ go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= golang.org/x/build v0.0.0-20230616225607-2f704e01b36f h1:+DJe+d0u+/q3cXpk/n2r4D7MmyECuvojksgwIkifUkw= golang.org/x/build v0.0.0-20230616225607-2f704e01b36f/go.mod h1:YAvpYk9SQ69z5Nt7IfnPJCr4/1XKsyE7gI/x9nKRqc4= -golang.org/x/crypto v0.0.0-20190219172222-a4c6cb3142f2/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4= golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM= @@ -508,6 +507,7 @@ golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81R golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= @@ -543,7 +543,6 @@ golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E= golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190221075227-b4e8571b14e0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -574,10 +573,10 @@ golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210304124612-50617c2ba197/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -770,13 +769,14 @@ google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqw gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU= -gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME= gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 81312f9077c2399bb8693c39747fce48550ae217 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 28 Jun 2023 13:54:48 +0400 Subject: [PATCH 31/46] Disable permissions monitoring for now --- .github/workflows/clean.yml | 1 + .github/workflows/conform-pr.yml | 1 + .github/workflows/debug.yml | 1 + .github/workflows/docs.yml | 2 ++ .github/workflows/format.yml | 1 + .github/workflows/go-trust.yml | 1 + .github/workflows/go.yml | 7 +++++++ .github/workflows/js.yml | 1 + .github/workflows/packages.yml | 1 + .github/workflows/security.yml | 1 + .github/workflows/website-preview.yml | 2 ++ 11 files changed, 19 insertions(+) diff --git a/.github/workflows/clean.yml b/.github/workflows/clean.yml index 11c799962dcf..0d9b641edffc 100644 --- a/.github/workflows/clean.yml +++ b/.github/workflows/clean.yml @@ -14,6 +14,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - uses: actions/checkout@v3 - name: Set up Go diff --git a/.github/workflows/conform-pr.yml b/.github/workflows/conform-pr.yml index df82a16fb20b..c5cb89f34b63 100644 --- a/.github/workflows/conform-pr.yml +++ b/.github/workflows/conform-pr.yml @@ -76,6 +76,7 @@ jobs: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Setup Go uses: FerretDB/github-actions/setup-go@main diff --git a/.github/workflows/debug.yml b/.github/workflows/debug.yml index 3a43e4e95f62..eade80277399 100644 --- a/.github/workflows/debug.yml +++ b/.github/workflows/debug.yml @@ -36,6 +36,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 863e8156eb79..ea5596a5a3b1 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -38,6 +38,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -118,6 +119,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Deploy to GitHub Pages id: deploy diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index a7c1bc0f604e..40c5f8a0f5ca 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -37,6 +37,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/go-trust.yml b/.github/workflows/go-trust.yml index bef199ea6ed6..e9cdbfebad2b 100644 --- a/.github/workflows/go-trust.yml +++ b/.github/workflows/go-trust.yml @@ -55,6 +55,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index e55c4272ea90..85e2b0dffbb2 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -37,6 +37,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -86,6 +87,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -184,6 +186,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -258,6 +261,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Submit coveralls uses: coverallsapp/github-action@v2 @@ -279,6 +283,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -324,6 +329,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -354,6 +360,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 25e4d85725e6..6fd3782ad0f4 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -39,6 +39,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/packages.yml b/.github/workflows/packages.yml index 723d67ba1126..6c3f1cbee636 100644 --- a/.github/workflows/packages.yml +++ b/.github/workflows/packages.yml @@ -62,6 +62,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code if: github.event_name != 'pull_request_target' diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index c109bb487623..11b70f2d68e0 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -37,6 +37,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 diff --git a/.github/workflows/website-preview.yml b/.github/workflows/website-preview.yml index 656d3a4d451b..4e0b1f5df198 100644 --- a/.github/workflows/website-preview.yml +++ b/.github/workflows/website-preview.yml @@ -41,6 +41,7 @@ jobs: steps: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 @@ -109,6 +110,7 @@ jobs: - name: Setup permissions monitoring uses: GitHubSecurityLab/actions-permissions/monitor@v1 + if: false - name: Checkout code uses: actions/checkout@v3 From ed330f74c46f77f9135b33120ebe52aba1f0ea21 Mon Sep 17 00:00:00 2001 From: Patryk Kwiatek Date: Wed, 28 Jun 2023 14:24:32 +0200 Subject: [PATCH 32/46] Fix integration tests after bumping deps (#2934) --- integration/basic_test.go | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/integration/basic_test.go b/integration/basic_test.go index 7906d97f4dc0..e86a4cad95fc 100644 --- a/integration/basic_test.go +++ b/integration/basic_test.go @@ -25,7 +25,6 @@ import ( "go.mongodb.org/mongo-driver/bson" "go.mongodb.org/mongo-driver/mongo" "go.mongodb.org/mongo-driver/mongo/options" - "go.mongodb.org/mongo-driver/x/mongo/driver" "github.com/FerretDB/FerretDB/integration/setup" "github.com/FerretDB/FerretDB/integration/shareddata" @@ -518,16 +517,6 @@ func TestDatabaseName(t *testing.T) { } }) - t.Run("Empty", func(t *testing.T) { - t.Parallel() - - ctx, collection := setup.Setup(t) - - err := collection.Database().Client().Database("").CreateCollection(ctx, collection.Name()) - expectedErr := driver.InvalidOperationError(driver.InvalidOperationError{MissingField: "Database"}) - assert.Equal(t, expectedErr, err) - }) - t.Run("63ok", func(t *testing.T) { ctx, collection := setup.Setup(t) From 6161e73086baf7600fc9f34a0c1edfdf8d912882 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 28 Jun 2023 16:28:59 +0400 Subject: [PATCH 33/46] Update benchmark to use cursors (#2932) --- integration/benchmarks_test.go | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/integration/benchmarks_test.go b/integration/benchmarks_test.go index 83bcbe543b0e..8b5fd046b8a6 100644 --- a/integration/benchmarks_test.go +++ b/integration/benchmarks_test.go @@ -51,25 +51,31 @@ func BenchmarkQuerySmallDocuments(b *testing.B) { }, } { b.Run(name, func(b *testing.B) { - var firstDocs, docs []bson.D + var firstDocs, docs int for i := 0; i < b.N; i++ { cursor, err := s.Collection.Find(s.Ctx, bc.filter) require.NoError(b, err) - docs = FetchAll(b, s.Ctx, cursor) - require.NotEmpty(b, docs) + docs = 0 + for cursor.Next(s.Ctx) { + docs++ + } + + require.NoError(b, cursor.Close(s.Ctx)) + require.NoError(b, cursor.Err()) + require.Positive(b, docs) - if firstDocs == nil { + if firstDocs == 0 { firstDocs = docs } } b.StopTimer() - require.Len(b, docs, len(firstDocs)) + require.Equal(b, firstDocs, docs) - b.ReportMetric(float64(len(docs)), "docs-returned") + b.ReportMetric(float64(docs), "docs-returned") }) } }) From f47f33dc90ab332028f48207a5493276fdd61a33 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 28 Jun 2023 16:29:12 +0400 Subject: [PATCH 34/46] Crush PNG images (#2931) --- website/static/img/blog/FerretDB-is-now-Beta.-1-980x551.png | 4 ++-- website/static/img/blog/Maintainer@5x-1-300x300.png | 4 ++-- website/static/img/blog/blog-img.png | 4 ++-- ...31-fa1b-4465-8277-e73da46127de-1650484034538-1-300x120.png | 4 ++-- website/static/img/blog/developer-preview.png | 4 ++-- website/static/img/blog/displaying-studio3t-data.png | 4 ++-- website/static/img/blog/ferret-rpi-1-1024x390.png | 4 ++-- website/static/img/blog/ferretdb-1-1-0-release.png | 4 ++-- website/static/img/blog/ferretdb-demo-creating-database.png | 4 ++-- website/static/img/blog/ferretdb-demo-page.png | 4 ++-- website/static/img/blog/image1-1.png | 4 ++-- website/static/img/blog/image2.png | 4 ++-- website/static/img/blog/image3.png | 4 ++-- website/static/img/blog/image5.png | 4 ++-- website/static/img/blog/image6.png | 4 ++-- website/static/img/blog/image7.png | 4 ++-- website/static/img/blog/mongodb-alternatives.png | 4 ++-- website/static/img/blog/percona-ferretdb.png | 4 ++-- website/static/img/blog/perconalive.png | 4 ++-- website/static/img/blog/postgresql.png | 4 ++-- website/static/img/guis/logo-light.png | 4 ++-- website/static/img/logo-light.png | 4 ++-- 22 files changed, 44 insertions(+), 44 deletions(-) diff --git a/website/static/img/blog/FerretDB-is-now-Beta.-1-980x551.png b/website/static/img/blog/FerretDB-is-now-Beta.-1-980x551.png index 513585f93850..011e5c0cc9fc 100644 --- a/website/static/img/blog/FerretDB-is-now-Beta.-1-980x551.png +++ b/website/static/img/blog/FerretDB-is-now-Beta.-1-980x551.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c866f611004fef277727618c454a11ec63b0c31ca09da1ef1df0738a348421d -size 523135 +oid sha256:0ccb002b41e3edf96239d96a07ae9eff727a1c02969c531e67b4f8f466973b34 +size 452516 diff --git a/website/static/img/blog/Maintainer@5x-1-300x300.png b/website/static/img/blog/Maintainer@5x-1-300x300.png index 050d421e6655..9f132ad2d728 100644 --- a/website/static/img/blog/Maintainer@5x-1-300x300.png +++ b/website/static/img/blog/Maintainer@5x-1-300x300.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:136ed5b4591f4a3916775decd2c3f60b15588074ec6850339a15fe278d7af3f7 -size 51839 +oid sha256:0fccaa9dcdff99b70b537138adee28af7c80f5a4568e2c3a4f4f79abf5ee9517 +size 35863 diff --git a/website/static/img/blog/blog-img.png b/website/static/img/blog/blog-img.png index 12ff2b27b0f5..7da2e3b63699 100644 --- a/website/static/img/blog/blog-img.png +++ b/website/static/img/blog/blog-img.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef004cfac8a9d452c295ce6962f4e4ed863cf75c4066fab01de9c2c39d903950 -size 13907 +oid sha256:36d639c969c0bd2427fff9723fa640eb7b7c9353df94fab3f0314fb9709f1bea +size 5840 diff --git a/website/static/img/blog/cf73bb31-fa1b-4465-8277-e73da46127de-1650484034538-1-300x120.png b/website/static/img/blog/cf73bb31-fa1b-4465-8277-e73da46127de-1650484034538-1-300x120.png index 342beb7d357e..e622ebf86db6 100644 --- a/website/static/img/blog/cf73bb31-fa1b-4465-8277-e73da46127de-1650484034538-1-300x120.png +++ b/website/static/img/blog/cf73bb31-fa1b-4465-8277-e73da46127de-1650484034538-1-300x120.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f49da1a57bc69f2f7cb76065991715c8a1f8bd34af93bebc17922b865e17e103 -size 4792 +oid sha256:8a6dcf1dc404a03ae575bb4bcdeae4bfc570e5ec949913827ece5c762688e62f +size 4138 diff --git a/website/static/img/blog/developer-preview.png b/website/static/img/blog/developer-preview.png index 03f661e2a27e..e4a262b53e67 100644 --- a/website/static/img/blog/developer-preview.png +++ b/website/static/img/blog/developer-preview.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d33ca60dd0af65b42e9d4e323da322da427de6a93143fac6525db521513321e -size 1422888 +oid sha256:76f5dda9da17da762f326406b5554cd96031e1cc75b0c3adcf01d588eeea4886 +size 1240540 diff --git a/website/static/img/blog/displaying-studio3t-data.png b/website/static/img/blog/displaying-studio3t-data.png index c8593d34644f..d85f43416211 100644 --- a/website/static/img/blog/displaying-studio3t-data.png +++ b/website/static/img/blog/displaying-studio3t-data.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fd3d77be3d71b972f08f4a70a87484cc18eb7b4f68ee397432a503a079e36ff -size 222290 +oid sha256:2a3edb47ef00cf8126bc29b710dd058aed7a6402e96a669778629937eaa4eea6 +size 130832 diff --git a/website/static/img/blog/ferret-rpi-1-1024x390.png b/website/static/img/blog/ferret-rpi-1-1024x390.png index 38a00eb1ed7e..562f6f8c1511 100644 --- a/website/static/img/blog/ferret-rpi-1-1024x390.png +++ b/website/static/img/blog/ferret-rpi-1-1024x390.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73335fcf9568da49491f87c5707823b5b85fe69cd6e1ab8ffa5a9f971afeaaa4 -size 54765 +oid sha256:b59064ff8e0e55b3fa0c5988571187fd0126f34319493f02c0d4ab2db5e031ae +size 39609 diff --git a/website/static/img/blog/ferretdb-1-1-0-release.png b/website/static/img/blog/ferretdb-1-1-0-release.png index 2a98d4abb884..c515aa957b4a 100644 --- a/website/static/img/blog/ferretdb-1-1-0-release.png +++ b/website/static/img/blog/ferretdb-1-1-0-release.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a58dc84518f89f93e79d637a23bcb2e87705fa5af1b158dcef955d2f70514e94 -size 697026 +oid sha256:10f409aae2a63fc4416eba3e66a96ec0328bd2be1aeb5804da89f45c17d22475 +size 661033 diff --git a/website/static/img/blog/ferretdb-demo-creating-database.png b/website/static/img/blog/ferretdb-demo-creating-database.png index 80917667f6aa..70a265276840 100644 --- a/website/static/img/blog/ferretdb-demo-creating-database.png +++ b/website/static/img/blog/ferretdb-demo-creating-database.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46ce5852e6d5afee47820fca39a16e915c61239626e330ea7c23c5fbf4095e85 -size 328374 +oid sha256:40e9581c97c0809139312c3a3878b36f99cc91a3c268d4cf2c07b679073e5ec7 +size 183287 diff --git a/website/static/img/blog/ferretdb-demo-page.png b/website/static/img/blog/ferretdb-demo-page.png index 2fd34ab4b3e0..f9287833e605 100644 --- a/website/static/img/blog/ferretdb-demo-page.png +++ b/website/static/img/blog/ferretdb-demo-page.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81b12d41d343d011f92d95682b447250a0ea5d828f8437a38d64b80c6d043103 -size 374166 +oid sha256:eb3152b119079638a1b1c6bd469dc602f33f2b84a511fa6b86ee6add37299c81 +size 219242 diff --git a/website/static/img/blog/image1-1.png b/website/static/img/blog/image1-1.png index 8792db6a351c..6743905625f0 100644 --- a/website/static/img/blog/image1-1.png +++ b/website/static/img/blog/image1-1.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f304ea855ce4f1351a1ef56daa8d6094be529587d1c2b75ecd4f7b3283962072 -size 139974 +oid sha256:d14786ff4f7cd969b276db5da1c80078e1e97a5e2a3d897b304cfc12509a6e37 +size 125278 diff --git a/website/static/img/blog/image2.png b/website/static/img/blog/image2.png index 1499787088fa..dfea1c26e23e 100644 --- a/website/static/img/blog/image2.png +++ b/website/static/img/blog/image2.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bffde14766980f0a5639d52f4e4ad24f3d5829ea3c4eda52c8812d3af95a3879 -size 447513 +oid sha256:4ff23afa26b9298b75f76b5e28fff19131967a5c628e2ca47b9cf0bf3ffa294a +size 388049 diff --git a/website/static/img/blog/image3.png b/website/static/img/blog/image3.png index 9a50167d663a..364f33fccd74 100644 --- a/website/static/img/blog/image3.png +++ b/website/static/img/blog/image3.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:079eddd46405f13a3aa6b007913f553ff1bcb0231cbdcb8dafc828513a4bd265 -size 251559 +oid sha256:348342c9587c1744eb17bff54b79e5c64f823db5c19552fffc20ac29596ace64 +size 221598 diff --git a/website/static/img/blog/image5.png b/website/static/img/blog/image5.png index 055906060789..ba71326f9f71 100644 --- a/website/static/img/blog/image5.png +++ b/website/static/img/blog/image5.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5fbd6719f60e770fd6688f3f016a21d195391c517dd7e8dc5cf31bb436abc432 -size 25706 +oid sha256:20e4ccfaceb9e1aff4f61b73bc992b1ac33fb3584d36eb6a622b3b63c536173b +size 15957 diff --git a/website/static/img/blog/image6.png b/website/static/img/blog/image6.png index 2deeb00b61e0..ad76056ae041 100644 --- a/website/static/img/blog/image6.png +++ b/website/static/img/blog/image6.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29a4776edc18d3c552566d351a83cac31afd822e45a3a1f94c6921eb686eb50f -size 22229 +oid sha256:9fa6d6f650af866cd9c8e4e29d838d1378897331118cadb037a42a6832130d69 +size 11970 diff --git a/website/static/img/blog/image7.png b/website/static/img/blog/image7.png index c254eba2e4f8..5ffe33ee93c0 100644 --- a/website/static/img/blog/image7.png +++ b/website/static/img/blog/image7.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94d50b4ba4447142712f3e1bb7cb54ef09ee6286648442edf867d66f5d941eee -size 134656 +oid sha256:f92a9ffefb302d9f3ccd95ae48ff6c145929e69cec0053f6771bc24c2676922d +size 133889 diff --git a/website/static/img/blog/mongodb-alternatives.png b/website/static/img/blog/mongodb-alternatives.png index 1bd9da3ebab9..14df93a4fc1b 100644 --- a/website/static/img/blog/mongodb-alternatives.png +++ b/website/static/img/blog/mongodb-alternatives.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ed4a7707b67f4571d4eb01bcc4e351c1d91f18b0940e4a9118492d0fbf7d33b -size 1710153 +oid sha256:d127af191eb426117502bc90c1d7c97a1d226d4aead9db08f0d7ad081598b77d +size 1710631 diff --git a/website/static/img/blog/percona-ferretdb.png b/website/static/img/blog/percona-ferretdb.png index 7b2be3935f1d..3caf72596aaa 100644 --- a/website/static/img/blog/percona-ferretdb.png +++ b/website/static/img/blog/percona-ferretdb.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be17587a9d1a6f9fab306f0f7f9916bd352dfc49857ddb618524014366fea39a -size 126198 +oid sha256:1721fb59d091e3795b0ccd3094fe63a6eb9327e4c4919b6936cf79e927cabb32 +size 84614 diff --git a/website/static/img/blog/perconalive.png b/website/static/img/blog/perconalive.png index 866a080f4e96..5de34dadd3f1 100644 --- a/website/static/img/blog/perconalive.png +++ b/website/static/img/blog/perconalive.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1efa9ee9cdd6428a979c7195caea5ba2d38ed3715f15f96aa0069249125ac714 -size 136671 +oid sha256:11a5f7505957dbd7265e23a529c4183b1cd772d8b3f9e74a9930ab035e46cd19 +size 115262 diff --git a/website/static/img/blog/postgresql.png b/website/static/img/blog/postgresql.png index b4d88baf6bf5..ef4752bcbc83 100644 --- a/website/static/img/blog/postgresql.png +++ b/website/static/img/blog/postgresql.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b7c82ae9b0346bf45d9f4f17a11b0511007bcac41af2451e3d65fbfc5d45f0e5 -size 1440761 +oid sha256:ae43619770e2ef645e6dde2624e74cc043b3d3f23717986d1c5ea5f4358887b0 +size 830167 diff --git a/website/static/img/guis/logo-light.png b/website/static/img/guis/logo-light.png index e51fed4f7d51..395246e7ba42 100644 --- a/website/static/img/guis/logo-light.png +++ b/website/static/img/guis/logo-light.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29723b3fedac1105d04a4156cb4599263bdacc04fb8bfc78a7a2592bb0eae17e -size 33547 +oid sha256:aa3801ae17ad6985e2bd7c538c190fb69f46ec9e32737aa2842b0b894d536af5 +size 25551 diff --git a/website/static/img/logo-light.png b/website/static/img/logo-light.png index 57bd3fe03177..395246e7ba42 100644 --- a/website/static/img/logo-light.png +++ b/website/static/img/logo-light.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ffb0beabeff569a076a70728d83ffdeec5091457466466bc9907d04d1de1ef9b -size 41167 +oid sha256:aa3801ae17ad6985e2bd7c538c190fb69f46ec9e32737aa2842b0b894d536af5 +size 25551 From 2d8f797e01304f5b805427556df8b29fe753b123 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Wed, 28 Jun 2023 20:57:32 +0400 Subject: [PATCH 35/46] Set `minWireVersion` to 0 (#2937) --- internal/handlers/common/common.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/handlers/common/common.go b/internal/handlers/common/common.go index d8b853e9cee1..38e4322e1aec 100644 --- a/internal/handlers/common/common.go +++ b/internal/handlers/common/common.go @@ -17,7 +17,7 @@ package common const ( // MinWireVersion is the minimal supported wire protocol version. - MinWireVersion = int32(13) // 5.0 + MinWireVersion = int32(0) // needed for some apps and drivers // MaxWireVersion is the maximal supported wire protocol version. MaxWireVersion = int32(17) From 7b6ccf3ac95333d07e6a2ce6059b8cc8a1ba1528 Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Thu, 29 Jun 2023 10:31:24 +0900 Subject: [PATCH 36/46] Run `getMore` integration test in one connection pool (#2878) Closes #1807. --- ferretdb/ferretdb.go | 6 +- integration/commands_diagnostic_test.go | 100 ++++++++++++++ integration/query_test.go | 173 +++++++++++++++++++++++- integration/setup/client.go | 48 ------- integration/setup/listener.go | 75 +++++++--- integration/setup/setup.go | 32 ++++- integration/setup/setup_compat.go | 3 +- integration/setup/test_helpers.go | 13 ++ internal/util/teststress/stress.go | 11 +- 9 files changed, 372 insertions(+), 89 deletions(-) diff --git a/ferretdb/ferretdb.go b/ferretdb/ferretdb.go index e3faee3589c4..43c4cdd70693 100644 --- a/ferretdb/ferretdb.go +++ b/ferretdb/ferretdb.go @@ -184,9 +184,9 @@ func (f *FerretDB) MongoDBURI() string { switch { case f.config.Listener.TLS != "": - q := make(url.Values) - - q.Set("tls", "true") + q := url.Values{ + "tls": []string{"true"}, + } u = &url.URL{ Scheme: "mongodb", diff --git a/integration/commands_diagnostic_test.go b/integration/commands_diagnostic_test.go index 7a1a2085b24c..b24a0242cf90 100644 --- a/integration/commands_diagnostic_test.go +++ b/integration/commands_diagnostic_test.go @@ -16,6 +16,7 @@ package integration import ( "net" + "net/url" "testing" "github.com/stretchr/testify/assert" @@ -28,6 +29,7 @@ import ( "github.com/FerretDB/FerretDB/integration/shareddata" "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/must" + "github.com/FerretDB/FerretDB/internal/util/teststress" "github.com/FerretDB/FerretDB/internal/util/testutil" ) @@ -404,3 +406,101 @@ func TestCommandsDiagnosticWhatsMyURI(t *testing.T) { assert.NotEqual(t, ports[0], ports[1]) } } + +// TestCommandWhatsMyURIConnection tests that integration test setup applies +// minPoolSize, maxPoolSize and maxIdleTimeMS correctly to the driver. +// It also tests that the driver behaves like we think it should. +func TestCommandWhatsMyURIConnection(t *testing.T) { + t.Parallel() + + // options are applied to create a client that uses single connection pool + s := setup.SetupWithOpts(t, &setup.SetupOpts{ + ExtraOptions: url.Values{ + "minPoolSize": []string{"1"}, + "maxPoolSize": []string{"1"}, + "maxIdleTimeMS": []string{"0"}, + }, + }) + + collection1 := s.Collection + databaseName := s.Collection.Database().Name() + collectionName := s.Collection.Name() + + t.Run("SameClientStress", func(t *testing.T) { + t.Parallel() + + ports := make(chan string, teststress.NumGoroutines) + + teststress.Stress(t, func(ready chan<- struct{}, start <-chan struct{}) { + ready <- struct{}{} + <-start + + var res bson.D + err := collection1.Database().RunCommand(s.Ctx, bson.D{{"whatsmyuri", int32(1)}}).Decode(&res) + require.NoError(t, err) + + doc := ConvertDocument(t, res) + v, _ := doc.Get("ok") + resOk, ok := v.(float64) + require.True(t, ok) + assert.Equal(t, float64(1), resOk) + + v, _ = doc.Get("you") + you, ok := v.(string) + require.True(t, ok) + + _, port, err := net.SplitHostPort(you) + require.NoError(t, err) + assert.NotEmpty(t, port) + ports <- port + }) + + close(ports) + + firstPort := <-ports + for port := range ports { + require.Equal(t, firstPort, port, "expected same client to use the same port") + } + }) + + t.Run("DifferentClient", func(t *testing.T) { + t.Parallel() + + u, err := url.Parse(s.MongoDBURI) + require.NoError(t, err) + + client2, err := mongo.Connect(s.Ctx, options.Client().ApplyURI(u.String())) + require.NoError(t, err) + + defer client2.Disconnect(s.Ctx) + + collection2 := client2.Database(databaseName).Collection(collectionName) + + var ports []string + + for _, collection := range []*mongo.Collection{collection1, collection2} { + var res bson.D + err := collection.Database().RunCommand(s.Ctx, bson.D{{"whatsmyuri", int32(1)}}).Decode(&res) + require.NoError(t, err) + + doc := ConvertDocument(t, res) + v, _ := doc.Get("ok") + resOk, ok := v.(float64) + require.True(t, ok) + assert.Equal(t, float64(1), resOk) + + v, _ = doc.Get("you") + you, ok := v.(string) + require.True(t, ok) + + _, port, err := net.SplitHostPort(you) + require.NoError(t, err) + assert.NotEmpty(t, port) + + ports = append(ports, port) + } + + require.Equal(t, 2, len(ports)) + assert.NotEqual(t, ports[0], ports[1]) + }) +} diff --git a/integration/query_test.go b/integration/query_test.go index 6cae94c6d811..daafc5d63d8f 100644 --- a/integration/query_test.go +++ b/integration/query_test.go @@ -16,6 +16,7 @@ package integration import ( "math" + "net/url" "testing" "time" @@ -29,6 +30,7 @@ import ( "github.com/FerretDB/FerretDB/integration/setup" "github.com/FerretDB/FerretDB/integration/shareddata" + "github.com/FerretDB/FerretDB/internal/types" ) func TestQueryBadFindType(t *testing.T) { @@ -956,7 +958,17 @@ func TestQueryBatchSize(t *testing.T) { func TestQueryCommandGetMore(t *testing.T) { t.Parallel() - ctx, collection := setup.Setup(t) + + // options are applied to create a client that uses single connection pool + s := setup.SetupWithOpts(t, &setup.SetupOpts{ + ExtraOptions: url.Values{ + "minPoolSize": []string{"1"}, + "maxPoolSize": []string{"1"}, + "maxIdleTimeMS": []string{"0"}, + }, + }) + + ctx, collection := s.Ctx, s.Collection // the number of documents is set above the default batchSize of 101 // for testing unset batchSize returning default batchSize @@ -1179,6 +1191,34 @@ func TestQueryCommandGetMore(t *testing.T) { Message: "BSON field 'getMore.collection' is missing but a required field", }, }, + "UnsetAllBatchSize": { + findBatchSize: nil, + getMoreBatchSize: nil, + collection: collection.Name(), + firstBatch: docs[:101], + nextBatch: docs[101:], + }, + "UnsetFindBatchSize": { + findBatchSize: nil, + getMoreBatchSize: 5, + collection: collection.Name(), + firstBatch: docs[:101], + nextBatch: docs[101:106], + }, + "UnsetGetMoreBatchSize": { + findBatchSize: 5, + getMoreBatchSize: nil, + collection: collection.Name(), + firstBatch: docs[:5], + nextBatch: docs[5:], + }, + "BatchSize": { + findBatchSize: 3, + getMoreBatchSize: 5, + collection: collection.Name(), + firstBatch: docs[:3], + nextBatch: docs[3:8], + }, } { name, tc := name, tc t.Run(name, func(t *testing.T) { @@ -1186,8 +1226,11 @@ func TestQueryCommandGetMore(t *testing.T) { t.Skip(tc.skip) } - // TODO: https://github.com/FerretDB/FerretDB/issues/1807 - // Do not run tests in parallel, MongoDB throws error that session and cursor do not match. + // Do not run subtests in t.Parallel() to eliminate the occurrence + // of session error. + // Supporting session would help us understand fix it + // https://github.com/FerretDB/FerretDB/issues/153. + // // > Location50738 // > Cannot run getMore on cursor 2053655655200551971, // > which was created in session 2926eea5-9775-41a3-a563-096969f1c7d5 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= - - , @@ -1280,3 +1323,127 @@ func TestQueryCommandGetMore(t *testing.T) { }) } } + +func TestQueryCommandGetMoreConnection(t *testing.T) { + t.Parallel() + + // options are applied to create a client that uses single connection pool + s := setup.SetupWithOpts(t, &setup.SetupOpts{ + ExtraOptions: url.Values{ + "minPoolSize": []string{"1"}, + "maxPoolSize": []string{"1"}, + "maxIdleTimeMS": []string{"0"}, + }, + }) + + ctx := s.Ctx + collection1 := s.Collection + databaseName := s.Collection.Database().Name() + collectionName := s.Collection.Name() + + docs := generateDocuments(0, 5) + _, err := collection1.InsertMany(ctx, docs) + require.NoError(t, err) + + t.Run("SameClient", func(t *testing.T) { + // Do not run subtests in t.Parallel() to eliminate the occurrence + // of session error. + // Supporting session would help us understand fix it + // https://github.com/FerretDB/FerretDB/issues/153. + // + // > Location50738 + // > Cannot run getMore on cursor 2053655655200551971, + // > which was created in session 2926eea5-9775-41a3-a563-096969f1c7d5 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= - - , + // > in session 774d9ac6-b24a-4fd8-9874-f92ab1c9c8f5 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= - - + + var res bson.D + err = collection1.Database().RunCommand( + ctx, + bson.D{ + {"find", collection1.Name()}, + {"batchSize", 2}, + }, + ).Decode(&res) + require.NoError(t, err) + + doc := ConvertDocument(t, res) + + v, _ := doc.Get("cursor") + require.NotNil(t, v) + + cursor, ok := v.(*types.Document) + require.True(t, ok) + + cursorID, _ := cursor.Get("id") + assert.NotNil(t, cursorID) + + err = collection1.Database().RunCommand( + ctx, + bson.D{ + {"getMore", cursorID}, + {"collection", collection1.Name()}, + }, + ).Decode(&res) + require.NoError(t, err) + }) + + t.Run("DifferentClient", func(t *testing.T) { + // The error returned from MongoDB is a session error, FerretDB does not + // return an error because db, collection and username are the same. + setup.SkipExceptMongoDB(t, "https://github.com/FerretDB/FerretDB/issues/153") + + // do not run subtest in parallel to avoid breaking another parallel subtest + + u, err := url.Parse(s.MongoDBURI) + require.NoError(t, err) + + client2, err := mongo.Connect(ctx, options.Client().ApplyURI(u.String())) + require.NoError(t, err) + + defer client2.Disconnect(ctx) + + collection2 := client2.Database(databaseName).Collection(collectionName) + + var res bson.D + err = collection1.Database().RunCommand( + ctx, + bson.D{ + {"find", collection1.Name()}, + {"batchSize", 2}, + }, + ).Decode(&res) + require.NoError(t, err) + + doc := ConvertDocument(t, res) + + v, _ := doc.Get("cursor") + require.NotNil(t, v) + + cursor, ok := v.(*types.Document) + require.True(t, ok) + + cursorID, _ := cursor.Get("id") + assert.NotNil(t, cursorID) + + err = collection2.Database().RunCommand( + ctx, + bson.D{ + {"getMore", cursorID}, + {"collection", collection2.Name()}, + }, + ).Decode(&res) + + // use AssertMatchesCommandError because message cannot be compared as it contains session ID + AssertMatchesCommandError( + t, + mongo.CommandError{ + Code: 50738, + Name: "Location50738", + Message: "Cannot run getMore on cursor 5720627396082469624, which was created in session " + + "95326129-ff9c-48a4-9060-464b4ea3ee06 - 47DEQpj8HBSa+/TImW+5JC\neuQeRkm5NMpJWZG3hSuFU= - - , " + + "in session 9e8902e9-338c-4156-9fd8-50e5d62ac992 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= - - ", + }, + err, + ) + }) +} diff --git a/integration/setup/client.go b/integration/setup/client.go index 91db78793b08..82d1bb58b21d 100644 --- a/integration/setup/client.go +++ b/integration/setup/client.go @@ -16,8 +16,6 @@ package setup import ( "context" - "net/url" - "path/filepath" "testing" "github.com/stretchr/testify/require" @@ -31,52 +29,6 @@ import ( "github.com/FerretDB/FerretDB/internal/util/observability" ) -// mongoDBURIOpts represents mongoDBURI's options. -type mongoDBURIOpts struct { - hostPort string // for TCP and TLS - unixSocketPath string - tlsAndAuth bool -} - -// mongoDBURI builds MongoDB URI with given options. -func mongoDBURI(tb testing.TB, opts *mongoDBURIOpts) string { - tb.Helper() - - var host string - - if opts.hostPort != "" { - require.Empty(tb, opts.unixSocketPath, "both hostPort and unixSocketPath are set") - host = opts.hostPort - } else { - host = opts.unixSocketPath - } - - var user *url.Userinfo - q := make(url.Values) - - if opts.tlsAndAuth { - require.Empty(tb, opts.unixSocketPath, "unixSocketPath cannot be used with TLS") - - // we don't separate TLS and auth just for simplicity of our test configurations - q.Set("tls", "true") - q.Set("tlsCertificateKeyFile", filepath.Join(CertsRoot, "client.pem")) - q.Set("tlsCaFile", filepath.Join(CertsRoot, "rootCA-cert.pem")) - q.Set("authMechanism", "PLAIN") - user = url.UserPassword("username", "password") - } - - // TODO https://github.com/FerretDB/FerretDB/issues/1507 - u := &url.URL{ - Scheme: "mongodb", - Host: host, - Path: "/", - User: user, - RawQuery: q.Encode(), - } - - return u.String() -} - // makeClient returns new client for the given working MongoDB URI. func makeClient(ctx context.Context, uri string) (*mongo.Client, error) { clientOpts := options.Client().ApplyURI(uri) diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 2456440660ef..28fe876b3159 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -17,6 +17,7 @@ package setup import ( "context" "errors" + "net/url" "os" "path/filepath" "strings" @@ -24,7 +25,6 @@ import ( "testing" "github.com/stretchr/testify/require" - "go.mongodb.org/mongo-driver/mongo" "go.opentelemetry.io/otel" "go.uber.org/zap" @@ -63,9 +63,50 @@ func unixSocketPath(tb testing.TB) string { return f.Name() } +// listenerMongoDBURI builds MongoDB URI for in-process FerretDB. +func listenerMongoDBURI(tb testing.TB, hostPort, unixSocketPath string, tlsAndAuth bool) string { + tb.Helper() + + var host string + + if hostPort != "" { + require.Empty(tb, unixSocketPath, "both hostPort and unixSocketPath are set") + host = hostPort + } else { + host = unixSocketPath + } + + var user *url.Userinfo + var q url.Values + + if tlsAndAuth { + require.Empty(tb, unixSocketPath, "unixSocketPath cannot be used with TLS") + + // we don't separate TLS and auth just for simplicity of our test configurations + q = url.Values{ + "tls": []string{"true"}, + "tlsCertificateKeyFile": []string{filepath.Join(CertsRoot, "client.pem")}, + "tlsCaFile": []string{filepath.Join(CertsRoot, "rootCA-cert.pem")}, + "authMechanism": []string{"PLAIN"}, + } + user = url.UserPassword("username", "password") + } + + // TODO https://github.com/FerretDB/FerretDB/issues/1507 + u := &url.URL{ + Scheme: "mongodb", + Host: host, + Path: "/", + User: user, + RawQuery: q.Encode(), + } + + return u.String() +} + // setupListener starts in-process FerretDB server that runs until ctx is canceled. -// It returns client and MongoDB URI of that listener. -func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mongo.Client, string) { +// It returns basic MongoDB URI for that listener. +func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) string { tb.Helper() _, span := otel.Tracer("").Start(ctx, "setupListener") @@ -170,20 +211,12 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mon l := clientconn.NewListener(&listenerOpts) - runCtx, runCancel := context.WithCancel(ctx) runDone := make(chan struct{}) - // that prevents the deadlock on failed client setup; see below - defer func() { - if tb.Failed() { - runCancel() - } - }() - go func() { defer close(runDone) - err := l.Run(runCtx) + err := l.Run(ctx) if err == nil || errors.Is(err, context.Canceled) { logger.Info("Listener stopped without error") } else { @@ -196,24 +229,22 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) (*mon <-runDone }) - var clientOpts mongoDBURIOpts + var hostPort, unixSocketPath string + var tlsAndAuth bool switch { case *targetTLSF: - clientOpts.hostPort = l.TLSAddr().String() - clientOpts.tlsAndAuth = true + hostPort = l.TLSAddr().String() + tlsAndAuth = true case *targetUnixSocketF: - clientOpts.unixSocketPath = l.UnixAddr().String() + unixSocketPath = l.UnixAddr().String() default: - clientOpts.hostPort = l.TCPAddr().String() + hostPort = l.TCPAddr().String() } - // those will fail the test if in-process FerretDB is not working; - // for example, when backend is down - uri := mongoDBURI(tb, &clientOpts) - client := setupClient(tb, ctx, uri) + uri := listenerMongoDBURI(tb, hostPort, unixSocketPath, tlsAndAuth) logger.Info("Listener started", zap.String("handler", handler), zap.String("uri", uri)) - return client, uri + return uri } diff --git a/integration/setup/setup.go b/integration/setup/setup.go index 6dc3ad076b75..b4820cc9b26e 100644 --- a/integration/setup/setup.go +++ b/integration/setup/setup.go @@ -19,6 +19,7 @@ import ( "context" "flag" "fmt" + "net/url" "path/filepath" "runtime/trace" "strings" @@ -86,6 +87,9 @@ type SetupOpts struct { // Benchmark data provider. If empty, collection is not created. BenchmarkProvider shareddata.BenchmarkProvider + + // ExtraOptions sets the options in MongoDB URI, when the option exists it overwrites that option. + ExtraOptions url.Values } // SetupResult represents setup results. @@ -132,16 +136,30 @@ func SetupWithOpts(tb testing.TB, opts *SetupOpts) *SetupResult { } logger := testutil.LevelLogger(tb, level) - var client *mongo.Client - var uri string + uri := *targetURLF + if uri == "" { + uri = setupListener(tb, setupCtx, logger) + } - if *targetURLF == "" { - client, uri = setupListener(tb, setupCtx, logger) - } else { - client = setupClient(tb, setupCtx, *targetURLF) - uri = *targetURLF + if opts.ExtraOptions != nil { + u, err := url.Parse(uri) + require.NoError(tb, err) + + q := u.Query() + + for k, vs := range opts.ExtraOptions { + for _, v := range vs { + q.Set(k, v) + } + } + + u.RawQuery = q.Encode() + uri = u.String() + tb.Logf("URI with extra options: %s", uri) } + client := setupClient(tb, setupCtx, uri) + // register cleanup function after setupListener registers its own to preserve full logs tb.Cleanup(cancel) diff --git a/integration/setup/setup_compat.go b/integration/setup/setup_compat.go index f6edfaccc867..b4e4278e37d1 100644 --- a/integration/setup/setup_compat.go +++ b/integration/setup/setup_compat.go @@ -94,7 +94,8 @@ func SetupCompatWithOpts(tb testing.TB, opts *SetupCompatOpts) *SetupCompatResul var targetClient *mongo.Client if *targetURLF == "" { - targetClient, _ = setupListener(tb, setupCtx, logger) + uri := setupListener(tb, setupCtx, logger) + targetClient = setupClient(tb, setupCtx, uri) } else { targetClient = setupClient(tb, setupCtx, *targetURLF) } diff --git a/integration/setup/test_helpers.go b/integration/setup/test_helpers.go index f16c5e8ab2e1..dce73c4b501d 100644 --- a/integration/setup/test_helpers.go +++ b/integration/setup/test_helpers.go @@ -33,6 +33,19 @@ func SkipForMongoDB(tb testing.TB, reason string) { } } +// SkipExceptMongoDB skips the current test for backends except MongoDB. +// +// This function should not be used lightly. +func SkipExceptMongoDB(tb testing.TB, reason string) { + tb.Helper() + + if *targetBackendF != "mongodb" { + require.NotEmpty(tb, reason, "reason must not be empty") + + tb.Skipf("Skipping for %s: %s.", *targetBackendF, reason) + } +} + // IsTigris returns true if tests are running against FerretDB with `ferretdb-tigris` backend. // // This function should not be used lightly. diff --git a/internal/util/teststress/stress.go b/internal/util/teststress/stress.go index 972c95ef0daa..794d730224e0 100644 --- a/internal/util/teststress/stress.go +++ b/internal/util/teststress/stress.go @@ -23,6 +23,9 @@ import ( "testing" ) +// NumGoroutines is the total count of goroutines created in Stress function. +var NumGoroutines = runtime.GOMAXPROCS(-1) * 10 + // Stress runs function f in multiple goroutines. // // Function f should do a needed setup, send a message to ready channel when it is ready to start, @@ -30,15 +33,13 @@ import ( func Stress(tb testing.TB, f func(ready chan<- struct{}, start <-chan struct{})) { tb.Helper() - n := runtime.GOMAXPROCS(-1) * 10 - // do a bit more work to reduce a chance that one goroutine would finish // before the other one is still being created var wg sync.WaitGroup - readyCh := make(chan struct{}, n) + readyCh := make(chan struct{}, NumGoroutines) startCh := make(chan struct{}) - for i := 0; i < n; i++ { + for i := 0; i < NumGoroutines; i++ { wg.Add(1) go func() { @@ -48,7 +49,7 @@ func Stress(tb testing.TB, f func(ready chan<- struct{}, start <-chan struct{})) }() } - for i := 0; i < n; i++ { + for i := 0; i < NumGoroutines; i++ { <-readyCh } From d3fd02a62523fbbb5eec977077d1ebaa9b70877d Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Thu, 29 Jun 2023 09:20:13 +0400 Subject: [PATCH 37/46] Use cursors in `find` command (#2933) Closes #1733. --- .github/workflows/js.yml | 2 +- Taskfile.yml | 8 +-- integration/Taskfile.yml | 8 +-- internal/handlers/pg/msg_drop.go | 7 +++ internal/handlers/pg/msg_dropdatabase.go | 7 +++ internal/handlers/pg/msg_find.go | 47 +++++++++-------- internal/handlers/pg/pgdb/pool.go | 4 +- internal/handlers/pg/pgdb/query_iterator.go | 7 +++ internal/handlers/pg/pgdb/transactions.go | 50 ++++++++++++------- .../handlers/pg/pgdb/transactions_test.go | 44 ++++++++++++++++ internal/handlers/sqlite/msg_drop.go | 5 +- internal/handlers/sqlite/msg_dropdatabase.go | 11 ++++ internal/handlers/sqlite/msg_find.go | 11 ++-- 13 files changed, 150 insertions(+), 61 deletions(-) diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 6fd3782ad0f4..d006eb0f48ab 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -75,7 +75,7 @@ jobs: --proxy-addr=127.0.0.1:47017 --mode=diff-normal --handler=pg - --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 2> ferretdb.log & - name: Run testjs for MongoDB diff --git a/Taskfile.yml b/Taskfile.yml index 2ac822676847..eacc301b4f53 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -197,7 +197,7 @@ tasks: -coverprofile=integration-pg.txt . -target-backend=ferretdb-pg -target-tls - -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 -compat-url='mongodb://username:password@127.0.0.1:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem' vars: SHARD_RUN: @@ -308,7 +308,7 @@ tasks: --proxy-addr=127.0.0.1:47017 --mode=diff-normal --handler=pg - --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 --test-records-dir=tmp/records run-sqlite: @@ -366,7 +366,7 @@ tasks: --proxy-addr=127.0.0.1:47017 --mode=diff-normal --handler=pg - --postgresql-url=postgres://127.0.0.1:5433/ferretdb + --postgresql-url=postgres://127.0.0.1:5433/ferretdb?pool_max_conns=50 --test-records-dir=tmp/records # set FERRETDB_TIGRIS_CLIENT_ID and FERRETDB_TIGRIS_CLIENT_SECRET environment variables to use it @@ -397,7 +397,7 @@ tasks: --proxy-addr=127.0.0.1:47017 --mode=diff-proxy --handler=pg - --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + --postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 --test-records-dir=tmp/records run-sqlite-proxy: diff --git a/integration/Taskfile.yml b/integration/Taskfile.yml index 534c5179ee09..b842a99307ac 100644 --- a/integration/Taskfile.yml +++ b/integration/Taskfile.yml @@ -17,12 +17,12 @@ tasks: go test -count=1 {{.RACE_FLAG}} -run=TestEnvData -tags=ferretdb_testenvdata . -target-backend=ferretdb-pg - -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_min_conns=1 + -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 - > go test -count=1 {{.RACE_FLAG}} -run=TestEnvData -tags=ferretdb_testenvdata . -target-backend=ferretdb-pg - -postgresql-url=postgres://username:password@127.0.0.1:5433/ferretdb?pool_min_conns=1 + -postgresql-url=postgres://username:password@127.0.0.1:5433/ferretdb?pool_max_conns=50 # no sqlite yet - > go test -count=1 {{.RACE_FLAG}} -run=TestEnvData @@ -61,7 +61,7 @@ tasks: -log-level=error -bench-docs={{.BENCH_DOCS}} -target-backend=ferretdb-pg - -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 | tee new-pg.txt - ../bin/benchstat{{exeExt}} old-pg.txt new-pg.txt @@ -74,7 +74,7 @@ tasks: -log-level=error -bench-docs={{.BENCH_DOCS}} -target-backend=ferretdb-pg - -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb + -postgresql-url=postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50 -- -disable-filter-pushdown | tee new-pg.txt - ../bin/benchstat{{exeExt}} old-pg.txt new-pg.txt diff --git a/internal/handlers/pg/msg_drop.go b/internal/handlers/pg/msg_drop.go index 3da9eeb56e71..52b8484a9a33 100644 --- a/internal/handlers/pg/msg_drop.go +++ b/internal/handlers/pg/msg_drop.go @@ -55,6 +55,13 @@ func (h *Handler) MsgDrop(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } + // PostgreSQL would block on `DropDatabase` below otherwise + for _, c := range h.cursors.All() { + if c.DB == db && c.Collection == collection { + c.Close() + } + } + err = dbPool.InTransaction(ctx, func(tx pgx.Tx) error { return pgdb.DropCollection(ctx, tx, db, collection) }) diff --git a/internal/handlers/pg/msg_dropdatabase.go b/internal/handlers/pg/msg_dropdatabase.go index de87f93319b5..af4bf6f0bcf1 100644 --- a/internal/handlers/pg/msg_dropdatabase.go +++ b/internal/handlers/pg/msg_dropdatabase.go @@ -47,6 +47,13 @@ func (h *Handler) MsgDropDatabase(ctx context.Context, msg *wire.OpMsg) (*wire.O return nil, err } + // PostgreSQL would block on `DropDatabase` below otherwise + for _, c := range h.cursors.All() { + if c.DB == db { + c.Close() + } + } + res := must.NotFail(types.NewDocument()) err = dbPool.InTransaction(ctx, func(tx pgx.Tx) error { diff --git a/internal/handlers/pg/msg_find.go b/internal/handlers/pg/msg_find.go index 58ee8b713b13..0cd27c30e850 100644 --- a/internal/handlers/pg/msg_find.go +++ b/internal/handlers/pg/msg_find.go @@ -52,13 +52,6 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er username, _ := conninfo.Get(ctx).Auth() - cancel := func() {} - if params.MaxTimeMS != 0 { - // It is not clear if maxTimeMS affects only find, or both find and getMore (as the current code does). - // TODO https://github.com/FerretDB/FerretDB/issues/1808 - ctx, cancel = context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) - } - qp := &pgdb.QueryParams{ DB: params.DB, Collection: params.Collection, @@ -68,7 +61,6 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er // get comment from query, e.g. db.collection.find({$comment: "test"}) if params.Filter != nil { if qp.Comment, err = common.GetOptionalParam(params.Filter, "$comment", qp.Comment); err != nil { - cancel() return nil, err } } @@ -81,18 +73,28 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er qp.Sort = params.Sort } - var resDocs []*types.Document - err = dbPool.InTransaction(ctx, func(tx pgx.Tx) error { - var iter types.DocumentsIterator - var queryRes pgdb.QueryResults + cancel := func() {} + if params.MaxTimeMS != 0 { + // It is not clear if maxTimeMS affects only find, or both find and getMore (as the current code does). + // TODO https://github.com/FerretDB/FerretDB/issues/1808 + ctx, cancel = context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) + } + + // closer accumulates all things that should be closed / canceled. + closer := iterator.NewMultiCloser(iterator.CloserFunc(cancel)) + var keepTx pgx.Tx + var iter types.DocumentsIterator + err = dbPool.InTransactionKeep(ctx, func(tx pgx.Tx) error { + keepTx = tx + + var queryRes pgdb.QueryResults iter, queryRes, err = pgdb.QueryDocuments(ctx, tx, qp) if err != nil { return lazyerrors.Error(err) } - closer := iterator.NewMultiCloser(iter) - defer closer.Close() + closer.Add(iter) iter = common.FilterIterator(iter, closer, params.Filter) @@ -121,22 +123,23 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return lazyerrors.Error(err) } - // TODO https://github.com/FerretDB/FerretDB/issues/1733 - resDocs, err = iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) - - return err + return nil }) if err != nil { - cancel() + closer.Close() return nil, lazyerrors.Error(err) } - iter := iterator.Values(iterator.ForSlice(resDocs)) - closer := iterator.NewMultiCloser(iter, iterator.CloserFunc(cancel)) + closer.Add(iterator.CloserFunc(func() { + // It does not matter if we commit or rollback the read transaction, + // but we should close it. + // ctx could be cancelled already. + _ = keepTx.Rollback(context.Background()) + })) cursor := h.cursors.NewCursor(ctx, &cursor.NewParams{ - Iter: iterator.WithClose(iter, closer.Close), + Iter: iterator.WithClose(iterator.Interface[struct{}, *types.Document](iter), closer.Close), DB: params.DB, Collection: params.Collection, Username: username, diff --git a/internal/handlers/pg/pgdb/pool.go b/internal/handlers/pg/pgdb/pool.go index cc652d066228..6ba1cea79ef6 100644 --- a/internal/handlers/pg/pgdb/pool.go +++ b/internal/handlers/pg/pgdb/pool.go @@ -57,8 +57,8 @@ func NewPool(ctx context.Context, uri string, logger *zap.Logger, p *state.Provi values := u.Query() if !values.Has("pool_max_conns") { - // it default to 4 which is too low for us - values.Set("pool_max_conns", "20") + // the default is too low + values.Set("pool_max_conns", "50") } values.Set("application_name", "FerretDB") diff --git a/internal/handlers/pg/pgdb/query_iterator.go b/internal/handlers/pg/pgdb/query_iterator.go index 4d86f30d0d5a..121eabd7e928 100644 --- a/internal/handlers/pg/pgdb/query_iterator.go +++ b/internal/handlers/pg/pgdb/query_iterator.go @@ -24,6 +24,7 @@ import ( "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/iterator" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" + "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/resource" ) @@ -70,6 +71,8 @@ func newIterator(ctx context.Context, rows pgx.Rows, p *iteratorParams) types.Do // // Otherwise, the next document is returned. func (iter *queryIterator) Next() (struct{}, *types.Document, error) { + defer observability.FuncCall(iter.ctx)() + iter.m.Lock() defer iter.m.Unlock() @@ -111,6 +114,8 @@ func (iter *queryIterator) Next() (struct{}, *types.Document, error) { // Close implements iterator.Interface. func (iter *queryIterator) Close() { + defer observability.FuncCall(iter.ctx)() + iter.m.Lock() defer iter.m.Unlock() @@ -121,6 +126,8 @@ func (iter *queryIterator) Close() { // // This should be called only when the caller already holds the mutex. func (iter *queryIterator) close() { + defer observability.FuncCall(iter.ctx)() + if iter.rows != nil { iter.rows.Close() iter.rows = nil diff --git a/internal/handlers/pg/pgdb/transactions.go b/internal/handlers/pg/pgdb/transactions.go index 0a96f892ba53..38179de7e059 100644 --- a/internal/handlers/pg/pgdb/transactions.go +++ b/internal/handlers/pg/pgdb/transactions.go @@ -54,6 +54,28 @@ func (e *transactionConflictError) Error() string { // so the caller needs to use errors.Is to check the error, // for example, errors.Is(err, ErrSchemaNotExist). func (pgPool *Pool) InTransaction(ctx context.Context, f func(pgx.Tx) error) (err error) { + var keepTx pgx.Tx + + err = pgPool.InTransactionKeep(ctx, func(tx pgx.Tx) error { + keepTx = tx + return f(tx) + }) + if err != nil { + err = lazyerrors.Error(err) + return + } + + err = keepTx.Commit(ctx) + if err != nil { + err = lazyerrors.Error(err) + _ = keepTx.Rollback(ctx) + } + + return +} + +// InTransactionKeep is a variant of InTransaction that keeps transaction open if there is no error. +func (pgPool *Pool) InTransactionKeep(ctx context.Context, f func(pgx.Tx) error) (err error) { var tx pgx.Tx if tx, err = pgPool.p.Begin(ctx); err != nil { @@ -61,7 +83,7 @@ func (pgPool *Pool) InTransaction(ctx context.Context, f func(pgx.Tx) error) (er return } - var committed bool + var done bool defer func() { // It is not enough to check `err == nil` there, @@ -69,22 +91,17 @@ func (pgPool *Pool) InTransaction(ctx context.Context, f func(pgx.Tx) error) (er // that call `runtime.Goexit()`, leaving `err` unset in `err = f(tx)` below. // This situation would hang a test. // - // As a bonus, checking a separate variable also handles any panics in `f`. - if committed { + // As a bonus, checking a separate variable also handles any panics in `f`, + // including `panic(nil)` that is problematic for tests too. + if done { return } - if rerr := tx.Rollback(ctx); rerr != nil { - pgPool.logger.Log( - ctx, tracelog.LogLevelError, "failed to perform rollback", - map[string]any{"err": rerr}, - ) - - // in case of `runtime.Goexit()` or `panic(nil)`; see above - if err == nil { - err = rerr - } + if err == nil { + err = lazyerrors.Errorf("transaction was not committed") } + + _ = tx.Rollback(ctx) }() if err = f(tx); err != nil { @@ -92,12 +109,7 @@ func (pgPool *Pool) InTransaction(ctx context.Context, f func(pgx.Tx) error) (er return } - if err = tx.Commit(ctx); err != nil { - err = lazyerrors.Error(err) - return - } - - committed = true + done = true return } diff --git a/internal/handlers/pg/pgdb/transactions_test.go b/internal/handlers/pg/pgdb/transactions_test.go index 15055554627c..d91313ae20e3 100644 --- a/internal/handlers/pg/pgdb/transactions_test.go +++ b/internal/handlers/pg/pgdb/transactions_test.go @@ -15,10 +15,12 @@ package pgdb import ( + "errors" "testing" "github.com/jackc/pgx/v5" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" "github.com/FerretDB/FerretDB/internal/util/testutil" ) @@ -39,3 +41,45 @@ func TestInTransaction(t *testing.T) { }) }) } + +func TestInTransactionKeep(t *testing.T) { + t.Parallel() + + ctx := testutil.Ctx(t) + pool := getPool(ctx, t) + + t.Run("Commit", func(t *testing.T) { + t.Parallel() + + var keepTx pgx.Tx + err := pool.InTransactionKeep(ctx, func(tx pgx.Tx) error { + keepTx = tx + return nil + }) + require.NoError(t, err) + + var res int + err = keepTx.QueryRow(ctx, "SELECT 1").Scan(&res) + require.NoError(t, keepTx.Commit(ctx)) + require.NoError(t, err) + assert.Equal(t, 1, res) + }) + + t.Run("Rollback", func(t *testing.T) { + t.Parallel() + + var keepTx pgx.Tx + err := pool.InTransactionKeep(ctx, func(tx pgx.Tx) error { + keepTx = tx + return errors.New("boom") + }) + require.Error(t, err) + + var res int + err = keepTx.QueryRow(ctx, "SELECT 1").Scan(&res) + require.Equal(t, pgx.ErrTxClosed, keepTx.Commit(ctx)) + require.Equal(t, pgx.ErrTxClosed, keepTx.Rollback(ctx)) + require.Equal(t, pgx.ErrTxClosed, err) + assert.Equal(t, 0, res) + }) +} diff --git a/internal/handlers/sqlite/msg_drop.go b/internal/handlers/sqlite/msg_drop.go index 7b9b00611265..88f6a88ae636 100644 --- a/internal/handlers/sqlite/msg_drop.go +++ b/internal/handlers/sqlite/msg_drop.go @@ -47,12 +47,11 @@ func (h *Handler) MsgDrop(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er return nil, err } - // This is mainly needed for SQLite and our cursor tests to avoid SQLITE_BUSY errors, - // but fine for other backends too. + // Most backends would block on `DropCollection` below otherwise. // // There is a race condition: another client could create a new cursor for that collection // after we closed all of them, but before we drop the collection itself. - // In that case, we expect the client to retry the operation. + // In that case, we expect the client to wait or to retry the operation. for _, c := range h.cursors.All() { if c.DB == dbName && c.Collection == collectionName { c.Close() diff --git a/internal/handlers/sqlite/msg_dropdatabase.go b/internal/handlers/sqlite/msg_dropdatabase.go index ae3cc22cc67e..68b196a77275 100644 --- a/internal/handlers/sqlite/msg_dropdatabase.go +++ b/internal/handlers/sqlite/msg_dropdatabase.go @@ -39,6 +39,17 @@ func (h *Handler) MsgDropDatabase(ctx context.Context, msg *wire.OpMsg) (*wire.O return nil, err } + // Most backends would block on `DropDatabase` below otherwise. + // + // There is a race condition: another client could create a new cursor for that database + // after we closed all of them, but before we drop the database itself. + // In that case, we expect the client to wait or to retry the operation. + for _, c := range h.cursors.All() { + if c.DB == dbName { + c.Close() + } + } + err = h.b.DropDatabase(ctx, &backends.DropDatabaseParams{ Name: dbName, }) diff --git a/internal/handlers/sqlite/msg_find.go b/internal/handlers/sqlite/msg_find.go index 233b383d96a6..f5dc33d46117 100644 --- a/internal/handlers/sqlite/msg_find.go +++ b/internal/handlers/sqlite/msg_find.go @@ -54,17 +54,16 @@ func (h *Handler) MsgFind(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, er ctx, cancel = context.WithTimeout(ctx, time.Duration(params.MaxTimeMS)*time.Millisecond) } + // closer accumulates all things that should be closed / canceled. + closer := iterator.NewMultiCloser(iterator.CloserFunc(cancel)) + queryRes, err := db.Collection(params.Collection).Query(ctx, nil) if err != nil { - cancel() + closer.Close() return nil, lazyerrors.Error(err) } - // closer accumulates all things that should be closed / canceled - closer := iterator.NewMultiCloser(queryRes.Iter) - - // to free maxTimeMS's context resources when they are not needed - closer.Add(iterator.CloserFunc(cancel)) + closer.Add(queryRes.Iter) iter := common.FilterIterator(queryRes.Iter, closer, params.Filter) From 5f0ddcbec884709343954cc386eacfd42eeb7570 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Thu, 29 Jun 2023 12:55:09 +0400 Subject: [PATCH 38/46] Add better metrics for connections (#2938) --- cmd/ferretdb/main.go | 24 +++++----- ferretdb/ferretdb.go | 2 +- integration/setup/listener.go | 7 +-- integration/setup/startup.go | 10 ++++- internal/clientconn/conn.go | 5 +++ .../clientconn/connmetrics/conn_metrics.go | 2 +- .../connmetrics/listener_metrics.go | 44 +++++++++++++------ internal/clientconn/cursor/registry.go | 24 +++++----- internal/clientconn/listener.go | 27 ++++++++---- internal/handlers/hana/hana.go | 2 +- internal/handlers/pg/msg_serverstatus.go | 2 +- internal/handlers/pg/pg.go | 2 +- internal/handlers/registry/hana.go | 2 +- internal/handlers/registry/pg.go | 2 +- internal/handlers/registry/registry.go | 2 +- internal/handlers/registry/sqlite.go | 2 +- internal/handlers/registry/tigris.go | 2 +- internal/handlers/sqlite/msg_serverstatus.go | 2 +- internal/handlers/sqlite/sqlite.go | 2 +- internal/handlers/tigris/msg_serverstatus.go | 2 +- internal/handlers/tigris/tigris.go | 2 +- internal/util/observability/funccall.go | 2 + 22 files changed, 103 insertions(+), 68 deletions(-) diff --git a/cmd/ferretdb/main.go b/cmd/ferretdb/main.go index ca9d3a716749..a21ed6f2b18f 100644 --- a/cmd/ferretdb/main.go +++ b/cmd/ferretdb/main.go @@ -203,17 +203,17 @@ func setupState() *state.Provider { // setupMetrics setups Prometheus metrics registerer with some metrics. func setupMetrics(stateProvider *state.Provider) prometheus.Registerer { - r := prometheus.WrapRegistererWith( - prometheus.Labels{"uuid": stateProvider.Get().UUID}, - prometheus.DefaultRegisterer, - ) - m := stateProvider.MetricsCollector(false) - - // Unless requested, don't add UUID to all metrics, but add it to one. - // See https://prometheus.io/docs/instrumenting/writing_exporters/#target-labels-not-static-scraped-labels - if !cli.MetricsUUID { - r = prometheus.DefaultRegisterer - m = stateProvider.MetricsCollector(true) + r := prometheus.DefaultRegisterer + m := stateProvider.MetricsCollector(true) + + // we don't do it by default due to + // https://prometheus.io/docs/instrumenting/writing_exporters/#target-labels-not-static-scraped-labels + if cli.MetricsUUID { + r = prometheus.WrapRegistererWith( + prometheus.Labels{"uuid": stateProvider.Get().UUID}, + prometheus.DefaultRegisterer, + ) + m = stateProvider.MetricsCollector(false) } r.MustRegister(m) @@ -355,7 +355,7 @@ func run() { h, err := registry.NewHandler(cli.Handler, ®istry.NewHandlerOpts{ Logger: logger, - Metrics: metrics.ConnMetrics, + ConnMetrics: metrics.ConnMetrics, StateProvider: stateProvider, PostgreSQLURL: pgFlags.PostgreSQLURL, diff --git a/ferretdb/ferretdb.go b/ferretdb/ferretdb.go index 43c4cdd70693..c0dee9d2f7af 100644 --- a/ferretdb/ferretdb.go +++ b/ferretdb/ferretdb.go @@ -122,7 +122,7 @@ func New(config *Config) (*FerretDB, error) { h, err := registry.NewHandler(config.Handler, ®istry.NewHandlerOpts{ Logger: logger, - Metrics: metrics.ConnMetrics, + ConnMetrics: metrics.ConnMetrics, StateProvider: p, PostgreSQLURL: config.PostgreSQLURL, diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 28fe876b3159..352c8ccc5f71 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -29,7 +29,6 @@ import ( "go.uber.org/zap" "github.com/FerretDB/FerretDB/internal/clientconn" - "github.com/FerretDB/FerretDB/internal/clientconn/connmetrics" "github.com/FerretDB/FerretDB/internal/handlers/registry" "github.com/FerretDB/FerretDB/internal/util/observability" "github.com/FerretDB/FerretDB/internal/util/state" @@ -157,11 +156,9 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) strin p, err := state.NewProvider("") require.NoError(tb, err) - metrics := connmetrics.NewListenerMetrics() - handlerOpts := ®istry.NewHandlerOpts{ Logger: logger, - Metrics: metrics.ConnMetrics, + ConnMetrics: listenerMetrics.ConnMetrics, StateProvider: p, PostgreSQLURL: *postgreSQLURLF, @@ -183,7 +180,7 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) strin listenerOpts := clientconn.NewListenerOpts{ ProxyAddr: *targetProxyAddrF, Mode: clientconn.NormalMode, - Metrics: metrics, + Metrics: listenerMetrics, Handler: h, Logger: logger, TestRecordsDir: filepath.Join("..", "tmp", "records"), diff --git a/integration/setup/startup.go b/integration/setup/startup.go index 124fb45714e8..7c7dd7f71bce 100644 --- a/integration/setup/startup.go +++ b/integration/setup/startup.go @@ -32,12 +32,16 @@ import ( "golang.org/x/exp/slices" "github.com/FerretDB/FerretDB/integration/shareddata" + "github.com/FerretDB/FerretDB/internal/clientconn/connmetrics" "github.com/FerretDB/FerretDB/internal/util/debug" "github.com/FerretDB/FerretDB/internal/util/logging" "github.com/FerretDB/FerretDB/internal/util/must" ) -// jaegerExporter is a global Jaeger exporter for tests. +// listenerMetrics are shared between tests. +var listenerMetrics = connmetrics.NewListenerMetrics() + +// jaegerExporter is a shared Jaeger exporter for tests. var jaegerExporter *jaeger.Exporter // sqliteDir is a fixed directory for SQLite backend tests. @@ -55,7 +59,9 @@ func Startup() { *debugSetupF = true } - // use any available port to allow running different configuration in parallel + prometheus.DefaultRegisterer.MustRegister(listenerMetrics) + + // use any available port to allow running different configurations in parallel go debug.RunHandler(context.Background(), "127.0.0.1:0", prometheus.DefaultRegisterer, zap.L().Named("debug")) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) diff --git a/internal/clientconn/conn.go b/internal/clientconn/conn.go index 0cbfd2cab02a..c220ad3ddee6 100644 --- a/internal/clientconn/conn.go +++ b/internal/clientconn/conn.go @@ -26,6 +26,7 @@ import ( "net" "os" "path/filepath" + "runtime/pprof" "sync/atomic" "time" @@ -555,6 +556,10 @@ func (c *conn) handleOpMsg(ctx context.Context, msg *wire.OpMsg, command string) // TODO move it to route, closer to Prometheus metrics defer observability.FuncCall(ctx)() + defer pprof.SetGoroutineLabels(ctx) + ctx = pprof.WithLabels(ctx, pprof.Labels("command", command)) + pprof.SetGoroutineLabels(ctx) + return cmd.Handler(c.h, ctx, msg) } } diff --git a/internal/clientconn/connmetrics/conn_metrics.go b/internal/clientconn/connmetrics/conn_metrics.go index 0c40a61ce661..a7bb56600278 100644 --- a/internal/clientconn/connmetrics/conn_metrics.go +++ b/internal/clientconn/connmetrics/conn_metrics.go @@ -24,7 +24,7 @@ import ( "github.com/FerretDB/FerretDB/internal/util/must" ) -// ConnMetrics represents conn metrics. +// ConnMetrics represents metrics of an individual conn or a collection of conns. type ConnMetrics struct { Requests *prometheus.CounterVec Responses *prometheus.CounterVec diff --git a/internal/clientconn/connmetrics/listener_metrics.go b/internal/clientconn/connmetrics/listener_metrics.go index aa487bd2a40a..f4bf74fbb7e2 100644 --- a/internal/clientconn/connmetrics/listener_metrics.go +++ b/internal/clientconn/connmetrics/listener_metrics.go @@ -14,7 +14,11 @@ package connmetrics -import "github.com/prometheus/client_golang/prometheus" +import ( + "time" + + "github.com/prometheus/client_golang/prometheus" +) const ( namespace = "ferretdb" @@ -23,22 +27,14 @@ const ( // ListenerMetrics represents listener metrics. type ListenerMetrics struct { - ConnectedClients prometheus.Gauge - Accepts *prometheus.CounterVec - ConnMetrics *ConnMetrics + Accepts *prometheus.CounterVec + Durations *prometheus.HistogramVec + ConnMetrics *ConnMetrics } // NewListenerMetrics creates new listener metrics. func NewListenerMetrics() *ListenerMetrics { return &ListenerMetrics{ - ConnectedClients: prometheus.NewGauge( - prometheus.GaugeOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "connected", - Help: "The current number of connected clients.", - }, - ), Accepts: prometheus.NewCounterVec( prometheus.CounterOpts{ Namespace: namespace, @@ -48,21 +44,41 @@ func NewListenerMetrics() *ListenerMetrics { }, []string{"error"}, ), + Durations: prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: namespace, + Subsystem: subsystem, + Name: "duration_seconds", + Help: "Client connection lifetime in seconds.", + Buckets: []float64{ + 1, + 5, + 10, + 30, + (1 * time.Minute).Seconds(), + (5 * time.Minute).Seconds(), + (10 * time.Minute).Seconds(), + (30 * time.Minute).Seconds(), + }, + }, + []string{"error"}, + ), + ConnMetrics: newConnMetrics(), } } // Describe implements prometheus.Collector. func (lm *ListenerMetrics) Describe(ch chan<- *prometheus.Desc) { - lm.ConnectedClients.Describe(ch) lm.Accepts.Describe(ch) + lm.Durations.Describe(ch) lm.ConnMetrics.Describe(ch) } // Collect implements prometheus.Collector. func (lm *ListenerMetrics) Collect(ch chan<- prometheus.Metric) { - lm.ConnectedClients.Collect(ch) lm.Accepts.Collect(ch) + lm.Durations.Collect(ch) lm.ConnMetrics.Collect(ch) } diff --git a/internal/clientconn/cursor/registry.go b/internal/clientconn/cursor/registry.go index aefa09614580..ca8497eae4b8 100644 --- a/internal/clientconn/cursor/registry.go +++ b/internal/clientconn/cursor/registry.go @@ -82,18 +82,18 @@ func NewRegistry(l *zap.Logger) *Registry { Name: "duration_seconds", Help: "Cursors lifetime in seconds.", Buckets: []float64{ - 1 * time.Millisecond.Seconds(), - 5 * time.Millisecond.Seconds(), - 10 * time.Millisecond.Seconds(), - 25 * time.Millisecond.Seconds(), - 50 * time.Millisecond.Seconds(), - 100 * time.Millisecond.Seconds(), - 250 * time.Millisecond.Seconds(), - 500 * time.Millisecond.Seconds(), - 1000 * time.Millisecond.Seconds(), - 2500 * time.Millisecond.Seconds(), - 5000 * time.Millisecond.Seconds(), - 10000 * time.Millisecond.Seconds(), + (1 * time.Millisecond).Seconds(), + (5 * time.Millisecond).Seconds(), + (10 * time.Millisecond).Seconds(), + (25 * time.Millisecond).Seconds(), + (50 * time.Millisecond).Seconds(), + (100 * time.Millisecond).Seconds(), + (250 * time.Millisecond).Seconds(), + (500 * time.Millisecond).Seconds(), + (1000 * time.Millisecond).Seconds(), + (2500 * time.Millisecond).Seconds(), + (5000 * time.Millisecond).Seconds(), + (10000 * time.Millisecond).Seconds(), }, }, []string{"db", "collection", "username"}, diff --git a/internal/clientconn/listener.go b/internal/clientconn/listener.go index 6eb674c41c02..0cce78f1db93 100644 --- a/internal/clientconn/listener.go +++ b/internal/clientconn/listener.go @@ -268,11 +268,18 @@ func acceptLoop(ctx context.Context, listener net.Listener, wg *sync.WaitGroup, wg.Add(1) l.Metrics.Accepts.WithLabelValues("0").Inc() - l.Metrics.ConnectedClients.Inc() go func() { + var connErr error + start := time.Now() + defer func() { - l.Metrics.ConnectedClients.Dec() + lv := "0" + if connErr != nil { + lv = "1" + } + + l.Metrics.Durations.WithLabelValues(lv).Observe(time.Since(start).Seconds()) netConn.Close() wg.Done() }() @@ -298,23 +305,25 @@ func acceptLoop(ctx context.Context, listener net.Listener, wg *sync.WaitGroup, mode: l.Mode, l: l.Logger.Named("// " + connID + " "), // derive from the original unnamed logger handler: l.Handler, - connMetrics: l.Metrics.ConnMetrics, + connMetrics: l.Metrics.ConnMetrics, // share between all conns proxyAddr: l.ProxyAddr, testRecordsDir: l.TestRecordsDir, } - conn, e := newConn(opts) - if e != nil { - logger.Warn("Failed to create connection", zap.String("conn", connID), zap.Error(e)) + + conn, connErr := newConn(opts) + if connErr != nil { + logger.Warn("Failed to create connection", zap.String("conn", connID), zap.Error(connErr)) return } logger.Info("Connection started", zap.String("conn", connID)) - e = conn.run(runCtx) - if errors.Is(e, wire.ErrZeroRead) { + connErr = conn.run(runCtx) + if errors.Is(connErr, wire.ErrZeroRead) { + connErr = nil logger.Info("Connection stopped", zap.String("conn", connID)) } else { - logger.Warn("Connection stopped", zap.String("conn", connID), zap.Error(e)) + logger.Warn("Connection stopped", zap.String("conn", connID), zap.Error(connErr)) } }() } diff --git a/internal/handlers/hana/hana.go b/internal/handlers/hana/hana.go index b6b48ecccbef..5a5f60e1743e 100644 --- a/internal/handlers/hana/hana.go +++ b/internal/handlers/hana/hana.go @@ -47,7 +47,7 @@ type Handler struct { // NewOpts represents handler configuration. type NewOpts struct { L *zap.Logger - Metrics *connmetrics.ConnMetrics + ConnMetrics *connmetrics.ConnMetrics StateProvider *state.Provider HANAURL string } diff --git a/internal/handlers/pg/msg_serverstatus.go b/internal/handlers/pg/msg_serverstatus.go index ed2ca2b1c456..3b287a5b1316 100644 --- a/internal/handlers/pg/msg_serverstatus.go +++ b/internal/handlers/pg/msg_serverstatus.go @@ -34,7 +34,7 @@ func (h *Handler) MsgServerStatus(ctx context.Context, msg *wire.OpMsg) (*wire.O return nil, lazyerrors.Error(err) } - res, err := common.ServerStatus(h.StateProvider.Get(), h.Metrics) + res, err := common.ServerStatus(h.StateProvider.Get(), h.ConnMetrics) if err != nil { return nil, lazyerrors.Error(err) } diff --git a/internal/handlers/pg/pg.go b/internal/handlers/pg/pg.go index 0c21060b1a3e..c337b61dc1f6 100644 --- a/internal/handlers/pg/pg.go +++ b/internal/handlers/pg/pg.go @@ -50,7 +50,7 @@ type NewOpts struct { PostgreSQLURL string L *zap.Logger - Metrics *connmetrics.ConnMetrics + ConnMetrics *connmetrics.ConnMetrics StateProvider *state.Provider // test options diff --git a/internal/handlers/registry/hana.go b/internal/handlers/registry/hana.go index 302ca9aba1fc..56ee05f7fffb 100644 --- a/internal/handlers/registry/hana.go +++ b/internal/handlers/registry/hana.go @@ -29,7 +29,7 @@ func init() { handlerOpts := &hana.NewOpts{ HANAURL: opts.HANAURL, L: opts.Logger, - Metrics: opts.Metrics, + ConnMetrics: opts.ConnMetrics, StateProvider: opts.StateProvider, } diff --git a/internal/handlers/registry/pg.go b/internal/handlers/registry/pg.go index e71f8fab2d52..143ca9f2d0f1 100644 --- a/internal/handlers/registry/pg.go +++ b/internal/handlers/registry/pg.go @@ -26,7 +26,7 @@ func init() { PostgreSQLURL: opts.PostgreSQLURL, L: opts.Logger, - Metrics: opts.Metrics, + ConnMetrics: opts.ConnMetrics, StateProvider: opts.StateProvider, DisableFilterPushdown: opts.DisableFilterPushdown, diff --git a/internal/handlers/registry/registry.go b/internal/handlers/registry/registry.go index a5a570781f9f..e898cd6bd40e 100644 --- a/internal/handlers/registry/registry.go +++ b/internal/handlers/registry/registry.go @@ -38,7 +38,7 @@ var registry = map[string]newHandlerFunc{} type NewHandlerOpts struct { // for all handlers Logger *zap.Logger - Metrics *connmetrics.ConnMetrics + ConnMetrics *connmetrics.ConnMetrics StateProvider *state.Provider // for `pg` handler diff --git a/internal/handlers/registry/sqlite.go b/internal/handlers/registry/sqlite.go index 246a05115237..e2bb55469378 100644 --- a/internal/handlers/registry/sqlite.go +++ b/internal/handlers/registry/sqlite.go @@ -28,7 +28,7 @@ func init() { Dir: opts.SQLiteURI, L: opts.Logger.Named("sqlite"), - Metrics: opts.Metrics, + ConnMetrics: opts.ConnMetrics, StateProvider: opts.StateProvider, DisableFilterPushdown: opts.DisableFilterPushdown, diff --git a/internal/handlers/registry/tigris.go b/internal/handlers/registry/tigris.go index 7da8c4de12e3..96561da2d50d 100644 --- a/internal/handlers/registry/tigris.go +++ b/internal/handlers/registry/tigris.go @@ -34,7 +34,7 @@ func init() { }, L: opts.Logger, - Metrics: opts.Metrics, + ConnMetrics: opts.ConnMetrics, StateProvider: opts.StateProvider, DisableFilterPushdown: opts.DisableFilterPushdown, diff --git a/internal/handlers/sqlite/msg_serverstatus.go b/internal/handlers/sqlite/msg_serverstatus.go index b5f2ebdfde3a..b30a70ae6759 100644 --- a/internal/handlers/sqlite/msg_serverstatus.go +++ b/internal/handlers/sqlite/msg_serverstatus.go @@ -26,7 +26,7 @@ import ( // MsgServerStatus implements HandlerInterface. func (h *Handler) MsgServerStatus(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - res, err := common.ServerStatus(h.StateProvider.Get(), h.Metrics) + res, err := common.ServerStatus(h.StateProvider.Get(), h.ConnMetrics) if err != nil { return nil, lazyerrors.Error(err) } diff --git a/internal/handlers/sqlite/sqlite.go b/internal/handlers/sqlite/sqlite.go index b01390ede4a4..c8ac3e77f916 100644 --- a/internal/handlers/sqlite/sqlite.go +++ b/internal/handlers/sqlite/sqlite.go @@ -54,7 +54,7 @@ type NewOpts struct { Dir string L *zap.Logger - Metrics *connmetrics.ConnMetrics + ConnMetrics *connmetrics.ConnMetrics StateProvider *state.Provider // test options diff --git a/internal/handlers/tigris/msg_serverstatus.go b/internal/handlers/tigris/msg_serverstatus.go index 615fff935181..ed1a0877536f 100644 --- a/internal/handlers/tigris/msg_serverstatus.go +++ b/internal/handlers/tigris/msg_serverstatus.go @@ -26,7 +26,7 @@ import ( // MsgServerStatus implements HandlerInterface. func (h *Handler) MsgServerStatus(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - res, err := common.ServerStatus(h.StateProvider.Get(), h.Metrics) + res, err := common.ServerStatus(h.StateProvider.Get(), h.ConnMetrics) if err != nil { return nil, lazyerrors.Error(err) } diff --git a/internal/handlers/tigris/tigris.go b/internal/handlers/tigris/tigris.go index 2d8ff602a80c..8da9016d14c4 100644 --- a/internal/handlers/tigris/tigris.go +++ b/internal/handlers/tigris/tigris.go @@ -51,7 +51,7 @@ type NewOpts struct { AuthParams L *zap.Logger - Metrics *connmetrics.ConnMetrics + ConnMetrics *connmetrics.ConnMetrics StateProvider *state.Provider // test options diff --git a/internal/util/observability/funccall.go b/internal/util/observability/funccall.go index a91ba0e7ed57..4bd9c65bacc9 100644 --- a/internal/util/observability/funccall.go +++ b/internal/util/observability/funccall.go @@ -56,6 +56,8 @@ func FuncCall(ctx context.Context) func() { } resource.Track(fc, fc.token) + // TODO add pprof labels + if trace.IsEnabled() { pc := make([]uintptr, 1) runtime.Callers(1, pc) From a5cc636fab2872bfd2099f7c7f0a58753f79c38c Mon Sep 17 00:00:00 2001 From: Chi Fujii Date: Fri, 30 Jun 2023 19:07:22 +0900 Subject: [PATCH 39/46] Use cursors with iterator in `aggregate` command (#2929) Closes #2877. Closes #1733. --- .../aggregate_documents_compat_test.go | 64 +++- integration/aggregate_documents_test.go | 31 +- integration/helpers.go | 11 +- integration/indexes_compat_test.go | 20 +- integration/query_test.go | 316 ++++++++++-------- .../common/aggregations/stages/collstats.go | 5 +- .../common/aggregations/stages/group.go | 5 +- .../common/aggregations/stages/unwind.go | 5 +- internal/handlers/pg/msg_aggregate.go | 72 ++-- 9 files changed, 328 insertions(+), 201 deletions(-) diff --git a/integration/aggregate_documents_compat_test.go b/integration/aggregate_documents_compat_test.go index d36cf7e673ae..5edea47e3401 100644 --- a/integration/aggregate_documents_compat_test.go +++ b/integration/aggregate_documents_compat_test.go @@ -19,10 +19,13 @@ import ( "testing" "time" + "github.com/AlekSi/pointer" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.mongodb.org/mongo-driver/bson" "go.mongodb.org/mongo-driver/bson/primitive" + "go.mongodb.org/mongo-driver/mongo" + "go.mongodb.org/mongo-driver/mongo/options" "github.com/FerretDB/FerretDB/integration/setup" "github.com/FerretDB/FerretDB/integration/shareddata" @@ -30,12 +33,13 @@ import ( // aggregateStagesCompatTestCase describes aggregation stages compatibility test case. type aggregateStagesCompatTestCase struct { - pipeline bson.A // required, unspecified $sort appends bson.D{{"$sort", bson.D{{"_id", 1}}}} for non empty pipeline. + pipeline bson.A // required, unspecified $sort appends bson.D{{"$sort", bson.D{{"_id", 1}}}} for non empty pipeline. + maxTime *time.Duration // optional, leave nil for unset maxTime + resultType compatTestCaseResultType // defaults to nonEmptyResult resultPushdown bool // defaults to false - - skip string // skip test for all handlers, must have issue number mentioned - skipForTigris string // skip test for Tigris handler, must have issue number mentioned + skip string // skip test for all handlers, must have issue number mentioned + skipForTigris string // skip test for Tigris handler, must have issue number mentioned } // testAggregateStagesCompat tests aggregation stages compatibility test cases with all providers. @@ -92,6 +96,12 @@ func testAggregateStagesCompatWithProviders(t *testing.T, providers shareddata.P pipeline = append(pipeline, bson.D{{"$sort", bson.D{{"_id", 1}}}}) } + opts := options.Aggregate() + + if tc.maxTime != nil { + opts.SetMaxTime(*tc.maxTime) + } + var nonEmptyResults bool for i := range targetCollections { targetCollection := targetCollections[i] @@ -114,8 +124,8 @@ func testAggregateStagesCompatWithProviders(t *testing.T, providers shareddata.P assert.Equal(t, tc.resultPushdown, explainRes.Map()["pushdown"], msg) - targetCursor, targetErr := targetCollection.Aggregate(ctx, pipeline) - compatCursor, compatErr := compatCollection.Aggregate(ctx, pipeline) + targetCursor, targetErr := targetCollection.Aggregate(ctx, pipeline, opts) + compatCursor, compatErr := compatCollection.Aggregate(ctx, pipeline, opts) if targetCursor != nil { defer targetCursor.Close(ctx) @@ -207,8 +217,13 @@ func testAggregateCommandCompat(t *testing.T, testCases map[string]aggregateComm t.Logf("Target error: %v", targetErr) t.Logf("Compat error: %v", compatErr) - // error messages are intentionally not compared - AssertMatchesCommandError(t, compatErr, targetErr) + if _, ok := targetErr.(mongo.CommandError); ok { //nolint:errorlint // do not inspect error chain + // error messages are intentionally not compared + AssertMatchesCommandError(t, compatErr, targetErr) + } else { + // driver sent an error + require.Equal(t, compatErr, targetErr) + } return } @@ -270,11 +285,44 @@ func TestAggregateCommandCompat(t *testing.T) { }, resultType: emptyResult, }, + "MaxTimeMSNegative": { + command: bson.D{ + {"aggregate", "collection-name"}, + {"pipeline", bson.A{}}, + {"maxTimeMS", int64(-1)}, + {"cursor", bson.D{}}, + }, + resultType: emptyResult, + // compat and target return an error from the driver + // > cannot decode document into []primitive.D + }, } testAggregateCommandCompat(t, testCases) } +func TestAggregateCompatOptions(t *testing.T) { + t.Parallel() + + providers := []shareddata.Provider{ + // one provider is sufficient to test aggregate options + shareddata.Unsets, + } + + testCases := map[string]aggregateStagesCompatTestCase{ + "MaxTimeZero": { + pipeline: bson.A{}, + maxTime: pointer.ToDuration(time.Duration(0)), + }, + "MaxTime": { + pipeline: bson.A{}, + maxTime: pointer.ToDuration(time.Second), + }, + } + + testAggregateStagesCompatWithProviders(t, providers, testCases) +} + func TestAggregateCompatStages(t *testing.T) { setup.SkipForTigrisWithReason(t, "https://github.com/FerretDB/FerretDB/issues/2523") diff --git a/integration/aggregate_documents_test.go b/integration/aggregate_documents_test.go index 84afdacfa52c..5c2e05b3978d 100644 --- a/integration/aggregate_documents_test.go +++ b/integration/aggregate_documents_test.go @@ -17,7 +17,6 @@ package integration import ( "testing" - "github.com/AlekSi/pointer" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.mongodb.org/mongo-driver/bson" @@ -669,8 +668,8 @@ func TestAggregateCommandCursor(t *testing.T) { // the number of documents is set above the default batchSize of 101 // for testing unset batchSize returning default batchSize - docs := generateDocuments(0, 110) - _, err := collection.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 110) + _, err := collection.InsertMany(ctx, arr) require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only @@ -684,11 +683,11 @@ func TestAggregateCommandCursor(t *testing.T) { }{ "Int": { cursor: bson.D{{"batchSize", 1}}, - firstBatch: docs[:1], + firstBatch: arr[:1], }, "Long": { cursor: bson.D{{"batchSize", int64(2)}}, - firstBatch: docs[:2], + firstBatch: arr[:2], }, "LongZero": { cursor: bson.D{{"batchSize", int64(0)}}, @@ -717,11 +716,11 @@ func TestAggregateCommandCursor(t *testing.T) { }, "DoubleFloor": { cursor: bson.D{{"batchSize", 1.9}}, - firstBatch: docs[:1], + firstBatch: arr[:1], }, "Bool": { cursor: bson.D{{"batchSize", true}}, - firstBatch: docs[:1], + firstBatch: arr[:1], err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -731,7 +730,7 @@ func TestAggregateCommandCursor(t *testing.T) { }, "Unset": { cursor: nil, - firstBatch: docs[:101], + firstBatch: arr[:101], err: &mongo.CommandError{ Code: 9, Name: "FailedToParse", @@ -740,11 +739,11 @@ func TestAggregateCommandCursor(t *testing.T) { }, "Empty": { cursor: bson.D{}, - firstBatch: docs[:101], + firstBatch: arr[:101], }, "String": { cursor: "invalid", - firstBatch: docs[:101], + firstBatch: arr[:101], err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -754,14 +753,14 @@ func TestAggregateCommandCursor(t *testing.T) { }, "LargeBatchSize": { cursor: bson.D{{"batchSize", 102}}, - firstBatch: docs[:102], + firstBatch: arr[:102], }, "LargeBatchSizeMatch": { pipeline: bson.A{ bson.D{{"$match", bson.D{{"_id", bson.D{{"$in", bson.A{0, 1, 2, 3, 4, 5}}}}}}}, }, cursor: bson.D{{"batchSize", 102}}, - firstBatch: docs[:6], + firstBatch: arr[:6], }, } { name, tc := name, tc @@ -826,14 +825,14 @@ func TestAggregateBatchSize(t *testing.T) { // The batchSize set by `aggregate` is used also by `getMore` unless // `aggregate` has default batchSize or 0 batchSize, then `getMore` has unlimited batchSize. // To test that, the number of documents is set to more than the double of default batchSize 101. - docs := generateDocuments(0, 220) - _, err := collection.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 220) + _, err := collection.InsertMany(ctx, arr) require.NoError(t, err) t.Run("SetBatchSize", func(t *testing.T) { t.Parallel() - cursor, err := collection.Aggregate(ctx, bson.D{}, &options.AggregateOptions{BatchSize: pointer.ToInt32(2)}) + cursor, err := collection.Aggregate(ctx, bson.D{}, options.Aggregate().SetBatchSize(2)) require.NoError(t, err) defer cursor.Close(ctx) @@ -873,7 +872,7 @@ func TestAggregateBatchSize(t *testing.T) { t.Run("ZeroBatchSize", func(t *testing.T) { t.Parallel() - cursor, err := collection.Aggregate(ctx, bson.D{}, &options.AggregateOptions{BatchSize: pointer.ToInt32(0)}) + cursor, err := collection.Aggregate(ctx, bson.D{}, options.Aggregate().SetBatchSize(0)) require.NoError(t, err) defer cursor.Close(ctx) diff --git a/integration/helpers.go b/integration/helpers.go index 8e25a7a0425d..c74a7f351fe8 100644 --- a/integration/helpers.go +++ b/integration/helpers.go @@ -428,12 +428,15 @@ func FindAll(t testing.TB, ctx context.Context, collection *mongo.Collection) [] } // generateDocuments generates documents with _id ranging from startID to endID. -// It returns bson.A containing bson.D documents. -func generateDocuments(startID, endID int32) bson.A { - var docs bson.A +// It returns bson.A and []bson.D both containing same bson.D documents. +func generateDocuments(startID, endID int32) (bson.A, []bson.D) { + var arr bson.A + var docs []bson.D + for i := startID; i < endID; i++ { + arr = append(arr, bson.D{{"_id", i}}) docs = append(docs, bson.D{{"_id", i}}) } - return docs + return arr, docs } diff --git a/integration/indexes_compat_test.go b/integration/indexes_compat_test.go index f6b9a10daa04..928cdb525c62 100644 --- a/integration/indexes_compat_test.go +++ b/integration/indexes_compat_test.go @@ -141,7 +141,7 @@ func TestCreateIndexesCompat(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"foo", 1}, {"bar", -1}}, - Options: new(options.IndexOptions).SetName("custom-name"), + Options: options.Index().SetName("custom-name"), }, }, }, @@ -200,11 +200,11 @@ func TestCreateIndexesCompat(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"v", -1}}, - Options: new(options.IndexOptions).SetName("foo"), + Options: options.Index().SetName("foo"), }, { Keys: bson.D{{"v", -1}}, - Options: new(options.IndexOptions).SetName("bar"), + Options: options.Index().SetName("bar"), }, }, resultType: emptyResult, @@ -213,11 +213,11 @@ func TestCreateIndexesCompat(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"foo", -1}}, - Options: new(options.IndexOptions).SetName("index-name"), + Options: options.Index().SetName("index-name"), }, { Keys: bson.D{{"bar", -1}}, - Options: new(options.IndexOptions).SetName("index-name"), + Options: options.Index().SetName("index-name"), }, }, resultType: emptyResult, @@ -830,7 +830,7 @@ func TestCreateIndexesCompatUnique(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"_id", 1}}, - Options: new(options.IndexOptions).SetUnique(true), + Options: options.Index().SetUnique(true), }, }, insertDoc: bson.D{{"_id", "int322"}}, @@ -839,7 +839,7 @@ func TestCreateIndexesCompatUnique(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"v", 1}}, - Options: new(options.IndexOptions).SetUnique(true), + Options: options.Index().SetUnique(true), }, }, insertDoc: bson.D{{"v", "value"}}, @@ -849,7 +849,7 @@ func TestCreateIndexesCompatUnique(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"not-existing-field", 1}}, - Options: new(options.IndexOptions).SetUnique(true), + Options: options.Index().SetUnique(true), }, }, insertDoc: bson.D{{"not-existing-field", "value"}}, @@ -859,7 +859,7 @@ func TestCreateIndexesCompatUnique(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"v", 1}, {"foo", 1}}, - Options: new(options.IndexOptions).SetUnique(true), + Options: options.Index().SetUnique(true), }, }, insertDoc: bson.D{{"v", "baz"}, {"foo", "bar"}}, @@ -868,7 +868,7 @@ func TestCreateIndexesCompatUnique(t *testing.T) { models: []mongo.IndexModel{ { Keys: bson.D{{"v", 1}}, - Options: new(options.IndexOptions).SetUnique(true), + Options: options.Index().SetUnique(true), }, }, insertDoc: bson.D{{"v", int32(42)}}, diff --git a/integration/query_test.go b/integration/query_test.go index daafc5d63d8f..ed582f19cfc0 100644 --- a/integration/query_test.go +++ b/integration/query_test.go @@ -31,6 +31,7 @@ import ( "github.com/FerretDB/FerretDB/integration/setup" "github.com/FerretDB/FerretDB/integration/shareddata" "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/must" ) func TestQueryBadFindType(t *testing.T) { @@ -623,8 +624,8 @@ func TestQueryCommandBatchSize(t *testing.T) { // the number of documents is set above the default batchSize of 101 // for testing unset batchSize returning default batchSize - docs := generateDocuments(0, 110) - _, err := collection.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 110) + _, err := collection.InsertMany(ctx, arr) require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only @@ -638,11 +639,11 @@ func TestQueryCommandBatchSize(t *testing.T) { }{ "Int": { batchSize: 1, - firstBatch: docs[:1], + firstBatch: arr[:1], }, "Long": { batchSize: int64(2), - firstBatch: docs[:2], + firstBatch: arr[:2], }, "LongZero": { batchSize: int64(0), @@ -671,11 +672,11 @@ func TestQueryCommandBatchSize(t *testing.T) { }, "DoubleFloor": { batchSize: 1.9, - firstBatch: docs[:1], + firstBatch: arr[:1], }, "Bool": { batchSize: true, - firstBatch: docs[:1], + firstBatch: arr[:1], err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -686,16 +687,16 @@ func TestQueryCommandBatchSize(t *testing.T) { "Unset": { // default batchSize is 101 when unset batchSize: nil, - firstBatch: docs[:101], + firstBatch: arr[:101], }, "LargeBatchSize": { batchSize: 102, - firstBatch: docs[:102], + firstBatch: arr[:102], }, "LargeBatchSizeFilter": { filter: bson.D{{"_id", bson.D{{"$in", bson.A{0, 1, 2, 3, 4, 5}}}}}, batchSize: 102, - firstBatch: docs[:6], + firstBatch: arr[:6], }, } { name, tc := name, tc @@ -752,8 +753,8 @@ func TestQueryCommandSingleBatch(t *testing.T) { t.Parallel() ctx, collection := setup.Setup(t) - docs := generateDocuments(0, 5) - _, err := collection.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 5) + _, err := collection.InsertMany(ctx, arr) require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only @@ -846,8 +847,8 @@ func TestQueryBatchSize(t *testing.T) { // The batchSize set by `find` is used also by `getMore` unless // `find` has default batchSize or 0 batchSize, then `getMore` has unlimited batchSize. // To test that, the number of documents is set to more than the double of default batchSize 101. - docs := generateDocuments(0, 220) - _, err := collection.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 220) + _, err := collection.InsertMany(ctx, arr) require.NoError(t, err) t.Run("SetBatchSize", func(t *testing.T) { @@ -972,34 +973,35 @@ func TestQueryCommandGetMore(t *testing.T) { // the number of documents is set above the default batchSize of 101 // for testing unset batchSize returning default batchSize - docs := generateDocuments(0, 110) - _, err := collection.InsertMany(ctx, docs) + bsonArr, arr := generateDocuments(0, 110) + + _, err := collection.InsertMany(ctx, bsonArr) require.NoError(t, err) for name, tc := range map[string]struct { //nolint:vet // used for testing only - findBatchSize any // optional, nil to leave findBatchSize unset + firstBatchSize any // optional, nil to leave firstBatchSize unset getMoreBatchSize any // optional, nil to leave getMoreBatchSize unset collection any // optional, nil to leave collection unset cursorID any // optional, defaults to cursorID from find() - firstBatch primitive.A // required, expected find firstBatch - nextBatch primitive.A // optional, expected getMore nextBatch + firstBatch []*types.Document // required, expected find firstBatch + nextBatch []*types.Document // optional, expected getMore nextBatch err *mongo.CommandError // optional, expected error from MongoDB altMessage string // optional, alternative error message for FerretDB, ignored if empty skip string // optional, skip test with a specified reason }{ "Int": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int32(1), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:2], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:2]), }, "IntNegative": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int32(-1), collection: collection.Name(), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 51024, Name: "Location51024", @@ -1007,24 +1009,24 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "IntZero": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int32(0), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:]), }, "Long": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int64(1), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:2], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:2]), }, "LongNegative": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int64(-1), collection: collection.Name(), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 51024, Name: "Location51024", @@ -1032,24 +1034,24 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "LongZero": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: int64(0), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:]), }, "Double": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: float64(1), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:2], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:2]), }, "DoubleNegative": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: float64(-1), collection: collection.Name(), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 51024, Name: "Location51024", @@ -1057,24 +1059,24 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "DoubleZero": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: float64(0), collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:]), }, "DoubleFloor": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1.9, collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:2], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:2]), }, "GetMoreCursorExhausted": { - findBatchSize: 200, + firstBatchSize: 200, getMoreBatchSize: int32(1), collection: collection.Name(), - firstBatch: docs[:110], + firstBatch: ConvertDocuments(t, arr[:110]), err: &mongo.CommandError{ Code: 43, Name: "CursorNotFound", @@ -1082,10 +1084,10 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "Bool": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: false, collection: collection.Name(), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -1094,26 +1096,26 @@ func TestQueryCommandGetMore(t *testing.T) { altMessage: "BSON field 'getMore.batchSize' is the wrong type 'bool', expected types '[long, int, decimal, double]'", }, "Unset": { - findBatchSize: 1, + firstBatchSize: 1, // unset getMore batchSize gets all remaining documents getMoreBatchSize: nil, collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:]), }, "LargeBatchSize": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 105, collection: collection.Name(), - firstBatch: docs[:1], - nextBatch: docs[1:106], + firstBatch: ConvertDocuments(t, arr[:1]), + nextBatch: ConvertDocuments(t, arr[1:106]), }, "StringCursorID": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: collection.Name(), cursorID: "invalid", - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -1122,11 +1124,11 @@ func TestQueryCommandGetMore(t *testing.T) { altMessage: "BSON field 'getMore.getMore' is the wrong type, expected type 'long'", }, "Int32CursorID": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: collection.Name(), cursorID: int32(1111), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -1135,11 +1137,11 @@ func TestQueryCommandGetMore(t *testing.T) { altMessage: "BSON field 'getMore.getMore' is the wrong type, expected type 'long'", }, "NotFoundCursorID": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: collection.Name(), cursorID: int64(1234), - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 43, Name: "CursorNotFound", @@ -1147,10 +1149,10 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "WrongTypeNamespace": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: bson.D{}, - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 14, Name: "TypeMismatch", @@ -1158,10 +1160,10 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "InvalidNamespace": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: "invalid", - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 13, Name: "Unauthorized", @@ -1170,10 +1172,10 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "EmptyCollectionName": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: "", - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 73, Name: "InvalidNamespace", @@ -1181,10 +1183,10 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "MissingCollectionName": { - findBatchSize: 1, + firstBatchSize: 1, getMoreBatchSize: 1, collection: nil, - firstBatch: docs[:1], + firstBatch: ConvertDocuments(t, arr[:1]), err: &mongo.CommandError{ Code: 40414, Name: "Location40414", @@ -1192,32 +1194,32 @@ func TestQueryCommandGetMore(t *testing.T) { }, }, "UnsetAllBatchSize": { - findBatchSize: nil, + firstBatchSize: nil, getMoreBatchSize: nil, collection: collection.Name(), - firstBatch: docs[:101], - nextBatch: docs[101:], + firstBatch: ConvertDocuments(t, arr[:101]), + nextBatch: ConvertDocuments(t, arr[101:]), }, "UnsetFindBatchSize": { - findBatchSize: nil, + firstBatchSize: nil, getMoreBatchSize: 5, collection: collection.Name(), - firstBatch: docs[:101], - nextBatch: docs[101:106], + firstBatch: ConvertDocuments(t, arr[:101]), + nextBatch: ConvertDocuments(t, arr[101:106]), }, "UnsetGetMoreBatchSize": { - findBatchSize: 5, + firstBatchSize: 5, getMoreBatchSize: nil, collection: collection.Name(), - firstBatch: docs[:5], - nextBatch: docs[5:], + firstBatch: ConvertDocuments(t, arr[:5]), + nextBatch: ConvertDocuments(t, arr[5:]), }, "BatchSize": { - findBatchSize: 3, + firstBatchSize: 3, getMoreBatchSize: 5, collection: collection.Name(), - firstBatch: docs[:3], - nextBatch: docs[3:8], + firstBatch: ConvertDocuments(t, arr[:3]), + nextBatch: ConvertDocuments(t, arr[3:8]), }, } { name, tc := name, tc @@ -1239,8 +1241,17 @@ func TestQueryCommandGetMore(t *testing.T) { require.NotNil(t, tc.firstBatch, "firstBatch must not be nil") var findRest bson.D - if tc.findBatchSize != nil { - findRest = append(findRest, bson.E{Key: "batchSize", Value: tc.findBatchSize}) + aggregateCursor := bson.D{} + + if tc.firstBatchSize != nil { + findRest = append(findRest, bson.E{Key: "batchSize", Value: tc.firstBatchSize}) + aggregateCursor = bson.D{{"batchSize", tc.firstBatchSize}} + } + + aggregateCommand := bson.D{ + {"aggregate", collection.Name()}, + {"pipeline", bson.A{}}, + {"cursor", aggregateCursor}, } findCommand := append( @@ -1248,78 +1259,107 @@ func TestQueryCommandGetMore(t *testing.T) { findRest..., ) - var res bson.D - err := collection.Database().RunCommand(ctx, findCommand).Decode(&res) - require.NoError(t, err) + for _, command := range []bson.D{findCommand, aggregateCommand} { + var res bson.D + err := collection.Database().RunCommand(ctx, command).Decode(&res) + require.NoError(t, err) - v, ok := res.Map()["cursor"] - require.True(t, ok) + doc := ConvertDocument(t, res) - cursor, ok := v.(bson.D) - require.True(t, ok) + v, _ := doc.Get("cursor") + require.NotNil(t, v) - cursorID := cursor.Map()["id"] - assert.NotNil(t, cursorID) + cursor, ok := v.(*types.Document) + require.True(t, ok) - firstBatch, ok := cursor.Map()["firstBatch"] - require.True(t, ok) - require.Equal(t, tc.firstBatch, firstBatch) + cursorID, _ := cursor.Get("id") + assert.NotNil(t, cursorID) - if tc.cursorID != nil { - cursorID = tc.cursorID - } + v, _ = cursor.Get("firstBatch") + require.NotNil(t, v) - var getMoreRest bson.D - if tc.getMoreBatchSize != nil { - getMoreRest = append(getMoreRest, bson.E{Key: "batchSize", Value: tc.getMoreBatchSize}) - } + firstBatch, ok := v.(*types.Array) + require.True(t, ok) - if tc.collection != nil { - getMoreRest = append(getMoreRest, bson.E{Key: "collection", Value: tc.collection}) - } + require.Equal(t, len(tc.firstBatch), firstBatch.Len(), "expected: %v, got: %v", tc.firstBatch, firstBatch) + for i, elem := range tc.firstBatch { + require.Equal(t, elem, must.NotFail(firstBatch.Get(i))) + } - getMoreCommand := append( - bson.D{ - {"getMore", cursorID}, - }, - getMoreRest..., - ) + if tc.cursorID != nil { + cursorID = tc.cursorID + } - err = collection.Database().RunCommand(ctx, getMoreCommand).Decode(&res) - if tc.err != nil { - AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) + var getMoreRest bson.D + if tc.getMoreBatchSize != nil { + getMoreRest = append(getMoreRest, bson.E{Key: "batchSize", Value: tc.getMoreBatchSize}) + } - // upon error response contains firstBatch field. - v, ok = res.Map()["cursor"] - require.True(t, ok) + if tc.collection != nil { + getMoreRest = append(getMoreRest, bson.E{Key: "collection", Value: tc.collection}) + } - cursor, ok = v.(bson.D) - require.True(t, ok) + getMoreCommand := append( + bson.D{ + {"getMore", cursorID}, + }, + getMoreRest..., + ) - cursorID = cursor.Map()["id"] - assert.NotNil(t, cursorID) + err = collection.Database().RunCommand(ctx, getMoreCommand).Decode(&res) + if tc.err != nil { + AssertEqualAltCommandError(t, *tc.err, tc.altMessage, err) - firstBatch, ok = cursor.Map()["firstBatch"] - require.True(t, ok) - require.Equal(t, tc.firstBatch, firstBatch) + // upon error response contains firstBatch field. + doc = ConvertDocument(t, res) - return - } + v, _ = doc.Get("cursor") + require.NotNil(t, v) - require.NoError(t, err) + cursor, ok = v.(*types.Document) + require.True(t, ok) - v, ok = res.Map()["cursor"] - require.True(t, ok) + cursorID, _ = cursor.Get("id") + assert.NotNil(t, cursorID) - cursor, ok = v.(bson.D) - require.True(t, ok) + v, _ = cursor.Get("firstBatch") + require.NotNil(t, v) - cursorID = cursor.Map()["id"] - assert.NotNil(t, cursorID) + firstBatch, ok = v.(*types.Array) + require.True(t, ok) - nextBatch, ok := cursor.Map()["nextBatch"] - require.True(t, ok) - require.Equal(t, tc.nextBatch, nextBatch) + require.Equal(t, len(tc.firstBatch), firstBatch.Len(), "expected: %v, got: %v", tc.firstBatch, firstBatch) + for i, elem := range tc.firstBatch { + require.Equal(t, elem, must.NotFail(firstBatch.Get(i))) + } + + return + } + + require.NoError(t, err) + + doc = ConvertDocument(t, res) + + v, _ = doc.Get("cursor") + require.NotNil(t, v) + + cursor, ok = v.(*types.Document) + require.True(t, ok) + + cursorID, _ = cursor.Get("id") + assert.NotNil(t, cursorID) + + v, _ = cursor.Get("nextBatch") + require.NotNil(t, v) + + nextBatch, ok := v.(*types.Array) + require.True(t, ok) + + require.Equal(t, len(tc.nextBatch), nextBatch.Len(), "expected: %v, got: %v", tc.nextBatch, nextBatch) + for i, elem := range tc.nextBatch { + require.Equal(t, elem, must.NotFail(nextBatch.Get(i))) + } + } }) } } @@ -1341,8 +1381,8 @@ func TestQueryCommandGetMoreConnection(t *testing.T) { databaseName := s.Collection.Database().Name() collectionName := s.Collection.Name() - docs := generateDocuments(0, 5) - _, err := collection1.InsertMany(ctx, docs) + arr, _ := generateDocuments(0, 5) + _, err := collection1.InsertMany(ctx, arr) require.NoError(t, err) t.Run("SameClient", func(t *testing.T) { diff --git a/internal/handlers/common/aggregations/stages/collstats.go b/internal/handlers/common/aggregations/stages/collstats.go index a23ac3558f1e..ed8761ba65b8 100644 --- a/internal/handlers/common/aggregations/stages/collstats.go +++ b/internal/handlers/common/aggregations/stages/collstats.go @@ -118,7 +118,10 @@ func (c *collStats) Process(ctx context.Context, iter types.DocumentsIterator, c panic("collStatsStage: Process: expected 1 document, got more") } - return iterator.Values(iterator.ForSlice([]*types.Document{res})), nil + iter = iterator.Values(iterator.ForSlice([]*types.Document{res})) + closer.Add(iter) + + return iter, nil } // Type implements Stage interface. diff --git a/internal/handlers/common/aggregations/stages/group.go b/internal/handlers/common/aggregations/stages/group.go index d12bdb316542..118a371e92b7 100644 --- a/internal/handlers/common/aggregations/stages/group.go +++ b/internal/handlers/common/aggregations/stages/group.go @@ -157,7 +157,10 @@ func (g *group) Process(ctx context.Context, iter types.DocumentsIterator, close res = append(res, doc) } - return iterator.Values(iterator.ForSlice(res)), nil + iter = iterator.Values(iterator.ForSlice(res)) + closer.Add(iter) + + return iter, nil } // groupDocuments groups documents by group expression. diff --git a/internal/handlers/common/aggregations/stages/unwind.go b/internal/handlers/common/aggregations/stages/unwind.go index aeb8d581177a..0be1f1a41999 100644 --- a/internal/handlers/common/aggregations/stages/unwind.go +++ b/internal/handlers/common/aggregations/stages/unwind.go @@ -154,7 +154,10 @@ func (u *unwind) Process(ctx context.Context, iter types.DocumentsIterator, clos } } - return iterator.Values(iterator.ForSlice(out)), nil + iter = iterator.Values(iterator.ForSlice(out)) + closer.Add(iter) + + return iter, nil } // Type implements Stage interface. diff --git a/internal/handlers/pg/msg_aggregate.go b/internal/handlers/pg/msg_aggregate.go index 613effc18aaa..04235f863198 100644 --- a/internal/handlers/pg/msg_aggregate.go +++ b/internal/handlers/pg/msg_aggregate.go @@ -57,7 +57,7 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs common.Ignored( document, h.L, - "allowDiskUse", "maxTimeMS", "bypassDocumentValidation", "readConcern", "hint", "comment", "writeConcern", + "allowDiskUse", "bypassDocumentValidation", "readConcern", "hint", "comment", "writeConcern", ) var db string @@ -86,6 +86,17 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs username, _ := conninfo.Get(ctx).Auth() + v, _ := document.Get("maxTimeMS") + if v == nil { + v = int64(0) + } + + maxTimeMS, err := commonparams.GetValidatedNumberParamWithMinValue(document.Command(), "maxTimeMS", v, 0) + if err != nil { + // unreachable for MongoDB GO driver, it validates maxTimeMS parameter + return nil, lazyerrors.Error(err) + } + pipeline, err := common.GetRequiredParam[*types.Array](document, "pipeline") if err != nil { return nil, commonerrors.NewCommandErrorMsgWithArgument( @@ -135,7 +146,7 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs } // validate cursor after validating pipeline stages to keep compatibility - v, _ := document.Get("cursor") + v, _ = document.Get("cursor") if v == nil { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrFailedToParse, @@ -166,7 +177,16 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs return nil, err } - var resDocs []*types.Document + cancel := func() {} + if maxTimeMS != 0 { + // It is not clear if maxTimeMS affects only aggregate, or both aggregate and getMore (as the current code does). + // TODO https://github.com/FerretDB/FerretDB/issues/1808 + ctx, cancel = context.WithTimeout(ctx, time.Duration(maxTimeMS)*time.Millisecond) + } + + closer := iterator.NewMultiCloser(iterator.CloserFunc(cancel)) + + var iter iterator.Interface[struct{}, *types.Document] // At this point we have a list of stages to apply to the documents or stats. // If stagesStats contains the same stages as stagesDocuments, we apply aggregation to documents fetched from the DB. @@ -188,23 +208,25 @@ func (h *Handler) MsgAggregate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMs qp.Sort = sort } - // TODO https://github.com/FerretDB/FerretDB/issues/1733 - resDocs, err = processStagesDocuments(ctx, &stagesDocumentsParams{dbPool, &qp, stagesDocuments}) + iter, err = processStagesDocuments(ctx, closer, &stagesDocumentsParams{dbPool, &qp, stagesDocuments}) } else { // stats stages are provided - fetch stats from the DB and apply stages to them statistics := stages.GetStatistics(stagesStats) - resDocs, err = processStagesStats(ctx, &stagesStatsParams{ + iter, err = processStagesStats(ctx, closer, &stagesStatsParams{ dbPool, db, collection, statistics, stagesStats, }) } if err != nil { + closer.Close() return nil, err } + closer.Add(iter) + cursor := h.cursors.NewCursor(ctx, &cursor.NewParams{ - Iter: iterator.Values(iterator.ForSlice(resDocs)), + Iter: iterator.WithClose(iter, closer.Close), DB: db, Collection: collection, Username: username, @@ -253,18 +275,20 @@ type stagesDocumentsParams struct { } // processStagesDocuments retrieves the documents from the database and then processes them through the stages. -func processStagesDocuments(ctx context.Context, p *stagesDocumentsParams) ([]*types.Document, error) { //nolint:lll // for readability - var docs []*types.Document +func processStagesDocuments(ctx context.Context, closer *iterator.MultiCloser, p *stagesDocumentsParams) (types.DocumentsIterator, error) { //nolint:lll // for readability + var keepTx pgx.Tx + var iter types.DocumentsIterator + + if err := p.dbPool.InTransactionKeep(ctx, func(tx pgx.Tx) error { + keepTx = tx - if err := p.dbPool.InTransaction(ctx, func(tx pgx.Tx) error { var err error - iter, _, err := pgdb.QueryDocuments(ctx, tx, p.qp) + iter, _, err = pgdb.QueryDocuments(ctx, tx, p.qp) if err != nil { - return err + return lazyerrors.Error(err) } - closer := iterator.NewMultiCloser(iter) - defer closer.Close() + closer.Add(iter) for _, s := range p.stages { if iter, err = s.Process(ctx, iter, closer); err != nil { @@ -272,13 +296,19 @@ func processStagesDocuments(ctx context.Context, p *stagesDocumentsParams) ([]*t } } - docs, err = iterator.ConsumeValues(iterator.Interface[struct{}, *types.Document](iter)) - return err + return nil }); err != nil { return nil, err } - return docs, nil + closer.Add(iterator.CloserFunc(func() { + // It does not matter if we commit or rollback the read transaction, + // but we should close it. + // ctx could be cancelled already. + _ = keepTx.Rollback(context.Background()) + })) + + return iter, nil } // stagesStatsParams contains the parameters for processStagesStats. @@ -291,7 +321,7 @@ type stagesStatsParams struct { } // processStagesStats retrieves the statistics from the database and then processes them through the stages. -func processStagesStats(ctx context.Context, p *stagesStatsParams) ([]*types.Document, error) { +func processStagesStats(ctx context.Context, closer *iterator.MultiCloser, p *stagesStatsParams) (types.DocumentsIterator, error) { //nolint:lll // for readability // Clarify what needs to be retrieved from the database and retrieve it. _, hasCount := p.statistics[stages.StatisticCount] _, hasStorage := p.statistics[stages.StatisticStorage] @@ -368,9 +398,7 @@ func processStagesStats(ctx context.Context, p *stagesStatsParams) ([]*types.Doc // Process the retrieved statistics through the stages. iter := iterator.Values(iterator.ForSlice([]*types.Document{doc})) - - closer := iterator.NewMultiCloser(iter) - defer closer.Close() + closer.Add(iter) for _, s := range p.stages { if iter, err = s.Process(ctx, iter, closer); err != nil { @@ -378,5 +406,5 @@ func processStagesStats(ctx context.Context, p *stagesStatsParams) ([]*types.Doc } } - return iterator.ConsumeValues(iter) + return iter, nil } From a2d3e43ff487d40d5b5c1f70515bc964c9ec6214 Mon Sep 17 00:00:00 2001 From: Elena Grahovac Date: Fri, 30 Jun 2023 16:32:09 +0200 Subject: [PATCH 40/46] Implement proper response for `createIndexes` (#2936) Closes #2845. --- integration/indexes_command_compat_test.go | 495 +++++++++++++++++++++ integration/indexes_compat_test.go | 378 +--------------- internal/handlers/pg/msg_createindexes.go | 23 +- internal/handlers/pg/msg_listindexes.go | 3 +- internal/handlers/pg/msg_update.go | 3 +- internal/handlers/pg/pgdb/collections.go | 14 +- internal/handlers/pg/pgdb/indexes.go | 18 +- internal/handlers/pg/pgdb/indexes_test.go | 17 +- internal/handlers/pg/pgdb/insert.go | 3 +- internal/handlers/pg/pgdb/pgdb_test.go | 15 +- 10 files changed, 567 insertions(+), 402 deletions(-) create mode 100644 integration/indexes_command_compat_test.go diff --git a/integration/indexes_command_compat_test.go b/integration/indexes_command_compat_test.go new file mode 100644 index 000000000000..9b9c946ed163 --- /dev/null +++ b/integration/indexes_command_compat_test.go @@ -0,0 +1,495 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package integration + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/mongo" + + "github.com/FerretDB/FerretDB/integration/setup" + "github.com/FerretDB/FerretDB/integration/shareddata" +) + +// TestCreateIndexesCommandCompat tests specific behavior for index creation that can be only provided through RunCommand. +func TestCreateIndexesCommandCompat(t *testing.T) { + setup.SkipForTigrisWithReason(t, "Indexes creation is not supported for Tigris") + + t.Parallel() + + ctx, targetCollections, compatCollections := setup.SetupCompat(t) + targetCollection := targetCollections[0] + compatCollection := compatCollections[0] + + for name, tc := range map[string]struct { //nolint:vet // for readability + collectionName any + indexName any + key any + unique any + resultType compatTestCaseResultType // defaults to nonEmptyResult + skip string // optional, skip test with a specified reason + }{ + "InvalidCollectionName": { + collectionName: 42, + key: bson.D{{"v", -1}}, + indexName: "custom-name", + resultType: emptyResult, + }, + "NilCollectionName": { + collectionName: nil, + key: bson.D{{"v", -1}}, + indexName: "custom-name", + resultType: emptyResult, + }, + "EmptyCollectionName": { + collectionName: "", + key: bson.D{{"v", -1}}, + indexName: "custom-name", + resultType: emptyResult, + }, + "IndexNameNotSet": { + collectionName: "test", + key: bson.D{{"v", -1}}, + indexName: nil, + resultType: emptyResult, + }, + "EmptyIndexName": { + collectionName: "test", + key: bson.D{{"v", -1}}, + indexName: "", + resultType: emptyResult, + }, + "NonStringIndexName": { + collectionName: "test", + key: bson.D{{"v", -1}}, + indexName: 42, + resultType: emptyResult, + }, + "ExistingNameDifferentKeyLength": { + collectionName: "test", + key: bson.D{{"_id", 1}, {"v", 1}}, + indexName: "_id_", // the same name as the default index + }, + "InvalidKey": { + collectionName: "test", + key: 42, + resultType: emptyResult, + }, + "EmptyKey": { + collectionName: "test", + key: bson.D{}, + resultType: emptyResult, + }, + "KeyNotSet": { + collectionName: "test", + resultType: emptyResult, + }, + "UniqueFalse": { + collectionName: "unique_false", + key: bson.D{{"v", 1}}, + indexName: "unique_false", + unique: false, + }, + "UniqueTypeDocument": { + collectionName: "test", + key: bson.D{{"v", 1}}, + indexName: "test", + unique: bson.D{}, + resultType: emptyResult, + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Helper() + t.Parallel() + + indexesDoc := bson.D{} + + if tc.key != nil { + indexesDoc = append(indexesDoc, bson.E{Key: "key", Value: tc.key}) + } + + if tc.indexName != nil { + indexesDoc = append(indexesDoc, bson.E{"name", tc.indexName}) + } + + if tc.unique != nil { + indexesDoc = append(indexesDoc, bson.E{Key: "unique", Value: tc.unique}) + } + + var targetRes bson.D + targetErr := targetCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", tc.collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&targetRes) + + var compatRes bson.D + compatErr := compatCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", tc.collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&compatRes) + + if targetErr != nil { + t.Logf("Target error: %v", targetErr) + t.Logf("Compat error: %v", compatErr) + + // error messages are intentionally not compared + AssertMatchesCommandError(t, compatErr, targetErr) + + return + } + require.NoError(t, compatErr, "compat error; target returned no error") + + if tc.resultType == emptyResult { + require.Nil(t, targetRes) + require.Nil(t, compatRes) + } + + assert.Equal(t, compatRes, targetRes) + + targetCursor, targetErr := targetCollection.Indexes().List(ctx) + compatCursor, compatErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + + require.NoError(t, targetErr) + require.NoError(t, compatErr) + + targetListRes := FetchAll(t, ctx, targetCursor) + compatListRes := FetchAll(t, ctx, compatCursor) + + assert.Equal(t, compatListRes, targetListRes) + + targetSpec, targetErr := targetCollection.Indexes().ListSpecifications(ctx) + compatSpec, compatErr := compatCollection.Indexes().ListSpecifications(ctx) + + require.NoError(t, compatErr) + require.NoError(t, targetErr) + + assert.Equal(t, compatSpec, targetSpec) + }) + } +} + +// TestCreateIndexesCommandCompatCheckFields check that the response contains response's fields +// such as numIndexBefore, numIndexAfter, createdCollectionAutomatically +// contain the correct values. +func TestCreateIndexesCommandCompatCheckFields(t *testing.T) { + setup.SkipForTigrisWithReason(t, "Indexes creation is not supported for Tigris") + + t.Parallel() + + ctx, targetCollections, compatCollections := setup.SetupCompat(t) + targetCollection := targetCollections[0] + compatCollection := compatCollections[0] + + // Create an index for a non-existent collection, expect createdCollectionAutomatically to be true. + collectionName := "newCollection" + indexesDoc := bson.D{{"key", bson.D{{"v", 1}}}, {"name", "v_1"}} + + var targetRes bson.D + targetErr := targetCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&targetRes) + + var compatRes bson.D + compatErr := compatCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&compatRes) + + require.NoError(t, compatErr) + require.NoError(t, targetErr, "target error; compat returned no error") + + compatDoc := ConvertDocument(t, compatRes) + createdCollectionAutomatically, err := compatDoc.Get("createdCollectionAutomatically") + require.NoError(t, err) + require.True(t, createdCollectionAutomatically.(bool)) // must be true because a new collection was created + + AssertEqualDocuments(t, compatRes, targetRes) + + // Now this collection exists, so we create another index and expect createdCollectionAutomatically to be false. + indexesDoc = bson.D{{"key", bson.D{{"foo", 1}}}, {"name", "foo_1"}} + + targetErr = targetCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&targetRes) + + compatErr = compatCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&compatRes) + + require.NoError(t, compatErr) + require.NoError(t, targetErr, "target error; compat returned no error") + + compatDoc = ConvertDocument(t, compatRes) + createdCollectionAutomatically, err = compatDoc.Get("createdCollectionAutomatically") + require.NoError(t, err) + require.False(t, createdCollectionAutomatically.(bool)) // must be false because the collection already exists + + AssertEqualDocuments(t, compatRes, targetRes) + + // Call index creation for the index that already exists, expect note to be set. + indexesDoc = bson.D{{"key", bson.D{{"foo", 1}}}, {"name", "foo_1"}} + + targetErr = targetCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&targetRes) + + compatErr = compatCollection.Database().RunCommand( + ctx, bson.D{ + {"createIndexes", collectionName}, + {"indexes", bson.A{indexesDoc}}, + }, + ).Decode(&compatRes) + + require.NoError(t, compatErr) + require.NoError(t, targetErr, "target error; compat returned no error") + + compatDoc = ConvertDocument(t, compatRes) + createdCollectionAutomatically, err = compatDoc.Get("note") + require.NoError(t, err) + + // note must be set because no new indexes were created: + require.Equal(t, "all indexes already exist", createdCollectionAutomatically.(string)) + + AssertEqualDocuments(t, compatRes, targetRes) +} + +func TestDropIndexesCommandCompat(t *testing.T) { + setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") + + t.Parallel() + + for name, tc := range map[string]struct { //nolint:vet // for readability + toCreate []mongo.IndexModel // optional, if set, create the given indexes before drop is called + toDrop any // required, index to drop + + resultType compatTestCaseResultType // optional, defaults to nonEmptyResult + skip string // optional, skip test with a specified reason + }{ + "MultipleIndexesByName": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"v", -1}}}, + {Keys: bson.D{{"v", 1}, {"foo", 1}}}, + {Keys: bson.D{{"v.foo", -1}}}, + }, + toDrop: bson.A{"v_-1", "v_1_foo_1"}, + }, + "MultipleIndexesByKey": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"v", -1}}}, + {Keys: bson.D{{"v.foo", -1}}}, + }, + toDrop: bson.A{bson.D{{"v", -1}}, bson.D{{"v.foo", -1}}}, + resultType: emptyResult, + }, + "NonExistentMultipleIndexes": { + toDrop: bson.A{"non-existent", "invalid"}, + resultType: emptyResult, + }, + "InvalidMultipleIndexType": { + toDrop: bson.A{1}, + resultType: emptyResult, + }, + "DocumentIndex": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"v", -1}}}, + }, + toDrop: bson.D{{"v", -1}}, + }, + "DropAllExpression": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"v", -1}}}, + {Keys: bson.D{{"foo.bar", 1}}}, + {Keys: bson.D{{"foo", 1}, {"bar", 1}}}, + }, + toDrop: "*", + }, + "WrongExpression": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"v", -1}}}, + {Keys: bson.D{{"foo.bar", 1}}}, + {Keys: bson.D{{"foo", 1}, {"bar", 1}}}, + }, + toDrop: "***", + resultType: emptyResult, + }, + "NonExistentDescendingID": { + toDrop: bson.D{{"_id", -1}}, + resultType: emptyResult, + }, + "MultipleKeyIndex": { + toCreate: []mongo.IndexModel{ + {Keys: bson.D{{"_id", -1}, {"v", 1}}}, + }, + toDrop: bson.D{ + {"_id", -1}, + {"v", 1}, + }, + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Helper() + t.Parallel() + + require.NotNil(t, tc.toDrop, "toDrop must not be nil") + + // It's enough to use a single provider for drop indexes test as indexes work the same for different collections. + s := setup.SetupCompatWithOpts(t, &setup.SetupCompatOpts{ + Providers: []shareddata.Provider{shareddata.Composites}, + AddNonExistentCollection: true, + }) + ctx, targetCollections, compatCollections := s.Ctx, s.TargetCollections, s.CompatCollections + + var nonEmptyResults bool + for i := range targetCollections { + targetCollection := targetCollections[i] + compatCollection := compatCollections[i] + + t.Run(targetCollection.Name(), func(t *testing.T) { + t.Helper() + + if tc.toCreate != nil { + _, targetErr := targetCollection.Indexes().CreateMany(ctx, tc.toCreate) + _, compatErr := compatCollection.Indexes().CreateMany(ctx, tc.toCreate) + require.NoError(t, compatErr) + require.NoError(t, targetErr) + + // List indexes to see they are identical after creation. + targetCursor, targetListErr := targetCollection.Indexes().List(ctx) + compatCursor, compatListErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + + require.NoError(t, targetListErr) + require.NoError(t, compatListErr) + + targetList := FetchAll(t, ctx, targetCursor) + compatList := FetchAll(t, ctx, compatCursor) + + require.Equal(t, compatList, targetList) + } + + targetCommand := bson.D{ + {"dropIndexes", targetCollection.Name()}, + {"index", tc.toDrop}, + } + + compatCommand := bson.D{ + {"dropIndexes", compatCollection.Name()}, + {"index", tc.toDrop}, + } + + var targetRes bson.D + targetErr := targetCollection.Database().RunCommand(ctx, targetCommand).Decode(&targetRes) + + var compatRes bson.D + compatErr := compatCollection.Database().RunCommand(ctx, compatCommand).Decode(&compatRes) + + if targetErr != nil { + t.Logf("Target error: %v", targetErr) + t.Logf("Compat error: %v", compatErr) + + // error messages are intentionally not compared + AssertMatchesCommandError(t, compatErr, targetErr) + + return + } + require.NoError(t, compatErr, "compat error; target returned no error") + + if tc.resultType == emptyResult { + require.Nil(t, targetRes) + require.Nil(t, compatRes) + } + + require.Equal(t, compatRes, targetRes) + + if compatErr == nil { + nonEmptyResults = true + } + + // List indexes to see they are identical after deletion. + targetCursor, targetListErr := targetCollection.Indexes().List(ctx) + compatCursor, compatListErr := compatCollection.Indexes().List(ctx) + + if targetCursor != nil { + defer targetCursor.Close(ctx) + } + if compatCursor != nil { + defer compatCursor.Close(ctx) + } + + require.NoError(t, targetListErr) + require.NoError(t, compatListErr) + + targetList := FetchAll(t, ctx, targetCursor) + compatList := FetchAll(t, ctx, compatCursor) + + assert.Equal(t, compatList, targetList) + }) + } + + switch tc.resultType { + case nonEmptyResult: + require.True(t, nonEmptyResults, "expected non-empty results (some indexes should be deleted)") + case emptyResult: + require.False(t, nonEmptyResults, "expected empty results (no indexes should be deleted)") + default: + t.Fatalf("unknown result type %v", tc.resultType) + } + }) + } +} diff --git a/integration/indexes_compat_test.go b/integration/indexes_compat_test.go index 928cdb525c62..e54149c406bd 100644 --- a/integration/indexes_compat_test.go +++ b/integration/indexes_compat_test.go @@ -311,179 +311,6 @@ func TestCreateIndexesCompat(t *testing.T) { } } -// TestCreateIndexesCommandCompat tests specific behavior for index creation that can be only provided through RunCommand. -func TestCreateIndexesCommandCompat(t *testing.T) { - setup.SkipForTigrisWithReason(t, "Indexes creation is not supported for Tigris") - - t.Parallel() - - ctx, targetCollections, compatCollections := setup.SetupCompat(t) - targetCollection := targetCollections[0] - compatCollection := compatCollections[0] - - for name, tc := range map[string]struct { //nolint:vet // for readability - collectionName any - indexName any - key any - unique any - resultType compatTestCaseResultType // defaults to nonEmptyResult - skip string // optional, skip test with a specified reason - }{ - "InvalidCollectionName": { - collectionName: 42, - key: bson.D{{"v", -1}}, - indexName: "custom-name", - resultType: emptyResult, - }, - "NilCollectionName": { - collectionName: nil, - key: bson.D{{"v", -1}}, - indexName: "custom-name", - resultType: emptyResult, - }, - "EmptyCollectionName": { - collectionName: "", - key: bson.D{{"v", -1}}, - indexName: "custom-name", - resultType: emptyResult, - }, - "IndexNameNotSet": { - collectionName: "test", - key: bson.D{{"v", -1}}, - indexName: nil, - resultType: emptyResult, - }, - "EmptyIndexName": { - collectionName: "test", - key: bson.D{{"v", -1}}, - indexName: "", - resultType: emptyResult, - }, - "NonStringIndexName": { - collectionName: "test", - key: bson.D{{"v", -1}}, - indexName: 42, - resultType: emptyResult, - }, - "ExistingNameDifferentKeyLength": { - collectionName: "test", - key: bson.D{{"_id", 1}, {"v", 1}}, - indexName: "_id_", // the same name as the default index - }, - "InvalidKey": { - collectionName: "test", - key: 42, - resultType: emptyResult, - }, - "EmptyKey": { - collectionName: "test", - key: bson.D{}, - resultType: emptyResult, - }, - "KeyNotSet": { - collectionName: "test", - resultType: emptyResult, - }, - "UniqueFalse": { - collectionName: "unique_false", - key: bson.D{{"v", 1}}, - indexName: "unique_false", - unique: false, - }, - "UniqueTypeDocument": { - collectionName: "test", - key: bson.D{{"v", 1}}, - indexName: "test", - unique: bson.D{}, - resultType: emptyResult, - }, - } { - name, tc := name, tc - t.Run(name, func(t *testing.T) { - if tc.skip != "" { - t.Skip(tc.skip) - } - - t.Helper() - t.Parallel() - - indexesDoc := bson.D{} - - if tc.key != nil { - indexesDoc = append(indexesDoc, bson.E{Key: "key", Value: tc.key}) - } - - if tc.indexName != nil { - indexesDoc = append(indexesDoc, bson.E{"name", tc.indexName}) - } - - if tc.unique != nil { - indexesDoc = append(indexesDoc, bson.E{Key: "unique", Value: tc.unique}) - } - - var targetRes bson.D - targetErr := targetCollection.Database().RunCommand( - ctx, bson.D{ - {"createIndexes", tc.collectionName}, - {"indexes", bson.A{indexesDoc}}, - }, - ).Decode(&targetRes) - - var compatRes bson.D - compatErr := compatCollection.Database().RunCommand( - ctx, bson.D{ - {"createIndexes", tc.collectionName}, - {"indexes", bson.A{indexesDoc}}, - }, - ).Decode(&compatRes) - - if targetErr != nil { - t.Logf("Target error: %v", targetErr) - t.Logf("Compat error: %v", compatErr) - - // error messages are intentionally not compared - AssertMatchesCommandError(t, compatErr, targetErr) - - return - } - require.NoError(t, compatErr, "compat error; target returned no error") - - if tc.resultType == emptyResult { - require.Nil(t, targetRes) - require.Nil(t, compatRes) - } - - assert.Equal(t, compatRes, targetRes) - - targetCursor, targetErr := targetCollection.Indexes().List(ctx) - compatCursor, compatErr := compatCollection.Indexes().List(ctx) - - if targetCursor != nil { - defer targetCursor.Close(ctx) - } - if compatCursor != nil { - defer compatCursor.Close(ctx) - } - - require.NoError(t, targetErr) - require.NoError(t, compatErr) - - targetListRes := FetchAll(t, ctx, targetCursor) - compatListRes := FetchAll(t, ctx, compatCursor) - - assert.Equal(t, compatListRes, targetListRes) - - targetSpec, targetErr := targetCollection.Indexes().ListSpecifications(ctx) - compatSpec, compatErr := compatCollection.Indexes().ListSpecifications(ctx) - - require.NoError(t, compatErr) - require.NoError(t, targetErr) - - assert.Equal(t, compatSpec, targetSpec) - }) - } -} - func TestDropIndexesCompat(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") @@ -618,202 +445,6 @@ func TestDropIndexesCompat(t *testing.T) { } } -func TestDropIndexesCommandCompat(t *testing.T) { - setup.SkipForTigrisWithReason(t, "Indexes are not supported for Tigris") - - t.Parallel() - - for name, tc := range map[string]struct { //nolint:vet // for readability - toCreate []mongo.IndexModel // optional, if set, create the given indexes before drop is called - toDrop any // required, index to drop - - resultType compatTestCaseResultType // optional, defaults to nonEmptyResult - skip string // optional, skip test with a specified reason - }{ - "MultipleIndexesByName": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"v", -1}}}, - {Keys: bson.D{{"v", 1}, {"foo", 1}}}, - {Keys: bson.D{{"v.foo", -1}}}, - }, - toDrop: bson.A{"v_-1", "v_1_foo_1"}, - }, - "MultipleIndexesByKey": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"v", -1}}}, - {Keys: bson.D{{"v.foo", -1}}}, - }, - toDrop: bson.A{bson.D{{"v", -1}}, bson.D{{"v.foo", -1}}}, - resultType: emptyResult, - }, - "NonExistentMultipleIndexes": { - toDrop: bson.A{"non-existent", "invalid"}, - resultType: emptyResult, - }, - "InvalidMultipleIndexType": { - toDrop: bson.A{1}, - resultType: emptyResult, - }, - "DocumentIndex": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"v", -1}}}, - }, - toDrop: bson.D{{"v", -1}}, - }, - "DropAllExpression": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"v", -1}}}, - {Keys: bson.D{{"foo.bar", 1}}}, - {Keys: bson.D{{"foo", 1}, {"bar", 1}}}, - }, - toDrop: "*", - }, - "WrongExpression": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"v", -1}}}, - {Keys: bson.D{{"foo.bar", 1}}}, - {Keys: bson.D{{"foo", 1}, {"bar", 1}}}, - }, - toDrop: "***", - resultType: emptyResult, - }, - "NonExistentDescendingID": { - toDrop: bson.D{{"_id", -1}}, - resultType: emptyResult, - }, - "MultipleKeyIndex": { - toCreate: []mongo.IndexModel{ - {Keys: bson.D{{"_id", -1}, {"v", 1}}}, - }, - toDrop: bson.D{ - {"_id", -1}, - {"v", 1}, - }, - }, - } { - name, tc := name, tc - t.Run(name, func(t *testing.T) { - if tc.skip != "" { - t.Skip(tc.skip) - } - - t.Helper() - t.Parallel() - - require.NotNil(t, tc.toDrop, "toDrop must not be nil") - - // It's enough to use a single provider for drop indexes test as indexes work the same for different collections. - s := setup.SetupCompatWithOpts(t, &setup.SetupCompatOpts{ - Providers: []shareddata.Provider{shareddata.Composites}, - AddNonExistentCollection: true, - }) - ctx, targetCollections, compatCollections := s.Ctx, s.TargetCollections, s.CompatCollections - - var nonEmptyResults bool - for i := range targetCollections { - targetCollection := targetCollections[i] - compatCollection := compatCollections[i] - - t.Run(targetCollection.Name(), func(t *testing.T) { - t.Helper() - - if tc.toCreate != nil { - _, targetErr := targetCollection.Indexes().CreateMany(ctx, tc.toCreate) - _, compatErr := compatCollection.Indexes().CreateMany(ctx, tc.toCreate) - require.NoError(t, compatErr) - require.NoError(t, targetErr) - - // List indexes to see they are identical after creation. - targetCursor, targetListErr := targetCollection.Indexes().List(ctx) - compatCursor, compatListErr := compatCollection.Indexes().List(ctx) - - if targetCursor != nil { - defer targetCursor.Close(ctx) - } - if compatCursor != nil { - defer compatCursor.Close(ctx) - } - - require.NoError(t, targetListErr) - require.NoError(t, compatListErr) - - targetList := FetchAll(t, ctx, targetCursor) - compatList := FetchAll(t, ctx, compatCursor) - - require.Equal(t, compatList, targetList) - } - - targetCommand := bson.D{ - {"dropIndexes", targetCollection.Name()}, - {"index", tc.toDrop}, - } - - compatCommand := bson.D{ - {"dropIndexes", compatCollection.Name()}, - {"index", tc.toDrop}, - } - - var targetRes bson.D - targetErr := targetCollection.Database().RunCommand(ctx, targetCommand).Decode(&targetRes) - - var compatRes bson.D - compatErr := compatCollection.Database().RunCommand(ctx, compatCommand).Decode(&compatRes) - - if targetErr != nil { - t.Logf("Target error: %v", targetErr) - t.Logf("Compat error: %v", compatErr) - - // error messages are intentionally not compared - AssertMatchesCommandError(t, compatErr, targetErr) - - return - } - require.NoError(t, compatErr, "compat error; target returned no error") - - if tc.resultType == emptyResult { - require.Nil(t, targetRes) - require.Nil(t, compatRes) - } - - require.Equal(t, compatRes, targetRes) - - if compatErr == nil { - nonEmptyResults = true - } - - // List indexes to see they are identical after deletion. - targetCursor, targetListErr := targetCollection.Indexes().List(ctx) - compatCursor, compatListErr := compatCollection.Indexes().List(ctx) - - if targetCursor != nil { - defer targetCursor.Close(ctx) - } - if compatCursor != nil { - defer compatCursor.Close(ctx) - } - - require.NoError(t, targetListErr) - require.NoError(t, compatListErr) - - targetList := FetchAll(t, ctx, targetCursor) - compatList := FetchAll(t, ctx, compatCursor) - - assert.Equal(t, compatList, targetList) - }) - } - - switch tc.resultType { - case nonEmptyResult: - require.True(t, nonEmptyResults, "expected non-empty results (some indexes should be deleted)") - case emptyResult: - require.False(t, nonEmptyResults, "expected empty results (no indexes should be deleted)") - default: - t.Fatalf("unknown result type %v", tc.resultType) - } - }) - } -} - func TestCreateIndexesCompatUnique(t *testing.T) { setup.SkipForTigrisWithReason(t, "Indexes creation is not supported for Tigris") @@ -855,6 +486,15 @@ func TestCreateIndexesCompatUnique(t *testing.T) { insertDoc: bson.D{{"not-existing-field", "value"}}, skip: "https://github.com/FerretDB/FerretDB/issues/2830", }, + "NotUniqueIndex": { + models: []mongo.IndexModel{ + { + Keys: bson.D{{"v", 1}}, + Options: options.Index().SetUnique(false), + }, + }, + insertDoc: bson.D{{"v", "value"}}, + }, "CompoundIndex": { models: []mongo.IndexModel{ { diff --git a/internal/handlers/pg/msg_createindexes.go b/internal/handlers/pg/msg_createindexes.go index c45dc311a2a8..bd417f1e55cb 100644 --- a/internal/handlers/pg/msg_createindexes.go +++ b/internal/handlers/pg/msg_createindexes.go @@ -109,7 +109,7 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. indexes := map[*types.Document]*pgdb.Index{} - var isUniqueSpecified bool + var collCreated bool var numIndexesBefore, numIndexesAfter int32 err = dbPool.InTransactionRetry(ctx, func(tx pgx.Tx) error { var indexesBefore []pgdb.Index @@ -212,7 +212,7 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. indexes[indexDoc] = index - err = pgdb.CreateIndexIfNotExists(ctx, tx, db, collection, index) + collCreated, err = pgdb.CreateIndexIfNotExists(ctx, tx, db, collection, index) if errors.Is(err, pgdb.ErrIndexKeyAlreadyExist) && index.Name == "_id_1" { // ascending _id index is created by default return nil @@ -221,8 +221,6 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. if err != nil { return err } - - isUniqueSpecified = index.Unique != nil } }) @@ -254,10 +252,13 @@ func (h *Handler) MsgCreateIndexes(ctx context.Context, msg *wire.OpMsg) (*wire. res := new(types.Document) - if isUniqueSpecified { - res.Set("numIndexesBefore", numIndexesBefore) - res.Set("numIndexesAfter", numIndexesAfter) - res.Set("createdCollectionAutomatically", true) + res.Set("numIndexesBefore", numIndexesBefore) + res.Set("numIndexesAfter", numIndexesAfter) + + if numIndexesBefore != numIndexesAfter { + res.Set("createdCollectionAutomatically", collCreated) + } else { + res.Set("note", "all indexes already exist") } res.Set("ok", float64(1)) @@ -375,7 +376,7 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { case "unique": v := must.NotFail(indexDoc.Get("unique")) - _, ok := v.(bool) + unique, ok := v.(bool) if !ok { return nil, commonerrors.NewCommandErrorMsgWithArgument( commonerrors.ErrTypeMismatch, @@ -401,7 +402,9 @@ func processIndexOptions(indexDoc *types.Document) (*pgdb.Index, error) { ) } - index.Unique = pointer.ToBool(true) + if unique { + index.Unique = pointer.ToBool(true) + } case "background": // ignore deprecated options diff --git a/internal/handlers/pg/msg_listindexes.go b/internal/handlers/pg/msg_listindexes.go index c146bd4a00d2..a86ce89b9a54 100644 --- a/internal/handlers/pg/msg_listindexes.go +++ b/internal/handlers/pg/msg_listindexes.go @@ -100,7 +100,8 @@ func (h *Handler) MsgListIndexes(ctx context.Context, msg *wire.OpMsg) (*wire.Op "name", index.Name, )) - if index.Unique != nil && index.Name != "_id_" { + // only non-default unique indexes should have unique field in the response + if index.Unique != nil && *index.Unique && index.Name != "_id_" { indexDoc.Set("unique", *index.Unique) } diff --git a/internal/handlers/pg/msg_update.go b/internal/handlers/pg/msg_update.go index 553e638535fa..3fcc3485b81e 100644 --- a/internal/handlers/pg/msg_update.go +++ b/internal/handlers/pg/msg_update.go @@ -48,7 +48,8 @@ func (h *Handler) MsgUpdate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, } err = dbPool.InTransactionRetry(ctx, func(tx pgx.Tx) error { - return pgdb.CreateCollectionIfNotExists(ctx, tx, params.DB, params.Collection) + _, err = pgdb.CreateCollectionIfNotExists(ctx, tx, params.DB, params.Collection) + return err }) switch { diff --git a/internal/handlers/pg/pgdb/collections.go b/internal/handlers/pg/pgdb/collections.go index dce2d369e6b3..4543907a40ed 100644 --- a/internal/handlers/pg/pgdb/collections.go +++ b/internal/handlers/pg/pgdb/collections.go @@ -143,7 +143,7 @@ func CreateCollection(ctx context.Context, tx pgx.Tx, db, collection string) err Unique: pointer.ToBool(true), } - if err := CreateIndexIfNotExists(ctx, tx, db, collection, indexParams); err != nil { + if _, err := CreateIndexIfNotExists(ctx, tx, db, collection, indexParams); err != nil { return lazyerrors.Error(err) } @@ -153,18 +153,22 @@ func CreateCollection(ctx context.Context, tx pgx.Tx, db, collection string) err // CreateCollectionIfNotExists ensures that given FerretDB database and collection exist. // If the database does not exist, it will be created. // +// It returns true if the collection was created, false if it already existed or an error occurred. +// // It returns possibly wrapped error: // - ErrInvalidDatabaseName - if the given database name doesn't conform to restrictions. // - ErrInvalidCollectionName - if the given collection name doesn't conform to restrictions. // - *transactionConflictError - if a PostgreSQL conflict occurs (the caller could retry the transaction). -func CreateCollectionIfNotExists(ctx context.Context, tx pgx.Tx, db, collection string) error { +func CreateCollectionIfNotExists(ctx context.Context, tx pgx.Tx, db, collection string) (bool, error) { err := CreateCollection(ctx, tx, db, collection) switch { - case err == nil, errors.Is(err, ErrAlreadyExist): - return nil + case err == nil: + return true, nil + case errors.Is(err, ErrAlreadyExist): + return false, nil default: - return lazyerrors.Error(err) + return false, lazyerrors.Error(err) } } diff --git a/internal/handlers/pg/pgdb/indexes.go b/internal/handlers/pg/pgdb/indexes.go index 1dc92d9fb324..e83b7790f635 100644 --- a/internal/handlers/pg/pgdb/indexes.go +++ b/internal/handlers/pg/pgdb/indexes.go @@ -64,16 +64,20 @@ func Indexes(ctx context.Context, tx pgx.Tx, db, collection string) ([]Index, er // CreateIndexIfNotExists creates a new index for the given params if such an index doesn't exist. // +// If index creation also caused the collection to be created, it returns true as the first return value. +// // If the index exists, it doesn't return an error. -// If the collection doesn't exist, it will be created and then the index will be created. -func CreateIndexIfNotExists(ctx context.Context, tx pgx.Tx, db, collection string, i *Index) error { - if err := CreateCollectionIfNotExists(ctx, tx, db, collection); err != nil { - return err +func CreateIndexIfNotExists(ctx context.Context, tx pgx.Tx, db, collection string, i *Index) (bool, error) { + var collCreated bool + var err error + + if collCreated, err = CreateCollectionIfNotExists(ctx, tx, db, collection); err != nil { + return false, err } pgTable, pgIndex, err := newMetadataStorage(tx, db, collection).setIndex(ctx, i.Name, i.Key, i.Unique) if err != nil { - return err + return false, err } var unique bool @@ -82,10 +86,10 @@ func CreateIndexIfNotExists(ctx context.Context, tx pgx.Tx, db, collection strin } if err := createPgIndexIfNotExists(ctx, tx, db, pgTable, pgIndex, i.Key, unique); err != nil { - return err + return false, err } - return nil + return collCreated, nil } // Equal returns true if the given index key is equal to the current one. diff --git a/internal/handlers/pg/pgdb/indexes_test.go b/internal/handlers/pg/pgdb/indexes_test.go index 21b7a86c8263..31b00e95e9d3 100644 --- a/internal/handlers/pg/pgdb/indexes_test.go +++ b/internal/handlers/pg/pgdb/indexes_test.go @@ -102,7 +102,7 @@ func TestCreateIndexIfNotExists(t *testing.T) { t.Run(name, func(t *testing.T) { t.Helper() err := pool.InTransaction(ctx, func(tx pgx.Tx) error { - if err := CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &tc.index); err != nil { + if _, err := CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &tc.index); err != nil { return err } return nil @@ -144,7 +144,8 @@ func TestDropIndexes(t *testing.T) { setupDatabase(ctx, t, pool, databaseName) err := pool.InTransaction(ctx, func(tx pgx.Tx) error { - return CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + _, err := CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + return err }) require.NoError(t, err) @@ -272,9 +273,13 @@ func TestDropIndexes(t *testing.T) { err := pool.InTransaction(ctx, func(tx pgx.Tx) error { for _, idx := range tc.toCreate { - if err := CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &idx); err != nil { + var collCreated bool + + if collCreated, err = CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &idx); err != nil { return err } + + assert.False(t, collCreated) } return nil @@ -371,7 +376,11 @@ func TestDropIndexesStress(t *testing.T) { Key: indexKeys, } - return CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &idx) + var collCreated bool + collCreated, err = CreateIndexIfNotExists(ctx, tx, databaseName, collectionName, &idx) + assert.False(t, collCreated) + + return err }) require.NoError(t, err) diff --git a/internal/handlers/pg/pgdb/insert.go b/internal/handlers/pg/pgdb/insert.go index 3d54c6f073b0..57175fd60430 100644 --- a/internal/handlers/pg/pgdb/insert.go +++ b/internal/handlers/pg/pgdb/insert.go @@ -44,8 +44,7 @@ func InsertDocument(ctx context.Context, tx pgx.Tx, db, collection string, doc * var err error - err = CreateCollectionIfNotExists(ctx, tx, db, collection) - if err != nil { + if _, err = CreateCollectionIfNotExists(ctx, tx, db, collection); err != nil { return lazyerrors.Error(err) } diff --git a/internal/handlers/pg/pgdb/pgdb_test.go b/internal/handlers/pg/pgdb/pgdb_test.go index a144e912ec26..d1e91fb7d347 100644 --- a/internal/handlers/pg/pgdb/pgdb_test.go +++ b/internal/handlers/pg/pgdb/pgdb_test.go @@ -260,7 +260,10 @@ func TestCreateCollectionIfNotExists(t *testing.T) { setupDatabase(ctx, t, pool, databaseName) err := pool.InTransaction(ctx, func(tx pgx.Tx) error { - return CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + created, err := CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + assert.True(t, created) + + return err }) require.NoError(t, err) }) @@ -277,7 +280,10 @@ func TestCreateCollectionIfNotExists(t *testing.T) { return err } - return CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + created, err := CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + assert.True(t, created) + + return err }) require.NoError(t, err) }) @@ -298,7 +304,10 @@ func TestCreateCollectionIfNotExists(t *testing.T) { return err } - return CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + created, err := CreateCollectionIfNotExists(ctx, tx, databaseName, collectionName) + assert.False(t, created) + + return err }) require.NoError(t, err) }) From e78942e996019e3dabae7a6c94b82b4b21f8b994 Mon Sep 17 00:00:00 2001 From: Dmitry Date: Fri, 30 Jun 2023 16:29:12 +0100 Subject: [PATCH 41/46] Re-implement `DELETE` for SQLite backend (#2907) Closes #2751. --- integration/delete_command_compat_test.go | 155 ++++++++++++++++++++++ integration/delete_compat_test.go | 16 +-- internal/backends/collection.go | 4 + internal/backends/sqlite/collection.go | 41 +++++- internal/handlers/common/delete.go | 1 + internal/handlers/pg/msg_delete.go | 7 +- internal/handlers/sqlite/msg_delete.go | 114 +++++++++++++++- internal/handlers/tigris/msg_delete.go | 7 +- 8 files changed, 326 insertions(+), 19 deletions(-) create mode 100644 integration/delete_command_compat_test.go diff --git a/integration/delete_command_compat_test.go b/integration/delete_command_compat_test.go new file mode 100644 index 000000000000..6a5e8f6a79b1 --- /dev/null +++ b/integration/delete_command_compat_test.go @@ -0,0 +1,155 @@ +// Copyright 2021 FerretDB Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package integration + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.mongodb.org/mongo-driver/bson" + + "github.com/FerretDB/FerretDB/integration/setup" + "github.com/FerretDB/FerretDB/integration/shareddata" +) + +// deleteCommandCompatTestCase describes delete compatibility test case. +type deleteCommandCompatTestCase struct { + skip string // optional, skip test with a specified reason + deletes bson.A // required +} + +func testDeleteCommandCompat(t *testing.T, testCases map[string]deleteCommandCompatTestCase) { + t.Helper() + + for name, tc := range testCases { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + t.Helper() + + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Parallel() + + // Use per-test setup because deletes modify data set. + ctx, targetCollections, compatCollections := setup.SetupCompat(t) + + for i := range targetCollections { + targetCollection := targetCollections[i] + compatCollection := compatCollections[i] + t.Run(targetCollection.Name(), func(t *testing.T) { + t.Helper() + + var targetRes, compatRes bson.D + targetErr := targetCollection.Database().RunCommand(ctx, + bson.D{ + {"delete", targetCollection.Name()}, + {"deletes", tc.deletes}, + }).Decode(&targetRes) + compatErr := compatCollection.Database().RunCommand(ctx, + bson.D{ + {"delete", compatCollection.Name()}, + {"deletes", tc.deletes}, + }).Decode(&compatRes) + + if targetErr != nil { + t.Logf("Target error: %v", targetErr) + t.Logf("Compat error: %v", compatErr) + + // error messages are intentionally not compared + AssertMatchesBulkException(t, compatErr, targetErr) + + return + } + require.NoError(t, compatErr, "compat error; target returned no error") + + t.Logf("Compat (expected) result: %v", compatRes) + t.Logf("Target (actual) result: %v", targetRes) + assert.Equal(t, compatRes, targetRes) + + targetDocs := FindAll(t, ctx, targetCollection) + compatDocs := FindAll(t, ctx, compatCollection) + + t.Logf("Compat (expected) IDs: %v", CollectIDs(t, compatDocs)) + t.Logf("Target (actual) IDs: %v", CollectIDs(t, targetDocs)) + AssertEqualDocumentsSlice(t, compatDocs, targetDocs) + }) + } + }) + } +} + +func TestDeleteCommandCompat(t *testing.T) { + t.Parallel() + + testCases := map[string]deleteCommandCompatTestCase{ + "OneLimited": { + deletes: bson.A{ + bson.D{{"q", bson.D{{"v", int32(0)}}}, {"limit", 1}}, + }, + skip: "https://github.com/FerretDB/FerretDB/issues/2935", + }, + "TwoLimited": { + deletes: bson.A{ + bson.D{{"q", bson.D{{"v", int32(42)}}}, {"limit", 1}}, + bson.D{{"q", bson.D{{"v", "foo"}}}, {"limit", 1}}, + }, + skip: "https://github.com/FerretDB/FerretDB/issues/2935", + }, + "DuplicateFilter": { + deletes: bson.A{ + bson.D{{"q", bson.D{{"v", "foo"}}}, {"limit", 1}}, + bson.D{{"q", bson.D{{"v", "foo"}}}, {"limit", 1}}, + }, + skip: "https://github.com/FerretDB/FerretDB/issues/2935", + }, + } + + testDeleteCommandCompat(t, testCases) +} + +func TestDeleteCommandCompatNotExistingDatabase(t *testing.T) { + t.Parallel() + + res := setup.SetupCompatWithOpts(t, &setup.SetupCompatOpts{ + Providers: []shareddata.Provider{shareddata.Int32s}, + }) + ctx, targetColl, compatColl := res.Ctx, res.TargetCollections[0], res.CompatCollections[0] + + targetDB := targetColl.Database().Client().Database("doesnotexist") + compatDB := compatColl.Database().Client().Database("doesnotexist") + + var targetRes, compatRes bson.D + + targetErr := targetDB.RunCommand(ctx, bson.D{ + {"delete", "test"}, + {"deletes", bson.A{ + bson.D{{"q", bson.D{{"v", "foo"}}}, {"limit", 1}}, + }}, + }).Decode(&targetRes) + compatErr := compatDB.RunCommand(ctx, bson.D{ + {"delete", "test"}, + {"deletes", bson.A{ + bson.D{{"q", bson.D{{"v", "foo"}}}, {"limit", 1}}, + }}, + }).Decode(&compatRes) + + assert.NoError(t, targetErr) + assert.NoError(t, compatErr) + + assert.Equal(t, compatRes, targetRes) +} diff --git a/integration/delete_compat_test.go b/integration/delete_compat_test.go index 6b9b74a9054b..752b50f06450 100644 --- a/integration/delete_compat_test.go +++ b/integration/delete_compat_test.go @@ -38,11 +38,6 @@ func TestDeleteCompat(t *testing.T) { t.Parallel() testCases := map[string]deleteCompatTestCase{ - "Empty": { - filters: []bson.D{}, - resultType: emptyResult, - }, - "One": { filters: []bson.D{ {{"v", int32(42)}}, @@ -166,10 +161,11 @@ func testDeleteCompat(t *testing.T, testCases map[string]deleteCompatTestCase) { if targetErr != nil { t.Logf("Target error: %v", targetErr) - targetErr = UnsetRaw(t, targetErr) - compatErr = UnsetRaw(t, compatErr) - assert.Equal(t, compatErr, targetErr) - } else { + t.Logf("Compat error: %v", compatErr) + + // error messages are intentionally not compared + AssertMatchesBulkException(t, compatErr, targetErr) + } else { // we have to check the results in case of error because some documents may be deleted require.NoError(t, compatErr, "compat error; target returned no error") } @@ -177,6 +173,8 @@ func testDeleteCompat(t *testing.T, testCases map[string]deleteCompatTestCase) { nonEmptyResults = true } + t.Logf("Compat (expected) result: %v", compatRes) + t.Logf("Target (actual) result: %v", targetRes) assert.Equal(t, compatRes, targetRes) targetDocs := FindAll(t, ctx, targetCollection) diff --git a/internal/backends/collection.go b/internal/backends/collection.go index 957261b98c31..33de8d5ecd22 100644 --- a/internal/backends/collection.go +++ b/internal/backends/collection.go @@ -68,6 +68,8 @@ type QueryResult struct { // Query executes a query against the collection. // +// If the collection does not exist it returns empty iterator. +// // The passed context should be used for canceling the initial query. // It also can be used to close the returned iterator and free underlying resources, // but doing so is not necessary - the handler will do that anyway. @@ -132,6 +134,8 @@ type DeleteResult struct { } // Delete deletes documents in collection. +// +// If requested database or collection does not exist it returns 0 deleted documents. func (cc *collectionContract) Delete(ctx context.Context, params *DeleteParams) (res *DeleteResult, err error) { defer observability.FuncCall(ctx)() defer checkError(err) diff --git a/internal/backends/sqlite/collection.go b/internal/backends/sqlite/collection.go index 244de22084c3..3dacc3b953f4 100644 --- a/internal/backends/sqlite/collection.go +++ b/internal/backends/sqlite/collection.go @@ -19,6 +19,9 @@ import ( "errors" "fmt" + "modernc.org/sqlite" + sqlitelib "modernc.org/sqlite/lib" + "github.com/FerretDB/FerretDB/internal/backends" "github.com/FerretDB/FerretDB/internal/backends/sqlite/metadata" "github.com/FerretDB/FerretDB/internal/handlers/sjson" @@ -59,6 +62,12 @@ func (c *collection) Query(ctx context.Context, params *backends.QueryParams) (* rows, err := db.QueryContext(ctx, query) if err != nil { + // No such table, return empty result. + var e *sqlite.Error + if errors.As(err, &e) && e.Code() == sqlitelib.SQLITE_ERROR { + return &backends.QueryResult{Iter: newQueryIterator(ctx, nil)}, nil + } + return nil, lazyerrors.Error(err) } @@ -119,7 +128,7 @@ func (c *collection) Update(ctx context.Context, params *backends.UpdateParams) tableName := c.r.CollectionToTable(c.name) - query := fmt.Sprintf(`UPDATE %q SET _ferretdb_sjson = ? WHERE json_extract(_ferretdb_sjson, '$._id') = ?`, tableName) + query := fmt.Sprintf(`UPDATE %q SET _ferretdb_sjson = ? WHERE _ferretdb_sjson -> '$._id' = ?`, tableName) var res backends.UpdateResult @@ -164,9 +173,35 @@ func (c *collection) Update(ctx context.Context, params *backends.UpdateParams) // Delete implements backends.Collection interface. func (c *collection) Delete(ctx context.Context, params *backends.DeleteParams) (*backends.DeleteResult, error) { - // TODO https://github.com/FerretDB/FerretDB/issues/2751 + db := c.r.DatabaseGetExisting(ctx, c.dbName) + if db == nil { + return &backends.DeleteResult{Deleted: 0}, nil + } + + tableName := c.r.CollectionToTable(c.name) + + query := fmt.Sprintf(`DELETE FROM %q WHERE _ferretdb_sjson -> '$._id' = ?`, tableName) + + var deleted int64 + + for _, id := range params.IDs { + idArg := string(must.NotFail(sjson.MarshalSingleValue(id))) + + res, err := db.ExecContext(ctx, query, idArg) + if err != nil { + return nil, lazyerrors.Error(err) + } + + rowsAffected, err := res.RowsAffected() + if err != nil { + return nil, lazyerrors.Error(err) + } + + deleted += rowsAffected + } + return &backends.DeleteResult{ - Deleted: 0, + Deleted: deleted, }, nil } diff --git a/internal/handlers/common/delete.go b/internal/handlers/common/delete.go index ba8f1faf5370..47cbb95e5abb 100644 --- a/internal/handlers/common/delete.go +++ b/internal/handlers/common/delete.go @@ -33,6 +33,7 @@ type DeleteParams struct { Let *types.Document `ferretdb:"let,unimplemented"` WriteConcern *types.Document `ferretdb:"writeConcern,ignored"` + LSID any `ferretdb:"lsid,ignored"` } // Delete represents single delete operation parameters. diff --git a/internal/handlers/pg/msg_delete.go b/internal/handlers/pg/msg_delete.go index 0b61ae78f91e..4d3d7babf157 100644 --- a/internal/handlers/pg/msg_delete.go +++ b/internal/handlers/pg/msg_delete.go @@ -81,14 +81,15 @@ func (h *Handler) MsgDelete(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, } replyDoc := must.NotFail(types.NewDocument( - "ok", float64(1), + "n", deleted, )) if delErrors.Len() > 0 { - replyDoc = delErrors.Document() + // "writeErrors" should be after "n" field + replyDoc.Set("writeErrors", must.NotFail(delErrors.Document().Get("writeErrors"))) } - replyDoc.Set("n", deleted) + replyDoc.Set("ok", float64(1)) var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ diff --git a/internal/handlers/sqlite/msg_delete.go b/internal/handlers/sqlite/msg_delete.go index 261b588d9f51..ad4cf4df2707 100644 --- a/internal/handlers/sqlite/msg_delete.go +++ b/internal/handlers/sqlite/msg_delete.go @@ -16,12 +16,124 @@ package sqlite import ( "context" + "errors" + "github.com/FerretDB/FerretDB/internal/backends" + "github.com/FerretDB/FerretDB/internal/handlers/common" + "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" + "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/iterator" + "github.com/FerretDB/FerretDB/internal/util/lazyerrors" "github.com/FerretDB/FerretDB/internal/util/must" "github.com/FerretDB/FerretDB/internal/wire" ) // MsgDelete implements HandlerInterface. func (h *Handler) MsgDelete(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { - return nil, notImplemented(must.NotFail(msg.Document()).Command()) + document, err := msg.Document() + if err != nil { + return nil, lazyerrors.Error(err) + } + + params, err := common.GetDeleteParams(document, h.L) + if err != nil { + return nil, lazyerrors.Error(err) + } + + var deleted int32 + var delErrors commonerrors.WriteErrors + + db := h.b.Database(params.DB) + defer db.Close() + + coll := db.Collection(params.Collection) + + // process every delete filter + for i, deleteParams := range params.Deletes { + del, err := execDelete(ctx, coll, deleteParams.Filter, deleteParams.Limited) + if err == nil { + deleted += del + continue + } + + delErrors.Append(err, int32(i)) + + if params.Ordered { + break + } + } + + replyDoc := must.NotFail(types.NewDocument( + "n", deleted, + )) + + if delErrors.Len() > 0 { + // "writeErrors" should be after "n" field + replyDoc.Set("writeErrors", must.NotFail(delErrors.Document().Get("writeErrors"))) + } + + replyDoc.Set("ok", float64(1)) + + var reply wire.OpMsg + must.NoError(reply.SetSections(wire.OpMsgSection{ + Documents: []*types.Document{replyDoc}, + })) + + return &reply, nil +} + +// execDelete fetches documents, filters them out, limits them (if needed) and deletes them. +// If limited is true, only the first matched document is chosen for deletion, otherwise all matched documents are chosen. +// It returns the number of deleted documents or an error. +func execDelete(ctx context.Context, coll backends.Collection, filter *types.Document, limited bool) (int32, error) { + // query documents here + res, err := coll.Query(ctx, nil) + if err != nil { + return 0, lazyerrors.Error(err) + } + + defer res.Iter.Close() + + var ids []any + var doc *types.Document + var matches bool + + for { + if _, doc, err = res.Iter.Next(); err != nil { + if errors.Is(err, iterator.ErrIteratorDone) { + break + } + + return 0, lazyerrors.Error(err) + } + + if matches, err = common.FilterDocument(doc, filter); err != nil { + return 0, lazyerrors.Error(err) + } + + if !matches { + continue + } + + ids = append(ids, must.NotFail(doc.Get("_id"))) + + // if limit is set, no need to fetch all the documents + if limited { + res.Iter.Close() // call Close() to release the underlying connection early + + break + } + } + + // if no documents matched, there is nothing to delete + if len(ids) == 0 { + return 0, nil + } + + deleteRes, err := coll.Delete(ctx, &backends.DeleteParams{IDs: ids}) + if err != nil { + return 0, err + } + + return int32(deleteRes.Deleted), nil } diff --git a/internal/handlers/tigris/msg_delete.go b/internal/handlers/tigris/msg_delete.go index 2b13a158c251..2ce511ce37fa 100644 --- a/internal/handlers/tigris/msg_delete.go +++ b/internal/handlers/tigris/msg_delete.go @@ -80,14 +80,15 @@ func (h *Handler) MsgDelete(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, } replyDoc := must.NotFail(types.NewDocument( - "ok", float64(1), + "n", deleted, )) if delErrors.Len() > 0 { - replyDoc = delErrors.Document() + // "writeErrors" should be after "n" field + replyDoc.Set("writeErrors", must.NotFail(delErrors.Document().Get("writeErrors"))) } - replyDoc.Set("n", deleted) + replyDoc.Set("ok", float64(1)) var reply wire.OpMsg must.NoError(reply.SetSections(wire.OpMsgSection{ From 11a2389b352d34e58dd5dcbed2dd3396f3d59997 Mon Sep 17 00:00:00 2001 From: Patryk Kwiatek Date: Fri, 30 Jun 2023 17:53:05 +0200 Subject: [PATCH 42/46] Validate database names for SQLite handler (#2924) --- integration/basic_test.go | 68 ++++++++++++++++++++++---- internal/handlers/pg/pgdb/databases.go | 2 +- internal/handlers/sqlite/msg_create.go | 13 ++++- 3 files changed, 71 insertions(+), 12 deletions(-) diff --git a/integration/basic_test.go b/integration/basic_test.go index e86a4cad95fc..c6d5a47dcead 100644 --- a/integration/basic_test.go +++ b/integration/basic_test.go @@ -446,6 +446,46 @@ func TestDatabaseName(t *testing.T) { t.Parallel() + t.Run("NoErr", func(t *testing.T) { + ctx, collection := setup.Setup(t) + for name, tc := range map[string]struct { + db string // database name, defaults to empty string + skip string // optional, skip test with a specified reason + }{ + "Dash": { + db: "--", + }, + "Underscore": { + db: "__", + }, + "Sqlite": { + db: "sqlite_", + }, + "Number": { + db: "0prefix", + }, + "63ok": { + db: strings.Repeat("a", 63), + }, + } { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + if tc.skip != "" { + t.Skip(tc.skip) + } + + t.Parallel() + + // there is no explicit command to create database, so create collection instead + err := collection.Database().Client().Database(tc.db).CreateCollection(ctx, collection.Name()) + require.NoError(t, err) + + err = collection.Database().Client().Database(tc.db).Drop(ctx) + require.NoError(t, err) + }) + } + }) + t.Run("Err", func(t *testing.T) { ctx, collection := setup.Setup(t) @@ -471,6 +511,25 @@ func TestDatabaseName(t *testing.T) { }, altMessage: fmt.Sprintf("Invalid namespace: %s.%s", dbName64, "TestDatabaseName-Err"), }, + "WithASlash": { + db: "/", + err: &mongo.CommandError{ + Name: "InvalidNamespace", + Code: 73, + Message: `Invalid namespace specified '/.TestDatabaseName-Err'`, + }, + altMessage: `Invalid namespace: /.TestDatabaseName-Err`, + }, + + "WithABackslash": { + db: "\\", + err: &mongo.CommandError{ + Name: "InvalidNamespace", + Code: 73, + Message: `Invalid namespace specified '\.TestDatabaseName-Err'`, + }, + altMessage: `Invalid namespace: \.TestDatabaseName-Err`, + }, "WithADollarSign": { db: "name_with_a-$", err: &mongo.CommandError{ @@ -516,15 +575,6 @@ func TestDatabaseName(t *testing.T) { }) } }) - - t.Run("63ok", func(t *testing.T) { - ctx, collection := setup.Setup(t) - - dbName63 := strings.Repeat("a", 63) - err := collection.Database().Client().Database(dbName63).CreateCollection(ctx, collection.Name()) - require.NoError(t, err) - collection.Database().Client().Database(dbName63).Drop(ctx) - }) } func TestDebugError(t *testing.T) { diff --git a/internal/handlers/pg/pgdb/databases.go b/internal/handlers/pg/pgdb/databases.go index aabcf59692d2..ba30acf5e516 100644 --- a/internal/handlers/pg/pgdb/databases.go +++ b/internal/handlers/pg/pgdb/databases.go @@ -29,7 +29,7 @@ import ( ) // validateDatabaseNameRe validates FerretDB database / PostgreSQL schema names. -var validateDatabaseNameRe = regexp.MustCompile("^[a-zA-Z_-][a-zA-Z0-9_-]{0,62}$") +var validateDatabaseNameRe = regexp.MustCompile("^[a-zA-Z0-9_-]{1,63}$") // Databases returns a sorted list of FerretDB databases / PostgreSQL schemas. func Databases(ctx context.Context, tx pgx.Tx) ([]string, error) { diff --git a/internal/handlers/sqlite/msg_create.go b/internal/handlers/sqlite/msg_create.go index a7c509842d78..64e42528a721 100644 --- a/internal/handlers/sqlite/msg_create.go +++ b/internal/handlers/sqlite/msg_create.go @@ -17,6 +17,7 @@ package sqlite import ( "context" "fmt" + "regexp" "github.com/FerretDB/FerretDB/internal/backends" "github.com/FerretDB/FerretDB/internal/handlers/common" @@ -27,6 +28,9 @@ import ( "github.com/FerretDB/FerretDB/internal/wire" ) +// validateDatabaseNameRe validates FerretDB database name. +var validateDatabaseNameRe = regexp.MustCompile("^[a-zA-Z0-9_-]{1,63}$") + // MsgCreate implements HandlerInterface. func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, error) { document, err := msg.Document() @@ -78,6 +82,11 @@ func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, return nil, err } + if !validateDatabaseNameRe.MatchString(dbName) { + msg := fmt.Sprintf("Invalid namespace: %s.%s", dbName, collectionName) + return nil, commonerrors.NewCommandErrorMsgWithArgument(commonerrors.ErrInvalidNamespace, msg, "create") + } + db := h.b.Database(dbName) defer db.Close() @@ -98,11 +107,11 @@ func (h *Handler) MsgCreate(ctx context.Context, msg *wire.OpMsg) (*wire.OpMsg, case backends.ErrorCodeIs(err, backends.ErrorCodeCollectionAlreadyExists): msg := fmt.Sprintf("Collection %s.%s already exists.", dbName, collectionName) - return nil, commonerrors.NewCommandErrorMsg(commonerrors.ErrNamespaceExists, msg) + return nil, commonerrors.NewCommandErrorMsgWithArgument(commonerrors.ErrNamespaceExists, msg, "create") case backends.ErrorCodeIs(err, backends.ErrorCodeCollectionNameIsInvalid): msg := fmt.Sprintf("Invalid namespace: %s.%s", dbName, collectionName) - return nil, commonerrors.NewCommandErrorMsg(commonerrors.ErrInvalidNamespace, msg) + return nil, commonerrors.NewCommandErrorMsgWithArgument(commonerrors.ErrInvalidNamespace, msg, "create") default: return nil, lazyerrors.Error(err) From 206abe8edb5366bd2dcaa554589320ab2a6ee38c Mon Sep 17 00:00:00 2001 From: Dmitry Date: Fri, 30 Jun 2023 17:13:23 +0100 Subject: [PATCH 43/46] Add `insert` documents type validation (#2946) --- integration/insert_command_test.go | 30 ++++++++++++++++++++++++++++++ internal/handlers/common/insert.go | 19 +++++++++++++++++++ internal/handlers/pg/msg_insert.go | 25 +++++++------------------ 3 files changed, 56 insertions(+), 18 deletions(-) diff --git a/integration/insert_command_test.go b/integration/insert_command_test.go index 07a173b5cefa..9e87a9bb5993 100644 --- a/integration/insert_command_test.go +++ b/integration/insert_command_test.go @@ -62,6 +62,32 @@ func TestInsertCommandErrors(t *testing.T) { }, altMessage: "E11000 duplicate key error collection: TestInsertCommandErrors-InsertDuplicateKey.TestInsertCommandErrors-InsertDuplicateKey", }, + "InsertDuplicateKeyOrdered": { + toInsert: []any{ + bson.D{{"foo", "bar"}}, + bson.D{{"_id", "double"}}, + }, + ordered: true, + werr: &mongo.WriteError{ + Code: 11000, + Index: 1, + Message: `E11000 duplicate key error collection: ` + + `TestInsertCommandErrors-InsertDuplicateKeyOrdered.TestInsertCommandErrors-InsertDuplicateKeyOrdered index: _id_ dup key: { _id: "double" }`, + }, + altMessage: `E11000 duplicate key error collection: TestInsertCommandErrors-InsertDuplicateKeyOrdered.TestInsertCommandErrors-InsertDuplicateKeyOrdered`, + }, + "InsertArray": { + toInsert: []any{ + bson.D{{"a", int32(1)}}, + bson.A{}, + }, + ordered: true, + cerr: &mongo.CommandError{ + Code: 14, + Name: "TypeMismatch", + Message: "BSON field 'insert.documents.1' is the wrong type 'array', expected type 'object'", + }, + }, } { name, tc := name, tc t.Run(name, func(t *testing.T) { @@ -87,11 +113,15 @@ func TestInsertCommandErrors(t *testing.T) { if tc.cerr != nil { AssertEqualAltCommandError(t, *tc.cerr, tc.altMessage, err) + return } if tc.werr != nil { AssertEqualAltWriteError(t, *tc.werr, tc.altMessage, err) + return } + + require.NoError(t, err) }) } } diff --git a/internal/handlers/common/insert.go b/internal/handlers/common/insert.go index 7109d77fd1cf..cbc30cb2e25c 100644 --- a/internal/handlers/common/insert.go +++ b/internal/handlers/common/insert.go @@ -15,10 +15,14 @@ package common import ( + "fmt" + "go.uber.org/zap" + "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" "github.com/FerretDB/FerretDB/internal/handlers/commonparams" "github.com/FerretDB/FerretDB/internal/types" + "github.com/FerretDB/FerretDB/internal/util/must" ) // InsertParams represents the parameters for an insert command. @@ -45,5 +49,20 @@ func GetInsertParams(document *types.Document, l *zap.Logger) (*InsertParams, er return nil, err } + for i := 0; i < params.Docs.Len(); i++ { + doc := must.NotFail(params.Docs.Get(i)) + + if _, ok := doc.(*types.Document); !ok { + return nil, commonerrors.NewCommandErrorMsg( + commonerrors.ErrTypeMismatch, + fmt.Sprintf( + "BSON field 'insert.documents.%d' is the wrong type '%s', expected type 'object'", + i, + commonparams.AliasFromType(doc), + ), + ) + } + } + return ¶ms, nil } diff --git a/internal/handlers/pg/msg_insert.go b/internal/handlers/pg/msg_insert.go index 9808c9cb210c..bbb4ca26c07b 100644 --- a/internal/handlers/pg/msg_insert.go +++ b/internal/handlers/pg/msg_insert.go @@ -23,7 +23,6 @@ import ( "github.com/FerretDB/FerretDB/internal/handlers/common" "github.com/FerretDB/FerretDB/internal/handlers/commonerrors" - "github.com/FerretDB/FerretDB/internal/handlers/commonparams" "github.com/FerretDB/FerretDB/internal/handlers/pg/pgdb" "github.com/FerretDB/FerretDB/internal/types" "github.com/FerretDB/FerretDB/internal/util/lazyerrors" @@ -87,7 +86,7 @@ func insertMany(ctx context.Context, dbPool *pgdb.Pool, qp *pgdb.QueryParams, do for i := 0; i < docs.Len(); i++ { doc := must.NotFail(docs.Get(i)) - err := insertDocument(ctx, tx, qp, doc) + err := insertDocument(ctx, tx, qp, doc.(*types.Document)) if err != nil { return err } @@ -98,7 +97,7 @@ func insertMany(ctx context.Context, dbPool *pgdb.Pool, qp *pgdb.QueryParams, do // try inserting one document at a time if err != nil { for i := 0; i < docs.Len(); i++ { - doc := must.NotFail(docs.Get(i)) + doc := must.NotFail(docs.Get(i)).(*types.Document) err := insertDocumentSeparately(ctx, dbPool, qp, doc) @@ -126,22 +125,12 @@ func insertMany(ctx context.Context, dbPool *pgdb.Pool, qp *pgdb.QueryParams, do } // insertDocument prepares and executes actual INSERT request to Postgres in provided transaction. -func insertDocument(ctx context.Context, tx pgx.Tx, qp *pgdb.QueryParams, doc any) error { - d, ok := doc.(*types.Document) - if !ok { - return commonerrors.NewCommandErrorMsg( - commonerrors.ErrBadValue, - fmt.Sprintf("document has invalid type %s", commonparams.AliasFromType(doc)), - ) - } - - toInsert := d - - if !toInsert.Has("_id") { - toInsert.Set("_id", types.NewObjectID()) +func insertDocument(ctx context.Context, tx pgx.Tx, qp *pgdb.QueryParams, doc *types.Document) error { + if !doc.Has("_id") { + doc.Set("_id", types.NewObjectID()) } - err := pgdb.InsertDocument(ctx, tx, qp.DB, qp.Collection, toInsert) + err := pgdb.InsertDocument(ctx, tx, qp.DB, qp.Collection, doc) switch { case err == nil: @@ -168,7 +157,7 @@ func insertDocument(ctx context.Context, tx pgx.Tx, qp *pgdb.QueryParams, doc an // insertDocumentSeparately prepares and executes actual INSERT request to Postgres in separate transaction. // // It should be used in places where we don't want to rollback previous inserted documents on error. -func insertDocumentSeparately(ctx context.Context, dbPool *pgdb.Pool, qp *pgdb.QueryParams, doc any) error { +func insertDocumentSeparately(ctx context.Context, dbPool *pgdb.Pool, qp *pgdb.QueryParams, doc *types.Document) error { return dbPool.InTransactionRetry(ctx, func(tx pgx.Tx) error { return insertDocument(ctx, tx, qp, doc) }) From 0b347ae4994bf70c6b1292e5fbb5f4e043e206ee Mon Sep 17 00:00:00 2001 From: Dmitry Date: Mon, 3 Jul 2023 08:49:59 +0100 Subject: [PATCH 44/46] Convert SQLite directory to URI (#2922) Co-authored-by: Chi Fujii Co-authored-by: Alexey Palazhchenko --- .golangci-new.yml | 6 + Taskfile.yml | 6 +- cmd/ferretdb/main.go | 6 +- ferretdb/ferretdb.go | 10 +- integration/setup/listener.go | 7 +- integration/setup/startup.go | 12 +- internal/backends/sqlite/backend.go | 23 +--- .../backends/sqlite/metadata/pool/pool.go | 102 +++++++++++--- .../sqlite/metadata/pool/pool_test.go | 129 +++++++++++++++++- internal/backends/sqlite/metadata/registry.go | 6 +- internal/handlers/registry/registry.go | 2 +- internal/handlers/registry/sqlite.go | 2 +- internal/handlers/sqlite/sqlite.go | 4 +- 13 files changed, 248 insertions(+), 67 deletions(-) diff --git a/.golangci-new.yml b/.golangci-new.yml index e072af77bdd7..4ff41ffe383d 100644 --- a/.golangci-new.yml +++ b/.golangci-new.yml @@ -36,6 +36,12 @@ linters-settings: - name: package-comments staticcheck: checks: ["all"] + tagalign: + sort: true + order: + - name + - default + - help wsl: # https://github.com/bombsimon/wsl/blob/master/doc/configuration.md strict-append: false diff --git a/Taskfile.yml b/Taskfile.yml index eacc301b4f53..b4f33bc975ff 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -322,7 +322,7 @@ tasks: --proxy-addr=127.0.0.1:47017 --mode=diff-normal --handler=sqlite - --sqlite-uri=tmp/sqlite + --sqlite-url=file:tmp/sqlite/ --test-records-dir=tmp/records --test-disable-filter-pushdown @@ -411,7 +411,7 @@ tasks: --proxy-addr=127.0.0.1:47017 --mode=diff-proxy --handler=sqlite - --sqlite-uri=tmp/sqlite + --sqlite-url=file:tmp/sqlite/ --test-records-dir=tmp/records --test-disable-filter-pushdown @@ -462,7 +462,7 @@ tasks: sqlite3: desc: "Run sqlite3" cmds: - - sqlite3 *.sqlite + - sqlite3 tmp/sqlite/*.sqlite mongosh: desc: "Run MongoDB shell (`mongosh`)" diff --git a/cmd/ferretdb/main.go b/cmd/ferretdb/main.go index a21ed6f2b18f..1be8dfe94986 100644 --- a/cmd/ferretdb/main.go +++ b/cmd/ferretdb/main.go @@ -104,7 +104,7 @@ var pgFlags struct { // // See main_sqlite.go. var sqliteFlags struct { - SQLiteURI string `name:"sqlite-uri" default:"." help:"Directory path or 'file' URI for 'sqlite' handler."` + SQLiteURL string `name:"sqlite-url" default:"file:data/" help:"SQLite URI (directory) for 'sqlite' handler."` } // The tigrisFlags struct represents flags that are used by the "tigris" handler. @@ -360,7 +360,7 @@ func run() { PostgreSQLURL: pgFlags.PostgreSQLURL, - SQLiteURI: sqliteFlags.SQLiteURI, + SQLiteURL: sqliteFlags.SQLiteURL, TigrisURL: tigrisFlags.TigrisURL, TigrisClientID: tigrisFlags.TigrisClientID, @@ -374,7 +374,7 @@ func run() { }, }) if err != nil { - logger.Fatal(err.Error()) + logger.Sugar().Fatalf("Failed to construct handler: %s.", err) } l := clientconn.NewListener(&clientconn.NewListenerOpts{ diff --git a/ferretdb/ferretdb.go b/ferretdb/ferretdb.go index c0dee9d2f7af..1f75c9abd4d8 100644 --- a/ferretdb/ferretdb.go +++ b/ferretdb/ferretdb.go @@ -49,11 +49,9 @@ type Config struct { // - https://pkg.go.dev/github.com/jackc/pgx/v5/pgconn#ParseConfig PostgreSQLURL string // For example: `postgres://hostname:5432/ferretdb`. - // SQLite directory path or `file:` URI for `sqlite` handler. - // See: - // - https://www.sqlite.org/c3ref/open.html - // - https://www.sqlite.org/uri.html - SQLiteURI string + // SQLite URI (directory) for `sqlite` handler. + // See https://www.sqlite.org/uri.html. + SQLiteURL string // For example: `file:data/`. // Tigris parameters for `tigris` handler. // See https://www.tigrisdata.com/docs/sdkstools/golang/getting-started/ @@ -127,7 +125,7 @@ func New(config *Config) (*FerretDB, error) { PostgreSQLURL: config.PostgreSQLURL, - SQLiteURI: config.SQLiteURI, + SQLiteURL: config.SQLiteURL, TigrisURL: config.TigrisURL, TigrisClientID: config.TigrisClientID, diff --git a/integration/setup/listener.go b/integration/setup/listener.go index 352c8ccc5f71..d955e6f37b64 100644 --- a/integration/setup/listener.go +++ b/integration/setup/listener.go @@ -115,7 +115,7 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) strin require.Empty(tb, *targetURLF, "-target-url must be empty for in-process FerretDB") - var handler, sqliteURI string + var handler string switch *targetBackendF { case "ferretdb-pg": @@ -130,9 +130,6 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) strin require.Empty(tb, *hanaURLF, "-hana-url must be empty for %q", *targetBackendF) handler = "sqlite" - // TODO https://github.com/FerretDB/FerretDB/issues/2753 - sqliteURI = sqliteDir - case "ferretdb-tigris": require.Empty(tb, *postgreSQLURLF, "-postgresql-url must be empty for %q", *targetBackendF) require.NotEmpty(tb, *tigrisURLSF, "-tigris-urls must be set for %q", *targetBackendF) @@ -163,7 +160,7 @@ func setupListener(tb testing.TB, ctx context.Context, logger *zap.Logger) strin PostgreSQLURL: *postgreSQLURLF, - SQLiteURI: sqliteURI, + SQLiteURL: sqliteURL.String(), TigrisURL: nextTigrisUrl(), diff --git a/integration/setup/startup.go b/integration/setup/startup.go index 7c7dd7f71bce..21826d96de7d 100644 --- a/integration/setup/startup.go +++ b/integration/setup/startup.go @@ -17,8 +17,8 @@ package setup import ( "context" "net/http" + "net/url" "os" - "path/filepath" "runtime" "time" @@ -44,10 +44,10 @@ var listenerMetrics = connmetrics.NewListenerMetrics() // jaegerExporter is a shared Jaeger exporter for tests. var jaegerExporter *jaeger.Exporter -// sqliteDir is a fixed directory for SQLite backend tests. +// sqliteURL is a URI for SQLite backend tests. // // We don't use testing.T.TempDir() or something to make debugging of failed tests easier. -var sqliteDir = filepath.Join("..", "tmp", "sqlite-tests") +var sqliteURL = must.NotFail(url.Parse("file:../tmp/sqlite-tests/")) // Startup initializes things that should be initialized only once. func Startup() { @@ -83,8 +83,10 @@ func Startup() { zap.S().Fatalf("Unknown target backend %q.", *targetBackendF) } - _ = os.Remove(sqliteDir) - must.NoError(os.MkdirAll(sqliteDir, 0o777)) + must.BeTrue(sqliteURL.Path == "") + must.BeTrue(sqliteURL.Opaque != "") + _ = os.Remove(sqliteURL.Opaque) + must.NoError(os.MkdirAll(sqliteURL.Opaque, 0o777)) if u := *targetURLF; u != "" { client, err := makeClient(ctx, u) diff --git a/internal/backends/sqlite/backend.go b/internal/backends/sqlite/backend.go index 25138800b3ed..0020a881240d 100644 --- a/internal/backends/sqlite/backend.go +++ b/internal/backends/sqlite/backend.go @@ -16,47 +16,34 @@ package sqlite import ( "context" - "os" "go.uber.org/zap" _ "modernc.org/sqlite" "github.com/FerretDB/FerretDB/internal/backends" "github.com/FerretDB/FerretDB/internal/backends/sqlite/metadata" - "github.com/FerretDB/FerretDB/internal/util/lazyerrors" ) // backend implements backends.Backend interface. type backend struct { - r *metadata.Registry - dir string + r *metadata.Registry } // NewBackendParams represents the parameters of NewBackend function. type NewBackendParams struct { - Dir string + URI string L *zap.Logger } // NewBackend creates a new SQLite backend. func NewBackend(params *NewBackendParams) (backends.Backend, error) { - fi, err := os.Stat(params.Dir) + r, err := metadata.NewRegistry(params.URI, params.L.Named("metadata")) if err != nil { - return nil, lazyerrors.Errorf("%q should be an existing directory: %w", params.Dir, err) - } - - if !fi.IsDir() { - return nil, lazyerrors.Errorf("%q should be an existing directory", params.Dir) - } - - r, err := metadata.NewRegistry(params.Dir, params.L.Named("metadata")) - if err != nil { - return nil, lazyerrors.Error(err) + return nil, err } return backends.BackendContract(&backend{ - r: r, - dir: params.Dir, + r: r, }), nil } diff --git a/internal/backends/sqlite/metadata/pool/pool.go b/internal/backends/sqlite/metadata/pool/pool.go index 377c25b2659b..6cf584926009 100644 --- a/internal/backends/sqlite/metadata/pool/pool.go +++ b/internal/backends/sqlite/metadata/pool/pool.go @@ -20,7 +20,10 @@ package pool import ( "context" "database/sql" + "fmt" + "net/url" "os" + "path" "path/filepath" "strings" "sync" @@ -40,7 +43,7 @@ const filenameExtension = ".sqlite" // //nolint:vet // for readability type Pool struct { - dir string + uri *url.URL l *zap.Logger rw sync.RWMutex @@ -50,17 +53,19 @@ type Pool struct { } // New creates a pool for SQLite databases in the given directory. -func New(dir string, l *zap.Logger) (*Pool, error) { - // TODO accept URI with a directory name, not just directory name - // https://github.com/FerretDB/FerretDB/issues/2753 +func New(u string, l *zap.Logger) (*Pool, error) { + uri, err := validateURI(u) + if err != nil { + return nil, fmt.Errorf("failed to parse SQLite URI %q: %s", u, err) + } - matches, err := filepath.Glob(filepath.Join(dir, "*"+filenameExtension)) + matches, err := filepath.Glob(filepath.Join(uri.Path, "*"+filenameExtension)) if err != nil { return nil, lazyerrors.Error(err) } p := &Pool{ - dir: dir, + uri: uri, l: l, dbs: make(map[string]*db, len(matches)), token: resource.NewToken(), @@ -68,27 +73,85 @@ func New(dir string, l *zap.Logger) (*Pool, error) { resource.Track(p, p.token) - for _, m := range matches { - db, err := openDB(m) + for _, f := range matches { + name := p.databaseName(f) + uri := p.databaseURI(name) + + p.l.Debug("Opening existing database.", zap.String("name", name), zap.String("uri", uri)) + + db, err := openDB(uri) if err != nil { p.Close() return nil, lazyerrors.Error(err) } - p.dbs[p.databaseName(m)] = db + p.dbs[name] = db } return p, nil } +// validateURI checks given URI value and returns parsed URL. +// URI should contain 'file' scheme and point to an existing directory. +// Path should end with '/'. Authority should be empty or absent. +// +// Returned URL contains path in both Path and Opaque to make String() method work correctly. +func validateURI(u string) (*url.URL, error) { + uri, err := url.Parse(u) + if err != nil { + return nil, err + } + + if uri.Scheme != "file" { + return nil, fmt.Errorf(`expected "file:" schema, got %q`, uri.Scheme) + } + + if uri.User != nil { + return nil, fmt.Errorf(`expected empty user info, got %q`, uri.User) + } + + if uri.Host != "" { + return nil, fmt.Errorf(`expected empty host, got %q`, uri.Host) + } + + if uri.Path == "" && uri.Opaque != "" { + uri.Path = uri.Opaque + } + uri.Opaque = uri.Path + + if !strings.HasSuffix(uri.Path, "/") { + return nil, fmt.Errorf(`expected path ending with "/", got %q`, uri.Path) + } + + fi, err := os.Stat(uri.Path) + if err != nil { + return nil, fmt.Errorf(`%q should be an existing directory, got %s`, uri.Path, err) + } + + if !fi.IsDir() { + return nil, fmt.Errorf(`%q should be an existing directory`, uri.Path) + } + + return uri, nil +} + // databaseName returns database name for given database file path. -func (p *Pool) databaseName(databasePath string) string { - return strings.TrimSuffix(filepath.Base(databasePath), filenameExtension) +func (p *Pool) databaseName(databaseFile string) string { + return strings.TrimSuffix(filepath.Base(databaseFile), filenameExtension) +} + +// databaseURI returns SQLite URI for the given database name. +func (p *Pool) databaseURI(databaseName string) string { + dbURI := *p.uri + dbURI.Path = path.Join(dbURI.Path, databaseName+filenameExtension) + dbURI.Opaque = dbURI.Path + + return dbURI.String() } -// databasePath returns database file path for the given database name. -func (p *Pool) databasePath(databaseName string) string { - return filepath.Join(p.dir, databaseName+filenameExtension) +// databaseFile returns database file path for the given database name. +func (p *Pool) databaseFile(databaseName string) string { + return filepath.Join(p.uri.Path, databaseName+filenameExtension) } // Close closes all databases in the pool and frees all resources. @@ -146,12 +209,13 @@ func (p *Pool) GetOrCreate(ctx context.Context, name string) (*sql.DB, bool, err return db.sqlDB, false, nil } - db, err := openDB(p.databasePath(name)) + uri := p.databaseURI(name) + db, err := openDB(uri) if err != nil { - return nil, false, lazyerrors.Error(err) + return nil, false, lazyerrors.Errorf("%s: %w", uri, err) } - p.l.Debug("Database created", zap.String("name", name)) + p.l.Debug("Database created.", zap.String("name", name), zap.String("uri", uri)) p.dbs[name] = db @@ -173,10 +237,10 @@ func (p *Pool) Drop(ctx context.Context, name string) bool { } _ = db.Close() - _ = os.Remove(p.databasePath(name)) + _ = os.Remove(p.databaseFile(name)) delete(p.dbs, name) - p.l.Debug("Database dropped", zap.String("name", name)) + p.l.Debug("Database dropped.", zap.String("name", name)) return true } diff --git a/internal/backends/sqlite/metadata/pool/pool_test.go b/internal/backends/sqlite/metadata/pool/pool_test.go index 350af70d56a5..fe5c6b037a71 100644 --- a/internal/backends/sqlite/metadata/pool/pool_test.go +++ b/internal/backends/sqlite/metadata/pool/pool_test.go @@ -15,8 +15,11 @@ package pool import ( + "net/url" + "os" "testing" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/FerretDB/FerretDB/internal/util/testutil" @@ -25,7 +28,7 @@ import ( func TestCreateDrop(t *testing.T) { ctx := testutil.Ctx(t) - p, err := New(t.TempDir(), testutil.Logger(t)) + p, err := New("file:"+t.TempDir()+"/", testutil.Logger(t)) require.NoError(t, err) defer p.Close() @@ -55,3 +58,127 @@ func TestCreateDrop(t *testing.T) { db = p.GetExisting(ctx, t.Name()) require.Nil(t, db) } + +func TestNewBackend(t *testing.T) { + t.Parallel() + + err := os.MkdirAll("tmp/dir", 0o777) + require.NoError(t, err) + + _, err = os.Create("tmp/file") + require.NoError(t, err) + + t.Cleanup(func() { + err := os.RemoveAll("tmp") + require.NoError(t, err) + }) + + testCases := map[string]struct { + value string + uri *url.URL + err string + }{ + "LocalDirectory": { + value: "file:./", + uri: &url.URL{ + Scheme: "file", + Opaque: "./", + Path: "./", + }, + }, + "LocalSubDirectory": { + value: "file:./tmp/", + uri: &url.URL{ + Scheme: "file", + Opaque: "./tmp/", + Path: "./tmp/", + }, + }, + "LocalSubSubDirectory": { + value: "file:./tmp/dir/", + uri: &url.URL{ + Scheme: "file", + Opaque: "./tmp/dir/", + Path: "./tmp/dir/", + }, + }, + "LocalDirectoryWithParameters": { + value: "file:./tmp/?mode=ro", + uri: &url.URL{ + Scheme: "file", + Opaque: "./tmp/", + Path: "./tmp/", + RawQuery: "mode=ro", + }, + }, + "AbsoluteDirectory": { + value: "file:/tmp/", + uri: &url.URL{ + Scheme: "file", + Opaque: "/tmp/", + Path: "/tmp/", + OmitHost: true, + }, + }, + "WithEmptyAuthority": { + value: "file:///tmp/", + uri: &url.URL{ + Scheme: "file", + Opaque: "/tmp/", + Path: "/tmp/", + }, + }, + "WithEmptyAuthorityAndQuery": { + value: "file:///tmp/?mode=ro", + uri: &url.URL{ + Scheme: "file", + Opaque: "/tmp/", + Path: "/tmp/", + RawQuery: "mode=ro", + }, + }, + "HostIsNotEmpty": { + value: "file://localhost/./tmp/?mode=ro", + err: `expected empty host, got "localhost"`, + }, + "UserIsNotEmpty": { + value: "file://user:pass@./tmp/?mode=ro", + err: `expected empty user info, got "user:pass"`, + }, + "NoDirectory": { + value: "file:./nodir/", + err: `"./nodir/" should be an existing directory, got stat ./nodir/: no such file or directory`, + }, + "PathIsNotEndsWithSlash": { + value: "file:./tmp/file", + err: `expected path ending with "/", got "./tmp/file"`, + }, + "FileInsteadOfDirectory": { + value: "file:./tmp/file/", + err: `"./tmp/file/" should be an existing directory, got stat ./tmp/file/: not a directory`, + }, + "MalformedURI": { + value: ":./tmp/", + err: `parse ":./tmp/": missing protocol scheme`, + }, + "NoScheme": { + value: "./tmp/", + err: `expected "file:" schema, got ""`, + }, + } + for name, tc := range testCases { + name, tc := name, tc + t.Run(name, func(t *testing.T) { + t.Parallel() + + u, err := validateURI(tc.value) + if tc.err != "" { + assert.EqualError(t, err, tc.err) + return + } + + require.NoError(t, err) + assert.Equal(t, u, tc.uri) + }) + } +} diff --git a/internal/backends/sqlite/metadata/registry.go b/internal/backends/sqlite/metadata/registry.go index be1f4c991f4e..fcbf08c277b8 100644 --- a/internal/backends/sqlite/metadata/registry.go +++ b/internal/backends/sqlite/metadata/registry.go @@ -41,10 +41,10 @@ type Registry struct { } // NewRegistry creates a registry for the given directory. -func NewRegistry(dir string, l *zap.Logger) (*Registry, error) { - p, err := pool.New(dir, l.Named("pool")) +func NewRegistry(u string, l *zap.Logger) (*Registry, error) { + p, err := pool.New(u, l.Named("pool")) if err != nil { - return nil, lazyerrors.Error(err) + return nil, err } return &Registry{ diff --git a/internal/handlers/registry/registry.go b/internal/handlers/registry/registry.go index e898cd6bd40e..b10b075c66bf 100644 --- a/internal/handlers/registry/registry.go +++ b/internal/handlers/registry/registry.go @@ -45,7 +45,7 @@ type NewHandlerOpts struct { PostgreSQLURL string // for `sqlite` handler - SQLiteURI string + SQLiteURL string // for `tigris` handler TigrisURL string diff --git a/internal/handlers/registry/sqlite.go b/internal/handlers/registry/sqlite.go index e2bb55469378..cd31daa93a65 100644 --- a/internal/handlers/registry/sqlite.go +++ b/internal/handlers/registry/sqlite.go @@ -25,7 +25,7 @@ func init() { opts.Logger.Warn("SQLite handler is in alpha. It is not supported yet.") handlerOpts := &sqlite.NewOpts{ - Dir: opts.SQLiteURI, + URI: opts.SQLiteURL, L: opts.Logger.Named("sqlite"), ConnMetrics: opts.ConnMetrics, diff --git a/internal/handlers/sqlite/sqlite.go b/internal/handlers/sqlite/sqlite.go index c8ac3e77f916..7fadc08fbd18 100644 --- a/internal/handlers/sqlite/sqlite.go +++ b/internal/handlers/sqlite/sqlite.go @@ -51,7 +51,7 @@ type Handler struct { // //nolint:vet // for readability type NewOpts struct { - Dir string + URI string L *zap.Logger ConnMetrics *connmetrics.ConnMetrics @@ -64,7 +64,7 @@ type NewOpts struct { // New returns a new handler. func New(opts *NewOpts) (handlers.Interface, error) { b, err := sqlite.NewBackend(&sqlite.NewBackendParams{ - Dir: opts.Dir, + URI: opts.URI, L: opts.L, }) if err != nil { From 7fbc583180abb57bf190adfa8e2aeac646210540 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Mon, 3 Jul 2023 11:52:09 +0400 Subject: [PATCH 45/46] Do not break fuzzing initialization (#2951) Closes #2940. Refs #1636. --- internal/handlers/sjson/sjson_test.go | 4 ++++ internal/wire/record.go | 11 ++++++++--- internal/wire/wire_test.go | 4 ++++ 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/internal/handlers/sjson/sjson_test.go b/internal/handlers/sjson/sjson_test.go index 66c97d566703..af67f824cbcc 100644 --- a/internal/handlers/sjson/sjson_test.go +++ b/internal/handlers/sjson/sjson_test.go @@ -169,6 +169,10 @@ func fuzzJSON(f *testing.F, testCases []testCase, newFunc func() sjsontype) { var n int for _, rec := range records { + if rec.Body == nil { + continue + } + var docs []*types.Document switch b := rec.Body.(type) { diff --git a/internal/wire/record.go b/internal/wire/record.go index b2d8da10cfe2..6b2822b423e0 100644 --- a/internal/wire/record.go +++ b/internal/wire/record.go @@ -27,8 +27,11 @@ import ( // Record represents a single recorded wire protocol message, loaded from a .bin file. type Record struct { - Header *MsgHeader - Body MsgBody + // those may be unset if message is invalid + Header *MsgHeader + Body MsgBody + + // those are always set HeaderB []byte BodyB []byte } @@ -96,8 +99,10 @@ func loadRecordFile(file string) ([]Record, error) { break } + // TODO we should still set HeaderB and BodyB + // https://github.com/FerretDB/FerretDB/issues/1636 if err != nil { - return nil, lazyerrors.Error(err) + break } headerB, err := header.MarshalBinary() diff --git a/internal/wire/wire_test.go b/internal/wire/wire_test.go index 5a6fbbf6e086..a99336203507 100644 --- a/internal/wire/wire_test.go +++ b/internal/wire/wire_test.go @@ -146,6 +146,10 @@ func fuzzMessages(f *testing.F, testCases []testCase) { require.NoError(f, err) for _, rec := range records { + if rec.HeaderB == nil || rec.BodyB == nil { + continue + } + b := make([]byte, 0, len(rec.HeaderB)+len(rec.BodyB)) b = append(b, rec.HeaderB...) b = append(b, rec.BodyB...) From f939bd2f8d06c4ab9b3ae365b6276848f76fb649 Mon Sep 17 00:00:00 2001 From: Alexey Palazhchenko Date: Mon, 3 Jul 2023 12:26:35 +0300 Subject: [PATCH 46/46] Prepare v1.5.0 release --- CHANGELOG.md | 80 +++++++++++++++++++++++++- internal/handlers/registry/sqlite.go | 2 +- internal/handlers/registry/tigris.go | 2 +- website/docs/configuration/flags.md | 16 ++---- website/docs/understanding-ferretdb.md | 11 +--- 5 files changed, 88 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b52e870ab50..05f47c862bac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,82 @@ +## [v1.5.0](https://github.com/FerretDB/FerretDB/releases/tag/v1.5.0) (2023-07-03) + +### What's Changed + +This release provides beta-level support for the SQLite backend. +There is some missing functionality, but it is ready for early adopters. + +This release provides improved cursor support, enabling commands like `find` and `aggregate` to return large data sets much more effectively. + +Tigris data users: Please note that this is the last release of FerretDB which includes support for the Tigris backend. +Starting from FerretDB v1.6.0, Tigris will not be supported. +If you wish to use Tigris, please do not update FerretDB beyond v1.5.0. +This and earlier versions of FerretDB with Tigris support will still be available on GitHub. + +### New Features 🎉 + +- Implement `count` for SQLite by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2865 +- Enable cursor support for PostgreSQL and SQLite by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2864 + +### Enhancements 🛠 + +- Support `find` `singleBatch` and validate `getMore` parameters by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2855 +- Support cursors for aggregation pipelines by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2861 +- Fix collection name starting with dot validation by @noisersup in https://github.com/FerretDB/FerretDB/pull/2912 +- Improve validation for `createIndexes` and `dropIndexes` by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2884 +- Use cursors in `find` command by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2933 + +### Documentation 📄 + +- Add blogpost on FerretDB v1.4.0 by @Fashander in https://github.com/FerretDB/FerretDB/pull/2858 +- Add blog post on "Meet FerretDB at Percona University in Casablanca and Belgrade" by @Fashander in https://github.com/FerretDB/FerretDB/pull/2870 +- Update supported commands by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2876 +- Add blog post "FerretDB Demo: Launch and Test a Database in Minutes" by @Fashander in https://github.com/FerretDB/FerretDB/pull/2851 +- Fix Github link for Dance repository by @Matthieu68857 in https://github.com/FerretDB/FerretDB/pull/2887 +- Add blog post on "How to Configure FerretDB to work on Percona Distribution for PostgreSQL" by @Fashander in https://github.com/FerretDB/FerretDB/pull/2911 +- Update incorrect blog post image by @Fashander in https://github.com/FerretDB/FerretDB/pull/2920 +- Crush PNG images by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2931 + +### Other Changes 🤖 + +- Add more validation and tests for `$unset` by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2853 +- Make it easier to debug GitHub Actions by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2860 +- Unify tests for indexes by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2866 +- Bump deps by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2875 +- Fix fuzzing corpus collection by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2879 +- Add basic tests for iterators by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2880 +- Implement basic `insert` support for SAP HANA by @polyal in https://github.com/FerretDB/FerretDB/pull/2732 +- Update contributing docs by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2828 +- Improve `wire` and `sjson` fuzzing by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2883 +- Add operators support for `$addFields` by @noisersup in https://github.com/FerretDB/FerretDB/pull/2850 +- Unskip test that passes now by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2885 +- Tweak contributing guidelines by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2886 +- Add handler's metrics registration by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2895 +- Clean-up some code and comments by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2904 +- Fix cancelation signals propagation by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2908 +- Bump deps, add permissions monitoring by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2930 +- Fix integration tests after bumping deps by @noisersup in https://github.com/FerretDB/FerretDB/pull/2934 +- Update benchmark to use cursors by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2932 +- Set `minWireVersion` to 0 by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2937 +- Test `getMore` integration test using one connection pool by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2878 +- Add better metrics for connections by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2938 +- Use cursors with iterator in `aggregate` command by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2929 +- Implement proper response for `createIndexes` by @rumyantseva in https://github.com/FerretDB/FerretDB/pull/2936 +- Re-implement `DELETE` for SQLite backend by @w84thesun in https://github.com/FerretDB/FerretDB/pull/2907 +- Validate database names for SQLite handler by @noisersup in https://github.com/FerretDB/FerretDB/pull/2924 +- Add `insert` documents type validation by @w84thesun in https://github.com/FerretDB/FerretDB/pull/2946 +- Convert SQLite directory to URI by @w84thesun in https://github.com/FerretDB/FerretDB/pull/2922 +- Do not break fuzzing initialization by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2951 + +### New Contributors + +- @Matthieu68857 made their first contribution in https://github.com/FerretDB/FerretDB/pull/2887 + +[All closed issues and pull requests](https://github.com/FerretDB/FerretDB/milestone/45?closed=1). +[All commits](https://github.com/FerretDB/FerretDB/compare/v1.4.0...v1.5.0). + ## [v1.4.0](https://github.com/FerretDB/FerretDB/releases/tag/v1.4.0) (2023-06-19) ### New Features 🎉 @@ -46,7 +122,7 @@ - Improve benchmarks by @AlekSi in https://github.com/FerretDB/FerretDB/pull/2833 - Handle `$type` aggregation operator errors properly by @noisersup in https://github.com/FerretDB/FerretDB/pull/2829 -## New Contributors +### New Contributors - @shibasisp made their first contribution in https://github.com/FerretDB/FerretDB/pull/2676 @@ -113,7 +189,7 @@ - Add more handler tests by @w84thesun in https://github.com/FerretDB/FerretDB/pull/2769 - Remove `findAndModify` integration tests with `$` prefixed key for MongoDB 6.0.6 compatibility by @chilagrow in https://github.com/FerretDB/FerretDB/pull/2785 -## New Contributors +### New Contributors - @jeremyphua made their first contribution in https://github.com/FerretDB/FerretDB/pull/2714 diff --git a/internal/handlers/registry/sqlite.go b/internal/handlers/registry/sqlite.go index cd31daa93a65..cb3f8f851174 100644 --- a/internal/handlers/registry/sqlite.go +++ b/internal/handlers/registry/sqlite.go @@ -22,7 +22,7 @@ import ( // init registers "sqlite" handler. func init() { registry["sqlite"] = func(opts *NewHandlerOpts) (handlers.Interface, error) { - opts.Logger.Warn("SQLite handler is in alpha. It is not supported yet.") + opts.Logger.Warn("SQLite handler is in beta.") handlerOpts := &sqlite.NewOpts{ URI: opts.SQLiteURL, diff --git a/internal/handlers/registry/tigris.go b/internal/handlers/registry/tigris.go index 96561da2d50d..5b7554f7643e 100644 --- a/internal/handlers/registry/tigris.go +++ b/internal/handlers/registry/tigris.go @@ -24,7 +24,7 @@ import ( // init registers "tigris" handler for Tigris when "ferretdb_tigris" build tag is provided. func init() { registry["tigris"] = func(opts *NewHandlerOpts) (handlers.Interface, error) { - opts.Logger.Warn("Tigris handler is in beta.") + opts.Logger.Warn("This is the last release with Tigris support.") handlerOpts := &tigris.NewOpts{ AuthParams: tigris.AuthParams{ diff --git a/website/docs/configuration/flags.md b/website/docs/configuration/flags.md index f3643110f71a..4f1df02ea3d0 100644 --- a/website/docs/configuration/flags.md +++ b/website/docs/configuration/flags.md @@ -55,18 +55,14 @@ Some default values are overridden in [our Docker image](quickstart-guide/docker | ------------------ | ------------------------------- | ------------------------- | ------------------------------------ | | `--postgresql-url` | PostgreSQL URL for 'pg' handler | `FERRETDB_POSTGRESQL_URL` | `postgres://127.0.0.1:5432/ferretdb` | - +### SQLite (beta) -### Tigris (beta) +[SQLite backend](../understanding-ferretdb.md#sqlite-beta) can be enabled by +`--handler=sqlite` flag or `FERRETDB_HANDLER=sqlite` environment variable. -[Tigris backend](../understanding-ferretdb.md#tigris-beta) can be enabled by -`--handler=tigris` flag or `FERRETDB_HANDLER=tigris` environment variable. - -| Flag | Description | Environment Variable | Default Value | -| ------------------------ | ------------------------------- | ------------------------------- | ---------------- | -| `--tigris-url` | Tigris URL for 'tigris' handler | `FERRETDB_TIGRIS_URL` | `127.0.0.1:8081` | -| `--tigris-client-id` | Tigris Client ID | `FERRETDB_TIGRIS_CLIENT_ID` | | -| `--tigris-client-secret` | Tigris Client secret | `FERRETDB_TIGRIS_CLIENT_SECRET` | | +| Flag | Description | Environment Variable | Default Value | +| -------------- | ------------------------------------------- | --------------------- | ------------- | +| `--sqlite-url` | SQLite URI (directory) for 'sqlite' handler | `FERRETDB_SQLITE_URL` | `file:data/` | ## Miscellaneous diff --git a/website/docs/understanding-ferretdb.md b/website/docs/understanding-ferretdb.md index 51017e9fcc92..79e35d564cc5 100644 --- a/website/docs/understanding-ferretdb.md +++ b/website/docs/understanding-ferretdb.md @@ -15,7 +15,6 @@ Before inserting data into a document, you do not need to declare a schema. That makes it ideal for applications and workloads requiring flexible schemas, such as blogs, chat apps, and video games. :::note -For Tigris, FerretDB requires you to declare a JSON schema in [Tigris format](https://docs.tigrisdata.com/overview/schema). Get more information on the key differences [here](diff.md). ::: @@ -168,15 +167,9 @@ MongoDB documents are mapped to rows with a single [JSONB](https://www.postgresq Those mappings might change as we work on improving compatibility and performance, but no breaking changes will be introduced without a major version bump. -### SQLite (alpha) +### SQLite (beta) -We are [working on](https://github.com/FerretDB/FerretDB/issues/2387) SQLite backend. -It is not officially supported yet. - -### Tigris (beta) - -We also support the [Tigris](https://www.tigrisdata.com) backend on a beta level. -Read more [here](https://www.tigrisdata.com/docs/concepts/mongodb-compatibility/). +We also support the [SQLite](https://www.sqlite.org/) backend on a beta level. ### SAP HANA (alpha)