Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store indexes metadata #3434

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
5f63251
wip
rumyantseva Sep 26, 2023
4632b38
wip
rumyantseva Sep 26, 2023
b4bac2d
Merge branch 'main' into issue-3375-pg-indexes-mentadata
Sep 26, 2023
9547f3a
wip
rumyantseva Sep 26, 2023
ada4498
wip
rumyantseva Sep 26, 2023
d31b7ba
wip
rumyantseva Sep 27, 2023
61bb540
wip
rumyantseva Sep 27, 2023
86e88c6
wip
rumyantseva Sep 27, 2023
ccc1b97
wip
rumyantseva Sep 27, 2023
97e0a21
fixed a strange thing
rumyantseva Sep 27, 2023
fa4684f
merge conflicts
rumyantseva Sep 28, 2023
e97422b
bugfix
rumyantseva Sep 28, 2023
cdaca09
fix test
rumyantseva Sep 28, 2023
a198a14
linter things
rumyantseva Sep 28, 2023
1d6f3fb
Merge branch 'main' into issue-3375-pg-indexes-mentadata
rumyantseva Sep 28, 2023
acb5d20
Merge branch 'main' into issue-3375-pg-indexes-mentadata
rumyantseva Sep 28, 2023
49ece35
wip
rumyantseva Sep 28, 2023
ed53d66
wip
rumyantseva Sep 29, 2023
184ec9a
wip
rumyantseva Sep 29, 2023
c1e2861
wip
rumyantseva Sep 29, 2023
0d7bf07
wip
rumyantseva Sep 29, 2023
586f314
wip
rumyantseva Sep 29, 2023
3661271
wip
rumyantseva Sep 29, 2023
9a9a5a8
wip
rumyantseva Sep 29, 2023
97afd35
wip
rumyantseva Sep 29, 2023
db39ee5
wip
rumyantseva Sep 29, 2023
05139bb
wip
rumyantseva Sep 29, 2023
650df0d
Merge branch 'main' into issue-3375-pg-indexes-mentadata
Sep 29, 2023
8019fe0
Merge branch 'main' into issue-3375-pg-indexes-mentadata
Sep 29, 2023
0fb411a
wip
rumyantseva Sep 29, 2023
71c8389
wip
rumyantseva Sep 29, 2023
e577da1
wip
rumyantseva Sep 29, 2023
fc0ad27
Merge branch 'main' into issue-3375-pg-indexes-mentadata
AlekSi Oct 2, 2023
ebe36f1
Simplify
AlekSi Oct 2, 2023
0f6cf2c
Tidy comments
AlekSi Oct 2, 2023
f268901
Extract settings
AlekSi Oct 2, 2023
402e731
Refactor
AlekSi Oct 2, 2023
3fc9156
Cleanup
AlekSi Oct 2, 2023
9819817
Remove tests for library functions
AlekSi Oct 2, 2023
800d3b7
Merge branch 'main' into issue-3375-pg-indexes-mentadata
AlekSi Oct 4, 2023
47de006
version without uuid
rumyantseva Oct 4, 2023
11958ae
simpler way to check for existing indexes
rumyantseva Oct 4, 2023
b03d927
wip
rumyantseva Oct 4, 2023
14ad43d
Merge branch 'main' into issue-3375-pg-indexes-mentadata
Oct 4, 2023
4ebd311
wip
rumyantseva Oct 4, 2023
ee8a461
wip
rumyantseva Oct 4, 2023
bc35680
wip
rumyantseva Oct 4, 2023
97e89f7
wip
rumyantseva Oct 4, 2023
335953f
Merge branch 'main' into issue-3375-pg-indexes-mentadata
Oct 4, 2023
13a7cf7
wip
rumyantseva Oct 5, 2023
bc1800a
Merge branch 'main' into issue-3375-pg-indexes-mentadata
AlekSi Oct 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions internal/backends/doc.go → internal/backends/backends.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,8 @@
// But there are some common tests for all backends that check corner cases in contracts.
// They also test that all backends adjusted to contract changes.
//
// Some backends may have their own tests.
//
// Both kinds of tests could be removed over time as they are replaced by integration tests.
// Prefer integration tests when possible.
package backends
67 changes: 0 additions & 67 deletions internal/backends/postgresql/collection_test.go

This file was deleted.

2 changes: 1 addition & 1 deletion internal/backends/postgresql/database_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func TestDatabaseStats(t *testing.T) {
require.NotZero(t, res.SizeCollections)
require.Zero(t, res.CountObjects)
require.Zero(t, res.CountIndexes)
require.Zero(t, res.SizeIndexes)
require.NotZero(t, res.SizeIndexes) // includes metadata table's indexes
})

t.Run("DatabaseWithCollections", func(t *testing.T) {
Expand Down
21 changes: 21 additions & 0 deletions internal/backends/postgresql/dummy_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// Copyright 2021 FerretDB Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package postgresql

import "testing"

func TestDummy(t *testing.T) {
// we need at least one test per package to correctly calculate coverage
}
139 changes: 139 additions & 0 deletions internal/backends/postgresql/metadata/indexes.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
// Copyright 2021 FerretDB Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package metadata

import (
"errors"

"golang.org/x/exp/slices"

"github.com/FerretDB/FerretDB/internal/types"
"github.com/FerretDB/FerretDB/internal/util/iterator"
"github.com/FerretDB/FerretDB/internal/util/lazyerrors"
"github.com/FerretDB/FerretDB/internal/util/must"
)

// Indexes represents information about all indexes in a collection.
type Indexes []IndexInfo

// IndexInfo represents information about a single index.
type IndexInfo struct {
Name string
PgIndex string
Key []IndexKeyPair
Unique bool
}

// IndexKeyPair consists of a field name and a sort order that are part of the index.
type IndexKeyPair struct {
Field string
Descending bool
}

// deepCopy returns a deep copy.
func (indexes Indexes) deepCopy() Indexes {
res := make(Indexes, len(indexes))

for i, index := range indexes {
res[i] = IndexInfo{
Name: index.Name,
PgIndex: index.PgIndex,
Key: slices.Clone(index.Key),
Unique: index.Unique,
}
}

return res
}

// marshal returns [*types.Array] for indexes.
func (indexes Indexes) marshal() *types.Array {
res := types.MakeArray(len(indexes))

for _, index := range indexes {
key := types.MakeDocument(len(index.Key))

for _, pair := range index.Key {
order := int32(1)
if pair.Descending {
order = int32(-1)
}

key.Set(pair.Field, order)
}

res.Append(must.NotFail(types.NewDocument(
"pgindex", index.PgIndex,
"name", index.Name,
"key", key,
"unique", index.Unique,
)))
}

return res
}

// unmarshal sets indexes from [*types.Array].
func (s *Indexes) unmarshal(a *types.Array) error {
res := make(Indexes, a.Len())

iter := a.Iterator()
defer iter.Close()

for {
i, v, err := iter.Next()
if errors.Is(err, iterator.ErrIteratorDone) {
break
}

if err != nil {
return lazyerrors.Error(err)
}

Check warning on line 103 in internal/backends/postgresql/metadata/indexes.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/indexes.go#L102-L103

Added lines #L102 - L103 were not covered by tests

index := v.(*types.Document)

keyDoc := must.NotFail(index.Get("key")).(*types.Document)
fields := keyDoc.Keys()
orders := keyDoc.Values()
key := make([]IndexKeyPair, keyDoc.Len())

for j, f := range fields {
descending := false
if orders[j].(int32) == -1 {
descending = true
}

key[j] = IndexKeyPair{
Field: f,
Descending: descending,
}
}

// it was possible for it to be null in pgdb
v, _ = index.Get("unique")
unique, _ := v.(bool)

res[i] = IndexInfo{
Name: must.NotFail(index.Get("name")).(string),
PgIndex: must.NotFail(index.Get("pgindex")).(string),
Key: key,
Unique: unique,
}
}

*s = res

return nil
}
90 changes: 83 additions & 7 deletions internal/backends/postgresql/metadata/metadata.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@
package metadata

import (
"database/sql"
"database/sql/driver"

"github.com/FerretDB/FerretDB/internal/handlers/sjson"
"github.com/FerretDB/FerretDB/internal/types"
"github.com/FerretDB/FerretDB/internal/util/lazyerrors"
"github.com/FerretDB/FerretDB/internal/util/must"
)

Expand All @@ -29,10 +34,13 @@
)

// Collection represents collection metadata.
//
// Collection value should be immutable to avoid data races.
// Use [deepCopy] to replace the whole value instead of modifying fields of existing value.
type Collection struct {
Name string
TableName string
// TODO https://github.com/FerretDB/FerretDB/issues/3375
Indexes Indexes
}

// deepCopy returns a deep copy.
Expand All @@ -44,21 +52,89 @@
return &Collection{
Name: c.Name,
TableName: c.TableName,
Indexes: c.Indexes.deepCopy(),
}
}

// Value implements driver.Valuer interface.
func (c Collection) Value() (driver.Value, error) {
b, err := sjson.Marshal(c.marshal())
if err != nil {
return nil, lazyerrors.Error(err)

Check warning on line 63 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L63

Added line #L63 was not covered by tests
}

return b, nil
}

// Marshal returns [*types.Document] for that collection.
func (c *Collection) Marshal() *types.Document {
// Scan implements sql.Scanner interface.
func (c *Collection) Scan(src any) error {
var doc *types.Document
var err error

switch src := src.(type) {
case nil:
*c = Collection{}
return nil
case []byte:
doc, err = sjson.Unmarshal(src)

Check warning on line 79 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L75-L79

Added lines #L75 - L79 were not covered by tests
case string:
doc, err = sjson.Unmarshal([]byte(src))
default:
panic("can't scan collection")

Check warning on line 83 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L82-L83

Added lines #L82 - L83 were not covered by tests
}

if err != nil {
return lazyerrors.Error(err)
}

Check warning on line 88 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L87-L88

Added lines #L87 - L88 were not covered by tests

if err = c.unmarshal(doc); err != nil {
return lazyerrors.Error(err)
}

Check warning on line 92 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L91-L92

Added lines #L91 - L92 were not covered by tests

return nil
}

// marshal returns [*types.Document] for that collection.
func (c *Collection) marshal() *types.Document {
return must.NotFail(types.NewDocument(
"_id", c.Name,
"table", c.TableName,
"indexes", c.Indexes.marshal(),
))
}

// Unmarshal sets collection metadata from [*types.Document].
func (c *Collection) Unmarshal(doc *types.Document) error {
c.Name = must.NotFail(doc.Get("_id")).(string)
c.TableName = must.NotFail(doc.Get("table")).(string)
// unmarshal sets collection metadata from [*types.Document].
func (c *Collection) unmarshal(doc *types.Document) error {
v, _ := doc.Get("_id")
c.Name, _ = v.(string)

if c.Name == "" {
return lazyerrors.New("collection name is empty")
}

Check warning on line 113 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L112-L113

Added lines #L112 - L113 were not covered by tests

v, _ = doc.Get("table")
c.TableName, _ = v.(string)

if c.TableName == "" {
return lazyerrors.New("table name is empty")
}

Check warning on line 120 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L119-L120

Added lines #L119 - L120 were not covered by tests

v, _ = doc.Get("indexes")
i, _ := v.(*types.Array)

if i == nil {
return lazyerrors.New("indexes are empty")
}

Check warning on line 127 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L126-L127

Added lines #L126 - L127 were not covered by tests

if err := c.Indexes.unmarshal(i); err != nil {
return lazyerrors.Error(err)
}

Check warning on line 131 in internal/backends/postgresql/metadata/metadata.go

View check run for this annotation

Codecov / codecov/patch

internal/backends/postgresql/metadata/metadata.go#L130-L131

Added lines #L130 - L131 were not covered by tests

return nil
}

// check interfaces
var (
_ driver.Valuer = Collection{}
_ sql.Scanner = (*Collection)(nil)
)
21 changes: 21 additions & 0 deletions internal/backends/postgresql/metadata/pool/dummy_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// Copyright 2021 FerretDB Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package pool

import "testing"

func TestDummy(t *testing.T) {
// we need at least one test per package to correctly calculate coverage
}
Loading
Loading