Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate database and collection names for SQLite handler #2868

Merged
merged 89 commits into from
Jul 26, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
6e3df55
wip
noisersup Jun 20, 2023
8a2af00
wip
noisersup Jun 21, 2023
ae491af
wip
noisersup Jun 21, 2023
3f84105
basic
noisersup Jun 22, 2023
a5ca6cb
remove CollectionToTable from struct
noisersup Jun 22, 2023
88182fb
Replace with collectionGet
noisersup Jun 22, 2023
2009ea8
it works???
noisersup Jun 22, 2023
07738c7
test case
noisersup Jun 23, 2023
004aa8e
wip
noisersup Jun 23, 2023
1178e65
prefix validation
noisersup Jun 23, 2023
019ceb7
Merge branch 'main' into validate-db-coll-2749
noisersup Jun 23, 2023
8c3714d
tweak
noisersup Jun 23, 2023
1012a42
wip
noisersup Jun 23, 2023
7230c38
fix dot for postgres
noisersup Jun 26, 2023
176cb05
fix dot for sqlite
noisersup Jun 26, 2023
f184458
Merge remote-tracking branch 'upstream/main' into validate-db-coll-2749
noisersup Jun 26, 2023
2fbbd64
Update sqlite comment
noisersup Jun 26, 2023
dac0fa9
wip
noisersup Jun 26, 2023
f97a473
add sqlite validation
noisersup Jun 26, 2023
c179a50
add comment + lint
noisersup Jun 27, 2023
4bfe05b
fmt
noisersup Jun 27, 2023
825b11b
Merge branch 'main' into validate-db-coll-2749
noisersup Jun 27, 2023
4d98281
wip
noisersup Jun 27, 2023
0332333
wip
noisersup Jun 27, 2023
257e866
make all table names lowercase
noisersup Jun 27, 2023
7505d9a
proper error handling
noisersup Jun 27, 2023
f05bf62
lot of changes
noisersup Jun 27, 2023
15d7c77
Merge remote-tracking branch 'upstream/main' into validate-db-coll-2749
noisersup Jun 27, 2023
1ef2680
wip
noisersup Jun 27, 2023
a16bb20
fix
noisersup Jun 27, 2023
12a80e4
wip
noisersup Jun 27, 2023
2db636d
wip
noisersup Jun 27, 2023
ef77cfe
wip
noisersup Jun 27, 2023
9463201
Merge branch 'main' into validate-db-coll-2749
Jun 28, 2023
494dcd7
pair programming
noisersup Jun 28, 2023
53518ae
improve prefix check
noisersup Jun 28, 2023
3b95fe1
wip
noisersup Jun 28, 2023
af07a45
Merge branch 'fix-after-bump' into validate-db-coll-2749
noisersup Jun 28, 2023
b3d66ab
Remove test
noisersup Jun 28, 2023
10dcbaa
Merge branch 'fix-after-bump' into validate-db-coll-2749
noisersup Jun 28, 2023
1c2f423
apply review comments
noisersup Jun 28, 2023
8f5667c
Merge branch 'main' into validate-db-coll-2749
Jun 28, 2023
c266454
Merge branch 'main' into validate-db-coll-2749
AlekSi Jun 29, 2023
c5fc06d
Update issue comment
AlekSi Jun 29, 2023
50694df
Simplify
AlekSi Jun 29, 2023
a8bb524
rename CollectionGet
noisersup Jun 29, 2023
c81bc84
remove comment
noisersup Jun 29, 2023
877d7d6
Merge branch 'main' into validate-db-coll-2749
Jun 30, 2023
cb80f54
wip
noisersup Jun 30, 2023
a3b9649
wip
noisersup Jun 30, 2023
7c075bc
wip
noisersup Jun 30, 2023
43c36ae
wip
noisersup Jun 30, 2023
22ff58d
lint
noisersup Jun 30, 2023
0d617a5
Merge branch 'main' into validate-db-coll-2749
Jun 30, 2023
6780ab3
remove unused errorcode
noisersup Jun 30, 2023
50397c1
remove empty collection error from update
noisersup Jun 30, 2023
af88550
Merge remote-tracking branch 'upstream/main' into validate-db-coll-2749
noisersup Jun 30, 2023
1b1a1f4
cleanup
noisersup Jun 30, 2023
122b3fe
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 2, 2023
6922636
Merge branch 'main' into pr/noisersup/2868-1
AlekSi Jul 14, 2023
88c3c77
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 18, 2023
5e519bd
Tweak comment
AlekSi Jul 19, 2023
81eb51d
Merge branch 'main' into pr/noisersup/2868
AlekSi Jul 19, 2023
75706fd
Update comments
AlekSi Jul 19, 2023
8346e0c
Merge branch 'main' into pr/noisersup/2868
AlekSi Jul 19, 2023
e4e77bf
Add linter configuration
AlekSi Jul 19, 2023
eaf50a9
Improve tests
AlekSi Jul 19, 2023
d95cca4
Tweaks
AlekSi Jul 19, 2023
5fef6b7
Do not import `commonerrors` in tests
AlekSi Jul 19, 2023
0b96a31
Merge branch 'commonerrors' into pr/noisersup/2868
AlekSi Jul 19, 2023
c875dd1
More work
AlekSi Jul 19, 2023
14e2bff
Add comment
AlekSi Jul 19, 2023
67f909b
Create unique index
AlekSi Jul 19, 2023
a500b14
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 20, 2023
b6d2961
Make SQLite URL configurable
AlekSi Jul 20, 2023
6a35453
Add validation
AlekSi Jul 20, 2023
a18b8f2
Merge branch 'main' into pr/noisersup/2868
AlekSi Jul 20, 2023
e070733
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 26, 2023
c5cc1d5
Update comments
AlekSi Jul 26, 2023
be93b2d
Tweak validation
AlekSi Jul 26, 2023
fe79866
Fix number
AlekSi Jul 26, 2023
1344019
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 26, 2023
1c72efc
Update comment
AlekSi Jul 26, 2023
cc083fa
Update comments
AlekSi Jul 26, 2023
296973f
Tiny tweak
AlekSi Jul 26, 2023
20f8200
Fix tests
AlekSi Jul 26, 2023
0cc1df7
Merge branch 'main' into validate-db-coll-2749
AlekSi Jul 26, 2023
c39f316
Lint
AlekSi Jul 26, 2023
e266e35
Add explicit panic
AlekSi Jul 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Improve tests
  • Loading branch information
AlekSi committed Jul 19, 2023
commit eaf50a95fccc4df43fb1efdaeb9a016c63b2d50e
8 changes: 6 additions & 2 deletions internal/backends/collection.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ type QueryResult struct {

// Query executes a query against the collection.
//
// If the collection does not exist it returns empty iterator.
// If database or collection does not exist it returns empty iterator.
//
// The passed context should be used for canceling the initial query.
// It also can be used to close the returned iterator and free underlying resources,
Expand Down Expand Up @@ -108,6 +108,8 @@ func (cc *collectionContract) Insert(ctx context.Context, params *InsertParams)

// UpdateParams represents the parameters of Collection.Update method.
type UpdateParams struct {
// TODO https://github.com/FerretDB/FerretDB/issues/3079
// that should be types.DocumentIterator
Docs *types.Array
}

Expand All @@ -117,6 +119,8 @@ type UpdateResult struct {
}

// Update updates documents in collection.
//
// Database or collection may not exist; that's not an error.
func (cc *collectionContract) Update(ctx context.Context, params *UpdateParams) (*UpdateResult, error) {
defer observability.FuncCall(ctx)()

Expand All @@ -138,7 +142,7 @@ type DeleteResult struct {

// Delete deletes documents in collection.
//
// If requested database or collection does not exist it returns 0 deleted documents.
// Database or collection may not exist; that's not an error.
func (cc *collectionContract) Delete(ctx context.Context, params *DeleteParams) (*DeleteResult, error) {
defer observability.FuncCall(ctx)()

Expand Down
2 changes: 1 addition & 1 deletion internal/backends/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ type CollectionInfo struct {

// ListCollections returns information about collections in the database.
//
// Database doesn't have to exist; that's not an error.
// Database may not exist; that's not an error.
func (dbc *databaseContract) ListCollections(ctx context.Context, params *ListCollectionsParams) (*ListCollectionsResult, error) {
defer observability.FuncCall(ctx)()

Expand Down
4 changes: 2 additions & 2 deletions internal/backends/error.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ const (

// Error represents a backend error returned by all Backend, Database and Collection methods.
type Error struct {
// this internal error can't be accessed by the caller;
// it may be nil
// This internal error can't be accessed by the caller; it exists only for debugging.
// It may be nil.
err error

code ErrorCode
Expand Down
1 change: 0 additions & 1 deletion internal/backends/sqlite/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import (
"context"

"go.uber.org/zap"
_ "modernc.org/sqlite"

"github.com/FerretDB/FerretDB/internal/backends"
"github.com/FerretDB/FerretDB/internal/backends/sqlite/metadata"
Expand Down
48 changes: 2 additions & 46 deletions internal/backends/sqlite/metadata/pool/pool.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ type Pool struct {
token *resource.Token
}

// New creates a pool for SQLite databases in the given directory.
// New creates a pool for SQLite databases in the directory specified by SQLite URI.
func New(u string, l *zap.Logger) (*Pool, error) {
uri, err := validateURI(u)
uri, err := parseURI(u)
if err != nil {
return nil, fmt.Errorf("failed to parse SQLite URI %q: %s", u, err)
}
Expand Down Expand Up @@ -91,50 +91,6 @@ func New(u string, l *zap.Logger) (*Pool, error) {
return p, nil
}

// validateURI checks given URI value and returns parsed URL.
// URI should contain 'file' scheme and point to an existing directory.
// Path should end with '/'. Authority should be empty or absent.
//
// Returned URL contains path in both Path and Opaque to make String() method work correctly.
func validateURI(u string) (*url.URL, error) {
uri, err := url.Parse(u)
if err != nil {
return nil, err
}

if uri.Scheme != "file" {
return nil, fmt.Errorf(`expected "file:" schema, got %q`, uri.Scheme)
}

if uri.User != nil {
return nil, fmt.Errorf(`expected empty user info, got %q`, uri.User)
}

if uri.Host != "" {
return nil, fmt.Errorf(`expected empty host, got %q`, uri.Host)
}

if uri.Path == "" && uri.Opaque != "" {
uri.Path = uri.Opaque
}
uri.Opaque = uri.Path

if !strings.HasSuffix(uri.Path, "/") {
return nil, fmt.Errorf(`expected path ending with "/", got %q`, uri.Path)
}

fi, err := os.Stat(uri.Path)
if err != nil {
return nil, fmt.Errorf(`%q should be an existing directory, got %s`, uri.Path, err)
}

if !fi.IsDir() {
return nil, fmt.Errorf(`%q should be an existing directory`, uri.Path)
}

return uri, nil
}

// databaseName returns database name for given database file path.
func (p *Pool) databaseName(databaseFile string) string {
return strings.TrimSuffix(filepath.Base(databaseFile), filenameExtension)
Expand Down
127 changes: 0 additions & 127 deletions internal/backends/sqlite/metadata/pool/pool_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,8 @@
package pool

import (
"net/url"
"os"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/FerretDB/FerretDB/internal/util/testutil"
Expand Down Expand Up @@ -58,127 +55,3 @@ func TestCreateDrop(t *testing.T) {
db = p.GetExisting(ctx, t.Name())
require.Nil(t, db)
}

func TestNewBackend(t *testing.T) {
t.Parallel()

err := os.MkdirAll("tmp/dir", 0o777)
require.NoError(t, err)

_, err = os.Create("tmp/file")
require.NoError(t, err)

t.Cleanup(func() {
err := os.RemoveAll("tmp")
require.NoError(t, err)
})

testCases := map[string]struct {
value string
uri *url.URL
err string
}{
"LocalDirectory": {
value: "file:./",
uri: &url.URL{
Scheme: "file",
Opaque: "./",
Path: "./",
},
},
"LocalSubDirectory": {
value: "file:./tmp/",
uri: &url.URL{
Scheme: "file",
Opaque: "./tmp/",
Path: "./tmp/",
},
},
"LocalSubSubDirectory": {
value: "file:./tmp/dir/",
uri: &url.URL{
Scheme: "file",
Opaque: "./tmp/dir/",
Path: "./tmp/dir/",
},
},
"LocalDirectoryWithParameters": {
value: "file:./tmp/?mode=ro",
uri: &url.URL{
Scheme: "file",
Opaque: "./tmp/",
Path: "./tmp/",
RawQuery: "mode=ro",
},
},
"AbsoluteDirectory": {
value: "file:/tmp/",
uri: &url.URL{
Scheme: "file",
Opaque: "/tmp/",
Path: "/tmp/",
OmitHost: true,
},
},
"WithEmptyAuthority": {
value: "file:///tmp/",
uri: &url.URL{
Scheme: "file",
Opaque: "/tmp/",
Path: "/tmp/",
},
},
"WithEmptyAuthorityAndQuery": {
value: "file:///tmp/?mode=ro",
uri: &url.URL{
Scheme: "file",
Opaque: "/tmp/",
Path: "/tmp/",
RawQuery: "mode=ro",
},
},
"HostIsNotEmpty": {
value: "file://localhost/./tmp/?mode=ro",
err: `expected empty host, got "localhost"`,
},
"UserIsNotEmpty": {
value: "file://user:pass@./tmp/?mode=ro",
err: `expected empty user info, got "user:pass"`,
},
"NoDirectory": {
value: "file:./nodir/",
err: `"./nodir/" should be an existing directory, got stat ./nodir/: no such file or directory`,
},
"PathIsNotEndsWithSlash": {
value: "file:./tmp/file",
err: `expected path ending with "/", got "./tmp/file"`,
},
"FileInsteadOfDirectory": {
value: "file:./tmp/file/",
err: `"./tmp/file/" should be an existing directory, got stat ./tmp/file/: not a directory`,
},
"MalformedURI": {
value: ":./tmp/",
err: `parse ":./tmp/": missing protocol scheme`,
},
"NoScheme": {
value: "./tmp/",
err: `expected "file:" schema, got ""`,
},
}
for name, tc := range testCases {
name, tc := name, tc
t.Run(name, func(t *testing.T) {
t.Parallel()

u, err := validateURI(tc.value)
if tc.err != "" {
assert.EqualError(t, err, tc.err)
return
}

require.NoError(t, err)
assert.Equal(t, u, tc.uri)
})
}
}
70 changes: 70 additions & 0 deletions internal/backends/sqlite/metadata/pool/uri.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// Copyright 2021 FerretDB Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package pool

import (
"fmt"
"net/url"
"os"
"strings"
)

// parseURI checks given SQLite URI and returns a parsed form.
//
// URI should contain 'file' scheme and point to an existing directory.
// Path should end with '/'. Authority should be empty or absent.
//
// Returned URL contains path in both Path and Opaque to make String() method work correctly.
// Callers should use Path.
func parseURI(u string) (*url.URL, error) {
uri, err := url.Parse(u)
if err != nil {
return nil, err
}

if uri.Scheme != "file" {
return nil, fmt.Errorf(`expected "file:" schema, got %q`, uri.Scheme)
}

if uri.User != nil {
return nil, fmt.Errorf(`expected empty user info, got %q`, uri.User)
}

if uri.Host != "" {
return nil, fmt.Errorf(`expected empty host, got %q`, uri.Host)
}

if uri.Path == "" {
uri.Path = uri.Opaque
}
uri.Opaque = uri.Path
uri.RawPath = ""
uri.OmitHost = true

if !strings.HasSuffix(uri.Path, "/") {
return nil, fmt.Errorf(`expected path ending with "/", got %q`, uri.Path)
}

fi, err := os.Stat(uri.Path)
if err != nil {
return nil, fmt.Errorf(`%q should be an existing directory, got %s`, uri.Path, err)
}

if !fi.IsDir() {
return nil, fmt.Errorf(`%q should be an existing directory`, uri.Path)
}

return uri, nil
}
Loading