Use filter and insert more documents in `BenchmarkReplaceSettingsDocument` #3343

chilagrow · 2023-09-12T07:18:28Z

Description

Closes #2340.

Readiness checklist

I added/updated unit tests (and they pass).
I added/updated integration/compatibility tests (and they pass).
I added/updated comments and checked rendering.
I made spot refactorings.
I updated user documentation.
I ran task all, and it passed.
I ensured that PR title is good enough for the changelog.
(for maintainers only) I set Reviewers (@FerretDB/core), Milestone (Next), Labels, Project and project's Sprint fields.
I marked all done items in this checklist.

codecov · 2023-09-12T07:22:04Z

Codecov Report

Merging #3343 (35a9918) into main (05fd6a8) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #3343      +/-   ##
==========================================
- Coverage   74.06%   74.03%   -0.03%     
==========================================
  Files         413      413              
  Lines       25315    25317       +2     
==========================================
- Hits        18750    18744       -6     
- Misses       5474     5481       +7     
- Partials     1091     1092       +1

Files Changed	Coverage
integration/setup/setup.go	`ø`
integration/setup/startup.go	`0.00%`
integration/shareddata/benchmark_provider.go	`0.00%`

Flag	Coverage Δ
hana-1	`?`
hana-2	`?`
hana-3	`?`
integration	`71.06% <0.00%> (-0.05%)`	⬇️
mongodb-1	`4.72% <0.00%> (-0.01%)`	⬇️
pg-1	`43.81% <0.00%> (-0.01%)`	⬇️
pg-2	`43.54% <0.00%> (-0.01%)`	⬇️
pg-3	`43.99% <0.00%> (-0.12%)`	⬇️
sqlite-1	`42.61% <0.00%> (-0.06%)`	⬇️
sqlite-2	`42.11% <0.00%> (-0.01%)`	⬇️
sqlite-3	`43.02% <0.00%> (+0.09%)`	⬆️
unit	`23.05% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

AlekSi · 2023-09-12T07:41:40Z

integration/benchmarks_test.go

+
+	insertDocs := make([]any, len(docs))
+	for i := range insertDocs {
+		insertDocs[i] = docs[i]


We could just use a setup like other benchmarks

chilagrow · 2023-09-13T10:42:45Z

Benchmark on sqlite

Inserting 10000 gives 291MB total size.

collStats [
{ns BenchmarkReplaceSettingsDocument.BenchmarkReplaceSettingsDocument} 
{size 291119104} 
{count 16707} 
{avgObjSize 17424} 
{storageSize 291119104} 
{nindexes 0} 
{totalIndexSize 0} 
{totalSize 291119104} // 291MB
{scaleFactor 1} 
{ok 1}
]

Oddly it inserted too many {count 16707}, I created #3355

bench-sqlite on this branch

$ task bench-sqlite BENCH_NAME=BenchmarkReplaceSettingsDocument BENCH_DOCS=10000

2023-09-13T19:34:10.271+0900	INFO	setup/startup.go:90	Target system: ferretdb-sqlite (built-in).
2023-09-13T19:34:10.271+0900	INFO	setup/startup.go:103	Compat system: none, compatibility tests will be skipped.
2023-09-13T19:34:10.272+0900	INFO	debug	debug/debug.go:92	Starting debug server on http://127.0.0.1:37543/
goos: linux
goarch: amd64
pkg: github.com/FerretDB/FerretDB/integration
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkReplaceSettingsDocument/Replace-16         	     151	  38163585 ns/op	38446824 B/op	   26750 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     166	  36680970 ns/op	38444189 B/op	   26748 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     169	  37578364 ns/op	38447371 B/op	   26748 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     156	  39770445 ns/op	38448202 B/op	   26748 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     180	  39827901 ns/op	38462472 B/op	   26758 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     128	  45731694 ns/op	38451232 B/op	   26759 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     138	  45367617 ns/op	38450878 B/op	   26756 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     144	  40828568 ns/op	38442355 B/op	   26757 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     142	  41641650 ns/op	38444676 B/op	   26757 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	     140	  41913568 ns/op	38447594 B/op	   26758 allocs/op
PASS
ok  	github.com/FerretDB/FerretDB/integration	139.140s

bench-sqlite-no-pushdown

$ task bench-sqlite-no-pushdown BENCH_NAME=BenchmarkReplaceSettingsDocument BENCH_DOCS=10000

2023-09-13T19:30:30.620+0900	INFO	setup/startup.go:90	Target system: ferretdb-sqlite (built-in).
2023-09-13T19:30:30.620+0900	INFO	setup/startup.go:103	Compat system: none, compatibility tests will be skipped.
2023-09-13T19:30:30.620+0900	INFO	debug	debug/debug.go:92	Starting debug server on http://127.0.0.1:35483/
goos: linux
goarch: amd64
pkg: github.com/FerretDB/FerretDB/integration
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12799343474 ns/op	9558929160 B/op	35550633 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12663396794 ns/op	9558913784 B/op	35550518 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12895047995 ns/op	9558931608 B/op	35550669 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12815387209 ns/op	9558965096 B/op	35551128 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12504455005 ns/op	9558948568 B/op	35550613 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12772391478 ns/op	9558982704 B/op	35550661 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12724095912 ns/op	9559067880 B/op	35550607 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	17386366640 ns/op	9559109688 B/op	35551977 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	16768867633 ns/op	9559183216 B/op	35552117 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	17931102349 ns/op	9559049296 B/op	35551722 allocs/op
PASS
ok  	github.com/FerretDB/FerretDB/integration	171.546s

bench-sqlite on v1.9.0 release tag

$ task bench-sqlite BENCH_NAME=BenchmarkReplaceSettingsDocument BENCH_DOCS=10000

2023-09-13T19:39:52.310+0900	INFO	setup/startup.go:90	Target system: ferretdb-sqlite (built-in).
2023-09-13T19:39:52.311+0900	INFO	setup/startup.go:103	Compat system: none, compatibility tests will be skipped.
2023-09-13T19:39:52.311+0900	INFO	debug	debug/debug.go:92	Starting debug server on http://127.0.0.1:44075/
goos: linux
goarch: amd64
pkg: github.com/FerretDB/FerretDB/integration
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12521495663 ns/op	9557172216 B/op	35550718 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12303573404 ns/op	9557196392 B/op	35550332 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12109764424 ns/op	9557268568 B/op	35550495 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12436747066 ns/op	9557347600 B/op	35551279 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12378462453 ns/op	9557102912 B/op	35550644 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12256648758 ns/op	9556987832 B/op	35550313 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	11841048312 ns/op	9557045688 B/op	35550385 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12492299455 ns/op	9557021464 B/op	35550451 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12629146974 ns/op	9557159544 B/op	35551235 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12337122948 ns/op	9557083600 B/op	35550633 allocs/op
PASS
ok  	github.com/FerretDB/FerretDB/integration	154.463s

Benchmark on pg

Inserting 10000 gives 84MB total size.

collStats [
{ns BenchmarkReplaceSettingsDocument.BenchmarkReplaceSettingsDocument} 
{size 83476480}
{count 10000} 
{avgObjSize 8347} 
{storageSize 83476480}
{nindexes 1} 
{totalIndexSize 778240}
{totalSize 84254720}  // 84MB 
{scaleFactor 1} 
{ok 1}
]

bench-pg

$ task bench-pg BENCH_NAME=BenchmarkReplaceSettingsDocument BENCH_DOCS=10000

2023-09-13T19:32:27.412+0900	INFO	setup/startup.go:90	Target system: ferretdb-pg (built-in).
2023-09-13T19:32:27.412+0900	INFO	setup/startup.go:103	Compat system: none, compatibility tests will be skipped.
2023-09-13T19:32:27.413+0900	INFO	debug	debug/debug.go:92	Starting debug server on http://127.0.0.1:36841/
goos: linux
goarch: amd64
pkg: github.com/FerretDB/FerretDB/integration
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkReplaceSettingsDocument/Replace-16         	      21	 268314176 ns/op	38233449 B/op	   27802 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      22	 268026482 ns/op	38234441 B/op	   27822 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      21	 361646590 ns/op	38228023 B/op	   27764 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      14	 404770008 ns/op	38237219 B/op	   27823 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      14	 362692815 ns/op	38237182 B/op	   27826 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      15	 358165936 ns/op	38231055 B/op	   27773 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      15	 374484274 ns/op	38239071 B/op	   27825 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      15	 365846874 ns/op	38231501 B/op	   27773 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      15	 379915652 ns/op	38237939 B/op	   27820 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	      15	 368829267 ns/op	38229273 B/op	   27767 allocs/op
PASS
ok  	github.com/FerretDB/FerretDB/integration	149.804s

bench-pg-no-pushdown

$ task bench-pg-no-pushdown BENCH_NAME=BenchmarkReplaceSettingsDocument BENCH_DOCS=10000

2023-09-13T19:35:29.948+0900	INFO	setup/startup.go:90	Target system: ferretdb-pg (built-in).
2023-09-13T19:35:29.948+0900	INFO	setup/startup.go:103	Compat system: none, compatibility tests will be skipped.
2023-09-13T19:35:29.948+0900	INFO	debug	debug/debug.go:92	Starting debug server on http://127.0.0.1:40511/
goos: linux
goarch: amd64
pkg: github.com/FerretDB/FerretDB/integration
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkReplaceSettingsDocument/Replace-16         	       1	13390856821 ns/op	9124864384 B/op	35541845 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12974494471 ns/op	9129032856 B/op	35541732 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12994821200 ns/op	9127398640 B/op	35541362 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	13099506601 ns/op	9130604616 B/op	35542748 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12870667404 ns/op	9130279432 B/op	35541900 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12789558078 ns/op	9128440480 B/op	35541936 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	12625968525 ns/op	9125260688 B/op	35541181 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	14751800086 ns/op	9129035776 B/op	35542814 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	13788438849 ns/op	9127930912 B/op	35541541 allocs/op
BenchmarkReplaceSettingsDocument/Replace-16         	       1	13412726255 ns/op	9127747696 B/op	35542795 allocs/op
PASS
ok  	github.com/FerretDB/FerretDB/integration	221.121s

rumyantseva

The changes look reasonable to me.

Great job with all the research and investigation!

add 100 documents

2c3c0da

mergify bot assigned chilagrow Sep 12, 2023

AlekSi reviewed Sep 12, 2023

View reviewed changes

chilagrow added 2 commits September 13, 2023 12:46

make replace settings document benchmark more standardised

ae4bcb2

use replaceOne with filter so filter is applied

7471e39

chilagrow changed the title ~~Insert more documents for BenchmarkReplaceSettingsDocument~~ Insert more documents for BenchmarkReplaceSettingsDocument and use filter Sep 13, 2023

chilagrow added the code/chore Code maintenance improvements label Sep 13, 2023

chilagrow added this to the Next milestone Sep 13, 2023

chilagrow requested review from a team, rumyantseva and noisersup September 13, 2023 10:46

chilagrow marked this pull request as ready for review September 13, 2023 10:46

chilagrow requested a review from a team as a code owner September 13, 2023 10:46

chilagrow requested a review from AlekSi September 13, 2023 10:46

rumyantseva previously approved these changes Sep 13, 2023

View reviewed changes

chilagrow changed the title ~~Insert more documents for BenchmarkReplaceSettingsDocument and use filter~~ Use filter and insert more documents in BenchmarkReplaceSettingsDocument Sep 14, 2023

chilagrow enabled auto-merge (squash) September 14, 2023 06:51

AlekSi assigned AlekSi and unassigned chilagrow Sep 15, 2023

AlekSi added 3 commits September 15, 2023 12:41

Merge branch 'main' into pr/chilagrow/3343

ffba69b

Check that -bench-docs is set

1000e3f

Merge branch 'main' into pr/chilagrow/3343

92531c0

AlekSi dismissed rumyantseva’s stale review via 92531c0 September 15, 2023 09:07

AlekSi added 3 commits September 15, 2023 14:16

Tweak benchmark

7055307

Lint

875c131

Improve names

4a66694

AlekSi previously approved these changes Sep 15, 2023

View reviewed changes

Fix

711a054

AlekSi dismissed their stale review via 711a054 September 15, 2023 14:24

Fix fix

35a9918

AlekSi disabled auto-merge September 18, 2023 05:23

AlekSi merged commit 58772a7 into FerretDB:main Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use filter and insert more documents in `BenchmarkReplaceSettingsDocument` #3343

Use filter and insert more documents in `BenchmarkReplaceSettingsDocument` #3343

chilagrow commented Sep 12, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading

AlekSi Sep 12, 2023

chilagrow Sep 13, 2023

chilagrow commented Sep 13, 2023 •

edited

Loading

rumyantseva left a comment

Use filter and insert more documents in BenchmarkReplaceSettingsDocument #3343

Use filter and insert more documents in BenchmarkReplaceSettingsDocument #3343

Conversation

chilagrow commented Sep 12, 2023 • edited Loading

Description

Readiness checklist

codecov bot commented Sep 12, 2023 • edited Loading

Codecov Report

AlekSi Sep 12, 2023

Choose a reason for hiding this comment

chilagrow Sep 13, 2023

Choose a reason for hiding this comment

chilagrow commented Sep 13, 2023 • edited Loading

Benchmark on sqlite

bench-sqlite on this branch

bench-sqlite-no-pushdown

bench-sqlite on v1.9.0 release tag

Benchmark on pg

bench-pg

bench-pg-no-pushdown

rumyantseva left a comment

Choose a reason for hiding this comment

Use filter and insert more documents in `BenchmarkReplaceSettingsDocument` #3343

Use filter and insert more documents in `BenchmarkReplaceSettingsDocument` #3343

chilagrow commented Sep 12, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading

chilagrow commented Sep 13, 2023 •

edited

Loading