Codecov Report

Merging #3363 (1e6dc92) into main (19f0906) will increase coverage by 0.00%.
The diff coverage is 75.00%.

@@           Coverage Diff           @@
##             main    #3363   +/-   ##
=======================================
  Coverage   74.07%   74.07%           
=======================================
  Files         413      413           
  Lines       25306    25315    +9     
=======================================
+ Hits        18746    18753    +7     
- Misses       5472     5473    +1     
- Partials     1088     1089    +1     

Flags with carried forward coverage won't be shown. Click here to find out more.

When I run the same benchmark #3355 (comment) the count is correct.
avgObjSize increased from 198 to 204, because storageSize (table size) is still the entire table size and now we use better approximation of rows.

$  task bench-sqlite BENCH_NAME=BenchmarkQuerySmallDocuments BENCH_DOCS=100
task: [bench-sqlite] go test -tags= -timeout=0 -run=XXX -count=10 -bench=BenchmarkQuerySmallDocuments -benchtime=5s -benchmem -log-level=error -bench-docs=100 -target-backend=ferretdb-sqlite -sqlite-url=file:../tmp/sqlite-tests/ | tee new-sqlite.txt

2023-09-15T12:32:16.471+0900    INFO    setup/startup.go:90     Target system: ferretdb-sqlite (built-in).
2023-09-15T12:32:16.471+0900    INFO    setup/startup.go:103    Compat system: none, compatibility tests will be skipped.
2023-09-15T12:32:16.471+0900    INFO    debug   debug/debug.go:92       Starting debug server on http://127.0.0.1:39451/
collStats [
{ns BenchmarkQuerySmallDocuments-SmallDocuments-Docs100-3947.BenchmarkQuerySmallDocuments-SmallDocuments-Docs100-3947} 
{size 20480} 
{count 100} 
{avgObjSize 204} 
{storageSize 20480} 
{nindexes 0} 
{totalIndexSize 0} 
{totalSize 20480} 
{scaleFactor 1} 
{ok 1}
]

LGTM, but we should understand that collStats's count field is an approximation. It might not be correct in MongoDB, too. If we need to know the exact count, we should run count, not collStats.

Yes indeed, for mongoDB as far as I can tell, there isn't an indication if it's exact or not. For us, both pg and sqlite are approximation.

mongosh method is called _estimated_DocumentCount: https://www.mongodb.com/docs/manual/reference/method/db.collection.estimatedDocumentCount/#mongodb-method-db.collection.estimatedDocumentCount

https://www.mongodb.com/docs/manual/reference/operator/aggregation/collStats/ says:

The count is based on the collection's metadata, which provides a fast but sometimes inaccurate count for sharded clusters.

Deprecated https://www.mongodb.com/docs/manual/reference/command/collStats/ says:

After an unclean shutdown: validate updates the count statistic in the collStats output with the latest value.

So it looks like that count does not count documents but returns the current value from the planner's metadata, that is an estimate.

mongosh method is called _estimated_DocumentCount: https://www.mongodb.com/docs/manual/reference/method/db.collection.estimatedDocumentCount/#mongodb-method-db.collection.estimatedDocumentCount

https://www.mongodb.com/docs/manual/reference/operator/aggregation/collStats/ says:

The count is based on the collection's metadata, which provides a fast but sometimes inaccurate count for sharded clusters.

Deprecated https://www.mongodb.com/docs/manual/reference/command/collStats/ says:

After an unclean shutdown: validate updates the count statistic in the [collStats](https://www.mongodb.com/docs/manual/reference/command/collStats/#mongodb-dbcommand-dbcmd.collStats output with the latest value.

So it looks like that count does not count documents but returns the current value from the planner's metadata, that is an estimate.

Thanks for looking into this in detail 🤗!