Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use iterator in aggregation stages #2480

Merged
merged 18 commits into from
Apr 26, 2023
Merged

Use iterator in aggregation stages #2480

merged 18 commits into from
Apr 26, 2023

Conversation

chilagrow
Copy link
Member

@chilagrow chilagrow commented Apr 19, 2023 •
edited
Loading

Description

Closes #1889.

  • $group needs to accumulate all values in the memory so it reads all document.
  • $unwind is a bit tricky to handle array iterator, it reads all document.

Readiness checklist

  • I added/updated unit tests.
  • I added/updated integration/compatibility tests.
  • I added/updated comments and checked rendering.
  • I made spot refactorings.
  • I updated user documentation.
  • I ran task all, and it passed.
  • I ensured that PR title is good enough for the changelog.
  • (for maintainers only) I set Reviewers (@FerretDB/core), Labels, Project and project's Sprint fields.
  • I marked all done items in this checklist.

@chilagrow chilagrow self-assigned this Apr 19, 2023
@codecov
Copy link

codecov bot commented Apr 19, 2023 •
edited
Loading

Codecov Report

Merging #2480 (734151c) into main (ce28270) will decrease coverage by 0.33%.
The diff coverage is 66.98%.

❗ Current head 734151c differs from pull request most recent head 66cb98d. Consider uploading reports for the commit 66cb98d to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2480      +/-   ##
==========================================
- Coverage   65.10%   64.78%   -0.33%     
==========================================
  Files         399      400       +1     
  Lines       19747    19769      +22     
==========================================
- Hits        12857    12808      -49     
- Misses       5968     6043      +75     
+ Partials      922      918       -4
Impacted Files Coverage Δ
...rnal/handlers/common/aggregations/stages/stages.go 20.00% <ø> (ø)
internal/handlers/common/limit_iterator.go 82.75% <0.00%> (-2.96%) ⬇️
internal/handlers/tigris/msg_aggregate.go 0.00% <0.00%> (ø)
...l/handlers/common/aggregations/stages/collstats.go 83.92% <25.00%> (-12.15%) ⬇️
...ernal/handlers/common/aggregations/stages/group.go 87.71% <57.14%> (-1.51%) ⬇️
...rnal/handlers/common/aggregations/stages/unwind.go 82.47% <62.50%> (-2.31%) ⬇️
internal/handlers/pg/msg_aggregate.go 83.18% <77.77%> (+1.78%) ⬆️
internal/handlers/common/count_iterator.go 91.17% <91.17%> (ø)
...ernal/handlers/common/aggregations/stages/count.go 100.00% <100.00%> (ø)
...ernal/handlers/common/aggregations/stages/limit.go 81.25% <100.00%> (+11.25%) ⬆️
... and 4 more

... and 4 files with indirect coverage changes

Flag Coverage Δ
integration 58.00% <66.98%> (+<0.01%) ⬆️
mongodb 5.11% <0.00%> (-0.01%) ⬇️
pg 57.92% <66.98%> (+<0.01%) ⬆️
unit 25.28% <0.00%> (-0.39%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@mergify
Copy link
Contributor

mergify bot commented Apr 19, 2023

@chilagrow this pull request has merge conflicts.

@mergify mergify bot added the conflict PRs that have merge conflicts label Apr 19, 2023
@chilagrow chilagrow force-pushed the issue-1889-aggregation-iterator branch from d9e5ff1 to 5363b18 Compare April 19, 2023 09:29
@mergify mergify bot removed the conflict PRs that have merge conflicts label Apr 19, 2023
@chilagrow chilagrow added the code/chore Code maintenance improvements label Apr 20, 2023
@chilagrow chilagrow changed the title Aggregation uses iterator Aggregation stages uses iterator Apr 20, 2023
@chilagrow chilagrow marked this pull request as ready for review April 20, 2023 08:03
@chilagrow chilagrow requested a review from a team as a code owner April 20, 2023 08:03
@chilagrow chilagrow requested review from AlekSi and rumyantseva April 20, 2023 08:03
@chilagrow chilagrow enabled auto-merge (squash) April 20, 2023 08:04
@chilagrow chilagrow requested review from a team, w84thesun and noisersup April 20, 2023 08:04
@chilagrow chilagrow mentioned this pull request Apr 20, 2023
Open
rumyantseva
rumyantseva suggested changes Apr 24, 2023
View reviewed changes
Copy link
Contributor

@rumyantseva rumyantseva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the implementation (it looks similar to how FilterIterator works), but I see quite a lot of things fail, so this PR needs some changes to make the linters and tests pass etc.

@chilagrow
Copy link
Member Author

Thanks @w84thesun for fixing the linter and merging main 🤗

@chilagrow chilagrow requested a review from rumyantseva April 25, 2023 03:38
@chilagrow
Copy link
Member Author

chilagrow commented Apr 25, 2023 •
edited
Loading

I like the implementation (it looks similar to how FilterIterator works), but I see quite a lot of things fail, so this PR needs some changes to make the linters and tests pass etc.

@rumyantseva thanks for noticing this, I think linter/failing tests were due to not being up to date with main branch and few conflicting change. Should be fine now, thanks to @w84thesun 🙏

w84thesun
w84thesun approved these changes Apr 25, 2023
View reviewed changes
Copy link
Contributor

@w84thesun w84thesun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

rumyantseva
rumyantseva approved these changes Apr 26, 2023
View reviewed changes
Copy link
Contributor

@rumyantseva rumyantseva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

noisersup
noisersup approved these changes Apr 26, 2023
View reviewed changes
Copy link
Member

@noisersup noisersup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chilagrow chilagrow merged commit ca200f5 into FerretDB:main Apr 26, 2023
@rumyantseva rumyantseva changed the title Aggregation stages uses iterator Use iterator in aggregation stages Apr 26, 2023
@AlekSi AlekSi added this to the v1.1.0 milestone May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@noisersup noisersup noisersup approved these changes

@rumyantseva rumyantseva rumyantseva approved these changes

@w84thesun w84thesun w84thesun approved these changes

@AlekSi AlekSi Awaiting requested review from AlekSi AlekSi is a code owner automatically assigned from FerretDB/leads

Assignees

@chilagrow chilagrow

Labels
code/chore Code maintenance improvements
Projects
Current Engineering
Archived in project
Milestone
v1.1.0
Development

Successfully merging this pull request may close these issues.

Use iterators in aggregation pipeline stages
5 participants
@chilagrow @w84thesun @rumyantseva @noisersup @AlekSi