Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify OP_MSG message checksum #2540

Merged
merged 54 commits into from
May 15, 2023
Merged

Conversation

adetunjii
Copy link
Contributor

@adetunjii adetunjii commented May 1, 2023

Description

Closes #1626.

This PR solves the issue with checksum verification for an OP_MSG

  • I added/updated unit tests.
  • I added/updated integration/compatibility tests.
  • I added/updated comments and checked rendering.
  • I made spot refactorings.
  • I updated user documentation.
  • I ran task all, and it passed.
  • I ensured that PR title is good enough for the changelog.
  • (for maintainers only) I set Reviewers (@FerretDB/core), Labels, Project and project's Sprint fields.
  • I marked all done items in this checklist.

@adetunjii adetunjii requested a review from a team as a code owner May 1, 2023 20:10
@adetunjii adetunjii requested review from rumyantseva and chilagrow May 1, 2023 20:10
@CLAassistant
Copy link

CLAassistant commented May 1, 2023

CLA assistant check
All committers have signed the CLA.

@adetunjii adetunjii marked this pull request as draft May 1, 2023 20:11
internal/wire/msg_body.go Outdated Show resolved Hide resolved
internal/wire/msg_body.go Outdated Show resolved Hide resolved
@adetunjii
Copy link
Contributor Author

adetunjii commented May 3, 2023

@AlekSi I cannot write the data directly into the crc32 hasher, I need to parse the whole message, check if the OpMsgChecksumPresent flag is present, detach the original checksum and compare it to the calculated checksum. See here https://www.mongodb.com/docs/manual/reference/mongodb-wire-protocol/#std-label-wire-msg-checksum

@adetunjii
Copy link
Contributor Author

For this implementation to work, the data can first be read into a TeeReader, then the checksum is calculated, then the data is then read again from the TeeReader into a bufio.Reader so that it can used for the rest of the implementation. However, this is duplicated work

@AlekSi
Copy link
Member

AlekSi commented May 4, 2023

Sorry, I don't follow. TeeReader, MultiWriter, and other IO helpers like those two do not keep read or written data in memory. So there is no duplication of work.

I also don't think we should parse the whole message to calculate the checksum. After all, we calculate the checksum of bytes, not parsed data structure.

I think the algorithm could be something like that:

  1. Read the message header.
  2. If it does not contain ChecksumPresent, do change anything.
  3. If it does contain that flag, create a TeeReader that wraps the existing reader and writes to the CRC hasher. Write read header bytes to hasher.
  4. Use that reader to read bytes for parsing while also hashing them.
  5. In the end, check the checksum.

@adetunjii
Copy link
Contributor Author

adetunjii commented May 5, 2023

@AlekSi I found a way around using a TeeReader and made use of the read bytes instead. Please kindly review and let me know what you think.

@adetunjii adetunjii changed the title verify message checksum Verify message checksum May 5, 2023
@AlekSi AlekSi self-requested a review May 5, 2023 11:15
@codecov
Copy link

codecov bot commented May 5, 2023

Codecov Report

Merging #2540 (15f94a8) into main (87b0a6e) will decrease coverage by 0.02%.
The diff coverage is 61.53%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2540      +/-   ##
==========================================
- Coverage   66.31%   66.30%   -0.02%     
==========================================
  Files         419      419              
  Lines       20444    20480      +36     
==========================================
+ Hits        13558    13579      +21     
- Misses       5972     5982      +10     
- Partials      914      919       +5     
Impacted Files Coverage Δ
internal/wire/op_msg.go 57.39% <ø> (-0.57%) ⬇️
internal/wire/msg_body.go 46.00% <61.53%> (+9.93%) ⬆️

... and 1 file with indirect coverage changes

Flag Coverage Δ
integration 59.16% <10.25%> (-0.09%) ⬇️
mongodb 5.42% <0.00%> (-0.01%) ⬇️
pg 59.07% <10.25%> (-0.09%) ⬇️
unit 27.27% <61.53%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@adetunjii
Copy link
Contributor Author

@AlekSi some of the tests are failing and I'm not sure why. My assumption is that some messages in the records do not contain valid checksums.

@AlekSi
Copy link
Member

AlekSi commented May 5, 2023

I will check it shortly

@adetunjii adetunjii marked this pull request as ready for review May 11, 2023 14:05
@noisersup noisersup self-requested a review May 12, 2023 14:05
@adetunjii adetunjii requested a review from rumyantseva May 12, 2023 15:15
Copy link
Member

@noisersup noisersup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calculation part looks good to me! I left some comments and questions :)

internal/wire/msg_body.go Show resolved Hide resolved
internal/wire/wire_test.go Outdated Show resolved Hide resolved
internal/wire/msg_body.go Show resolved Hide resolved
internal/wire/op_msg.go Show resolved Hide resolved
@noisersup noisersup self-requested a review May 15, 2023 12:19
Copy link
Member

@noisersup noisersup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, LGTM!

Copy link
Contributor

@rumyantseva rumyantseva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing! Great job calculating the checksum and figuring out the specifics of the wire package!

@rumyantseva rumyantseva enabled auto-merge (squash) May 15, 2023 12:28
@rumyantseva rumyantseva changed the title Verify message checksum Verify OP_MSG message checksum May 15, 2023
@rumyantseva rumyantseva merged commit 89c2880 into FerretDB:main May 15, 2023
@rumyantseva rumyantseva changed the title Verify OP_MSG message checksum Verify OP_MSG message checksum May 15, 2023
@AlekSi AlekSi added this to the Next milestone May 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/chore Code maintenance improvements
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Check crc32 checksums in the wire package
5 participants