Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add category and rollup stats #576

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Conversation

PJColombo
Copy link
Member

@PJColombo PJColombo commented Sep 19, 2024

Checklist

  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Resolves #518

This PR introduces changes to aggregate statistics by category and rollups, and stores them in the daily and overall stats tables. To achieve this, it adds optional category and rollup columns to these tables, allowing for grouped statistics.

If either of these columns is null, it indicates that the row reflects total stats for that grouping. Specifically, when rollup is null, the row holds the stats for all rollups. When both category and rollup are null, the row represents the overall total stats.

The PR also leverages Prisma’s new TypedSQL feature to implement type-safe raw SQL queries.

Additionally, it updates the unit tests to verify the validity and correctness of each aggregated value by comparing them against the aggregation of the fixture data, rather than checking against a snapshot, as we were doing so far.

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

Copy link

changeset-bot bot commented Sep 19, 2024

🦋 Changeset detected

Latest commit: d1a30e6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@blobscan/db Minor
@blobscan/dayjs Minor
@blobscan/stats-aggregation-cli Minor
@blobscan/syncers Minor
@blobscan/api Minor
@blobscan/web Patch
@blobscan/blob-propagation-jobs-cli Patch
@blobscan/auth Patch
@blobscan/blob-propagator Patch
@blobscan/blob-storage-manager Patch
@blobscan/rollups Patch
@blobscan/rest-api-server Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link

vercel bot commented Sep 19, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
blobscan-staging ✅ Ready (Inspect) Visit Preview 💬 Add feedback Sep 25, 2024 0:47am
5 Skipped Deployments
Name Status Preview Comments Updated (UTC)
blobscan-docs ⬜️ Ignored (Inspect) Visit Preview Sep 25, 2024 0:47am
blobscan-gnosis ⬜️ Ignored (Inspect) Visit Preview Sep 25, 2024 0:47am
blobscan-holesky ⬜️ Ignored (Inspect) Visit Preview Sep 25, 2024 0:47am
blobscan-mainnet ⬜️ Ignored (Inspect) Visit Preview Sep 25, 2024 0:47am
blobscan-sepolia ⬜️ Ignored (Inspect) Visit Preview Sep 25, 2024 0:47am

Copy link

codecov bot commented Sep 19, 2024

Codecov Report

Attention: Patch coverage is 90.96990% with 27 lines in your changes missing coverage. Please review.

Project coverage is 89.38%. Comparing base (3507a88) to head (36381b4).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...kages/api/src/routers/stats/getBlobOverallStats.ts 76.00% 6 Missing ⚠️
...pi/src/routers/stats/getTransactionOverallStats.ts 79.31% 6 Missing ⚠️
packages/db/prisma/extensions/stats.ts 85.71% 6 Missing ⚠️
clis/stats-aggregation-cli/src/commands/daily.ts 76.47% 4 Missing ⚠️
packages/syncers/src/syncers/DailyStatsSyncer.ts 95.23% 2 Missing ⚠️
packages/test/src/fixtures/index.ts 96.77% 2 Missing ⚠️
...ages/api/src/routers/stats/getBlockOverallStats.ts 50.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #576   +/-   ##
=======================================
  Coverage   89.38%   89.38%           
=======================================
  Files         150      150           
  Lines        9933     9933           
  Branches     1035     1035           
=======================================
  Hits         8879     8879           
  Misses       1054     1054           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@PabloCastellano PabloCastellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job 💪 . I've left some comments

});
const result = await dailyStatsFn(operationParam);

if (typeof result === "number") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a comment here making clearer what are the expected types of result for such a logic

"day": 2023-08-20T00:00:00.000Z,
"rollup": null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rollup field shouldn't be null when category is "ROLLUP", right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rollup field shouldn't be null when category is "ROLLUP", right?

For rows in the stats table, this can occur because the row represents an aggregation that includes all rollups, not just a single one. That’s why the rollup value is null.

The same applies to rows where both category and rollup are null; these rows represent total aggregations that include all elements.

@@ -33,7 +33,7 @@ describe("Transaction router", async () => {
caller.tx.getAll(input).then(({ transactions }) => transactions)
);

it("should get the total number of transactions", async () => {
it.skip("should get the total number of transactions", async () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why skip?

packages/test/src/fixtures/index.ts Outdated Show resolved Hide resolved
model Address {
address String @id
firstBlockNumberAsSender Int? @map("first_block_number_as_sender")
model AddressHistory {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the name tbh. How about naming it AddressData, AddressCategoryInfo or AddressByCategoryInfo? 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like AddressCategoryInfo

Co-authored-by: Pablo Castellano <pablo@anche.no>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Calculate stats by category and rollup
2 participants