Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for detecting unused Bazel dependencies #3709

Open
BenHenning opened this issue Aug 19, 2021 · 1 comment
Open

Add support for detecting unused Bazel dependencies #3709

BenHenning opened this issue Aug 19, 2021 · 1 comment
Labels
enhancement End user-perceivable enhancements. Impact: Low Low perceived user impact (e.g. edge cases). Z-ibt Temporary label for Ben to keep track of issues he's triaged.

Comments

@BenHenning
Copy link
Member

Since Kotlin rules today utilizing exporting & don't properly export deps, we can't rely on Bazel buildtools unused deps check to automatically find unnecessary deps. bazelbuild/rules_kotlin#37 is tracking fixing this, but it seems like it may be a while before it's available.

The consequence of this is that the build graph will continue to grow unnecessarily over time slowing down incremental builds and making non-incremental builds less efficient.

I propose we introduce our own unused deps mechanism since it seems straightforward given other patterns/solutions in the codebase (particularly the compute affected tests) by:

  • Introducing a script that computes all new & changed targets in a PR; output this as a matrix to the next script to kick off N jobs (probably rate-limited at 1/s since they're lower priority than the test targets)
  • For each target, first verify that it individually builds (which also primes the local build cache for that action)
  • Leave a comment & short-circuit if the target doesn't build (which is a nice improvement over the existing workflow since people have to look at test & build targets to see specific failures today)
  • Run a script which methodically removes one dependency at a time & verifies that it builds without it*
  • Leave a comment outlining all dependencies that are unneeded

*Note that removal isn't as simple as iterating through the list, removing a dep, and checking if it builds. It's multi-step: after each unused dep is successfully verified, the whole remaining list must be checked again since some deps might be needed to build given another unused dep being present in the list. This is a O(N^2) operation, but subsequent incremental builds should be fast for most targets.

@BenHenning
Copy link
Member Author

This is related to supporting the notion of "strict deps" or "strict dependencies" (partly leaving this comment so that my future search queries work better when trying to find this issue).

@Broppia Broppia added the Impact: Low Low perceived user impact (e.g. edge cases). label Jul 7, 2022
@BenHenning BenHenning added the Z-ibt Temporary label for Ben to keep track of issues he's triaged. label Sep 16, 2022
@seanlip seanlip added enhancement End user-perceivable enhancements. and removed issue_type_infrastructure labels Mar 28, 2023
BenHenning added a commit that referenced this issue Jun 12, 2024
## Explanation
Fixes part of #1719 and #3709

This PR introduces a new script & CI workflow for computing build stats
to compare both AABs and universal APKs between develop and the changes
in a given PR, as part of fixing #1719 (though this PR doesn't cover
everything outlined in that PR). This information is then detailed and
uploaded as a CI build artifact, and summarized & posted as a comment in
the PR. Some details included in the summary report:
- APK file/download size differences
- Method count differences
- Feature/permission differences
- New/removed resources & assets

The script supports computing differences for multiple "profiles" at the
same time, and the CI workflow has been set up to compute four:
1. dev
2. alpha
3. beta
4. GA

This workflow will be optional since it's very expensive to run (it has
to assemble 8 builds, 6 of which are Proguarded). It also doesn't really
need to be run in order to approve a PR, though reviewers may insist on
waiting for large or suspicious changes
(such as PRs introducing new dependencies) to ensure the actual affected
changes are as expected.

In order to mitigate this expense, the CI workflow runs on a scheduled
cron job off of develop across all open PRs and checks them in a group.
It runs at most once per day (based on
https://github.com/orgs/community/discussions/55768#discussioncomment-5941720)
so multiple changes to a PR will be picked up with a single check in the
next cron run. Currently, it will run even for a PR that hasn't changed
since the last run (but this is something that can be improved in the
future if it needs to be). It's being scheduled for 2:30am (02:30) UTC
which seems to have a few specific benefits:
- Per GitHub documentation, initiating the workflow outside the start of
the hour should reduce likelihood of cancellation (since the start of
the hour tends to use the most resources):
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule.
- This corresponds to 7:30pm PT, 2:30am GMT, 8:00am IST, 5:30am EAT, and
12:30pm AEST (just as a basis for a different part of the world). It's
actually a very nice time that shouldn't overlap with almost any
development in main locations around the world, so it hopefully won't
impact Oppia organization GitHub CI resources.

The example output from these workflows can be observed in a few places:
- Later in this PR (for back when the PR was configured to run the new
workflow per PR change).
- In #4261 which demonstrates the large math PRs and how they changed
the builds back before those were merged.
-
https://github.com/BenHenning/oppia-android/actions/workflows/stats.yml
and https://github.com/BenHenning/oppia-android/pulls (specifically:
BenHenning#14,
BenHenning#13,
BenHenning#12) which demonstrates
the workflow running correctly from a scheduled cron
(https://github.com/BenHenning/oppia-android/actions/runs/9232187176)
and posting the updates to open PRs.

Beyond that, implementing this utility involved several significant
changes to various systems, including the build graph:
- Three new utilities were added for the script: Aapt2Client,
ApkAnalyzerClient, and BundleToolClient. Each of these correspond to
Android CLI utilities, but each required special considerations:
- Aapt2Client requires direct access to the Android SDK, but fortunately
android_sdk_repository exposes this as a target so it's trivial to pass
it in & call it. Some build information is needed, too (see next outer
point).
- ApkAnalyzerClient couldn't use the apkanalyzer CLI contained within
the SDK since it's not exported by android_sdk_repository. Instead, we
needed to depend on the CLI's internal implementation library (which I
suspect is what Android Studio probably uses for its own APK Analyzer
tool). This required some new implementation.
- BundleToolClient fortunately can call right into the bundle tool
library that we use when building AABs, but unfortunately that tool
appears to not be designed to be called multiple times in the same
process. Because Java doesn't support forking, we actually needed to
fake a fork function by starting a new Java process using the current
process's classpath in order to re-run bundle tool for each needed
routine. Additionally, bundle tool required
https://github.com/oppia/archive-patcher (which needed new BUILD files
since it only supported Gradle building previously) and a non-Android
version of Guava (see below for the changes this has caused).
- A new build_vars.bzl was introduced to define the build SDK & build
tools versions (this is done in a way where they can actually be passed
to the new script's utilities since it needs to access aapt2).
- rules_kotlin had a bug where resources wouldn't be pulled in properly
for kt_jvm_library (see
bazelbuild/rules_kotlin#281), but this was
mitigated in a previous PR by upgrading rules_kotlin past alpha 2.
- The new functionality required the JRE-compatible version of Guava
(over the Android-constrained library used in the codebase today), but
this introduces a one-version issue. The solution ended up being
isolating the JRE-compatible Guava library to its own library with a
slightly hacky direct reference to it in BundleToolClient. Some of the
other attempts at solving this resulted in some Maven reference cleanups
in existing script documentation. This functionality will be improved in
downstream PRs, but other attempts that were originally made to isolate
this cleanly were:
- Introduce multiple maven_install files and isolate dependencies into:
production, tests, scripts. This has a number of nice benefits (more
correct licenses and faster Maven dependency fetches for production),
but it results in very tricky one-version violations for test targets
that cross dependencies between production and tests.
- Isolated maven_install just for scripts. This is closer to the
solution we'll want long-term, but it was too much complexity to fully
introduce in this PR so it's been reworked into a downstream PR that can
focus on cleaning up third-party dependency management across the whole
codebase.

This PR is introducing a few new dependencies that, in turn, pull in a
*bunch* of transitive dependencies. These are all due to the new
``apkanalyzer`` dependency. While it will affect licenses for this
specific PR, once third-party dependencies for scripts are cleaned up in
a downstream PR they will be moved out (since they are script-only
dependencies).

Separately, also note that the AAPT2 utility requires stdout to be
processed continuously in order for the process to finish. This was one
of the primary reasons CommandExecutorImpl was reworked in #4929.

For testing: most of the changes in this PR have been extensively
manually tested. However, the new utilities are lacking significant
automated tests. Since this utility is a nice-to-have for the rest of
the Bazel PR chain, it's being prioritized to be merged in spite of
lacking code coverage. #4971 has been filed to track adding these
missing tests in the long-term.

## Essential Checklist
- [x] The PR title and explanation each start with "Fix #bugnum: " (If
this PR fixes part of an issue, prefix the title with "Fix part of
#bugnum: ...".)
- [x] Any changes to
[scripts/assets](https://github.com/oppia/oppia-android/tree/develop/scripts/assets)
files have their rationale included in the PR explanation.
- [x] The PR follows the [style
guide](https://github.com/oppia/oppia-android/wiki/Coding-style-guide).
- [x] The PR does not contain any unnecessary code changes from Android
Studio
([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#undo-unnecessary-changes)).
- [x] The PR is made from a branch that's **not** called "develop" and
is up-to-date with "develop".
- [x] The PR is **assigned** to the appropriate reviewers
([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#clarification-regarding-assignees-and-reviewers-section)).

## For UI-specific PRs only
N/A -- This only affects CI workflows & the build system. Technically,
some dependency changes in the build system could have UI effects, but
there should be no such changes in this PR.

---------

Co-authored-by: Adhiambo Peres <59600948+adhiamboperes@users.noreply.github.com>
Co-authored-by: Sean Lip <sean@seanlip.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement End user-perceivable enhancements. Impact: Low Low perceived user impact (e.g. edge cases). Z-ibt Temporary label for Ben to keep track of issues he's triaged.
Projects
Development

No branches or pull requests

3 participants