Add support and docs for an auto-publish based local dev workflow.

Our docs currently recommend two ways of building consuming apps against local changes to this repo: an automated-but-perpetually-buggy composite build workflow, and a reliable-but-tedious manual workflow of publishing to a local maven repo. This commit removes them both and replaces them with something similar to the workflow used by android-components: some scripting to automate publishing to and consuming from a local maven repo.
Sajjon · Mar 12, 2020 · 0e31668 · 0e31668
1 parent 04fd44f
commit 0e31668
Show file tree

Hide file tree

Showing 10 changed files with 185 additions and 129 deletions.
diff --git a/CHANGES_UNRELEASED.md b/CHANGES_UNRELEASED.md
@@ -4,6 +4,14 @@
 
 [Full Changelog](https://github.com/mozilla/application-services/compare/v0.54.0...master)
 
+## Android
+
+### What's changed
+
+- There is now preliminary support for an "autoPublish" local-development workflow similar
+  to the one used when working with Fenix and android-components; see
+  [this howto guide](./docs/howtos/locally-published-components-in-fenix.md) for details.
+
 ## Places
 
 ### What's fixed

diff --git a/build-scripts/substitute-local-appservices.gradle b/build-scripts/substitute-local-appservices.gradle
@@ -0,0 +1,38 @@
+// This script is designed to be used by consuming apps that want to support locally-published
+// development versions of application-services components. A build that `apply from`'s this script
+// will be configured to look for development versions published to the local maven repository.
+//
+// There is a companion gradle command `autoPublishForLocalDevelopment` that can be used to publish
+// the local development versions that are targetted by this script.
+
+logger.lifecycle("[local-appservices] adjusting ${project} to use locally published application-services modules")
+
+// Inject mavenLocal repository. This is where we're expected to publish modules.
+repositories {
+    mavenLocal()
+}
+
+configurations.all { config ->
+    if (config.isCanBeResolved()) {
+        config.resolutionStrategy { strategy ->
+            dependencySubstitution {
+                all { dependency ->
+                    // We only care about substituting for a module, not a project.
+                    if (!(dependency.requested instanceof ModuleComponentSelector)) {
+                        return
+                    }
+
+                    // For every org.mozilla.appservices.* module, substitute its version for one
+                    // formatted like `0.0.1-SNAPSHOT-*`. This syntax will pick the latest such version
+                    // published locally by the `autoPublishForLocalDevelopment` command.
+                    def group = dependency.requested.group
+                    if (group == 'org.mozilla.appservices') {
+                        def name = dependency.requested.module
+                        dependency.useTarget([group: group, name: name, version: '[0.0.1-SNAPSHOT,0.0.1['])
+                    }
+                }
+            }
+        }
+    }
+}
+
diff --git a/build.gradle b/build.gradle
@@ -277,3 +277,15 @@ task ktlintFormat(type: JavaExec, group: "formatting") {
     main = "com.github.shyiko.ktlint.Main"
     args "-F", "${projectDir}/components/**/*.kt", "${projectDir}/gradle-plugin/**/*.kt", "buildSrc/**/*.kt", "!**/build"
 }
+
+// Extremely unsophisticated way to publish a local development version while hiding implementation details.
+//
+// For now we just run `publishToMavenLocal` while setting the special property `local`, and have some logic
+// in our gradle scripts to construct an auto-incrementing version number when this property is set.
+//
+// I hope to grow this into something more sophisticated over time (e.g. to avoid re-publishing when nothing
+// has changed, perhaps cleanup old versions, etc) while keeping a stable `./gradlew autoPublishForLocalDevelopment`
+// interface  for consumers.
+task autoPublishForLocalDevelopment(type: Exec) {
+  commandLine "./gradlew", "publishToMavenLocal", "-Plocal=true"
+}
diff --git a/components/support/android/src/main/java/mozilla/appservices/support/native/Helpers.kt b/components/support/android/src/main/java/mozilla/appservices/support/native/Helpers.kt
@@ -232,6 +232,10 @@ internal fun checkFullMegazord(componentName: String, componentVersion: String):
         true
     } catch (e: UnsatisfiedLinkError) {
         Log.e("RustNativeSupport", "Default megazord not found: ${e.localizedMessage}")
+        if (componentVersion.startsWith("0.0.1-SNAPSHOT")) {
+            Log.i("RustNativeSupport", "It looks like you're using a local development build.")
+            Log.i("RustNativeSupport", "You may need to check that `rust.targets` contains the appropriate platforms.")
+        }
         false
     }
 }
diff --git a/components/tabs/android/build.gradle b/components/tabs/android/build.gradle
@@ -100,6 +100,7 @@ dependencies {
     androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
 }
 
+evaluationDependsOn(":full-megazord")
 afterEvaluate {
     // The `cargoBuild` task isn't available until after evaluation.
     android.libraryVariants.all { variant ->

diff --git a/docs/howtos/locally-published-components-in-fenix.md b/docs/howtos/locally-published-components-in-fenix.md
@@ -1,15 +1,38 @@
 # Using locally-published components in Fenix
 
-Note: This is a bit tedious, and you might like to try the substitution-based
-approach documented in [Development with the Reference Browser](./working-with-reference-browser.md).
-That approach is still fairly new, and the local-publishing approach in this document
-is necessary if it fails.
+It's often important to test work-in-progress changes to this repo against a real-world
+consumer project. The most reliable method of performing such testing is to publish your
+components to a local Maven repository, and adjust the consuming project to install them
+from there.
 
-Note 2: This is fenix-specific only in that some links on the page go to the
-`mozilla-mobile/fenix` repository, and that I'm describing `fenix`, however
-these steps should work for e.g. `reference-browser`, as well. (Same goes for
-lockwise, or any other consumer of our components, but they may use a different
-structure -- lockwise has no Dependencies.kt, for example)
+With support from the upstream project, it's possible to do this in a single step using
+our auto-publishing workflow.
+
+## Using the auto-publishing workflow
+
+Some consumers (notably [Fenix](https://github.com/mozilla-mobile/fenix/)) have support for
+automatically publishing and including a local development version of application-services
+in their build. The workflow is:
+
+1. Check out the consuming project.
+1. Edit (or create) the file `local.properties` *in the consuming project* and tell it where to
+   find your local checkout of application-services, by adding a line like:
+
+   `autoPublish.application-services.dir=path/to/your/checkout/of/application-services`
+
+1. Build the consuming project following its usual build procedure, e.g. via `./gradlew assembleDebug` or `./gradlew
+   test`.
+
+If all goes well, this should automatically build your checkout of application-servies, publish it
+to a local maven repository, and configure the consuming project to install from there instead of
+from our published releases.
+
+## Using a manual workflow
+
+Note: This is a bit tedious, and you should first try the auto-publishing workflow described
+above. But if the auto-publishing workflow bitrots then it's important to know how to do it
+by hand. Since most consuming apps get their copy of application-services via a dependency
+on android-components, this procedure involves three separate repos:
 
 1. Inside the `application-services` repository root:
     1. In [`.buildconfig-android.yml`](app-services-yaml), change
@@ -21,6 +44,7 @@ structure -- lockwise has no Dependencies.kt, for example)
        testing on the emulator, `rust.targets=arm` if you're testing on 32-bit
        arm (arm64 for 64-bit arm, etc). This will make the build that's done in
        the next step much faster.
+
     3. Run `./gradlew publishToMavenLocal`. This may take between 5 and 10 minutes.
 
 2. Inside the `android-components` repository root:
@@ -31,46 +55,37 @@ structure -- lockwise has no Dependencies.kt, for example)
        Example: `componentsVersion: 0.51.0-TESTING3`
     2. Inside [`buildSrc/src/main/java/Dependencies.kt`](android-components-deps),
        change `mozilla_appservices` to reference the `libraryVersion` you
-       published in step 2 part 1.
+       published in step 1 part 1.
 
        Example: `const val mozilla_appservices = "0.27.0-TESTING3"`
 
     3. Inside [`build.gradle`](android-components-build-gradle), add
        `mavenLocal()` inside `allprojects { repositories { <here> } }`.
 
-    4. Inside the android-component's `local.properties` file, ensure
-       `substitutions.application-services.dir` is *NOT* set.
+    4. Inside the android-components `local.properties` file, ensure
+       `autoPublish.application-services.dir` is *NOT* set.
 
     5. Run `./gradlew publishToMavenLocal`.
 
-3. Inside the `fenix` repository root:
+3. Inside the consuming project repository root:
     1. Inside [`build.gradle`](fenix-build-gradle-1), add
        `mavenLocal()` inside `allprojects { repositories { <here> } }`.
-        1. If you added a new project to the megazord (e.g. you went through the
-           parts of step 1) you must also add `mavenLocal()` to
-           [`buildscript { ... dependencies { <here> }}`](fenix-build-gradle-2)
 
-    2. Inside fenix's `local.properties` file, ensure
-       `substitutions.application-services.dir` is *NOT* set.
+    2. Ensure that `local.properties` does not contain any configuration to
+       related to auto-publishing the application-services repo.
 
-    3. Inside [`buildSrc/src/main/java/Dependencies.kt`](fenix-deps), change
-       `mozilla_android_components` to the version you defined in step 3 part 1.
+    3. Inside [`buildSrc/src/main/java/Dependencies.kt`](fenix-deps), change the
+       version numbers for android-components and/or application-services to
+       match the new versions you defined above.
 
        Example: `const val mozilla_android_components = "0.51.0-TESTING3"`
-        1. If you added a new project to the megazord (e.g. you went through the
-           parts of step 1) you must also change `appservices_gradle_plugin` to
-           the version you defined in step 1 part 1.
 
-            Example: `const val appservices_gradle_plugin = "0.4.4-TESTING3"`
-        2. If there are any direct dependencies on application services (at the
-           moment there are not, but there have been in the past and may be in
-           the future), change it's version here to the one defined in step 2
-           part 1.
+       Example: `const val mozilla_appservices = "0.27.0-TESTING3"`
 
-You should now be able to build and run fenix (assuming you could before all
-this).
+You should now be able to build and run the consuming application (assuming you could
+do so before all this).
 
-## Caveats
+### Caveats
 
 1. This assumes you have followed the [android/rust build setup](./setup-android-build-environment.md)
 2. Make sure you're fully up to date in all repos, unless you know you need to
@@ -80,8 +95,58 @@ this).
    understandable to fix, you usually should be able to find a PR with the fixes
    somewhere in the android-component's list of pending PRs (or, failing that, a
    description of what to do in the application-services changelog).
-4. Ask in #rust-components slack (or #sync on mozilla IRC if you are an
-   especially brave external contributor) if you get stuck.
+4. Ask in #sync if you get stuck.
+
+
+## Adding support for the auto-publish workflow
+
+If you had to use the manual workflow above and found it incredibly tedious, you might like to
+try adding support for the auto-publish workflow to the consuming project! The details will differ
+depending on the specifics of the project's build setup, but at a high level you will need to:
+
+1. In your [settings.gradle](fenix-settings), locate (or add) the code for parsing the `local.properties` file,
+   and add support for loading a directory path from the property `autoPublish.application-services.dir`.
+
+   If this property is present, spawn a subprocess to run `./gradlew autoPublishForLocalDevelopment`
+   in the specified directory. This automates step (1) of the manual workflow above, publishing your
+   changes to application-services into a local maven repository under a unique version number.
+
+1. In your [build.gradle]([fenix-build-gradle-1]), if the `autoPublish.application-services.dir` property
+   is present, have each project apply the build script from `./build-scripts/substitute-local-appservices.gradle`
+   in the specified directory.
+
+   This automates steps (2) and (3) of the manual workflow above, using gradle's dependency substitution
+   capabilities to override the verion requirements for application-services components. It may be necessary
+   to experiment with the ordering of this relative to other build configuration steps, in order for the
+   dependency substitution to work correctly.
+
+   For a single-project build this would look something like:
+
+   ```groovy
+   if (gradle.hasProperty('localProperties.autoPublish.application-services.dir')) {
+      ext.appServicesSrcDir = gradle."localProperties.autoPublish.application-services.dir"
+      apply from: "${appServicesSrcDir}/build-scripts/substitute-local-appservices.gradle"
+   }
+   ```
+
+   For a multi-project build it should be applied to all subprojects, like:
+
+   ```groovy
+   subprojects {
+      if (gradle.hasProperty('localProperties.autoPublish.application-services.dir')) {
+         ext.appServicesSrcDir = gradle."localProperties.autoPublish.application-services.dir"
+         apply from: "${rootProject.projectDir}/${appServicesSrcDir}/build-scripts/substitute-local-appservices.gradle"
+      }
+   }
+   ```
+
+1. Confirm that the setup is working, by adding `autoPublish.application-services.dir` to your
+   `local.properties` file and running `./gradlew dependencies` for the project.
+
+   You should be able to see gradle checking the build status of the various application-services
+   dependencies as part of its setup phase. When the command completes, it should print the resolved
+   versions of all dependencies, and you should see that application-services components have a version
+   number in the format `0.0.1-SNAPSHOT-{TIMESTAMP}`.
 
 ---
 
@@ -98,5 +163,6 @@ I always try to use the same number).
 [android-components-deps]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/buildSrc/src/main/java/Dependencies.kt#L37
 [android-components-build-gradle]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/build.gradle#L28
 [fenix-build-gradle-1]: https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/build.gradle#L26
-[fenix-build-gradle-2]: https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/build.gradle#L6
-[fenix-deps]: https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/buildSrc/src/main/java/Dependencies.kt#L28
+[fenix-deps]:
+https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/buildSrc/src/main/java/Dependencies.kt#L28
+[fenix-settings]: https://github.com/mozilla-mobile/fenix/blob/0d398f7d44f877a61cd243ee9fac587a9d5c0a1f/settings.gradle#L31
diff --git a/docs/howtos/setup-android-build-environment.md b/docs/howtos/setup-android-build-environment.md
@@ -2,9 +2,11 @@
 
 This document describes how to make local builds of the Android components in
 this repository. Most consumers of these components *do not* need to follow
-this process, but will instead use pre-built components [todo: link to this]
+this process, but will instead use pre-built components published by the
+[android-components](https://github.com/mozilla-mobile/android-components/) project.
 
-This document, and the build process itself, is a work-in-progress - please file issues (or just update the wiki!) if you notice errors or omissions.
+This document, and the build process itself, is a work-in-progress - please file issues
+(or even better, pull requests!) if you notice errors or omissions.
 
 ## Prepare your build environment
 
@@ -84,7 +86,7 @@ You can also publish single projects - eg:
 
     ./gradlew service-sync-places:publishToMavenLocal
 
-For more information about using the local maven repo, see this [android components guide](https://mozilla-mobile.github.io/android-components/contributing/testing-components-inside-app)
+For more information about using the local maven repo, see this [android components guide](https://mozilla-mobile.github.io/android-components/contributing/testing-components-inside-app).
 
 ### Other build types
 

diff --git a/docs/howtos/working-with-reference-browser.md b/docs/howtos/working-with-reference-browser.md