docs: add troubleshooting and version policy docs

GoogleChrome · Nov 11, 2019 · 0d5544f · 0d5544f
1 parent 2825b46
commit 0d5544f
Show file tree

Hide file tree

Showing 3 changed files with 83 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -11,9 +11,11 @@
 Lighthouse CI is a set of commands that make running, asserting, saving, and retrieving [Lighthouse](https://github.com/GoogleChrome/lighthouse) results as easy as possible.
 
 - [Getting Started](./docs/getting-started.md)
+- [Troubleshooting FAQs](./docs/troubleshooting.md)
 - [CLI Documentation](./docs/cli.md)
 - [Assertions Documentation](./docs/assertions.md)
 - [Server Documentation](./docs/recipes/docker-server/README.md)
+- [Versioning Policy](./docs/version-policy.md)
 
 <img src="https://user-images.githubusercontent.com/39191/68522269-7917b680-025e-11ea-8d96-2774c0a0b04c.png" width="48%">
 
@@ -26,7 +28,7 @@ language: node_js
 node_js:
   - 10 # use Node 10 LTS or later
 before_install:
-  - npm install -g @lhci/cli
+  - npm install -g @lhci/cli@0.3.0-alpha.0
 script:
   - npm run build # build your site
   - lhci autorun # run lighthouse CI

diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -0,0 +1,45 @@
+# Troubleshooting FAQs
+
+## I can't get it working with my CI provider. Help!
+
+First, make sure you're using `lhci autorun` or invoking `lhci healthcheck --fatal` in your script. Check the output logs of the healthcheck for anything that's failing and try resolving those problems first using the guide below before filing an issue.
+
+## The GitHub App/LHCI server upload isn't working on PRs from external contributors.
+
+Most CI systems prevent secrets from leaking into the environment of untrusted pull requests from forks. This is a helpful security measure to prevent privileged information from falling into malicious hands.
+
+If your LHCI server is already publicly accessible and you don't mind allowing contributors to potentially spam data into it, then you can safely make the `LHCI_TOKEN` public as well. It is an additive-write-only credential that cannot destroy historical data.
+
+## LHCI server isn't finding the correct base branch ancestor.
+
+Some CI systems download a shallow copy of the repository to improve cloning time, but this prevents contextual information like the shared base ancestor commit from being picked up by Lighthouse CI. Ensure you configure your provider's git depth to a value large enough to cover most of your feature branches.
+
+**.travis.yml**
+
+```yaml
+git:
+  #### Example: clone with the past 20 commits ######
+  depth: 20
+  #### Example: disable git depth entirely ######
+  depth: false
+```
+
+## LHCI server won't let me upload a rerun of a build.
+
+This is a precautionary measure to prevent the effective deletion of historical data from the server. As workaround, you can create empty commits to rerun builds `git commit --allow-empty -m 'rerun CI' && git push`.
+
+## My URLs change between builds and LHCI server can't link them.
+
+If the URLs that you audit contain random components (ports, UUIDs, hashes, etc), then you'll need to tell LHCI how to normalize your data when you upload it with the `--url-replacement-patterns` option.
+
+When using this option, you'll lose the default `:PORT` and `UUID` replacements, so be sure to copy those into your configuration if necessary. [See the example in the lighthuose-ci repo itself](https://github.com/GoogleChrome/lighthouse-ci/blob/v0.3.0-alpha.0/lighthouserc.json) for how to configure these patterns.
+
+## I'm seeing differences in the results even though I didn't change anything.
+
+Webpages are fickle beasts and variance is a common problem when measuring properties of pages in a real browser as Lighthouse does. The Lighthouse team has [several](https://github.com/GoogleChrome/lighthouse/blob/v5.0.0/docs/variability.md) good [documents](https://docs.google.com/document/d/1AujmeKvBhzr-d8IsB7zPeS-vOtxCdw2GnspKpxJ7d_I/edit) on the [subject](https://docs.google.com/document/d/1BqtL-nG53rxWOI5RO0pItSRPowZVnYJ_gBEQCJ5EeUE/edit?usp=sharing), but the tl;dr is...
+
+- Run many times (LHCI runs 3 by default, use the `--numberOfRuns=X` option to increase this).
+- Use reliable hardware (avoid underpowered CI systems like Appveyor and burstable instances, use dedicate machines with a minimum 2 cores and 4GB of RAM).
+- Eliminate non-determinism (remove random `setTimeout` calls, A/B tests, etc).
+- Avoid external dependencies (disable variable third-party integrations like ads, YouTube embeds, analytics, etc).
+- Assert facts over conclusions (start with assertions on the number and size of your JavaScript requests rather than the value of TTI).
diff --git a/docs/version-policy.md b/docs/version-policy.md
@@ -0,0 +1,35 @@
+# Version Policy
+
+## Overview
+
+LHCI follows semver on the 0.x.x track where breaking changes are not introduced in a patch version with the exception of critical security or stability patches. Because what constitutes a breaking change can be somewhat ambiguous, this document outlines what is considered a breaking change, the expected release cadence of breaking changes, and the responsibilities of the LHCI team in communicating breaking changes.
+
+## Communication
+
+When a breaking change has been introduced, the relevant commit will contain a line of the form `BREAKING CHANGE: brief description of the break`. These breaking change descriptions will be compiled into the release notes of the next release.
+
+## Cadence
+
+Breaking changes come from many sources within LHCI and have a much stricter definition than those in core Lighthouse. Accordinly, breaking change releases will be more frequent, but should also be less impactful on average. See the examples below for estimated frequency of each impact.
+
+## Example Breaking Changes
+
+| Change                                                    | Impact   | Expected Frequency |
+| --------------------------------------------------------- | -------- | ------------------ |
+| Lighthouse minor version bump that introduces a new audit | Low      | 3-6/year           |
+| Lighthouse major version bump                             | Moderate | 1-2/year           |
+| LHCI server frontend URL format change                    | High     | 1-2/year           |
+| Change in an assertion preset (stricter)                  | High     | 1-2/year           |
+| Removal of a flag or option                               | High     | 1-2/year           |
+| Change of the default value for a flag or option          | High     | 1-2/year           |
+
+## Example Non-Breaking Changes
+
+| Change                                        | Impact   | Expected Frequency |
+| --------------------------------------------- | -------- | ------------------ |
+| Lighthouse minor version bump that fixes bugs | Low      | 3-6/year           |
+| Addition of a new flag or option              | Low      | 2-4/year           |
+| LHCI server frontend design change            | Moderate | 1-2/year           |
+| Change in an assertion preset (more lenient)  | Moderate | 1-2/year           |
+| Fix of a critical security vulnerability      | High     | 0/year ;)          |
+| Fix of a fatal startup bug                    | High     | 0/year ;)          |