Port back compat tests to Bazel (sourcegraph#50932)

Clones and patches Sourcegraph git repository under @sourcegraph_back_compat so we can run tests targets from that particular release against the new database schema. Flakes can be defined to skip known problematic tests which are either flaky or simply cannot run against the new schema in case of a breaking change. See //dev/backcompat:flakes.bzl for more details about how to define them. The final result is the definition of a @sourcegraph_back_compat target, whose test targets are exactly the same as back then, but with instead a new schema. Example: `bazel test @sourcegraph_back_compat//enterprise/internal/batches/...`. See https://github.com/sourcegraph/sourcegraph/pull/50932/files#diff-2f07315ec320aa4080768fec54f32ebb2cbf4e3e6df7c51a314beda827c48c41R104 for the command generating the mandatory patch file in CI for these tests to run. If the patch file were to be missing, a placeholder diff is in place to make it explicit (in the eventuality of someone running those locally). A new CI job has been added because this one is fully independent from the other targets to build, and will be cached most of the time. Depending how it goes, I might bring that one over the main bazel job to avoid getting an agent just for that. Because it's cached, we can run this within PRs and have it take 10s max for 99% of the PRs and main builds. TODO: The flakefiles are still there, they're just defined in different place. I need to port the annotation as well, that has proven to be really useful. Also right now it's a bit manual as the 5.0.0 target is a bit peculiar, but that'll get much simpler as we progress, as the next old release will be requiring less and less patching. This turns about a dozen of 10m individual jobs into a single one ranging from 30s if there's not database changes to 10m if there's some of them and cold cache. ## Test plan  QA: fails on old code when a new migration breaks something: - https://buildkite.com/sourcegraph/sourcegraph/builds/214283#01879f2d-67be-4caf-b5d5-93f045e19348/118-126
Demolicity · Apr 21, 2023 · 1e44799 · 1e44799
1 parent 923da0e
commit 1e44799
Show file tree

Hide file tree

Showing 30 changed files with 227 additions and 1,213 deletions.
diff --git a/.gitignore b/.gitignore
@@ -189,3 +189,7 @@ index.scip
 /an
 /tr
 /cache-*.tar
+
+# Backward compatibility database tests patch, generated on the fly in CI
+# for `bazel test @sourcegraph_back_compat//...` targets.
+/dev/backcompat/back_compat_migrations.patch
diff --git a/WORKSPACE b/WORKSPACE
@@ -251,3 +251,6 @@ crates_repository(
 load("@crate_index//:defs.bzl", "crate_repositories")
 
 crate_repositories()
+
+load("//dev/backcompat:defs.bzl", "back_compat_defs")
+back_compat_defs()
diff --git a/dev/backcompat/BUILD.bazel b/dev/backcompat/BUILD.bazel
diff --git a/dev/backcompat/defs.bzl b/dev/backcompat/defs.bzl
@@ -0,0 +1,133 @@
+"""Database schema backward compatibility definitions.
+
+Clones and patches Sourcegraph git repository under @sourcegraph_back_compat so we can run tests targets
+from that particular release against the new database schema.
+
+Flakes can be defined to skip known problematic tests which are either flaky or simply cannot run against
+the new schema in case of a breaking change. See //dev/backcompat:flakes.bzl for more details about
+how to define them.
+
+The final result is the definition of a @sourcegraph_back_compat target, whose test targets are exactly
+the same as back then, but with instead a new schema.
+
+Example: `bazel test @sourcegraph_back_compat//enterprise/internal/batches/...`.
+
+See https://sourcegraph.com/search?q=context:global+dev/backcompat/patches/back_compat_migrations.patch+repo:github.com/sourcegraph/sourcegraph+lang:Go&patternType=standard&sm=0&groupBy=repo
+for the command generating the mandatory patch file in CI for these tests to run.
+
+If the patch file were to be missing, a placeholder diff is in place to make it explicit (in the
+eventuality of someone running those locally).
+"""
+
+load("test_release_version.bzl", "MINIMUM_UPGRADEABLE_VERSION", "MINIMUM_UPGRADEABLE_VERSION_REF")
+load("flakes.bzl", "FLAKES")
+
+load("@bazel_gazelle//:deps.bzl", "go_repository")
+load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
+
+# Shell snippet to disable a test on the fly. Needs to be formatted before being used.
+#
+# OSX ships BSD sed and the GNU sed that is traditionally available on Linux is named gsed instead.
+PATCH_GO_TEST = """_sed_binary="sed"
+if [ "$(uname)" == "Darwin" ]; then
+    _sed_binary="gsed"
+fi
+$_sed_binary -i "s/func {}/func _{}/g" {}/*.go
+"""
+
+# Assemble go test patches, based on the currently defined version.
+# See //dev/backcompat:test_release_version.bzl for the version definition.
+PATCH_GO_TEST_CMDS = [
+    PATCH_GO_TEST.format(test["prefix"], test["prefix"], test["path"], test["prefix"], test["reason"])
+    for test in FLAKES[MINIMUM_UPGRADEABLE_VERSION]
+]
+
+# Join all individual go test patches into a single shell snippet.
+PATCH_ALL_GO_TESTS_CMD = "\n".join(PATCH_GO_TEST_CMDS)
+
+# Replaces all occurences of @com_github_sourcegraph_(scip|conc) by @back_compat_com_github_sourcegraph_(scip|conc).
+PATCH_BUILD_FIXES_CMD = """_sed_binary="sed"
+if [ "$(uname)" == "Darwin" ]; then
+    _sed_binary="gsed"
+fi
+find . -type f -name "*.bazel" -exec $_sed_binary -i 's|@com_github_sourcegraph_conc|@back_compat_com_github_sourcegraph_conc|g' {} +
+find . -type f -name "*.bazel" -exec $_sed_binary -i 's|@com_github_sourcegraph_scip|@back_compat_com_github_sourcegraph_scip|g' {} +
+"""
+
+def back_compat_defs():
+    # github.com/sourcegraph/scip and github.com/sourcegraph/conc both rely on a few
+    # internal libraries from github.com/sourcegraph/sourcegraph/lib and their
+    # respective go_repository rules are annoted with build directives for Gazelle
+    # that fixes package resolution.
+    #
+    # When we're cloning the git repository of sourcegraph/sourcegraph on the release version
+    # we're testing against, those directives are not working as intended, as they appear to be
+    # evaluate target resolution in the global scope instead of @sourcegraph_back_compat, leading
+    # to compiling these two packages with the right targets, but linking them against the ones
+    # from our root workspace.
+    #
+    # So to work around that, we introduce two additional go_repository rules, which are the one that
+    # @sourcegraph_back_compat declares, but we rewrite the directives to resolve toward
+    # @sourcegraph_back_compat explicitly. The version/sum are exactly the same ones as defined
+    # by @sourcegraph_back_compat on the commit tagged by the release.
+    #
+    # And to make this work, we patch on the fly buildfiles in @sourcegraph_back_compat to reference
+    # the newly declared repos, by replacing all occurences of @com_github_sourcegraph_scip by
+    # @back_compat_com_github_sourcegraph_scip (and same thing for sourcegraph/conc).
+    go_repository(
+        name = "back_compat_com_github_sourcegraph_scip",
+        # This fixes the build for sourcegraph/scip which depends on sourcegraph/sourcegraph/lib but
+        # gazelle doesn't know how to resolve those packages from within sourcegraph/scip.
+        build_directives = [
+            "gazelle:resolve go github.com/sourcegraph/sourcegraph/lib/errors @sourcegraph_back_compat//lib/errors",
+            "gazelle:resolve go github.com/sourcegraph/sourcegraph/lib/codeintel/lsif/protocol @sourcegraph_back_compat//lib/codeintel/lsif/protocol",
+            "gazelle:resolve go github.com/sourcegraph/sourcegraph/lib/codeintel/lsif/protocol/reader @sourcegraph_back_compat//lib/codeintel/lsif/protocol/reader",
+            "gazelle:resolve go github.com/sourcegraph/sourcegraph/lib/codeintel/lsif/protocol/writer @sourcegraph_back_compat//lib/codeintel/lsif/protocol/writer",
+        ],
+        build_file_proto_mode = "disable_global",
+        importpath = "github.com/sourcegraph/scip",
+        sum = "h1:fWPxLkDObzzKTGe9vb6wpzK0FYkwcfSxmxUBvAOc8aw=", # Need to be manually updated when bumping the back compat release target.
+        version = "v0.2.4-0.20221213205653-aa0e511dcfef", # Need to be manually updated when bumping the back compat release target.
+    )
+
+    # Same logic for this repository.
+    go_repository(
+        name = "back_compat_com_github_sourcegraph_conc",
+        build_directives = [
+            "gazelle:resolve go github.com/sourcegraph/sourcegraph/lib/errors @sourcegraph_back_compat//lib/errors",
+        ],
+        build_file_proto_mode = "disable_global",
+        importpath = "github.com/sourcegraph/conc",
+        sum = "h1:96VpOCAtXDCQ8Oycz0ftHqdPyMi8w12ltN4L2noYg7s=", # Need to be manually updated when bumping the back compat release target.
+        version = "v0.2.0", # Need to be manually updated when bumping the back compat release target.
+    )
+
+
+    # Now that we have declared a replacement for the two problematic go packages that
+    # @sourcegraph_back_compat depends on, we can define the repository itself. Because it
+    # comes with its Bazel rules (logical, that's just the current repository but with a different
+    # commit), we can simply use git_repository to fetch it and apply patches on the fly to
+    # inject migrations and build fixes.
+    git_repository(
+        name = "sourcegraph_back_compat",
+        remote = "https://github.com/sourcegraph/sourcegraph.git",
+        patches = ["//dev/backcompat/patches:back_compat_migrations.patch"],
+        patch_args = ["-p1"],
+        commit = MINIMUM_UPGRADEABLE_VERSION_REF,
+        patch_cmds = [
+            # webpack rules are complaining about a missing entry point, which is irrelevant as we're
+            # simply running go tests only. Therefore, we can simply drop the client folder.
+            #
+            # Because the target release at the time of writing this comment is 5.0.0 which was
+            # before we fully switched to Bazel, it's not exactly in a buildable state, and we're using
+            # a backported fix  to make it buildable. That fix is merely about running `bazel configure`
+            # and dropping the client folder.
+            #
+            # "rm -Rf client",
+            PATCH_ALL_GO_TESTS_CMD,
+            PATCH_BUILD_FIXES_CMD,
+            # Seems to be affecting the root workspace somehow.
+            # TODO(JH) Look into bzlmod.
+            "rm .bazelversion",
+        ],
+    )
diff --git a/dev/backcompat/flakes.bzl b/dev/backcompat/flakes.bzl
@@ -0,0 +1,17 @@
+"""Defines flaky tests to disable when running tests from the release we're testing the new database schema against.
+"""
+
+FLAKES = {
+    "5.0.0": [
+        {
+            "path": "enterprise/cmd/frontend/internal/batches/resolvers",
+            "prefix": "TestRepositoryPermissions",
+            "reason": "Test was having incomplete data, fails now that constraints are in place"
+        },
+        {
+            "path": "dev/sg/linters",
+            "prefix": "TestLibLogLinter",
+            "reason": "Test was having incomplete data, fails now that constraints are in place"
+        }
+    ]
+}
diff --git a/dev/backcompat/patches/BUILD.bazel b/dev/backcompat/patches/BUILD.bazel
diff --git a/dev/backcompat/patches/back_compat_migrations.patch b/dev/backcompat/patches/back_compat_migrations.patch
@@ -0,0 +1,3 @@
+diff --git a/failme b/failme
+--- This patch to port new migrations onto old release needs to be generated to insert the back compat tests, see docs in dev backcompat
++++ Read the docs in dev/backcompat
diff --git a/dev/backcompat/test_release_version.bzl b/dev/backcompat/test_release_version.bzl
@@ -0,0 +1,13 @@
+"""Defines the minimum ugradeable version of Sourcegraph.
+
+This designates the mininum version from which we guarantees the newest database
+schema can run.
+
+See https://docs.sourcegraph.com/dev/background-information/sql/migrations
+"""
+
+# Defines which version we target with the backward compatibility tests.
+MINIMUM_UPGRADEABLE_VERSION = "5.0.0"
+
+# Defines a reproducible reference to clone Sourcegraph at to run those tests.
+MINIMUM_UPGRADEABLE_VERSION_REF = "177663e4329d712f3493787410f71da60fe5dc7f"
diff --git a/dev/ci/go-backcompat/BUILD.bazel b/dev/ci/go-backcompat/BUILD.bazel
diff --git a/dev/ci/go-backcompat/README.md b/dev/ci/go-backcompat/README.md
diff --git a/dev/ci/go-backcompat/flakefiles/.gitkeep b/dev/ci/go-backcompat/flakefiles/.gitkeep
diff --git a/dev/ci/go-backcompat/flakefiles/v3.35.0.json b/dev/ci/go-backcompat/flakefiles/v3.35.0.json
diff --git a/dev/ci/go-backcompat/flakefiles/v3.37.0.json b/dev/ci/go-backcompat/flakefiles/v3.37.0.json
diff --git a/dev/ci/go-backcompat/flakefiles/v3.38.0.json b/dev/ci/go-backcompat/flakefiles/v3.38.0.json
diff --git a/dev/ci/go-backcompat/flakefiles/v3.39.0.json b/dev/ci/go-backcompat/flakefiles/v3.39.0.json
diff --git a/dev/ci/go-backcompat/flakefiles/v3.40.0.json b/dev/ci/go-backcompat/flakefiles/v3.40.0.json