Build a zip containing all protobuf files #5412
Conversation
This builds a fat zip that contains the DAML-LF protobuf files and
the Ledger API protobuf files for a given release.
I’ll tackle the Azure config for uploading this to github releases afterwards.
To ease reviewing this is how the resulting zip looks like:
```
Archive: bazel-bin/release/protobufs.zip
Length Date Time Name
--------- ---------- ----- ----
3593 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/package_service.proto
1355 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/ledger_identity_service.proto
6262 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/event.proto
2282 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/testing/reset_service.proto
1908 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/testing/time_service.proto
886 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/ledger_offset.proto
1327 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/transaction_filter.proto
2380 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/command_submission_service.proto
6208 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/transaction_service.proto
3800 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/transaction.proto
1506 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/completion.proto
1893 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/ledger_configuration_service.proto
5109 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/admin/party_management_service.proto
3562 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/admin/package_management_service.proto
3559 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/admin/config_management_service.proto
5542 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/commands.proto
4286 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/command_completion_service.proto
2989 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/active_contracts_service.proto
3393 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/command_service.proto
5868 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/value.proto
1810 2010-01-01 00:00 protos-0.0.0/com/digitalasset/ledger/api/v1/trace_context.proto
558 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_6/daml_lf_0.proto
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_6/daml_lf.proto
27265 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_6/daml_lf_1.proto
558 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_7/daml_lf_0.proto
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_7/daml_lf.proto
35657 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_7/daml_lf_1.proto
558 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/daml_lf_0.proto
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/daml_lf.proto
37168 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/daml_lf_1.proto
596 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_dev/daml_lf_0.proto
1804 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_dev/daml_lf.proto
39864 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_dev/daml_lf_1.proto
```
changelog_begin
changelog_end
cocreature
force-pushed
the
protobuf-tarball
branch
2 times, most recently
from
d72edbf to
4aa1bb3
Compare
cocreature
force-pushed
the
protobuf-tarball
branch
from
4aa1bb3 to
599026d
Compare
| tools = [ctx.executable._tar, ctx.executable._gzip] | ||
| ctx.actions.run_shell( | ||
| inputs = [ctx.file._ledger_api_tarball] + ctx.files._daml_lf_tarballs, | ||
| outputs = [tmp_dir], |
There was a problem hiding this comment.
Do we need to run this on Windows? If not maybe just disable it there, given how directory outputs can be problematic.
There was a problem hiding this comment.
Disabled on Windows.
| arguments = ["cC", ctx.outputs.out.path, "@" + zipper_args_file.path], | ||
| ) | ||
|
|
||
| protos_zip = rule( |
There was a problem hiding this comment.
I'm starting to think that we should implement our own versions of pkg_tar that can create tar and zip and don't have the leading ./ issue.
There was a problem hiding this comment.
I vaguely recall seeing issues that pkg_zip also has other problems where it breaks directory structures. But haven’t looked too deeply into it.
There was a problem hiding this comment.
One has to be a bit careful about how one uses it. As far as I understand the right intuition for these rules is that they originate from generating docker images. There you want to distinguish between paths that should always be placed at the filesystem root (e.g. /usr/bin/...) and those that should be placed under some path prefix. They use leading ./ for that distinction. I.e. paths that don't lead with ./ will always remain at the root of the archive independent of the package_dir attribute. Paths that do lead with ./ will be placed underneath package_dir. AFAIK pkg_tar|zip always apply the ./ prefrix, but this distinction matters on the deps attribute. If you pull in an archive that doesn't follow that convention you'll end up with surprises.
release/util.bzl
Outdated
| outputs = [zipper_args_file], | ||
| inputs = [tmp_dir], | ||
| command = """ | ||
| {find} -L {tmp_dir} -type f | {sed} -E 's#^{tmp_dir}/(.*)$#protos-{version}/\\1={tmp_dir}/\\1#' > {args_file} |
There was a problem hiding this comment.
You could also use something like
{find} -L daml-lf/archive/src/main/protobuf -type f -printf "protos-{version}/%P=%p\n" > {args_file}
instead of using sed. Note the upper and lower-case p.
There was a problem hiding this comment.
Much nicer, thank you! Your find skillz are clearly ahead of mine 🙂
release/util.bzl
Outdated
| "_daml_lf_tarballs": attr.label_list( | ||
| allow_files = True, | ||
| default = [ | ||
| Label("//daml-lf/archive:daml_lf_{}_archive_proto_tarball.tar.gz".format(version)) | ||
| for version in LF_VERSIONS | ||
| ], | ||
| ), | ||
| "_ledger_api_tarball": attr.label(allow_single_file = True, default = Label("//ledger-api/grpc-definitions:ledger-api-protos.tar.gz")), |
There was a problem hiding this comment.
These look more like regular attributes than hidden attributes. But, given that this is only used once, it's probably fine.
There was a problem hiding this comment.
I’ve changed it to regular attributes.
This builds a fat zip that contains the DAML-LF protobuf files and
the Ledger API protobuf files for a given release.
I’ll tackle the Azure config for uploading this to github releases afterwards.
To ease reviewing this is how the resulting zip looks like:
changelog_begin
changelog_end
Pull Request Checklist
CHANGELOG_BEGINandCHANGELOG_ENDtagsNOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with
/AzurePipelines runtotrigger the build.