Collect build-event-stream #9831

aherrmann-da · 2021-05-28T11:22:32Z

Collect Bazel metrics and copy the data to a GCS bucket.

Collects Bazel metrics and build metadata:
- build event stream as JSON dump from the bazel build and bazel test steps in build.sh and build.ps1
- JSON profile from the bazel build and bazel test steps in build.sh and build.ps1
- Metadata about the current job
Uploads this data to the GCS bucket gs://daml-data/bazel-metrics under a directory of the form TIMESTAMP-JOBNAME-JOBID-JOBATTEMPT
Installs jq in Windows dev-env

The uploaded data ranges between 20M to 50M in size per CI Job.

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

ci/upload-bazel-metrics.yml

cocreature

Nice work!

.dadew

ci/job-variables.yml

ci/build.yml

cocreature · 2021-06-10T15:54:14Z

ci/upload-bazel-metrics.yml

+  - template: bash-lib.yml
+    parameters:
+      var_name: bash_lib
+  - bash: |


Answering my own question: this takes less than a minute so not an issue for CI times.

changelog_begin changelog_end

@stefanobaghino-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @stefanobaghino-da is in charge of this release. Commit log: ``` adeba20 DPP-403 log flyway progress (#9964) 401069e prepare for upgrade to Scalaz 7.3.3 (#9997) 3ad9cd5 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005) af144e8 LF: Structure package loading errors (#9980) 3583fd1 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007) e49c7c8 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882) acb7733 Release 1.14 RC 2 (#10006) 3d2d4b9 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990) 0c32cf2 Clarify the semantics of DIV_INT64 in LF spec. (#10003) b79e739 Fix Type (#10004) 998f106 Drop pkgIds argument from Engine.validatePackages (#9986) ae5010f [Mutable state cache] Start cache from latest ledger head (#9998) dd42a5a update maven_install_2.12.json (#9999) 66d38cd Storage query abstraction first sketch (#9959) c750136 Just print out the operator name for the operation logging in CommandService.scala (#9989) b6faace Changing oracle backend from using varray to json array (#9943) 603a4a4 Add date to log statements in the HTTP JSON service (#9988) 836a82a Add proper error handling for missing contract keys (#9972) 85667af Address npm vulnerabilities (#9978) d7d6e37 LF: Structure Engine Errors. (#9971) 0adbbdc update Daml Hub auth in create-daml-app (#9973) 997a7d4 Use symlinks on Windows (#9969) e4258e6 docs: tweak Exception intro (#9967) b2c4ef8 rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933) de7a462 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940) 630d021 Collect build-event-stream (#9831) 26e199a Rerelease 1.11.2 snapshot (#9965) 8399039 Remove broken dep from release job (#9963) e1db529 DPP-390 configurable event decoding parallelism (#9931) cc7af93 [docs] Remove link to the #daml-contributors (#9928) f7ac3ba Release 1.11.2 snapshot attempt 2 (#9962) 4634147 Adds Indexer state to GRPC health checks [DPP-434] (#9951) 28c567b Release SDK 1.11.2 (#9957) 5f0f575 Define IncompleteTransaction for use by scenario-service (#9952) b7fd833 Increase `eventually` timeout when waiting for queue to be killed (#9955) ca0fb16 Fix collect_build_data job (#9956) 42030b2 Skip Scala_2_12 job on releases (#9954) 5e1ddfb LF: Fix specification of ArithmeticError (#9949) 3093a98 Forward port additional contract keys validation tests (#9944) e541558 update NOTICES file (#9946) db792a9 Proxy ledger API health endpoint in JSON API health endpoint (#9945) 13767b5 Load the correct logback file for the http json service respecting the deployment situation (#9938) 4e49cf6 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857) fce8e54 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932) ab55fb2 Handle module-prefixes in Java codegen (#9929) 7375e28 Force resolution for css what to 5.0.1 (#9925) ``` Changelog: ``` - [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level) - [JSON-API] The source and the path for incoming http requests are now logged - [JSON-API] The http response for a request is now logged - [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise) - [JSON API] log statements now include the date next to the time - [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx [Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint - [JSON API] The healthcheck endpoint on the JSON API now proxies the health check endpoint on the underlying ledger. Previously it only queried for ledger end to determine ledger health. [HTTP-JSON] - fixed that json log output could not be enabled via cli options except via usage of env vars - [Integration Kit] - ledger-api-bench-tool - reading active contract streams - [Integration Kit] - ledger-api-bench-tool - Reading completions stream - [Java Codegen] The Java codegen will now pick up the `module-prefixes` field from `daml.yaml` which can be used to handle module name collisions between different DALFs. ``` CHANGELOG_BEGIN CHANGELOG_END

@stefanobaghino-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @stefanobaghino-da is in charge of this release. Commit log: ``` adeba20 DPP-403 log flyway progress (#9964) 401069e prepare for upgrade to Scalaz 7.3.3 (#9997) 3ad9cd5 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005) af144e8 LF: Structure package loading errors (#9980) 3583fd1 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007) e49c7c8 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882) acb7733 Release 1.14 RC 2 (#10006) 3d2d4b9 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990) 0c32cf2 Clarify the semantics of DIV_INT64 in LF spec. (#10003) b79e739 Fix Type (#10004) 998f106 Drop pkgIds argument from Engine.validatePackages (#9986) ae5010f [Mutable state cache] Start cache from latest ledger head (#9998) dd42a5a update maven_install_2.12.json (#9999) 66d38cd Storage query abstraction first sketch (#9959) c750136 Just print out the operator name for the operation logging in CommandService.scala (#9989) b6faace Changing oracle backend from using varray to json array (#9943) 603a4a4 Add date to log statements in the HTTP JSON service (#9988) 836a82a Add proper error handling for missing contract keys (#9972) 85667af Address npm vulnerabilities (#9978) d7d6e37 LF: Structure Engine Errors. (#9971) 0adbbdc update Daml Hub auth in create-daml-app (#9973) 997a7d4 Use symlinks on Windows (#9969) e4258e6 docs: tweak Exception intro (#9967) b2c4ef8 rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933) de7a462 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940) 630d021 Collect build-event-stream (#9831) 26e199a Rerelease 1.11.2 snapshot (#9965) 8399039 Remove broken dep from release job (#9963) e1db529 DPP-390 configurable event decoding parallelism (#9931) cc7af93 [docs] Remove link to the #daml-contributors (#9928) f7ac3ba Release 1.11.2 snapshot attempt 2 (#9962) 4634147 Adds Indexer state to GRPC health checks [DPP-434] (#9951) 28c567b Release SDK 1.11.2 (#9957) 5f0f575 Define IncompleteTransaction for use by scenario-service (#9952) b7fd833 Increase `eventually` timeout when waiting for queue to be killed (#9955) ca0fb16 Fix collect_build_data job (#9956) 42030b2 Skip Scala_2_12 job on releases (#9954) 5e1ddfb LF: Fix specification of ArithmeticError (#9949) 3093a98 Forward port additional contract keys validation tests (#9944) e541558 update NOTICES file (#9946) db792a9 Proxy ledger API health endpoint in JSON API health endpoint (#9945) 13767b5 Load the correct logback file for the http json service respecting the deployment situation (#9938) 4e49cf6 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857) fce8e54 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932) ab55fb2 Handle module-prefixes in Java codegen (#9929) 7375e28 Force resolution for css what to 5.0.1 (#9925) ``` Changelog: ``` - [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level) - [JSON-API] The source and the path for incoming http requests are now logged - [JSON-API] The http response for a request is now logged - [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise) - [JSON API] log statements now include the date next to the time - [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx [Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint - [JSON API] The healthcheck endpoint on the JSON API now proxies the health check endpoint on the underlying ledger. Previously it only queried for ledger end to determine ledger health. [HTTP-JSON] - fixed that json log output could not be enabled via cli options except via usage of env vars - [Integration Kit] - ledger-api-bench-tool - reading active contract streams - [Integration Kit] - ledger-api-bench-tool - Reading completions stream - [Java Codegen] The Java codegen will now pick up the `module-prefixes` field from `daml.yaml` which can be used to handle module name collisions between different DALFs. ``` CHANGELOG_BEGIN CHANGELOG_END Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>

@stefanobaghino-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @stefanobaghino-da is in charge of this release. Commit log: ``` adeba20 DPP-403 log flyway progress (#9964) 401069e prepare for upgrade to Scalaz 7.3.3 (#9997) 3ad9cd5 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005) af144e8 LF: Structure package loading errors (#9980) 3583fd1 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007) e49c7c8 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882) acb7733 Release 1.14 RC 2 (#10006) 3d2d4b9 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990) 0c32cf2 Clarify the semantics of DIV_INT64 in LF spec. (#10003) b79e739 Fix Type (#10004) 998f106 Drop pkgIds argument from Engine.validatePackages (#9986) ae5010f [Mutable state cache] Start cache from latest ledger head (#9998) dd42a5a update maven_install_2.12.json (#9999) 66d38cd Storage query abstraction first sketch (#9959) c750136 Just print out the operator name for the operation logging in CommandService.scala (#9989) b6faace Changing oracle backend from using varray to json array (#9943) 603a4a4 Add date to log statements in the HTTP JSON service (#9988) 836a82a Add proper error handling for missing contract keys (#9972) 85667af Address npm vulnerabilities (#9978) d7d6e37 LF: Structure Engine Errors. (#9971) 0adbbdc update Daml Hub auth in create-daml-app (#9973) 997a7d4 Use symlinks on Windows (#9969) e4258e6 docs: tweak Exception intro (#9967) b2c4ef8 rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933) de7a462 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940) 630d021 Collect build-event-stream (#9831) 26e199a Rerelease 1.11.2 snapshot (#9965) 8399039 Remove broken dep from release job (#9963) e1db529 DPP-390 configurable event decoding parallelism (#9931) cc7af93 [docs] Remove link to the #daml-contributors (#9928) f7ac3ba Release 1.11.2 snapshot attempt 2 (#9962) 4634147 Adds Indexer state to GRPC health checks [DPP-434] (#9951) 28c567b Release SDK 1.11.2 (#9957) 5f0f575 Define IncompleteTransaction for use by scenario-service (#9952) b7fd833 Increase `eventually` timeout when waiting for queue to be killed (#9955) ca0fb16 Fix collect_build_data job (#9956) 42030b2 Skip Scala_2_12 job on releases (#9954) 5e1ddfb LF: Fix specification of ArithmeticError (#9949) 3093a98 Forward port additional contract keys validation tests (#9944) e541558 update NOTICES file (#9946) db792a9 Proxy ledger API health endpoint in JSON API health endpoint (#9945) 13767b5 Load the correct logback file for the http json service respecting the deployment situation (#9938) 4e49cf6 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857) fce8e54 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932) ab55fb2 Handle module-prefixes in Java codegen (#9929) 7375e28 Force resolution for css what to 5.0.1 (#9925) ``` Changelog: ``` - [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level) - [JSON-API] The source and the path for incoming http requests are now logged - [JSON-API] The http response for a request is now logged - [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise) - [JSON API] log statements now include the date next to the time - [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx [Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint - [JSON API] The healthcheck endpoint on the JSON API now proxies the health check endpoint on the underlying ledger. Previously it only queried for ledger end to determine ledger health. [HTTP-JSON] - fixed that json log output could not be enabled via cli options except via usage of env vars - [Integration Kit] - ledger-api-bench-tool - reading active contract streams - [Integration Kit] - ledger-api-bench-tool - Reading completions stream - [Java Codegen] The Java codegen will now pick up the `module-prefixes` field from `daml.yaml` which can be used to handle module name collisions between different DALFs. ``` CHANGELOG_BEGIN CHANGELOG_END

Commit log: ``` adeba20 DPP-403 log flyway progress (#9964) 401069e prepare for upgrade to Scalaz 7.3.3 (#9997) 3ad9cd5 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005) af144e8 LF: Structure package loading errors (#9980) 3583fd1 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007) e49c7c8 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882) acb7733 Release 1.14 RC 2 (#10006) 3d2d4b9 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990) 0c32cf2 Clarify the semantics of DIV_INT64 in LF spec. (#10003) b79e739 Fix Type (#10004) 998f106 Drop pkgIds argument from Engine.validatePackages (#9986) ae5010f [Mutable state cache] Start cache from latest ledger head (#9998) dd42a5a update maven_install_2.12.json (#9999) 66d38cd Storage query abstraction first sketch (#9959) c750136 Just print out the operator name for the operation logging in CommandService.scala (#9989) b6faace Changing oracle backend from using varray to json array (#9943) 603a4a4 Add date to log statements in the HTTP JSON service (#9988) 836a82a Add proper error handling for missing contract keys (#9972) 85667af Address npm vulnerabilities (#9978) d7d6e37 LF: Structure Engine Errors. (#9971) 0adbbdc update Daml Hub auth in create-daml-app (#9973) 997a7d4 Use symlinks on Windows (#9969) e4258e6 docs: tweak Exception intro (#9967) b2c4ef8 rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933) de7a462 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940) 630d021 Collect build-event-stream (#9831) 26e199a Rerelease 1.11.2 snapshot (#9965) 8399039 Remove broken dep from release job (#9963) e1db529 DPP-390 configurable event decoding parallelism (#9931) cc7af93 [docs] Remove link to the #daml-contributors (#9928) f7ac3ba Release 1.11.2 snapshot attempt 2 (#9962) 4634147 Adds Indexer state to GRPC health checks [DPP-434] (#9951) 28c567b Release SDK 1.11.2 (#9957) 5f0f575 Define IncompleteTransaction for use by scenario-service (#9952) b7fd833 Increase `eventually` timeout when waiting for queue to be killed (#9955) ca0fb16 Fix collect_build_data job (#9956) 42030b2 Skip Scala_2_12 job on releases (#9954) 5e1ddfb LF: Fix specification of ArithmeticError (#9949) 3093a98 Forward port additional contract keys validation tests (#9944) e541558 update NOTICES file (#9946) db792a9 Proxy ledger API health endpoint in JSON API health endpoint (#9945) 13767b5 Load the correct logback file for the http json service respecting the deployment situation (#9938) 4e49cf6 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857) fce8e54 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932) ab55fb2 Handle module-prefixes in Java codegen (#9929) 7375e28 Force resolution for css what to 5.0.1 (#9925) ``` Changelog: ``` - [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level) - [JSON-API] The source and the path for incoming http requests are now logged - [JSON-API] The http response for a request is now logged - [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise) - [JSON API] log statements now include the date next to the time - [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx [Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint - [JSON API] The healthcheck endpoint on the JSON API now proxies the health check endpoint on the underlying ledger. Previously it only queried for ledger end to determine ledger health. [HTTP-JSON] - fixed that json log output could not be enabled via cli options except via usage of env vars - [Integration Kit] - ledger-api-bench-tool - reading active contract streams - [Integration Kit] - ledger-api-bench-tool - Reading completions stream - [Java Codegen] The Java codegen will now pick up the `module-prefixes` field from `daml.yaml` which can be used to handle module name collisions between different DALFs. ``` CHANGELOG_BEGIN CHANGELOG_END

aherrmann-da force-pushed the ci-monitor branch 3 times, most recently from 8478803 to 189d6de Compare June 7, 2021 09:40

garyverhaegen-da reviewed Jun 7, 2021

View reviewed changes

ci/upload-bazel-metrics.yml Outdated Show resolved Hide resolved

garyverhaegen-da reviewed Jun 7, 2021

View reviewed changes

ci/upload-bazel-metrics.yml Outdated Show resolved Hide resolved

aherrmann-da force-pushed the ci-monitor branch from b672ff5 to 863cd85 Compare June 10, 2021 09:33

aherrmann-da marked this pull request as ready for review June 10, 2021 13:26

aherrmann-da force-pushed the ci-monitor branch from 863cd85 to c173c48 Compare June 10, 2021 13:31

aherrmann-da requested review from garyverhaegen-da and cocreature June 10, 2021 14:59

cocreature approved these changes Jun 10, 2021

View reviewed changes

Generate Bazel logs and upload to GCS

7f14d15

changelog_begin changelog_end

aherrmann-da force-pushed the ci-monitor branch from c173c48 to 7f14d15 Compare June 11, 2021 09:07

Move git_*_sha into variables template

73357cd

aherrmann-da merged commit 630d021 into main Jun 11, 2021

aherrmann-da deleted the ci-monitor branch June 11, 2021 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collect build-event-stream #9831

Collect build-event-stream #9831

aherrmann-da commented May 28, 2021 •

edited

Loading

cocreature left a comment

cocreature Jun 10, 2021

Collect build-event-stream #9831

Collect build-event-stream #9831

Conversation

aherrmann-da commented May 28, 2021 • edited Loading

Pull Request Checklist

cocreature left a comment

Choose a reason for hiding this comment

cocreature Jun 10, 2021

Choose a reason for hiding this comment

aherrmann-da commented May 28, 2021 •

edited

Loading