add blackduck scan to run on master #6130

dasormeter · 2020-05-27T21:49:31Z

THIS PR

Enable blackduck scan to run only on linux
Include bazel haskell library scan with current snapshot verison of hub detect
bazel jvm and npm and all other build tool scans runs as a second round but to the same version and project

TODO (follow-on PRs)

Daily binary/signature scan -- time consuming and takes hours so only need to run daily or weekly as a cron job Enable daily or weekly full Blackduck binary/signature scan #6420
Enable GO_MOD scanning (need to know location of GO exe on CI) Enable Blackduck GO scanning #6419

garyverhaegen-da · 2020-05-27T21:56:48Z

I don't think our CI has access to DACH-NY.

ci/build-unix.yml

dasormeter · 2020-05-27T22:07:01Z

I don't think our CI has access to DACH-NY.

that particular repo in DACH-NY is public so should be okay

garyverhaegen-da · 2020-05-27T22:26:21Z

Also note that despite the description of the PR, as it stands it would run on every PR and master commit, and on both Linux and macOS.

dasormeter · 2020-05-27T22:30:44Z

Also note that despite the description of the PR, as it stands it would run on every PR and master commit, and on both Linux and macOS.

yeah just testing the run first, will then move it to the daily job. i'll mark it as DRAFT so it is clear it is not yet to be merged

ci/build-unix.yml

Pipfile

garyverhaegen-da · 2020-06-18T18:35:37Z

ci/build-unix.yml

+    env:
+      BLACKDUCK_HUBDETECT_TOKEN: $(BLACKDUCK_HUBDETECT_TOKEN)
+
+


Do these depend on the build having run? If not, could we move this to a separate job?

@garyverhaegen-da yeah it does depend on the build having run since we need the bazel build resolution and node modules and other build output to be interrogated to get the full list of libraries used

cocreature

Requesting changes until I understand how I can see results and why we want this to break our master builds but not our PR builds.

garyverhaegen-da

I really don't like the idea that we have a check that can fail on master but is not checked on PRs.

garyverhaegen-da · 2020-06-19T15:20:03Z

I'm a bit confused by the change to use $(System.PullRequest.SourceBranch). I thought the conclusion of our discussions was to instead move this to a scheduled (daily) run on master, which would thus not be in the context of a PR and therefore not have that variable set.

dasormeter · 2020-06-23T16:01:43Z

I'm a bit confused by the change to use $(System.PullRequest.SourceBranch). I thought the conclusion of our discussions was to instead move this to a scheduled (daily) run on master, which would thus not be in the context of a PR and therefore not have that variable set.

experimenting with runs that fail on policy violation on PRs to see how long it takes

run on PRs and master

…for bazel jvm and node and other langs

ensure haskell processing complete before full scan

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

…to wrapper script

garyverhaegen-da · 2020-12-03T17:27:23Z

ci/cron/daily-compat.yml

+      - template: ../bash-lib.yml
+        parameters:
+          var_name: bash_lib


Suggested change

- template: ../bash-lib.yml

parameters:

var_name: bash_lib

You're not using that library.

* add blackduck scan * disable go scanning exclude entire language-support/ts directory for node scanning break to multiple lines to make command line params easier to parse * Increase timeout for blackduck binary scan * update blackduck scan config * remove some exclusions, force python3 * exclude GO until path to go executable can be resolved * added readme explanation of why we want this file * fail in case of policy violation * ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs * trigger notices file gen to ensure BOM complete * remove trailing end of lines * run with latest detect version and unique code location name changes to wrapper script * Add blackduck to daily compat job * DO NOT MERGE: condition false to disable other jobs for testing * remove parameters not available to cronjob

* add blackduck scan to run on master (#6130) * add blackduck scan * disable go scanning exclude entire language-support/ts directory for node scanning break to multiple lines to make command line params easier to parse * Increase timeout for blackduck binary scan * update blackduck scan config * remove some exclusions, force python3 * exclude GO until path to go executable can be resolved * added readme explanation of why we want this file * fail in case of policy violation * ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs * trigger notices file gen to ensure BOM complete * remove trailing end of lines * run with latest detect version and unique code location name changes to wrapper script * Add blackduck to daily compat job * DO NOT MERGE: condition false to disable other jobs for testing * remove parameters not available to cronjob * Revert changes to regular CI pipeline CHANGELOG_BEGIN CHANGELOG_END Signed-off-by: Brian Healey <brian.healey@digitalasset.com> * Do not get branch name from variable * Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability * Remove disabling of other jobs, set to branch to be used on prod runs * Apply suggestions from code review Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com> * Address code review comments * Updated NOTICES file * Run bazel build, update NOTICES file * Correct dade-assist * do not have perms to pipe to dev/null * Add md file explaining how to update NOTICES file * Add instructions for running blackduck locally * Add a link to full security-blackduck readme Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

dasormeter requested a review from garyverhaegen-da as a code owner May 27, 2020 21:49

dasormeter requested review from kareem-DA and nycnewman May 27, 2020 21:49

garyverhaegen-da reviewed May 27, 2020

View reviewed changes

ci/build-unix.yml Outdated Show resolved Hide resolved

garyverhaegen-da reviewed May 27, 2020

View reviewed changes

ci/build-unix.yml Outdated Show resolved Hide resolved

dasormeter force-pushed the enable-blackduck branch from 9b7d8a6 to d0b2375 Compare May 27, 2020 22:08

dasormeter mentioned this pull request May 27, 2020

blackduck in azure pipeline #461

Closed

5 tasks

dasormeter marked this pull request as draft May 27, 2020 22:30

dasormeter force-pushed the enable-blackduck branch from de79943 to 7bb3050 Compare June 2, 2020 14:55

dasormeter self-assigned this Jun 2, 2020

dasormeter force-pushed the enable-blackduck branch from fea2dc3 to 7cbf818 Compare June 18, 2020 15:55

kareem-DA approved these changes Jun 18, 2020

View reviewed changes

dasormeter commented Jun 18, 2020

View reviewed changes

ci/build-unix.yml Outdated Show resolved Hide resolved

ci/build-unix.yml Outdated Show resolved Hide resolved

ci/build-unix.yml Outdated Show resolved Hide resolved

ci/build-unix.yml Outdated Show resolved Hide resolved

dasormeter marked this pull request as ready for review June 18, 2020 16:55

dasormeter changed the title ~~add blackduck scan~~ add blackduck scan to run on master Jun 18, 2020

dasormeter commented Jun 18, 2020

View reviewed changes

Pipfile Show resolved Hide resolved

dasormeter mentioned this pull request Jun 18, 2020

Enable Blackduck GO scanning #6419

Open

garyverhaegen-da reviewed Jun 18, 2020

View reviewed changes

dasormeter mentioned this pull request Jun 18, 2020

Enable daily or weekly full Blackduck binary/signature scan #6420

Closed

cocreature requested changes Jun 18, 2020

View reviewed changes

garyverhaegen-da suggested changes Jun 18, 2020

View reviewed changes

dasormeter force-pushed the enable-blackduck branch 3 times, most recently from 4fc7d3b to c71d956 Compare June 25, 2020 14:54

dasormeter added 20 commits December 3, 2020 11:31

Update blackduck with version that support haskell

fb40754

exclude GO until path to go executable can be resolved

cc5e6d2

change branch name

542088d

add condition to only run on specific branch

64e7b69

correct conditions

6c94491

use Build.SourceBranchName

329aa02

add guard to only run blackduck scan on master branch

7c814e6

sha reference to synopsys-detect script

f30ab53

run on PRs and master

added readme explanation of why we want this file

0619ff8

mention the word blackduck

b18c068

use System.PullRequest.SourceBranch

3a347d3

fail in case of policy violation

7b667b5

ensure haskell bazel scan completes before running second round scan …

870d446

…for bazel jvm and node and other langs

.

01fd101

increase timeout

f8fd35e

trigger notices file gen to ensure BOM complete

c16d03e

first we take haskell then we take berlin

5fe11f1

ensure haskell processing complete before full scan

remove trailing end of lines

4ec58e0

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

run with latest detect version and unique code location name changes …

a45a9fe

…to wrapper script

Run latest version of detect

f60717d

dasormeter force-pushed the enable-blackduck branch from 4c2cc6e to f60717d Compare December 3, 2020 16:32

dasormeter added 3 commits December 3, 2020 11:49

Add blackduck to daily compat job

4adfe27

DO NOT MERGE: condition false to disable other jobs for testing

230004c

remove parameters not available to cronjob

0204735

garyverhaegen-da reviewed Dec 3, 2020

View reviewed changes

dasormeter changed the base branch from master to enable-blackduck-daily-scan December 3, 2020 18:58

dasormeter merged commit fbea7d5 into enable-blackduck-daily-scan Dec 3, 2020

dasormeter deleted the enable-blackduck branch December 3, 2020 18:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add blackduck scan to run on master #6130

add blackduck scan to run on master #6130

dasormeter commented May 27, 2020 •

edited

Loading

garyverhaegen-da commented May 27, 2020

dasormeter commented May 27, 2020

garyverhaegen-da commented May 27, 2020

dasormeter commented May 27, 2020

garyverhaegen-da Jun 18, 2020

dasormeter Sep 11, 2020

cocreature left a comment

garyverhaegen-da left a comment

garyverhaegen-da commented Jun 19, 2020 •

edited

Loading

dasormeter commented Jun 23, 2020

garyverhaegen-da Dec 3, 2020

add blackduck scan to run on master #6130

add blackduck scan to run on master #6130

Conversation

dasormeter commented May 27, 2020 • edited Loading

garyverhaegen-da commented May 27, 2020

dasormeter commented May 27, 2020

garyverhaegen-da commented May 27, 2020

dasormeter commented May 27, 2020

garyverhaegen-da Jun 18, 2020

Choose a reason for hiding this comment

dasormeter Sep 11, 2020

Choose a reason for hiding this comment

cocreature left a comment

Choose a reason for hiding this comment

garyverhaegen-da left a comment

Choose a reason for hiding this comment

garyverhaegen-da commented Jun 19, 2020 • edited Loading

dasormeter commented Jun 23, 2020

garyverhaegen-da Dec 3, 2020

Choose a reason for hiding this comment

dasormeter commented May 27, 2020 •

edited

Loading

garyverhaegen-da commented Jun 19, 2020 •

edited

Loading