Skip to content

[cdk-pipelines] Infinite-loop in self-mutating pipeline #32008

Open
@BwL1289

Description

Describe the bug

The pipeline's 'UpdatePipeline' stage succeeds and the pipeline restarts, which it is expected to do once when infrastructure is updated. However, when it restarts, it updates itself again. This loops infinitely, and the pipeline never reaches the Assets stage. It appears to happen on V2, with SUPERSEDED mode, and restart_execution_on_update=True.

I could not reproduce it on V2 with PARALLEL mode and restart_execution_on_update=False.

This appears to be caused by the docker asset hashes changing while the rest of the template stays the same. Synth is not introducing nondeterminism and the dockerfiles and directories are exactly the same between runs, but the hashes keep changing.

I initially reported this here and here.

This may be a regression of #9766 and the fix here.

#9080 is likely also related.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The pipeline to self mutate once and then continue to Assets stage at the next cycle

Current Behavior

The pipeline continuously self mutates in an infinite loop.

Reproduction Steps

On codepipeline V2, use restart_execution_on_update=True in SUPERSEDED mode.

Possible Solution

No response

Additional Information/Context

I reverted to PARALLEL mode from SUPERSEDED and restart_execution_on_update=False.

It's now able to progress to Assets stage. I'm on version 2.164.1.

For context:

  1. For months I was using SUPERSEDED mode with restart_execution_on_update=True on codepipeline V1.
  2. I recently migrated to V2, using PARALLEL mode and restart_execution_on_update=False.
    1. This worked
  3. Then I switched to SUPERSEDED mode with restart_execution_on_update=False.
    1. This did not work as the build would get cancelled (expected) after self-mutate, and I'd have to restart it manually, and then it would get cancelled again (unexpected)
  4. Then I switched to SUPERSEDED mode with restart_execution_on_update=True.
    1. This did not work as it would enter an endless loop between synth and self-mutate
  5. Then I switched back to PARALLEL mode and restart_execution_on_update=False.
    1. This worked, again

I would like to switch back to SUPERSEDED mode with restart_execution_on_update=True as SUPERSEDED mode supports building 50 docker assets in parallel while PARALLEL only supports 5, and I'd like to not worry about the pipeline restarting after infra changes.

CDK CLI Version

2.164.1

Framework Version

No response

Node.js Version

v20.15.1

OS

MacOS

Language

Python

Language Version

No response

Other information

No response

Metadata

Assignees

Labels

@aws-cdk/pipelinesCDK Pipelines librarybugThis issue is a bug.investigatingThis issue is being investigated and/or work is in progress to resolve the issue.p3potential-regressionMarking this issue as a potential regression to be checked by team member

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions