Step not getting latest overwritten global artifact #1085
Description
Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
What happened:
A step creates a global artifact global-art with value "A". The next step is a nested workflow with a loop, where in each iteration, there is a step creates another artifact with globalName global-art with value "B", and another step reads the global-art. As "B" is written after "A", I expect to see the reader got "B", but it got "A".
(simplified graph)
✔ workflow
├---✔ A # write A to global-art
└-·-✔ loop(0:B)
| ├---✔ B # write B to global-art
| └---✔ read-B # expected to read "B" but got "A"
└-✔ loop(1:C)
├---✔ C # write C to global-art
└---✔ read-C # expected to read "C" but got "A"
Besides, in my use case, the loop is sequential (parallelism=1), so I can definitely say I expect to see read-B gets B and read-C gets C. But in other use cases where many parallel steps are writing to the same global artifact, I wonder how we define what we should expect to get. "The latest generated" global artifact is a reasonable one. Another possible option is "the most nearly generated" global artifact. Is there a defined behavior currently?
In the "latest generated" case, if the loop is parallel, read-B and read-C may read (B,B), (B,C), (C,C) [(C,B) is not possible]. while in the "most nearly generated" case, they always read (B,C).
What you expected to happen:
The steps inside nested workflows read the latest written global artifact.
How to reproduce it (as minimally and precisely as possible):
https://gist.github.com/WeiTang114/ed3813ea801180c3cf860a3caa42eac9
# This example shows that the consume-artifact steps in the loop will
# wrongly get "AAA" as global-art output, while it should get "BBB" and
# "CCC" which are generated by loop-echo later than "AAA" generated by
# generate-artifact-A.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: global-artifact-passing-
spec:
entrypoint: global-artifact-example
templates:
- name: global-artifact-example
parallelism: 1
steps:
- - name: generate-artifact-A
template: echo
arguments:
parameters:
- name: msg
value: "AAA"
- - name: loop
template: loop
arguments:
parameters:
- name: msg
value: "{{item}}"
withItems:
- "BBB"
- "CCC"
- name: loop
inputs:
parameters:
- name: msg
steps:
- - name: loop-echo
template: echo
arguments:
parameters:
- name: msg
value: "{{inputs.parameters.msg}}"
- - name: consume-artifact
template: print-message
arguments:
artifacts:
- name: my-art
from: "{{workflow.outputs.artifacts.global-art}}"
- name: echo
inputs:
parameters:
- name: msg
container:
image: alpine:latest
command: [sh, -c]
args: ["echo {{inputs.parameters.msg}} | tee /tmp/echo.txt"]
outputs:
artifacts:
- name: echo-art
path: /tmp
globalName: global-art
- name: print-message
inputs:
artifacts:
- name: my-art
path: /tmp
container:
image: alpine:latest
command: [sh, -c]
args: ["cat /tmp/echo.txt"]
Anything else we need to know?:
Environment:
- Argo version:
argo: v2.2.0
BuildDate: 2018-08-30T08:51:40Z
GitCommit: af636ddd8455660f307d835814d3112b90815dfd
GitTreeState: clean
GitTag: v2.2.0
GoVersion: go1.10.3
Compiler: gc
Platform: darwin/amd64
- Kubernetes version :
clientVersion:
buildDate: 2018-09-10T11:44:26Z
compiler: gc
gitCommit: a4529464e4629c21224b3d52edfe0ea91b072862
gitTreeState: clean
gitVersion: v1.11.3
goVersion: go1.11
major: "1"
minor: "11"
platform: darwin/amd64
serverVersion:
buildDate: 2018-03-26T16:44:10Z
compiler: gc
gitCommit: fc32d2f3698e36b93322a3465f63a14e9f0eaead
gitTreeState: clean
gitVersion: v1.10.0
goVersion: go1.9.3
major: "1"
minor: "10"
platform: linux/amd64
Other debugging information (if applicable):
- workflow result:
Name: global-artifact-passing-zc2rh
Namespace: default
ServiceAccount: default
Status: Succeeded
Created: Sun Nov 11 02:30:10 +0800 (37 seconds ago)
Started: Sun Nov 11 02:30:10 +0800 (37 seconds ago)
Finished: Sun Nov 11 02:30:38 +0800 (9 seconds ago)
Duration: 28 seconds
Output Artifacts:
global-art: http://storage.googleapis.com/aip-argo-artifacts/global-artifact-passing-zc2rh/global-artifact-passing-zc2rh-4273248531/echo-art.tgz
STEP PODNAME DURATION MESSAGE
✔ global-artifact-passing-zc2rh
├---✔ generate-artifact-A global-artifact-passing-zc2rh-3249287693 4s
└-·-✔ loop(0:BBB)
| ├---✔ loop-echo global-artifact-passing-zc2rh-2356867109 4s
| └---✔ consume-artifact global-artifact-passing-zc2rh-1779394067 5s
└-✔ loop(1:CCC)
├---✔ loop-echo global-artifact-passing-zc2rh-4273248531 4s
└---✔ consume-artifact global-artifact-passing-zc2rh-1867768273 5s
$ argo logs global-artifact-passing-zc2rh-1779394067
AAA
$ argo logs global-artifact-passing-zc2rh-1867768273
AAA