Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Artifact registry support #65

Closed
wants to merge 8 commits into from
Closed

Artifact registry support #65

wants to merge 8 commits into from

Conversation

SteveLasker
Copy link
Contributor

Additions to enhance distribution for supporting additional artifact types, such as Helm Charts, Singularity Images for High Performance Computing workloads, Open Policy Agent Bundles

SteveLasker and others added 7 commits March 12, 2019 12:35
@SteveLasker
Add Slack info for discussions
2e73cd3 
Signed-off-by: Steve Lasker <StevenLasker@Hotmail.com>
@SteveLasker
Add Slack info for discussions
8e2b263 
Signed-off-by: Steve Lasker <StevenLasker@Hotmail.com>
Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
@SteveLasker
Merge branch 'Contributing' of https://github.com/AzureCR/distributio…
ae645f2 
…n-spec into Contributing

Signed-off-by: Steve Lasker <stevenlasker@hotmail.com>
@SteveLasker
First draft of OCI Artifact support
39a804b
@SteveLasker
Details on artifact mediaTypes
4b2dbc6
@SteveLasker
Breakup of artifact defs: fb from Joey Schorr
08d1f8e
@SteveLasker
OCI Artifact Definition
cafc292
@jzelinskie
Copy link
Member

I'll reiterate what I've said in every meeting on this topic so far: even though I strongly believe in this use case, I'm not sure that a specification for artifact registries should live in the Open Containers Initiative. However, in the meantime this still might be a good place to work on the idea.

I'm also still not convinced that we've come to a consensus on a cohesive strategy for reusing the existing distribution spec in a way that satisfies everyone. From our previous discussion, it seemed like we still require alignment from Docker's side of CNAB to have confidence in a model that everyone can properly leverage.

glyn
glyn reviewed Jun 19, 2019
View reviewed changes
artifacts.md Outdated Show resolved Hide resolved
glyn
glyn reviewed Jun 19, 2019
View reviewed changes
spec.md

## OCI Artifact Extensibility

The distribution spec was initially based on docker images and the [OCI Image spec](https://github.com/opencontainers/image-spec/). As new artifact types surfaced, it became a logical use case to leverage the common elements of distribution to store and retrieve additional artifacts. This spec will reference *images* as one type of artifact, with specifics on how to differentiate different types of artifacts.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to explain why it is desirable to store non-image artefacts in a registry. Does this only really make sense for artefacts which reference images?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a spec, I struggled with how much "pitch" to put into the spec, as opposed to speaking to the details. I've written a few different posts, done presentations to set the overall context:

Do you think we should put more of the pitch here? Are you looking for what type of artifacts to store?

Copy link

@glyn glyn Jun 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a spec, I struggled with how much "pitch" to put into the spec, as opposed to speaking to the details. I've written a few different posts, done presentations to set the overall context:

Thanks but I don't think it would be appropriate to reference these in the spec.

Do you think we should put more of the pitch here?

Yes, but not framed as a sales pitch. I'm looking for rationale of when it makes sense to use this feature.

Are you looking for what type of artifacts to store?

Yes. In particular, does it make any sense to store an artefact that does not reference images?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree we shouldn't reference these within the spec. I was more providing context on how I was framing things, outside the spec. To provide a funnel of the scenarios, to the specifics.

As for when and what artifact types, I don't know if we should attempt to limit. The question I ask in the above Cloud Native Rejekts talk is; why would you create Yet Another Storage Solution (YASS).

Registries have become ubiquitous. It's hard to see modern workloads that don't use containers. Even IoT devices and cars are running containers because of their packaging, isolation, patching, testing and deployment models. But, lets assume you have an optimized environment that doesn't require container images. Could you justify the engineering effort to build YASS? Or, would you just implement one of the current distribution solutions, and store the artifact types in it?

@jdolitsky
Copy link
Member

Can somebody from Docker/DockerHub please weigh in on this approach and any potential shortcomings? Quay? ECR? GCR?

Would love to see the hubs come together on a standard approach for mulitartifact support, whether it is how Steve has proposed, or some alternation.

Here is probably the most significant piece, which is on how registries reason over artifact types on the manifest level:

Artifacts are defined by setting the manifest.config.mediaType to a globally unique value. The following format is used to differentiate the type of artifact:

application/vnd.[org|company].[objectType].[optionalSubType].config.[version]+json

The rest of the changes are about new file artifactMapping.json, which is just a format projects can use to establish their custom artifact type (that registries can then support "officially").

@jdolitsky jdolitsky mentioned this pull request Jul 8, 2019
Closed
@dmcgowan
Copy link
Member

I'll sum up my recommendation from the OCI call earlier

  • Remove specific mention of artifact types. Propose a new repository under opencontainers Github org to the OCI TOB which defines the artifact use case and known examples of artifacts used by OCI manifests. The first example of this is the OCI image with clarification about the history which lead the OCI image type and the OCI manifest (a generic envelope type) to live within the same specification.
  • Remove wording within the distribution specification which mentions the distribution spec as used for distributing images. We can use the word artifact there as this PR does, just don't want that word overused. I have a slight preference for using the word manifest/manifest-index generically in this spec to refer to the objects stored by registries, but I understand that may be confusing.
  • Replace the parts of this PR which update the spec to discuss artifacts with links to the above artifact repository once it is created. That repository can be a place for these discussions while allowing the distribution spec to stay generic and versioned separately from the types that use it.

@caniszczyk
Copy link
Contributor

re: new project for @opencontainers/tob, have you decided what you will call this and who will put forth the proposal? happy to work with them

@mikebrow
Copy link
Member

I'll sum up my recommendation from the OCI call earlier

  • Remove specific mention of artifact types. Propose a new repository under opencontainers Github org to the OCI TOB which defines the artifact use case and known examples of artifacts used by OCI manifests. The first example of this is the OCI image with clarification about the history which lead the OCI image type and the OCI manifest (a generic envelope type) to live within the same specification.
  • Remove wording within the distribution specification which mentions the distribution spec as used for distributing images. We can use the word artifact there as this PR does, just don't want that word overused. I have a slight preference for using the word manifest/manifest-index generically in this spec to refer to the objects stored by registries, but I understand that may be confusing.
  • Replace the parts of this PR which update the spec to discuss artifacts with links to the above artifact repository once it is created. That repository can be a place for these discussions while allowing the distribution spec to stay generic and versioned separately from the types that use it.

SGTM

@SteveLasker
Copy link
Contributor Author

SteveLasker commented Jul 12, 2019 via email •
edited
Loading

@SteveLasker SteveLasker mentioned this pull request Jul 17, 2019
Closed
@SteveLasker
Copy link
Contributor Author

I've create the new TOB issue here: opencontainers/tob#59
If we can agree to the repo, I can start on the content.

@SteveLasker SteveLasker mentioned this pull request Jul 25, 2019
Merged
@vbatts
Copy link
Member

vbatts commented Dec 16, 2019

I think this effort has sufficiently become its own thing. Any references can be done in a separate PR now.

@vbatts vbatts closed this Dec 16, 2019
@SteveLasker SteveLasker mentioned this pull request Jul 13, 2023
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@glyn glyn glyn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@SteveLasker @jzelinskie @jdolitsky @dmcgowan @caniszczyk @mikebrow @vbatts @glyn