Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(artifacts) Implement S3 artifacts #2468

Merged
merged 9 commits into from
Apr 2, 2018

Conversation

benjaminws
Copy link
Contributor

Implement S3 artifacts in clouddriver to support spinnaker/spinnaker#2595

Looking for feedback. The logic for determining if the artifact reference is a valid bucket/path was lifted directly from the GCS implementation.

@benjaminws benjaminws changed the title Implement S3 artifacts feat(artifacts) Implement S3 artifacts Mar 29, 2018
Copy link
Member

@lwander lwander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, one AWS creds question

@Data
public class S3ArtifactAccount extends ArtifactAccount
{
private String name = "s3-account";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want a default name here.

Also -- should this list a role to assume? To allow for multiple accounts w/ different creds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great question. I was hoping for input on that. I think that would be useful, then we could tweak halyard to take the roles in the cli setup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, following up here. I want to make sure I understand before implementing it. So the behavior of the code as is will be to use the default credential provider, resolving whatever it can from the instance, the env, or configuration, etc. You're saying we could add a list of roles to assume, adding those as potential credentials to use?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar enough with AWS creds to give a firm answer, but I know that AWS accounts (even on running on GCE) can assume control over multiple other accounts using accountId and assumeRule like done for the AWS provider. https://github.com/spinnaker/halyard/blob/master/halyard-config/src/main/java/com/netflix/spinnaker/halyard/config/model/v1/providers/aws/AwsAccount.java#L36

Maybe we can implement this as is, and then defer to @imosquera to implement the multi-account logic?

@@ -0,0 +1,28 @@
/*
* Copyright 2018 Netflix, Inc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I would make these "Datadog"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 going to adjust. silly gradle plugins.

@benjaminws
Copy link
Contributor Author

I could use an assist.

Using the code in this PR I can't seem to make AmazonS3ClientBuilder authenticate using an instance profile via the .defaultClient() factory method.

Given the behavior of this method is to try various other auth mechanisms, I've ensured there are none available in the environment or otherwise for the user running clouddriver. I've also ensured that all the necessary IAM bits are in place, and I can access objects in the requested bucket using aws cli from a host with the given instance profile.

Additionally, I've also attempted to use the .getInstance() factory method explicitly. but had the same result. Each time I'm given a 403 for an object I can download with aws cli otherwise.

Thoughts?

@benjaminws
Copy link
Contributor Author

I got this working by moving the AmazonS3ClientBuilder to the download method and calling it inline, instead of creating an object. Not 100% sure why that worked.

@lwander
Copy link
Member

lwander commented Apr 2, 2018

Looks good! Can you file an issue that we need to support more than just default credentials?

@lwander lwander merged commit 0a8d99e into spinnaker:master Apr 2, 2018
@benjaminws benjaminws deleted the benjaminws/s3-artifacts branch April 3, 2018 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants