1.48.1 gives Credentials error. #6414

ryryfasch · 2019-07-19T21:24:02Z

This is a Bug Report

Description

What went wrong?
- Get this error now when deploying through an AWS pipeline:

Serverless Error --------------------------------------- 
  
Could not locate deployment bucket. Error: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: <http://bit.ly/aws-creds-setup>.

Our set up uses a serverless.yml file to deploy stacks. The pipeline were successfully building before the release of 1.48.1. I verified that this was an issue with serverless by installing 1.48.0 instead of the latest release using this command: npm install -g serverless@1.48.0. When I use 1.48.0 the builds pass as expected.

What did you expect should have happened?
- I expected the build to pass
What was the config you used?
What stacktrace or error message from your provider did you see?

Similar or dependent issues:

Error when using sls-invoke local with docker image #12345

Additional Data

Serverless Framework Version you're using: 1.48.1
Operating System:
Stack Trace:
Provider Error messages:

The text was updated successfully, but these errors were encountered:

dsciacca · 2019-07-19T21:30:17Z

I am having the same issue

lopezdp · 2019-07-19T22:32:25Z

What ever changes you merged onto master caused my CodePipeline to break also, as I have the same error. Im rolling back to 1.48 in my BuildSpec.yml for the time being but I would like to figure out what's going on here so I can update my environment when possible. Thanks!

ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: <http://bit.ly/aws-creds-setup>.

VinQbator · 2019-07-20T14:27:18Z

Same issue when trying to deploy in AWS CodePipeline.
Piece of log:

Serverless: Excluding development dependencies... 
Serverless: Excluding development dependencies... 
Serverless: Excluding development dependencies... 
Serverless: Excluding development dependencies... 
Serverless: Excluding development dependencies... 
Serverless: Excluding development dependencies... 
Serverless:  
User stats error: Request network error: { FetchError: network timeout at: https://tracking.serverlessteam.com/v1/track 
    at Timeout._onTimeout (/usr/local/lib/node_modules/serverless/node_modules/node-fetch/index.js:126:13) 
    at ontimeout (timers.js:436:11) 
    at tryOnTimeout (timers.js:300:5) 
    at listOnTimeout (timers.js:263:5) 
    at Timer.processTimers (timers.js:223:10) 
  name: 'FetchError', 
  message: 
   'network timeout at: https://tracking.serverlessteam.com/v1/track', 
  type: 'request-timeout' } 
Serverless: Invoke aws:package:finalize 
Serverless: Invoke aws:common:moveArtifactsToPackage 
Serverless: Invoke aws:common:validate 
Serverless: Invoke aws:deploy:deploy 
  
  Serverless Error --------------------------------------- 
  
  ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: <http://bit.ly/aws-creds-setup>. 
  
  Stack Trace -------------------------------------------- 
  
ServerlessError: ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: <http://bit.ly/aws-creds-setup>. 
    at AwsDeploy.BbPromise.bind.then.catch.e (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/lib/createStack.js:91:15) 
From previous event: 
    at AwsDeploy.createStack (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/lib/createStack.js:83:13) 
From previous event: 
    at Object.aws:deploy:deploy:createStack [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:99:67) 
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55) 
From previous event: 
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22) 
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:484:17) 
    at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:93:48) 
From previous event: 
    at Object.deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:89:30) 
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55) 
From previous event: 
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22) 
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:496:17) 
    at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:116:33) 
    at runCallback (timers.js:705:18) 
    at tryOnImmediate (timers.js:676:5) 
    at processImmediate (timers.js:658:5) 
    at process.topLevelDomainCallback (domain.js:126:23) 
From previous event: 
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:103:74) 
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless.js:52:28) 
    at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16 
    at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10 
    at FSReqWrap.args [as oncomplete] (fs.js:140:20) 
From previous event: 
    at initializeErrorReporter.then (/usr/local/lib/node_modules/serverless/bin/serverless.js:52:6) 
    at runCallback (timers.js:705:18) 
    at tryOnImmediate (timers.js:676:5) 
    at processImmediate (timers.js:658:5) 
    at process.topLevelDomainCallback (domain.js:126:23) 
From previous event: 
    at Object.<anonymous> (/usr/local/lib/node_modules/serverless/bin/serverless.js:38:39) 
    at Module._compile (internal/modules/cjs/loader.js:776:30) 
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 
    at Module.load (internal/modules/cjs/loader.js:653:32) 
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
    at Function.Module._load (internal/modules/cjs/loader.js:585:3) 
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12) 
    at startup (internal/bootstrap/node.js:283:19) 
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3) 
  
  Get Support -------------------------------------------- 
     Docs:          docs.serverless.com 
     Bugs:          github.com/serverless/serverless/issues 
     Issues:        forum.serverless.com 
  
  Your Environment Information --------------------------- 
     Operating System:          linux 
     Node Version:              10.16.0 
     Serverless Version:        1.48.2 
     Enterprise Plugin Version: 1.3.1 
     Platform SDK Version:      2.0.3 ```

serverless/serverless#6414

sha2nkprasad · 2019-07-22T09:46:01Z

Still facing the same issue with 1.48.1, had to restore back to 1.48.0 in BuildSpec.yml.

pmuens · 2019-07-22T10:21:36Z

Hey @ryryfasch, thanks for opening this issue and thanks everyone for chiming in.

Apparently this PR broke causes the problem: #6393

It seems like all of you use CodePipeline, so my hunch is that there's something going on with the credentials in the CI / CD environment. Can anyone shed some light into the way CodePipeline handles AWS credentials to carry out the deployments? With that information we can provide a fix ASAP. Thanks!

RaissaBergamini · 2019-07-22T12:02:31Z

Serverless Error ---------------------------------------

ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: http://bit.ly/aws-creds-setup.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: linux
Node Version: 10.16.0
Serverless Version: 1.48.2
Enterprise Plugin Version: 1.3.1
Platform SDK Version: 2.0.3

Had the same problem in codebuild stage in codepipeline. Serveless doesn't seem to be able to get the role associated with the codebuild instance.

Went back to 1.48.0 and everything is working.

rpgreen · 2019-07-22T12:34:48Z

+1 on AWS CodeBuild. Rolled back to 1.48.0 for now.

Serverless Error ---------------------------------------

ServerlessError: AWS provider credentials not found. Learn how to set up AWS provider credentials in our docs here: http://bit.ly/aws-creds-setup.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: linux
Node Version: 10.16.0
Serverless Version: 1.48.2
Enterprise Plugin Version: 1.3.1
Platform SDK Version: 2.0.3

rpgreen · 2019-07-22T12:43:39Z

@pmuens I believe CodeBuild sets environment variables for AWS credentials based on the configured pipeline role.

The variables in question should be:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

I'm guessing serverless isn't falling back to these environment variables anymore when looking for credentials.

dci-aloughran · 2019-07-22T12:48:19Z

It's also not falling back to the metadata API. That's the mech that I've used by default. I'll comment on the PR and see if I can work out what broke.

dci-aloughran · 2019-07-22T12:49:56Z

Ok, so PR #6393 implies that you'll have a default profile and some credentials set. That's not the case, as AWS falls back to metadata API and the IAM role of the machine executing. I guess that's the case with CodePipeline too (uses IAM attached to the job to set permissions).

dschep · 2019-07-22T13:40:41Z

Hmm. @dci-aloughran I attempted to check the creds in #6427 using AWS.EC2MetadataCredentials but to no avail 😖

dschep · 2019-07-22T13:43:41Z

Ahah, per https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html#troubleshooting-versions CodeBuild uses AWS.RemoteCredentials, gonna try that.

Ali-Dalal · 2019-07-23T03:57:36Z

Run into the same problem in codebuild.

I think we shouldn't force to use AWS keys in order to deploy serverless apps.

in codebuild, I have managed role which has the right access for deployment without using any AWS keys.

marcinkowalczyk · 2019-07-23T11:57:46Z

1.48.2 is broken too.

KowalskiTom · 2019-07-23T12:02:03Z

We are experiencing this issue on our Jenkins box as well which uses EC2 based IAM role permissions.

rpgreen · 2019-07-23T12:07:39Z

This issue will likely affect any EC2/ECS based environment which rely on the metadata service to access instance profile credentials.

curryfury-slalom · 2019-07-23T12:44:19Z

This is a pretty major bug that will block a lot of CD pipelines that we have setup. Is there a timeframe of when this bug will be addressed?

karlbanke · 2019-07-23T13:29:28Z

Encountered this issue today using 1.48.2. We can temporarily stick with 1.48.0 but this should be resolved as soon as possible.

Fix #6414 by checking for ec2 metadata

AnthonyWC · 2019-07-23T17:51:42Z

Also affecting Buildkite on 1.48.2

dschep · 2019-07-23T18:40:04Z

I've released 1.48.3 which should address this issue

dci-aloughran · 2019-07-23T18:58:02Z

Excellent work. Really appreciate the dedication of maintainers. Thank you.

nickolashkraus · 2019-07-23T18:59:16Z

1.48.3 working as expected! Thank you!

Edit: Running Jenkins with Amazon Elastic Container Service Plugin

wuelcas · 2019-07-23T19:35:22Z

1.48.3 Working now, thanks for the support!

curryfury-slalom · 2019-07-23T20:09:43Z

Thanks for the fix. Can confirm 1.48.3 is working in aws now! Moving forward I think we are going to lock it at this version till we test and validate.

dschep · 2019-07-23T20:15:41Z

@curryfury-slalom never a bad idea to pin dependencies ;)

ebaioni · 2019-07-24T00:43:29Z

Thank you, works on aws codebuild too

pmuens added bug cat/providers labels Jul 21, 2019

72636c added a commit to seek-oss/rynovate that referenced this issue Jul 22, 2019

Block npm:serverless@1.48

960e0e5

serverless/serverless#6414

pmuens mentioned this issue Jul 22, 2019

Could not locate deployment bucket. Error: AWS provider credentials not found. #6421

Closed

dschep added a commit that referenced this issue Jul 22, 2019

Fix #6414 by checking for ecs metadata

5464c5c

dschep added a commit that referenced this issue Jul 22, 2019

Fix #6414 by checking for ecs metadata

5148176

dschep added a commit that referenced this issue Jul 22, 2019

Fix #6414 by checking for ecs metadata

3261163

dschep mentioned this issue Jul 22, 2019

Fix #6414 by checking for ec2 metadata #6427

Merged

8 tasks

dschep added a commit that referenced this issue Jul 22, 2019

Fix #6414 by checking for ecs metadata

adb66ce

dschep closed this as completed in #6427 Jul 23, 2019

dschep added a commit that referenced this issue Jul 23, 2019

Fix #6414 by checking for ec2 metadata (#6427)

15ff4cb

Fix #6414 by checking for ec2 metadata

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.48.1 gives Credentials error. #6414

1.48.1 gives Credentials error. #6414

ryryfasch commented Jul 19, 2019

dsciacca commented Jul 19, 2019

lopezdp commented Jul 19, 2019

VinQbator commented Jul 20, 2019

sha2nkprasad commented Jul 22, 2019

pmuens commented Jul 22, 2019

RaissaBergamini commented Jul 22, 2019

rpgreen commented Jul 22, 2019

rpgreen commented Jul 22, 2019

dci-aloughran commented Jul 22, 2019

dci-aloughran commented Jul 22, 2019

dschep commented Jul 22, 2019

dschep commented Jul 22, 2019

Ali-Dalal commented Jul 23, 2019

marcinkowalczyk commented Jul 23, 2019

KowalskiTom commented Jul 23, 2019

rpgreen commented Jul 23, 2019

curryfury-slalom commented Jul 23, 2019

karlbanke commented Jul 23, 2019

AnthonyWC commented Jul 23, 2019

dschep commented Jul 23, 2019

dci-aloughran commented Jul 23, 2019

nickolashkraus commented Jul 23, 2019 •

edited

Loading

wuelcas commented Jul 23, 2019

curryfury-slalom commented Jul 23, 2019

dschep commented Jul 23, 2019

ebaioni commented Jul 24, 2019

1.48.1 gives Credentials error. #6414

1.48.1 gives Credentials error. #6414

Comments

ryryfasch commented Jul 19, 2019

This is a Bug Report

Description

Additional Data

dsciacca commented Jul 19, 2019

lopezdp commented Jul 19, 2019

VinQbator commented Jul 20, 2019

sha2nkprasad commented Jul 22, 2019

pmuens commented Jul 22, 2019

RaissaBergamini commented Jul 22, 2019

rpgreen commented Jul 22, 2019

rpgreen commented Jul 22, 2019

dci-aloughran commented Jul 22, 2019

dci-aloughran commented Jul 22, 2019

dschep commented Jul 22, 2019

dschep commented Jul 22, 2019

Ali-Dalal commented Jul 23, 2019

marcinkowalczyk commented Jul 23, 2019

KowalskiTom commented Jul 23, 2019

rpgreen commented Jul 23, 2019

curryfury-slalom commented Jul 23, 2019

karlbanke commented Jul 23, 2019

AnthonyWC commented Jul 23, 2019

dschep commented Jul 23, 2019

dci-aloughran commented Jul 23, 2019

nickolashkraus commented Jul 23, 2019 • edited Loading

wuelcas commented Jul 23, 2019

curryfury-slalom commented Jul 23, 2019

dschep commented Jul 23, 2019

ebaioni commented Jul 24, 2019

nickolashkraus commented Jul 23, 2019 •

edited

Loading