The Problem

1. Expected behavior for aws_profile is that it would use the same credential discovery that the AWS CLI does (or any AWS SDK).
2. In places where the aws_profile is not available (in CI or a production server, for instance), the config file is not decryptable without removing the aws_profile first.

Most of the aws_profile related github issues in this repo revolve around expected behavior vs observed behavior - most of which would be solved by following the same behavior that AWS enforces for their own tooling.

Proposal

Introduce the correct priority ordering for credential discovery. Instead of forcing the aws_profile to be the only method of authentication, we should use the value of aws_profile to override the AWS_PROFILE environment variable. This would allow the normal credential discovery built into the aws sdk to run its proper course.

In the case where the aws_profile is not available in the Credentials file or Config file, or those files do not exist at all, the aws_profile should be ignored entirely to allow those files to still be decrypted.

I am happy to work on this feature if permitted and needed.

Explanation

AWS Documentation on Credential Discovery

When using the AWS CLI, there is a priority to which credentials are chosen (I've listed them all here with the relevant ones retaining their descriptions):

1. Command line options – Overrides settings in any other location, such as the --region, --output, and --profile parameters.
2. Environment variables – You can store values in your system's environment variables.
3. Assume role - ...
4. Assume role with web identity - ...
5. AWS IAM Identity Center - ...
6. Credentials file – The credentials and config file are updated when you run the command aws configure. The credentials file is located at ~/.aws/credentials on Linux or macOS, or at C:\Users\USERNAME.aws\credentials on Windows.
7. Custom process - ...
8. Configuration file – The credentials and config file are updated when you run the command aws configure. The config file is located at ~/.aws/config on Linux or macOS, or at C:\Users\USERNAME.aws\config on Windows.
9. Container credentials - ...
10. Amazon EC2 instance profile credentials - ...
11. IAM Roles for Amazon EC2 - ...

From the AWS Documentation on CLI Credential configuration using environment variables:

AWS_PROFILE
Specifies the name of the AWS CLI profile with the credentials and options to use. This can be the name of a profile stored in a credentials or config file, or the value default to use the default profile.

If defined, this environment variable overrides the behavior of using the profile named [default] in the configuration file. You can override this environment variable by using the --profile command line parameter.

Stated extremely clearly on this documentation page:

If AWS_PROFILE environment variable is set and the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, then the credentials provided by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override the credentials located in the profile provided by AWS_PROFILE.

Sops aws_profile Functionality

When a file is encrypted with sops, you have the option to specify --aws-profile, and that profile is written into the config file as a sort of default profile when decrypting. The most obvious use case of the embedded aws_profile appears to be that it allows the use of multiple AWS accounts in a repo without requiring changing environment variables just to edit encrypted files that are encrypted with keys from different accounts. This is a very nice boon for setups where, for instance, non-prod and prod are in separate accounts. Very convenient!

In sops, when the aws_profile field is set up originally, it is used as a command line argument through --aws-profile. Command line arguments are priority 1, so it should override all other settings. This is done correctly by the AWS docs.

When decrypting, it is not a command line argument anymore. It's an implicit override for the AWS_PROFILE environment variable. Therefore, if AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables happen to be available, they should take precedence over it. This can (and has been) observed to not be the case, as sops forces the aws_profile to be used regardless of any other available credentials - even if the profile is not found at all and there are other valid credentials available.

See also

This extremely similar issue in the serverless repo, illustrating this issue of parity with the AWS tooling: serverless/serverless#9290