[BUG] cfn-guard validate JSON response is not standardised across CFN data and non-CFN data schemas #350
Description
Describe the bug
When running cfn-guard validate with -o json parameter, depending on the type of data payload used (e.g: CloudFormation template and AWS Config), the JSON response has a completely different structure.
To Reproduce: CloudFormation Validation
Rule (file: s3-cfn-1.guard):
let s3_buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]
let allowed_algos = ["aws:kms"]
rule s3_buckets_allowed_sse_algorithm when %s3_buckets !empty {
let encryption = %s3_buckets.Properties.BucketEncryption
%encryption exists
%encryption.ServerSideEncryptionConfiguration[*].ServerSideEncryptionByDefault.SSEAlgorithm in %allowed_algos
}
rule s3_buckets_allowed_name when %s3_buckets !empty {
%s3_buckets.Properties.BucketName != "my-bucket"
}
CloudFormation template (file: s3-cfn.yaml)
AWSTemplateFormatVersion: '2010-09-09'
Resources:
s3Valid:
Type: "AWS::S3::Bucket"
Properties:
BucketName: my-bucket
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: 'aws:kms'
KMSMasterKeyID: 'arn:aws:kms:us-east-1:123456789:key/056ea50b-1013-3907-8617-c93e474e400'
Command:
cfn-guard validate -r s3-cfn-1.guard -d s3-cfn.yaml -o json
Output:
s3-cfn.yaml Status = FAIL
FAILED rules
s3-cfn-1.guard/s3_buckets_allowed_name FAIL
---
{
"name": "",
"metadata": {},
"status": "FAIL",
"not_compliant": [
{
"Rule": {
"name": "s3_buckets_allowed_name",
"metadata": {},
"messages": {
"custom_message": null,
"error_message": null
},
"checks": [
{
"Clause": {
"Binary": {
"context": " %s3_buckets[*].Properties.BucketName not EQUALS \"my-bucket\"",
"messages": {
"custom_message": "",
"error_message": "Check was not compliant as property value [Path=/Resources/s3Valid/Properties/BucketName[L:5,C:18] Value=\"my-bucket\"] equal to value [Path=[L:0,C:0] Value=\"my-bucket\"]."
},
"check": {
"Resolved": {
"from": {
"path": "/Resources/s3Valid/Properties/BucketName",
"value": "my-bucket"
},
"to": {
"path": "",
"value": "my-bucket"
},
"comparison": [
"Eq",
true
]
}
}
}
}
}
]
}
}
],
"not_applicable": [],
"compliant": [
"s3_buckets_allowed_sse_algorithm"
]
}
To Reproduce: Config Validation
Rule (file: s3-config.guard)
let allowed_algos = ["aws:kms"]
rule s3_buckets_allowed_sse_algorithm when resourceType == "AWS::S3::Bucket" {
supplementaryConfiguration.ServerSideEncryptionConfiguration.rules[*].applyServerSideEncryptionByDefault.sseAlgorithm in %allowed_algos
supplementaryConfiguration.ServerSideEncryptionConfiguration.rules[*].bucketKeyEnabled == true
}
rule s3_buckets_allowed_name when resourceType == "AWS::S3::Bucket" {
configuration.name != "my-bucket"
}
Config Data Payload (file: s3-config.json)
{
"resourceType": "AWS::S3::Bucket",
"configuration": {
"name": "my-bucket"
},
"supplementaryConfiguration": {
"ServerSideEncryptionConfiguration": {
"rules": [
{
"applyServerSideEncryptionByDefault": {
"sseAlgorithm": "aws:kms",
"KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789:key/056ea50b-1013-3907-8617-c93e474e400"
},
"bucketKeyEnabled": true
}
]
}
}
}
Command:
cfn-guard validate -r s3-config.guard -d s3-config.json -o json
Output:
{
"data_from": "s3-config.json",
"rules_from": "s3-config.guard",
"not_compliant": {
"s3_buckets_allowed_name": [
{
"rule": "s3_buckets_allowed_name",
"path": "/configuration/name",
"provided": "my-bucket",
"expected": "my-bucket",
"comparison": {
"operator": "Eq",
"not_operator_exists": true
},
"message": "",
"error": null
}
]
},
"not_applicable": [],
"compliant": [
"s3_buckets_allowed_sse_algorithm"
]
}
Expected behavior
- I would expect a consistent response JSON structure irrespectively if I'm validating data files using CloudFormation or AWS Config Schema.
Please note that when validating Config file, there is a data_from attribute in the JSON response which is extremely useful to have it in the JSON payload, while when validating a CloudFormation file, the data file name comes outside of the JSON structure.
-
Also, when there is a non-compliant rule, the response structure is different as per the two outputs above.
-
It seems that the debug info is not correct when evaluating != for CFN as per below:
For CFN evaluation, it returns a comparison as per below. It's not clear what "true" is referring to. When I first saw I thought it was the comparison result. However I believe it refers to the "Not operator" (equivalent of not_operator_exists for the Config evaluation)
"comparison": [
"Eq",
true
]
While when evaluating Config payload, it returns
"comparison": {
"operator": "Eq",
"not_operator_exists": true
}
Operating System:
Amazon Linux 2
OS Version
Amazon Linux 2
Guard Version
2.1.3