Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Acceptance test config files proof of concept (mq) #36719

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

YakDriver
Copy link
Member

@YakDriver YakDriver commented Apr 3, 2024

Description

TL;DR

"github.com/hashicorp/terraform-plugin-testing/helper/resource" supports acceptance testing using configuration files rather than strings. This PR takes an entire service, MQ, and converts all acceptance tests to use configuration files and shows what using config files entails. The tests run perfectly fine like this with a small hiccup (provider blocks).

Tests are only as good as their ability to reflect real-world usage.

Advantages of Config Files:

  1. Config files align with the natural workflow of Terraform users, providing an intuitive environment for configuration management.
  2. Creating, parsing, and maintaining test configurations through config files is straightforward and does not necessitate familiarity with Go programming.
  3. Test config files serve as comprehensive and easily understandable examples, enriching the documentation and aiding in onboarding new users.
  4. Utilizing Terraform modules within config files promotes modularity without the complexity of integrating modular Go code.

Challenges with Config Files:

  1. The sheer volume of tests poses a challenge in transitioning to a config file-based approach.
  2. Migrating existing configurations to config files requires careful planning and execution due to the non-trivial nature of the process.
  3. While currently, we leverage Go functions to achieve modularity and reduce redundancy, transitioning to config files necessitates a shift towards utilizing Terraform modules, which introduces a learning curve and potential reengineering effort.

Relations

Closes #0000

References

Output from Acceptance Testing

% make t T=TestAccMQConfig K=mq
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.21.8 test ./internal/service/mq/... -v -count 1 -parallel 20 -run='TestAccMQConfig'  -timeout 360m
=== RUN   TestAccMQConfiguration_basic
=== PAUSE TestAccMQConfiguration_basic
=== RUN   TestAccMQConfiguration_activeMQData
=== PAUSE TestAccMQConfiguration_activeMQData
=== RUN   TestAccMQConfiguration_activeMQLDAPData
=== PAUSE TestAccMQConfiguration_activeMQLDAPData
=== RUN   TestAccMQConfiguration_rabbitMQData
=== PAUSE TestAccMQConfiguration_rabbitMQData
=== RUN   TestAccMQConfiguration_tags
=== PAUSE TestAccMQConfiguration_tags
=== CONT  TestAccMQConfiguration_basic
=== CONT  TestAccMQConfiguration_rabbitMQData
=== CONT  TestAccMQConfiguration_activeMQLDAPData
=== CONT  TestAccMQConfiguration_tags
=== CONT  TestAccMQConfiguration_activeMQData
--- PASS: TestAccMQConfiguration_rabbitMQData (24.80s)
--- PASS: TestAccMQConfiguration_activeMQData (24.81s)
--- PASS: TestAccMQConfiguration_activeMQLDAPData (24.81s)
--- PASS: TestAccMQConfiguration_basic (36.27s)
--- PASS: TestAccMQConfiguration_tags (45.20s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/mq	47.509s
% make t T=TestAccMQBrokerEng K=mq
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.21.8 test ./internal/service/mq/... -v -count 1 -parallel 20 -run='TestAccMQBrokerEng'  -timeout 360m
=== RUN   TestAccMQBrokerEngineTypesDataSource_basic
=== PAUSE TestAccMQBrokerEngineTypesDataSource_basic
=== CONT  TestAccMQBrokerEngineTypesDataSource_basic
--- PASS: TestAccMQBrokerEngineTypesDataSource_basic (10.22s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/mq	12.399s
% make t T=TestAccMQBrokerInst K=mq
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.21.8 test ./internal/service/mq/... -v -count 1 -parallel 20 -run='TestAccMQBrokerInst'  -timeout 360m
=== RUN   TestAccMQBrokerInstanceTypeOfferingsDataSource_basic
=== PAUSE TestAccMQBrokerInstanceTypeOfferingsDataSource_basic
=== CONT  TestAccMQBrokerInstanceTypeOfferingsDataSource_basic
--- PASS: TestAccMQBrokerInstanceTypeOfferingsDataSource_basic (12.83s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/mq	14.980s
% make t T=TestAccMQBroker_ K=mq P=10
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.21.8 test ./internal/service/mq/... -v -count 1 -parallel 10 -run='TestAccMQBroker_'  -timeout 360m
=== RUN   TestAccMQBroker_basic
=== PAUSE TestAccMQBroker_basic
=== RUN   TestAccMQBroker_disappears
=== PAUSE TestAccMQBroker_disappears
=== RUN   TestAccMQBroker_tags
=== PAUSE TestAccMQBroker_tags
=== RUN   TestAccMQBroker_throughputOptimized
=== PAUSE TestAccMQBroker_throughputOptimized
=== RUN   TestAccMQBroker_AllFields_defaultVPC
=== PAUSE TestAccMQBroker_AllFields_defaultVPC
=== RUN   TestAccMQBroker_AllFields_customVPC
=== PAUSE TestAccMQBroker_AllFields_customVPC
=== RUN   TestAccMQBroker_EncryptionOptions_kmsKeyID
=== PAUSE TestAccMQBroker_EncryptionOptions_kmsKeyID
=== RUN   TestAccMQBroker_EncryptionOptions_managedKeyDisabled
=== PAUSE TestAccMQBroker_EncryptionOptions_managedKeyDisabled
=== RUN   TestAccMQBroker_EncryptionOptions_managedKeyEnabled
=== PAUSE TestAccMQBroker_EncryptionOptions_managedKeyEnabled
=== RUN   TestAccMQBroker_Update_users
=== PAUSE TestAccMQBroker_Update_users
=== RUN   TestAccMQBroker_Update_securityGroup
=== PAUSE TestAccMQBroker_Update_securityGroup
=== RUN   TestAccMQBroker_Update_engineVersion
=== PAUSE TestAccMQBroker_Update_engineVersion
=== RUN   TestAccMQBroker_Update_hostInstanceType
=== PAUSE TestAccMQBroker_Update_hostInstanceType
=== RUN   TestAccMQBroker_RabbitMQ_basic
=== PAUSE TestAccMQBroker_RabbitMQ_basic
=== RUN   TestAccMQBroker_RabbitMQ_config
=== PAUSE TestAccMQBroker_RabbitMQ_config
=== RUN   TestAccMQBroker_RabbitMQ_logs
=== PAUSE TestAccMQBroker_RabbitMQ_logs
=== RUN   TestAccMQBroker_RabbitMQ_validationAuditLog
=== PAUSE TestAccMQBroker_RabbitMQ_validationAuditLog
=== RUN   TestAccMQBroker_RabbitMQ_cluster
=== PAUSE TestAccMQBroker_RabbitMQ_cluster
=== RUN   TestAccMQBroker_ldap
=== PAUSE TestAccMQBroker_ldap
=== RUN   TestAccMQBroker_dataReplicationMode
=== PAUSE TestAccMQBroker_dataReplicationMode
=== CONT  TestAccMQBroker_basic
=== CONT  TestAccMQBroker_Update_securityGroup
=== CONT  TestAccMQBroker_AllFields_customVPC
=== CONT  TestAccMQBroker_RabbitMQ_logs
=== CONT  TestAccMQBroker_EncryptionOptions_managedKeyEnabled
=== CONT  TestAccMQBroker_throughputOptimized
=== CONT  TestAccMQBroker_AllFields_defaultVPC
=== CONT  TestAccMQBroker_RabbitMQ_basic
=== CONT  TestAccMQBroker_Update_hostInstanceType
=== CONT  TestAccMQBroker_tags
--- PASS: TestAccMQBroker_RabbitMQ_basic (694.39s)
=== CONT  TestAccMQBroker_RabbitMQ_config
--- PASS: TestAccMQBroker_RabbitMQ_config (920.67s)
=== CONT  TestAccMQBroker_Update_users
--- PASS: TestAccMQBroker_RabbitMQ_logs (713.71s)
=== CONT  TestAccMQBroker_disappears
--- PASS: TestAccMQBroker_throughputOptimized (834.44s)
=== CONT  TestAccMQBroker_ldap
--- PASS: TestAccMQBroker_basic (857.06s)
=== CONT  TestAccMQBroker_dataReplicationMode
    broker_test.go:1852: Test validation error: TestStep 1/3 validation error: Providers must only be specified either at the TestCase or TestStep level
--- FAIL: TestAccMQBroker_dataReplicationMode (0.08s)
=== CONT  TestAccMQBroker_EncryptionOptions_managedKeyDisabled
--- PASS: TestAccMQBroker_EncryptionOptions_managedKeyEnabled (929.08s)
=== CONT  TestAccMQBroker_RabbitMQ_cluster
--- PASS: TestAccMQBroker_tags (940.33s)
=== CONT  TestAccMQBroker_Update_engineVersion
--- PASS: TestAccMQBroker_Update_securityGroup (1142.41s)
=== CONT  TestAccMQBroker_RabbitMQ_validationAuditLog
--- PASS: TestAccMQBroker_Update_hostInstanceType (1289.17s)
=== CONT  TestAccMQBroker_EncryptionOptions_kmsKeyID
--- PASS: TestAccMQBroker_disappears (802.80s)
--- PASS: TestAccMQBroker_AllFields_defaultVPC (1541.51s)
--- PASS: TestAccMQBroker_AllFields_customVPC (1610.90s)
--- PASS: TestAccMQBroker_EncryptionOptions_managedKeyDisabled (846.65s)
--- PASS: TestAccMQBroker_ldap (873.34s)
--- PASS: TestAccMQBroker_RabbitMQ_cluster (824.53s)
--- PASS: TestAccMQBroker_RabbitMQ_validationAuditLog (699.74s)
--- PASS: TestAccMQBroker_Update_users (1317.69s)
--- PASS: TestAccMQBroker_Update_engineVersion (1116.19s)
--- PASS: TestAccMQBroker_EncryptionOptions_kmsKeyID (833.88s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/mq	2125.425s

Copy link

github-actions bot commented Apr 3, 2024

Community Note

Voting for Prioritization

  • Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

  • Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
  • For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
  • Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

@github-actions github-actions bot added size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/mq Issues and PRs that pertain to the mq service. labels Apr 3, 2024
@terraform-aws-provider terraform-aws-provider bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Apr 3, 2024
Copy link

github-actions bot commented Apr 3, 2024

Thank you for your contribution! 🚀

A new usage of AWS SDK for Go V1 was detected. Please prefer AWS SDK for Go V2 for all net-new services. If this is an enhancement or bug fix to an existing AWS SDK Go V1 based resource, this comment can be safely ignored.

For additional information refer to the AWS SDK for Go Versions page in the contributor guide.

@YakDriver YakDriver changed the title acctest: Acctest files proof of concept (mq) Acceptance test files proof of concept (mq) Apr 3, 2024
@YakDriver YakDriver changed the title Acceptance test files proof of concept (mq) Acceptance test config files proof of concept (mq) Apr 3, 2024
Copy link
Contributor

@gdavison gdavison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the general direction of using Terraform configurations. I have a couple thoughts

@@ -0,0 +1,120 @@
variable "exclude_zone_ids" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit, the standard name for this file is variables.tf

Comment on lines +234 to +235
//Config: testAccConfigurationConfig_tags1(rName, "key1", "value1"),
ConfigDirectory: config.TestStepDirectory(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In cases like this, where a particular configuration is reused, it might be worthwhile using a named configuration, with config.StaticDirectory or a custom config.TestStepConfigFunc

Comment on lines +37 to +62
ConfigVariables: map[string]config.Variable{
"exclude_zone_ids": config.ListVariable(config.StringVariable("usw2-az4"), config.StringVariable("usgw1-az2")),
"state": config.StringVariable("available"),
"name": config.StringVariable("opt-in-status"),
"values": config.ListVariable(config.StringVariable("opt-in-not-required")),
"cidr_block": config.StringVariable("10.0.0.0/16"),
"random_name": config.StringVariable(rName),
"vcount": config.IntegerVariable(2),
"cidr_block_2": config.StringVariable("0.0.0.0/0"),
"engine_type": config.StringVariable("ActiveMQ"),
"engine_version": config.StringVariable("5.17.6"),
"auto_minor_version_upgrade": config.BoolVariable(true),
"apply_immediately": config.BoolVariable(true),
"deployment_mode": config.StringVariable("ACTIVE_STANDBY_MULTI_AZ"),
"host_instance_type": config.StringVariable("mq.t2.micro"),
"day_of_week": config.StringVariable("TUESDAY"),
"time_of_day": config.StringVariable("02:00"),
"time_zone": config.StringVariable("CET"),
"publicly_accessible": config.BoolVariable(true),
"username": config.StringVariable("Ender"),
"password": config.StringVariable("AndrewWiggin"),
"username_2": config.StringVariable("Petra"),
"password_2": config.StringVariable("PetraArkanian"),
"console_access": config.BoolVariable(true),
"groups": config.ListVariable(config.StringVariable("dragon"), config.StringVariable("salamander"), config.StringVariable("leopard")),
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need all of these variables, especially since the config is not being reused? The original testAccBrokerDataSourceConfig_byID took one parameter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. service/mq Issues and PRs that pertain to the mq service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants