[Test Proxy] Documentation updates and improvements (Azure#27036)

danigm · Oct 25, 2022 · 47dfd59 · 47dfd59
1 parent defaa79
commit 47dfd59
Show file tree

Hide file tree

Showing 2 changed files with 108 additions and 39 deletions.
diff --git a/doc/dev/test_proxy_migration_guide.md b/doc/dev/test_proxy_migration_guide.md
@@ -95,8 +95,8 @@ Resource preparers need a management client to function, so test classes that us
 
 ### Perform one-time setup
 
-1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install].
-2. After installing, make sure Docker is running and is using Linux containers before running tests.
+1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
+2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
 3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
 order to communicate with the test proxy over a secure connection.
 
@@ -110,6 +110,7 @@ In a `conftest.py` file for your package's tests, add a session-level fixture th
 `devtools_testutils.test_proxy` as a parameter (and has `autouse` set to `True`):
 
 ```python
+import pytest
 from devtools_testutils import test_proxy
 
 # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
@@ -119,9 +120,7 @@ def start_proxy(test_proxy):
 ```
 
 The `test_proxy` fixture will fetch the test proxy Docker image and create a new container called
-`ambitious_azsdk_test_proxy` if one doesn't exist already. If the container already exists, the fixture will start the
-container if it's currently stopped. The container will be stopped after tests finish running, but will stay running if
-test execution is interrupted.
+`ambitious_azsdk_test_proxy`, which will be deleted after test execution unless interrupted.
 
 If your tests already use an `autouse`d, session-level fixture for tests, you can accept the `test_proxy` parameter in
 that existing fixture instead of adding a new one. For an example, see the [Register sanitizers](#register-sanitizers)
@@ -144,26 +143,22 @@ need old `.yml` recordings.
 > **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been
 > deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
 
-> **Note:** the recording storage location is determined when the proxy Docker container is created. If there are
-> multiple local copies of the `azure-sdk-for-python` repo on your machine, you will need to delete any existing
-> `ambitious_azsdk_test_proxy` container before recordings can be stored in a different repo copy.
-
 ### Register sanitizers
 
 Since the test proxy doesn't use [`vcrpy`][vcrpy], tests don't use a scrubber to sanitize values in recordings.
 Instead, sanitizers (as well as matchers and transforms) can be registered on the proxy as detailed in
 [this][sanitizers] section of the proxy documentation. Sanitizers can be registered via `add_*_sanitizer` methods in
 `devtools_testutils`. For example, the general-use method for sanitizing recording bodies, headers, and URIs is
-`add_general_regex_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at
+`add_general_string_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at
 [devtools_testutils/sanitizers.py][py_sanitizers].
 
 Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that
 are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to
 [pytest's scoped fixture documentation][pytest_fixtures] for more details.
 
 As a simple example, to emulate the effect registering a name pair with a `vcrpy` scrubber, you can provide the exact
-value you want to sanitize from recordings as the `regex` in the general regex sanitizer. With `vcrpy`, you would likely
-do something like the following:
+value you want to sanitize from recordings as the `target` in the general string sanitizer. With `vcrpy`, you would
+likely do something like the following:
 
 ```python
 import os
@@ -180,19 +175,21 @@ To do the same sanitization with the test proxy, you could add something like th
 
 ```python
 import os
-from devtools_testutils import add_general_regex_sanitizer, test_proxy
+from devtools_testutils import add_general_string_sanitizer, test_proxy
 
 # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
 @pytest.fixture(scope="session", autouse=True)
 def add_sanitizers(test_proxy):
-    add_general_regex_sanitizer(regex=os.getenv("AZURE_KEYVAULT_NAME"), value="fake-vault")
+    # The default value for the environment variable should be the value you use in playback
+    vault_name = os.getenv("AZURE_KEYVAULT_NAME", "fake-vault")
+    add_general_string_sanitizer(target=vault_name, value="fake-vault")
 ```
 
 Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started
 beforehand.
 
 For a more advanced scenario, where we want to sanitize the account names of all Tables endpoints in recordings, we
-could instead call
+could instead use the `add_general_regex_sanitizer` method:
 
 ```python
 add_general_regex_sanitizer(
@@ -267,32 +264,40 @@ This loader is meant to be paired with the PowerShell test resource management c
 [/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for live test
 resource management.
 
-For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The
-CosmosPreparer and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the
-EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These
-preparers can be used to decorate test methods directly; for example:
+The EnvironmentVariableLoader accepts a positional `directory` argument and arbitrary keyword-only arguments:
+- `directory` is the name of your package's service as it appears in the Python repository; i.e. `service` in `azure-sdk-for-python/sdk/service/azure-service-package`.
+  - For example, for `azure-keyvault-keys`, the value of `directory` is `keyvault`.
+- For each environment variable you want to provide to tests, pass in a keyword argument with the pattern `environment_variable_name="sanitized-value"`.
+  - For example, to fetch the value of `STORAGE_ENDPOINT` and sanitize this value in recordings as `fake-endpoint`, provide `storage_endpoint="fake-endpoint"` to the EnvironmentVariableLoader constructor.
+
+Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these
+values will automatically have sanitizers registered with the test proxy. More specifically, the true values of
+requested variables will be provided to tests in live mode, and the sanitized values of these variables will be provided
+in playback mode.
+
+The most common way to use the EnvironmentVariableLoader is to declare a callable specifying arguments by using
+`functools.partial` and then decorate test methods with that callable. For example:
 
 ```python
-from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
-from .preparers import TablesPreparer
+import functools
+from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
+
+ServicePreparer = functools.partial(
+    EnvironmentVariableLoader,
+    "service",
+    service_endpoint="fake-endpoint",
+    service_account_name="fake-account-name",
+)
 
 class TestExample(AzureRecordedTestCase):
 
-    @TablesPreparer()
+    @ServicePreparer()
     @recorded_by_proxy
     def test_example_with_preparer(self, **kwargs):
-        tables_storage_account_name = kwargs.pop("tables_storage_account_name")
-        tables_primary_storage_account_key = kwargs.pop("tables_primary_storage_account_key")
+        service_endpoint = kwargs.pop("service_endpoint")
         ...
 ```
 
-Or, they can be used in a custom decorator, as they are in the `cosmos_decorator` and `tables_decorator` defined in
-[preparers.py][tables_preparers]. `@tables_decorator`, for instance, is then used in place of `@TablesPreparer()` for
-the example above (note that the method-style `tables_decorator` is used without parentheses).
-
-Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these
-values will automatically have sanitizers registered with the test proxy.
-
 ### Record test variables
 
 To run recorded tests successfully when there's an element of non-secret randomness to them, the test proxy provides a
@@ -420,6 +425,64 @@ container if it's not already running.
 
 For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].
 
+### Use `pytest.mark.parametrize` with migrated tests
+
+Migrating tests to use basic `pytest` tools allows us to take advantage of helpful features such as
+[parametrization][parametrize]. Parametrization allows you to share test code by re-running the same test with varying
+inputs. For example, [`azure-keyvault-keys` tests][parametrize_example] are parametrized to run with multiple API
+versions and multiple Key Vault configurations.
+
+Because of how the `pytest.mark.parametrize` mechanism works, the `recorded_by_proxy(_async)` decorators aren't
+compatible without an additional decorator that handles the arguments we want to parametrize. The callable that
+`pytest.mark.parametrize` decorates needs to have positional parameters that match the arguments we're parametrizing;
+for example:
+
+```python
+import pytest
+from devtools_testutils import recorded_by_proxy
+
+test_values = [
+    ("first_value_a", "first_value_b"),
+    ("second_value_a", "second_value_b"),
+]
+
+# Works because `parametrize` decorates a method with positional `a` and `b` parameters
+@pytest.mark.parameterize("a, b", test_values)
+def test_function(a, b, **kwargs):
+    ...
+
+# Doesn't work; raises collection error
+# `recorded_by_proxy`'s wrapping function doesn't accept positional `a` and `b` parameters
+@pytest.mark.parameterize("a, b", test_values)
+@recorded_by_proxy
+def test_recorded_function(a, b, **kwargs):
+    ...
+```
+
+To parametrize recorded tests, we need a decorator between `pytest.mark.parametrize` and `recorded_by_proxy` that
+accepts the expected arguments. We can do this by declaring a class with a custom `__call__` method:
+
+```python
+class ArgumentPasser:
+    def __call__(self, fn):
+        # _wrapper accepts the `a` and `b` arguments we want to parametrize with
+        def _wrapper(test_class, a, b, **kwargs):
+            fn(test_class, a, b, **kwargs)
+        return _wrapper
+
+# Works because `ArgumentPasser.__call__`'s return value has the expected parameters
+@pytest.mark.parameterize("a, b", test_values)
+@ArgumentPasser()
+@recorded_by_proxy
+def test_recorded_function(a, b, **kwargs):
+    ...
+```
+
+You can also introduce additional logic into the `__call__` method of your intermediate decorator. In the aforementioned
+[`azure-keyvault-keys` test example][parametrize_example], the decorator between `parametrize` and `recorded_by_proxy`
+is actually a [client preparer][parametrize_class] that creates a client based on the parametrized input and passes this
+client to the test.
+
 
 [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
 [docker_install]: https://docs.docker.com/get-docker/
@@ -431,8 +494,12 @@ For more details on proxy startup, please refer to the [proxy documentation][det
 
 [mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py
 
+[parametrize]: https://docs.pytest.org/latest/example/parametrize.html
+[parametrize_example]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/test_key_client.py#L182
+[parametrize_class]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/_test_case.py#L59
 [pipelines_ci]: https://github.com/Azure/azure-sdk-for-python/blob/5ba894966ed6b0e1ee8d854871f8c2da36a73d79/sdk/eventgrid/ci.yml#L30
 [pipelines_live]: https://github.com/Azure/azure-sdk-for-python/blob/e2b5852deaef04752c1323d2ab0958f83b98858f/sdk/textanalytics/tests.yml#L26-L27
+[podman]: https://podman.io/
 [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md
 [py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
 [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery

diff --git a/doc/dev/tests.md b/doc/dev/tests.md
@@ -151,8 +151,8 @@ To migrate an existing test suite to use the test proxy, or to learn more about
 
 ### Perform one-time test proxy setup
 
-1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install].
-2. After installing, make sure Docker is running and is using Linux containers before running tests.
+1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
+2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
 3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
 order to communicate with the test proxy over a secure connection.
 
@@ -213,6 +213,7 @@ Create a `conftest.py` file within your package's test directory (`sdk/{service}
 session-level fixture that accepts `devtools_testutils.test_proxy` as a parameter (and has `autouse` set to `True`):
 
 ```python
+import pytest
 from devtools_testutils import test_proxy
 
 # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
@@ -358,27 +359,27 @@ There are two primary ways to keep secrets from being written into recordings:
 1. The `EnvironmentVariableLoader` will automatically sanitize the values of captured environment variables with the
   provided fake values.
 2. Sanitizers can be registered via `add_*_sanitizer` methods in `devtools_testutils`. For example, the general-use
-  method for sanitizing recording bodies, headers, and URIs is `add_general_regex_sanitizer`. Other sanitizers are
+  method for sanitizing recording bodies, headers, and URIs is `add_general_string_sanitizer`. Other sanitizers are
   available for more specific scenarios and can be found at [devtools_testutils/sanitizers.py][py_sanitizers].
 
 As a simple example of registering a sanitizer, you can provide the exact value you want to sanitize from recordings as
-the `regex` in the general regex sanitizer. To replace all instances of the string "my-key-vault" with "fake-vault" in
+the `target` in the general string sanitizer. To replace all instances of the string "my-key-vault" with "fake-vault" in
 recordings, you could add something like the following in the package's `conftest.py` file:
 
 ```python
-from devtools_testutils import add_general_regex_sanitizer, test_proxy
+from devtools_testutils import add_general_string_sanitizer, test_proxy
 
 # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
 @pytest.fixture(scope="session", autouse=True)
 def add_sanitizers(test_proxy):
-    add_general_regex_sanitizer(regex="my-key-vault", value="fake-vault")
+    add_general_string_sanitizer(target="my-key-vault", value="fake-vault")
 ```
 
 Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started
 beforehand (see [Start the test proxy server](#start-the-test-proxy-server)).
 
 For a more advanced scenario, where we want to sanitize the account names of all storage endpoints in recordings, we
-could instead call
+could instead use `add_general_regex_sanitizer`:
 
 ```python
 add_general_regex_sanitizer(
@@ -640,7 +641,7 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
 [engsys_wiki]: https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/48/Create-a-new-Live-Test-pipeline?anchor=test-resources.json
 [env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
 
-[generate_sas]: https://github.com/Azure/azure-sdk-for-python/blob/6e1f7c02af0c28d5725a532ebe4fc7125256858c/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L200
+[generate_sas]: https://github.com/Azure/azure-sdk-for-python/blob/bf4749babb363e2dc972775f4408036e31f361b4/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L196
 [generate_sas_example]: https://github.com/Azure/azure-sdk-for-python/blob/3e3fbe818eb3c80ffdf6f9f1a86affd7e879b6ce/sdk/tables/azure-data-tables/tests/test_table_entity.py#L1691
 
 [kv_test_resources]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/keyvault/test-resources.json
@@ -650,6 +651,7 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
 [mgmt_settings_fake]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py
 
 [packaging]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md
+[podman]: https://podman.io/
 [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md
 [proxy_general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
 [proxy_migration_guide]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md