Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parameters): review types and exports #1528

Merged
merged 13 commits into from
Jun 22, 2023
Merged

Conversation

dreamorosi
Copy link
Contributor

@dreamorosi dreamorosi commented Jun 20, 2023

Description of your changes

This PR touches most of the files in the Parameters utility with the goal of streamlining the types of the inputs & outputs of each provider, as well as finalizing the exports that the utility will make available to customers. Additionally, it also adds a few types/type guards in the commons utility which are currently used by the parameters utility but that will be used by others as well.

Exports

As mentioned in previous issues/PRs/discussions, we want customers to be able to tree-shake the library as much as possible and be able to build the project without the need of installing dependencies that might needed by parts of the code that they are not using.

For example, in Parameters, customers using the SecretsProvider might not be interested in bringing into their project @aws-sdk/client-appconfigdata, a dependency needed by the AppConfigProvider. To allow for this, we need to create granular exports in the package.json of the utility.

The exports are governed by the exports and typeVersions sections of the package.json of the utility. The first one tells Node.js where to resolve a certain path in the node_modules entry of the utility, whereas the typeVersions section tells TypeScript where to find the types for that same path.

The exports that we are offering are:

import { Transform } from '@aws-lambda-powertools/parameters';
import { GetParameterError } from '@aws-lambda-powertools/parameters/errors';
import { ... } from '@aws-lambda-powertools/parameters/<PROVIDER>';
import type { ... } from '@aws-lambda-powertools/parameters/<PROVIDER>/types';

The first export, and only top-level one, is a readonly object that contains the different transform available so that customers can use it like Transform.JSON or Transform.AUTO and not have to worry about spelling or case (although the regular string is still allowed).

The second export, under /errors exposes the types of errors that can be thrown by the utility. This can be used by customers to catch & handle errors or during tests.

The other exports are all specific to the providers, for example, the SSM provider, exports the following things:

import {
  getParameter,
  getParameters,
  getParametersByName,
  SSMProvider,
 } from '@aws-lambda-powertools/parameters/ssm';
import type { ... } from '@aws-lambda-powertools/parameters/ssm/types'; // all the types specific to SSMProvider & functions

The other providers follow the same logic. We are also exposing the BaseProvider under the /base path, this way customers will be able to extend this class to create their own custom provider.

In order to support these exports I had to partially reorganize the folder structure of the utility; this is the main driver behind the large diff.

Base Provider

Before this PR, the BaseProvider was quite strict in the return types for the get and getMultiple methods. Specifically, it allowed only to return string, Uint8Array, Record<string, unknown>, and undefined. These types were overly restrictive and would have hindered the extensibility of the provider.

When researching HashiCorp Vault for #1320 I discovered that this particular store returned types that were incompatible with the types described above, this sparked also the discussion in aws-powertools/powertools-lambda-python#2250.

Starting from this PR, the BaseProvider is free to return a promise that resolves to unknown. The BaseProvider handles caching and transformations only, so it doesn't need to know the types it's handling when returning. The actual value retrieval is done by the _get & _getMultiple methods, which are abstract and leave the actual implementation to more specific providers.

With this in mind, BaseProvider can return unknown and not care about the return types as long as the downstream providers (i.e. SecretsProvider, DynamoDBProvider, or VaultProvider) handle and validate the types they retrieve and return as needed.

Once made this change I had to do some light rework of the types & add some runtime type-checks to the transform function so that we are able to work with unknown types most of the time and still apply the transform only on types we want.

Adaptive types for DynamoDBProvider and AppConfigProvider

Continuing the work started in previous releases for the other providers, I have applied generics to these two providers so that we can make some assumptions on the return type based on the transformations applied, or allow customers to specify a type if they know more than the compiler.

This closes #1425, and closes #1424

Updates to JSON transform types

Before this PR I mistakenly assumed that when applying a JSON transform the result type would always be an object ({} / Record<string, unknown>. This was a naïve assumption.

In reality, the JSON specification allows to parse and considers many/most primitive types as valid JSON documents. For instance, "hello", 5, true, and [] among others, are all valid JSON documents that can be processed with JSON.parse().

For this reason I introduced some new types in the commons utility that represent these primitives. The new JSONValue, which includes the combinations of primitives as well as arrays and objects containing these primitives, is now used as default return type whenever a JSON transform is applied.

As a side note, the types just discussed will be useful also for the Idempotency utility and for other utilities handling unknown payloads that can be JSON parsed/stringified.

Miscellaneous

While working on the PR I also had to do changes here & there, these don't fit in any specific category:

  • Shortened/standardized the names of most customer-facing types/interfaces (i.e. SSMGetOptionsUnion becomes simply SSMGetOptions)
  • Fixed some formatting issues in the docstrings as I encountered & updated the typedoc.json config to reflect the new exports
  • Updates to unit & integration tests to reflect the changes above (mostly imports & types-related)

Related issues, RFCs

Issue number: #1424

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my change is effective and works
  • The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@dreamorosi dreamorosi self-assigned this Jun 20, 2023
@github-actions
Copy link
Contributor

No related issues found. Please ensure there is an open issue related to this change to avoid significant delays or closure.

@github-actions github-actions bot added do-not-merge This item should not be merged need-issue This PR needs an issue before it can be reviewed/worked on further feature PRs that introduce new features or minor changes labels Jun 20, 2023
@dreamorosi dreamorosi removed do-not-merge This item should not be merged need-issue This PR needs an issue before it can be reviewed/worked on further labels Jun 20, 2023
@boring-cyborg boring-cyborg bot added commons This item relates to the Commons Utility parameters This item relates to the Parameters Utility dependencies Changes that touch dependencies, e.g. Dependabot, etc. tests PRs that add or change tests labels Jun 21, 2023
@pull-request-size pull-request-size bot added the size/XXL PRs with 1K+ LOC, largely documentation related label Jun 21, 2023
@dreamorosi dreamorosi marked this pull request as ready for review June 21, 2023 23:07
@dreamorosi dreamorosi requested a review from am29d June 21, 2023 23:07
@dreamorosi
Copy link
Contributor Author

Apologies in advance for the big diff, the PR body explains why it had to be this big (changes in exports & types). This should be the last major PR for this utility though.

Copy link
Contributor

@am29d am29d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor question on the void return type for getMultiple

_options?: unknown
): Promise<undefined | Record<string, unknown>> {
return super.getMultiple(path);
public async getMultiple(path: string, _options?: unknown): Promise<void> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the return type here be Promise<unknown> ?

Also, can't we provide the same option for the caller to pass explicit types to be expected, similar to get? I see, that it's not trivial to handle mixed return types, but getting list of the same type is possible.

I see you did it for DynamoDBProvider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

void is used when there's no return keyword. unknown would be a wider type in this case.

For AppConfigProvider specifically, the getMultiple operation is not supported at all and this method, when called, is guaranteed to throw (see _getMultiple).

So since we are not returning anything, we use void.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I wasn't aware of that, so it means user can call getMultiple and nothing happens and there will be no error or indication, except method doc. Seems like there is no way to prevent the method call 😕

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no, that's not what I meant, sorry.

If the user calls the getMultiple() method on the AppConfigProvider it will definitely throw, we have a test that verifies this: https://github.com/aws-powertools/powertools-lambda-typescript/blob/main/packages/parameters/tests/unit/AppConfigProvider.test.ts#L277-L292

What I meant is that we are telling TS not to expect any return value (because we know it'll always throw). AFAIK there's no way to tell TS that a method will throw, so we tell it to expect nothing.

packages/parameters/src/base/transformValue.ts Outdated Show resolved Hide resolved
@dreamorosi dreamorosi requested a review from a team as a code owner June 22, 2023 12:06
@dreamorosi dreamorosi requested a review from am29d June 22, 2023 12:09
@dreamorosi dreamorosi merged commit 6f96711 into main Jun 22, 2023
@dreamorosi dreamorosi deleted the feat/parameters/types branch June 22, 2023 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
commons This item relates to the Commons Utility dependencies Changes that touch dependencies, e.g. Dependabot, etc. feature PRs that introduce new features or minor changes parameters This item relates to the Parameters Utility size/XXL PRs with 1K+ LOC, largely documentation related tests PRs that add or change tests
Projects
None yet
2 participants