Issue32/event source concurrency #34
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Address dependabot alert, Unintended leak of Proxy-Authorization header in requests, by upgrading requests to 2.31.0.
Address dependabot alert: ReDoS in py library when used with subversion. Although we do not use subversion, this alert served as a good prompt to upgrade our pytest dependencies to their latest versions.
Version 11.15 is no longer available for new deployments, resulting in the following error during deployment: "Cannot find version 11.15 for aurora-postgresql"
| @@ -80,7 +79,7 @@ def __init__( | |||
| self, | |||
| id=f"{identifier}-downloader-rds", | |||
| engine=aws_rds.DatabaseClusterEngine.aurora_postgres( | |||
| version=aws_rds.AuroraPostgresEngineVersion.VER_11_15 | |||
| version=aws_rds.AuroraPostgresEngineVersion.of("11.21", "11") | |||
There was a problem hiding this comment.
@chuckwondo Will this trigger an upgrade of our Postgres engine version on upgrade?
There was a problem hiding this comment.
That's definitely a good question. Since it's a minor version bump, that shouldn't be a problem, particularly since the system is not continually running, as long as deployment occurs after the daily cron job completes, so there's no DB activity during deployment.
However, I suggest adding 1 additional step prior to deployment: create a DB snapshot after the cron job finishes.
There was a problem hiding this comment.
I updated the steps in the PR comment to include taking a DB snapshot.
sharkinsspatial
approved these changes
Nov 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I am changing
How I did it
I removed reserved concurrency on the downloader lambda in the CDK stack, and replaced it with maximum concurrency on invocation from an SQS event source, as described in Introducing maximum concurrency of AWS Lambda functions when using Amazon SQS as an event source
To fix the broken integration test stack deployment, I had to restore the previously removed
Pipfilefrom the db layer. We should address comprehensive replacement ofpipenvin an isolated issue to avoid having partial replacement causing odd interactions, such as the one that broke integration test stack deployment.To fix the broken integration test, I updated the test to match the latest Copernicus API URL syntax.
To address the following dependabot security alerts, I upgraded
requests,pytest,pytest-cov, andpytest-docker:Although we don't use subversion, I took the opportunity to upgrade
pytestand friends to their latest versions.How you can test it
CRITICAL: Due to how the CDK manages resources, before deploying the changes in this PR, you must first manually remove the existing event source trigger on the downloader lambda function. Otherwise, deployment will fail, indicating that you are attempting to create a trigger between the lambda and the queue, but such a trigger already exists. (See aws/aws-cdk#18706)
Specifically, before deploying, you should do the following:
Create a snapshot of the DB, just in case the minor version bump goes awry (Aurora Postgres 11.15 is no longer available for new deployments)
Remove the existing trigger via the AWS Console as follows:
Once the existing trigger is deleted, you should be able to deploy the new code, which will recreate the trigger, and will set the trigger's maximum concurrency to 14: