What I am changing

This PR adds an API endpoint to handle ESA's "push" subscription notification for new data. This endpoint is intended to handle new granule "created" event notifications for Sentinel-2. This PR also adds a CLI script to list, create, and terminate subscriptions. This script defines the filters that we wish to apply for new granules, which at the present is limited to filtering to only Sentinel-2 imagery.

For more background on ESA's subscription system, the design of this handler, and preliminary results, see this ticket, https://github.com/NASA-IMPACT/hls_development/issues/300

This subscription handling endpoint is intended to be run alongside the existing scheduled "link fetcher" that runs once a day. Once we have more evidence that this is working as expected we can remove the old "link fetcher" and associated machinery (date generator Lambda, StepFunction, and the GranuleCount tracking table).

How I did it

At a high level one of the implementation choices was "where should this live?" especially given that both the event and polling based link fetching was to live side-by-side for a time. In other words they will share some code related to the message payload, state tracking, and sending messages to the queue. I had thought of two possible alternatives,

1. Create a new module under lambdas/ and either,
   • Copy/paste "common" code
   • OR, migrate "common" code into some shared location (i.e., a new Lambda layer)
2. Reuse the same module lambdas/link_fetcher,
   • Refactor "common" code into a common.py

Either way we still have 2x sets of Lambda functions, log groups, and alerts! I picked option 2 because sharing the same directory didn't cause any blockers (i.e., Lambda zip getting too large), was relatively less complex than migrating to another directory, avoided setting up yet another workspace in CI and Makefiles, and eventually we'll be deleting the old link fetcher so this shared living arrangement will only be temporary.

With that high level decision made, there are a few other steps I want to call out for review,

1. I created the hls-s2-downloader-serverless/{stage}/esa-subscription-credentials secret
   in AWS SecretsManager and populated it with the user/password that I used when creating the 'push' subscription on ESA's side.
2. Once the endpoint was deployed I used the manage_subscription.py script to create the "push" subscription to our endpoint.
3. I added a "date observed" filter to the event handler to mimic the 30 day look back in the schedule based "link fetcher". I did this after noticing that ESA has been in the middle of a reprocessing campaign to process data from 2022 with the newer baseline 05.10 software (see announcement, https://dataspace.copernicus.eu/news/2024-9-2-sentinel-2-collection-1-products-availability).
   • Right now the filter is simply ignoring results that were acquired > 30 days since today. In the future we could handle these differently, perhaps adding them into some reprocessing download queue that is processed with lower priority than new granules.

How you can test it

There's a few ways to test this,

1. Unit tests can be run as usual make unit-tests from root, or make test from the lambdas/link_fetcher module
2. Integration tests are run from CI/CD in this PR
3. I've deployed this to our AWS environment using the IDENTIIFIER=event-subs for the past ~2 weeks. I've purged the queue a few times as I've made changes, for example to ignore reprocessed granules that were acquired in ~2022. Still though this has been useful to check in bulk how many granule messages we're getting. Below is a screenshot of the queue depth for the last 2 days that shows we're getting approximately the ~5-7k messages per day that we usually get from the schedule based link fetcher.