Problem description

Our async processing mixes API and DB interaction in odd ways. This mixture requires us to keep extra configuration of secrets lying around and also makes it unclear what we can add to different jobs, since it's tough to remember which jobs are configured for what. For example, recently, the upload processing job was updated to add a database interaction step via the batch jar, but it didn't have the configured env variables to be able to talk to the database.

Historically our python-based interaction facilitated easier work with AWS resources, like EMR. The downside was that it required additional copies of sections of the datamodel. With the mixture we went to after that, we now have additional copies of the datamodel in python and additional configuration so we can do two kinds of interaction.

We should choose a side and remove superfluous configuration / powers.

Expected Behavior or Output

Async jobs run in python and interact only with the API or run in Scala and interact only with the database