Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy failed tifs to debug data bucket #5446

Merged
merged 8 commits into from
Jul 15, 2020

Conversation

jisantuc
Copy link
Contributor

@jisantuc jisantuc commented Jul 13, 2020

Overview

This PR uploads tifs that fail to process to a debug data bucket so we can figure out what went wrong even if
people sensibly follow our advice about deleting their failed projects.

Checklist

  • Description of PR is in an appropriate section of the changelog and grouped with similar changes if possible

Testing Instructions

  • raise an exception (any exception) at the beginning of the GeoTiffS3SceneFactory
  • rebuild the batch container -- docker-compose build batch
  • if you haven't in a while, rebuild the batch jar from sbt: batch/assembly
  • fire up groundwork and your RF servers and create an upload with the network tab open
  • copy your upload's id
  • try to process your upload, pretending that this is the third job attempt: AWS_BATCH_JOB_ATTEMPT=3 ./scripts/console batch 'rf process-upload d64e5be4-bf8c-4c82-90bd-d7ea9ad4082d'
  • you should see a message beginning Skipping debug data upload
  • set a debug data bucket that you can write to (any bucket will do but I have a suggestion)
  • run the upload processing command again
  • You should see a message ending It will be available in the debug bucket for 14 days.
  • check the bucket / key that that message reported -- there should be a tif there
  • confirm that the environment variable for the debug data bucket matches what's in azavea/raster-foundry-deployment#255

Closes #1049

@jisantuc jisantuc requested a review from notthatbreezy July 13, 2020 20:47
Copy link

@notthatbreezy notthatbreezy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great after I fixed up my typo in my environment variables.

Could you update the .env.template file for the new environment variable and maybe our own development .env too?

@jisantuc jisantuc merged commit ac61948 into develop Jul 15, 2020
@jisantuc jisantuc deleted the feature/js/copy-failed-tifs-to-debug-data-bucket branch July 15, 2020 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants