Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hook up async tasks to new AnnotationProject.status field #5350

Merged

Conversation

Lknechtli
Copy link
Contributor

@Lknechtli Lknechtli commented Mar 6, 2020

Overview

Add a more informative status field to annotation projects
Migrate from the ready field to a status field
Set statuses correctly in upload processing

Checklist

  • Description of PR is in an appropriate section of the changelog and grouped with similar changes if possible
  • Swagger specification updated
  • New tables and queries have appropriate indices added (N/A)
  • Any content changes are properly templated using BUILDCONFIG.APP_NAME
  • Any new SQL strings have tests
  • Any new endpoints have scope validation and are included in the integration test csv

Demo

Statuses as set by raising exceptions in the appropriate sections of the upload batch process:
image

Notes

Error handling does not encompass larger scale batch issues (unable to contact API, etc), since we don't have a good way of handling that right now. This should be handled in a later issue

Testing Instructions

  • build your batch jar
  • build your batch container
  • run migrations
  • start the API server, etc.
  • start the annotate frontend on the accompanying annotate branch
  • create a project with a small upload to make things to quickly
  • find the upload id in your database: select id from uploads where annotation_project_id = '<annotation_project_id>';
  • run the batch command to ingest the upload: ./scripts/console batch 'AWS_BATCH_JOB_ATTEMPT=3 rf process-upload 819c2ce5-e48a-40c3-8a27-f274b1ec4e59'
  • refresh the project list and verify that the status is set to "ready" as expected
  • stick a normal exception inside the main try catch block, rebuild the batch container, and re-run the batch command. Verify that it sets the status to IMAGE_INGESTION_FAILURE
  • stick a TaskGridError there instead and verify it sets the status to TASK_GRID_FAILURE

Closes https://github.com/raster-foundry/annotate/issues/670

@Lknechtli Lknechtli requested a review from jisantuc March 6, 2020 21:08
@Lknechtli Lknechtli force-pushed the feature/lk/annotation-project-processing-improvements branch from db53fce to 65957fa Compare March 6, 2020 21:19
changelog

wip

address comments

formatting, add .env.local to gitignore
@Lknechtli Lknechtli force-pushed the feature/lk/annotation-project-processing-improvements branch from 9909382 to 4cbfb32 Compare March 10, 2020 12:35
Copy link
Contributor

@jisantuc jisantuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few small changes around how the api interaction with python works (objects instead of strings for the failures) and how the decoder for the success cases works, but it's small stuff and I don't need to review again after CI passes (only change to make outside of suggestions is importing scala.util.Try). Works great otherwise 🎉

app-tasks/rf/src/rf/commands/process_upload.py Outdated Show resolved Hide resolved
app-tasks/rf/src/rf/commands/process_upload.py Outdated Show resolved Hide resolved
Lknechtli and others added 3 commits March 10, 2020 16:16
Co-Authored-By: James Santucci <james.santucci@gmail.com>
@Lknechtli Lknechtli merged commit ad61184 into develop Mar 11, 2020
@Lknechtli Lknechtli deleted the feature/lk/annotation-project-processing-improvements branch March 11, 2020 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants