Skip to content

REST API can create task but won't add GCP cloud storage to it #4593

Closed
@nlandau-divert

Description

@nlandau-divert

Expected Behaviour

Based on the commit found here.
I am able to create a new cloud storage by sending a POST request to http://:/api/cloudstoreages with the following command body:

{
  "provider_type": "GOOGLE_CLOUD_STORAGE",
  "resource": "my_bucket",
  "display_name": "My Bucket",
  "credentials_type": "KEY_FILE_PATH",
  "key_file_path": "/credentials/gcs.json",
  "specific_attributes": "region=US-EAST1",
  "manifests": ["test_manifest.jsonl"]
}

Then I should be able to create a new task with that cloud storage attached by sending a POST request to http://:/api/tasks the following body:

{
  "name": "Test Task",
  "project_id": 1,
  "chunk_size": 1,
  "image_quality": 70,
  "server_files": [
     "test/img1.jpg",
     "test/img2.jpg",
     "test/manifest.jsonl"
  ],
  "cloud_storage_id": 1,
  "use_zip_chunks": true,
  "use_cache": true
}

Current Behaviour

With my requests the cloud storage is created and the manifest is attached. I am able to manually create a labeling task with data from this new cloud store, but when I try to use the API a task is created with no dataset attached. I can go to the tasks page and see a new task is there but there are no images and no jobs attached to it.

I have tried the POST request with "remote_files" and "client_files" instead of server files with no change in result as well as manually creating the task and trying to use a PATCH request to update an existing task with more photos.

When I then go inspect the status of the newly created task at http://:/api/task//status, I see the following error

{
    "state": "Failed",
    "message": "FileNotFoundError: [Errno 2] No such file or directory: '/home/django/data/storages/1/test/manfiest.jsonl'",
    "progress": 0.0
}

I went into the docker container and manually added the test files, both manifest and images into the directory from the error above and the request then worked.

Why is this searching a directory within the docker container for files and not the cloud storage?

Context

I need to use the API to add new labeling jobs as new photos are collected.

Your Environment

Using the following CVAT versions:

Server version: 2.1
Core version: 5.0.1
Canvas version: 2.13.2
UI version: 1.37.0

Interacting with the API using python requests library in python 3.7.

Activity

kaninaba94

kaninaba94 commented on Sep 8, 2022

@kaninaba94

Push! This happened to me, too

nmanovic

nmanovic commented on Sep 8, 2022

@nmanovic
Contributor

@Marishka17 , could you please look?

Marishka17

Marishka17 commented on Sep 8, 2022

@Marishka17
Contributor

Hi, @nlandau-divert , @kaninaba94,

  1. When sending the POST request tasks/{id}/data, you should specify a manifest file from the list of files that you specified when attaching the cloud storage (but in your case I see test_manifest.jsonl and test/manifest.jsonl).
  2. I see that there is one missed required parameter from the body of the request that is sent to tasks/{id}/data endpoint:
    "storage": "cloud_storage",
  3. Please use the latest CVAT develop branch because there was one fix with this functionality.

I will close this issue now because I've checked that it works correctly. Don't hesitate to reopen it if the issue is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    REST API can create task but won't add GCP cloud storage to it · Issue #4593 · cvat-ai/cvat