Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve task creation with cloud storage and share data #6074

Merged
merged 110 commits into from
May 20, 2023
Merged
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
29e43d0
Make CS manifest optional && support task creating with CS data wwith…
Marishka17 Apr 25, 2023
2913162
Small fixes
Marishka17 Apr 25, 2023
347ad62
Extend task creation with cloud storage data tests
Marishka17 Apr 25, 2023
7e77083
Resolve conflicts
Marishka17 Apr 25, 2023
5e3e5a0
Move key to consts
Marishka17 Apr 25, 2023
ba90936
Fix list azure bucket content
Marishka17 Apr 25, 2023
93d49f0
Add file limit
Marishka17 Apr 25, 2023
3f17dfc
Fix pylint issue
Marishka17 Apr 25, 2023
9e24954
Fix typos
Marishka17 Apr 25, 2023
67432e6
Move methods to parent class
Marishka17 Apr 25, 2023
9f602f6
Update changelog
Marishka17 Apr 26, 2023
ad585d5
[server] Support emulation of hierarchical structure for Azure, GCS, …
Marishka17 Apr 26, 2023
4820bbc
Use FileInfoSerializer in CloudStorageContentSerializer
Marishka17 Apr 27, 2023
114a986
Support task creation with CS dirs && small fixes && added emulation …
Marishka17 May 1, 2023
be42b4f
Optional manifest UI (#6089)
bsekachev May 1, 2023
d38aa2f
Remove slash for dirs from manifest content
Marishka17 May 1, 2023
d7c932b
Fix filtration
Marishka17 May 1, 2023
ac5f00b
Make cloudstorages/id/content identical to share api
Marishka17 May 1, 2023
82b74ca
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 1, 2023
3a0e2ab
Fixed several issues
bsekachev May 1, 2023
2cf02c3
Shared storage using new component
bsekachev May 1, 2023
c9e47d2
Adjusted feature
bsekachev May 1, 2023
1aed852
Fixed page size
bsekachev May 1, 2023
a886511
Removed old types
bsekachev May 1, 2023
94a0b51
Fixed UI tests
bsekachev May 1, 2023
023a2f1
Adjusted code & behaviour
bsekachev May 1, 2023
512c1a5
Applied some comments
bsekachev May 2, 2023
e133379
Fixed multiple issues
bsekachev May 2, 2023
4f8f4fa
Adjusted selection algorithms
bsekachev May 2, 2023
2e35e34
Couple of minor fixes
bsekachev May 2, 2023
43eaac8
Select parent if all children selected
bsekachev May 2, 2023
d952dac
Support new version of /cloudstorages/id/content REST API Endpoint
Marishka17 May 2, 2023
15aa6c2
Update server schema
Marishka17 May 2, 2023
01015bf
Use new version in UI
Marishka17 May 2, 2023
dd8949f
Add documentation
Marishka17 May 2, 2023
c05326c
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 2, 2023
ba6ef45
Fix schema
Marishka17 May 2, 2023
501ccfc
Merge branch 'develop' into mk/make_cs_manifest_optional
Marishka17 May 2, 2023
1a08da4
Adjusted algorithm
bsekachev May 2, 2023
0c9196f
Updated headers
bsekachev May 3, 2023
45e8d4d
Fixed case when all the videos selected in a directory
bsekachev May 3, 2023
527fb6b
Added manifest content pagination && added support for excluded data…
Marishka17 May 5, 2023
b53dec0
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 5, 2023
19a3ca6
Add page_size option for cloudstorages/id/content-v2 endpoint
Marishka17 May 7, 2023
b5ed9aa
Add REST API tests for CS content endpoints
Marishka17 May 7, 2023
5048bba
Small fix
Marishka17 May 7, 2023
880db86
Fix manifest content filtering with prefix
Marishka17 May 7, 2023
ff285ce
Extend test cases
Marishka17 May 7, 2023
cce311d
Applid comments
bsekachev May 8, 2023
ae82bae
Added mime type
bsekachev May 8, 2023
3e8bd8b
Extend task creation tests
Marishka17 May 8, 2023
609e248
Update server schema
Marishka17 May 8, 2023
6c9fecc
Update changelog
Marishka17 May 8, 2023
f3aa4af
Fix black issue
Marishka17 May 8, 2023
6c6abfa
Fix pylint issue
Marishka17 May 8, 2023
d7cb4b0
Fix remark issue
Marishka17 May 8, 2023
dc6e35a
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 8, 2023
3adf514
Fix typo
Marishka17 May 8, 2023
7db19af
Fix task creation case: share + manifest + copy_data=True
Marishka17 May 8, 2023
eb6fd6b
[UI] Fix task creation with manifest
Marishka17 May 8, 2023
4a819e7
Minor fix
bsekachev May 8, 2023
4bb307a
Fix actions_tasks/case_65_create_task_with_manifest.js
Marishka17 May 8, 2023
40a3a31
Update license header
Marishka17 May 8, 2023
dcc80e0
Increase chunk_size for partial image downloading
Marishka17 May 8, 2023
875fd8f
Added ending for directory paths
bsekachev May 8, 2023
7e17ed2
Merge branch 'mk/make_cs_manifest_optional' of github.com:opencv/cvat…
bsekachev May 8, 2023
816df13
Resolved conflicts
bsekachev May 9, 2023
3728968
Apply part of comments
Marishka17 May 9, 2023
8cd5b0c
Update changelog
Marishka17 May 9, 2023
78fc785
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 9, 2023
153ef34
Fix linter
Marishka17 May 9, 2023
61f6bdf
Simplify logic
Marishka17 May 9, 2023
02fa045
Update documentation
Marishka17 May 9, 2023
fbdb305
Update server schema
Marishka17 May 9, 2023
68b3971
Fix test
Marishka17 May 10, 2023
a2b107b
Apply comments
Marishka17 May 10, 2023
34209b8
Fix cpu count
Marishka17 May 11, 2023
96a515c
t
Marishka17 May 11, 2023
b199b8f
Merge branch 'develop' into mk/make_cs_manifest_optional
bsekachev May 11, 2023
7ebaa23
Update deprecation header
Marishka17 May 11, 2023
151f76a
Fix cpu number count
Marishka17 May 11, 2023
4150ec2
Add server_files_exclude to SDK
Marishka17 May 11, 2023
e51c46a
Improve performance
Marishka17 May 11, 2023
636e1b7
Fixed keys endings
bsekachev May 12, 2023
f6fd9ac
reset page when change current path
bsekachev May 12, 2023
c04fb73
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 12, 2023
20c356d
Fixed couple of minor issues
bsekachev May 12, 2023
d6e0396
Fixed dependency
bsekachev May 12, 2023
3dc4eed
Merge branch 'mk/make_cs_manifest_optional' of github.com:opencv/cvat…
bsekachev May 12, 2023
2dad233
Merge branch 'develop' into mk/make_cs_manifest_optional
Marishka17 May 12, 2023
288577f
Fix documentation
Marishka17 May 14, 2023
692cccc
Remove delimiter
Marishka17 May 14, 2023
3b334c1
Rename variable
Marishka17 May 14, 2023
389c9b6
Support filename_pattern for CS data without manifest
Marishka17 May 14, 2023
987f7dd
Add tests for using filename pattern without manifest file
Marishka17 May 14, 2023
1f93859
Extend documentation
Marishka17 May 14, 2023
906b3e5
Merge branch 'mk/make_cs_manifest_optional' of https://github.com/ope…
Marishka17 May 14, 2023
1cbf9f2
Use path.glob
Marishka17 May 15, 2023
3875380
Merge branch 'develop' into mk/make_cs_manifest_optional
Marishka17 May 15, 2023
befe6d6
Merge branch 'develop' into mk/make_cs_manifest_optional
Marishka17 May 15, 2023
d308004
Update tests/python/rest_api/test_cloud_storages.py
Marishka17 May 16, 2023
431a6dd
Small fixes
Marishka17 May 16, 2023
04bb6fe
Merge branch 'develop' into mk/make_cs_manifest_optional
Marishka17 May 16, 2023
6ece66a
t
Marishka17 May 16, 2023
c63bd07
Mime type required
bsekachev May 17, 2023
3e01ef4
Allow creating multiple tasks from cloud storage
bsekachev May 17, 2023
9328acd
Aborted change
bsekachev May 17, 2023
0ac43a0
Minor improvements
Marishka17 May 17, 2023
96f9470
Merge branch 'mk/make_cs_manifest_optional' of github.com:opencv/cvat…
bsekachev May 18, 2023
62468e9
Merge branch 'develop' into mk/make_cs_manifest_optional
nmanovic May 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Extend task creation with cloud storage data tests
  • Loading branch information
Marishka17 committed Apr 25, 2023
commit 347ad62ef9debbbe3bd8b58b5cf5cd228ea66397
32 changes: 21 additions & 11 deletions tests/python/rest_api/test_tasks.py
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@
from itertools import chain, product
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import List
from typing import List, Optional

import pytest
from cvat_sdk import Client, Config, exceptions
@@ -44,11 +44,11 @@
)


def get_cloud_storage_content(username, cloud_storage_id, manifest):
def get_cloud_storage_content(username: str, cloud_storage_id: int, manifest: Optional[str] = None):
with make_api_client(username) as api_client:
(data, _) = api_client.cloudstorages_api.retrieve_content(
cloud_storage_id, manifest_path=manifest
)
kwargs = {"manifest_path": manifest} if manifest else {}

(data, _) = api_client.cloudstorages_api.retrieve_content(cloud_storage_id, **kwargs)
return data


@@ -714,22 +714,32 @@ def test_can_get_annotations_from_new_task_with_skeletons(self):

@pytest.mark.with_external_services
@pytest.mark.parametrize(
"cloud_storage_id, manifest, use_bucket_content, org",
"use_cache, cloud_storage_id, manifest, use_bucket_content, org",
[
(1, "manifest.jsonl", False, ""), # public bucket
(2, "sub/manifest.jsonl", True, "org2"), # private bucket
(True, 1, "manifest.jsonl", False, ""), # public bucket
(True, 2, "sub/manifest.jsonl", True, "org2"), # private bucket
(True, 1, None, False, ""),
(True, 2, None, True, "org2"),
(False, 1, None, False, ""),
(False, 2, None, True, "org2"),
],
)
def test_create_task_with_cloud_storage_files(
self, cloud_storage_id, manifest, use_bucket_content, org
self,
use_cache: bool,
cloud_storage_id: int,
manifest: str,
use_bucket_content: bool,
org: str,
):
if use_bucket_content:
cloud_storage_content = get_cloud_storage_content(
self._USERNAME, cloud_storage_id, manifest
)
else:
cloud_storage_content = ["image_case_65_1.png", "image_case_65_2.png"]
cloud_storage_content.append(manifest)
if manifest:
cloud_storage_content.append(manifest)

task_spec = {
"name": f"Task with files from cloud storage {cloud_storage_id}",
@@ -742,7 +752,7 @@ def test_create_task_with_cloud_storage_files(

data_spec = {
"image_quality": 75,
"use_cache": True,
"use_cache": use_cache,
"cloud_storage_id": cloud_storage_id,
"server_files": cloud_storage_content,
}