add contribution instructions for prototype datasets #5133

pmeier · 2021-12-28T08:48:58Z

This is a preliminary guide to foster early contributions to the prototype datasets outside of the core team. You can find the rendered version here. In the future this should probably moved to CONTRIBUTING.md or the into the documentation.

@ reviewers: All the changes apart from README.md are minor convenience changes that I found while writing the guide.

cc @pmeier @bjuncek

facebook-github-bot · 2021-12-28T08:49:05Z

💊 CI failures summary and remediations

As of commit 55efdbb (more details on the Dr. CI page):

5/5 failures introduced in this PR

🕵️ 4 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

unittest_linux_cpu_py3.7 (1/4)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

/root/project/torchvision/io/video.py:406: Runt...log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found


test/test_internet.py::TestDatasetUtils::test_download_url_dont_exist
  /root/project/env/lib/python3.7/site-packages/pluggy/_callers.py:23: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
    args = [caller_kwargs[argname] for argname in hook_impl.argnames]

test/test_internet.py::TestDatasetUtils::test_download_url_dont_exist
  /root/project/env/lib/python3.7/site-packages/pluggy/_callers.py:23: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=13, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
    args = [caller_kwargs[argname] for argname in hook_impl.argnames]

test/test_io.py::TestVideo::test_read_video_timestamps_corrupted_file
  /root/project/torchvision/io/video.py:406: RuntimeWarning: Failed to open container for /tmp/tmp9qhcodls.mp4; Caught error: [Errno 1094995529] Invalid data found when processing input: '/tmp/tmp9qhcodls.mp4'; last error log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found
    warnings.warn(msg, RuntimeWarning)

test/test_models.py::test_memory_efficient_densenet[densenet121]
test/test_models.py::test_memory_efficient_densenet[densenet169]
test/test_models.py::test_memory_efficient_densenet[densenet201]
test/test_models.py::test_memory_efficient_densenet[densenet161]
  /root/project/env/lib/python3.7/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
    warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

test/test_models.py::test_inception_v3_eval

unittest_linux_cpu_py3.8 (2/4)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

/root/project/torchvision/io/video.py:406: Runt...log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found


test/test_image.py::test_encode_jpeg[grace_hopper_517x606.jpg]
  /root/project/test/test_image.py:493: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1641197134237/work/torch/csrc/utils/tensor_new.cpp:998.)
    encoded_jpeg_pil = torch.frombuffer(buf.getvalue(), dtype=torch.uint8)

test/test_internet.py::TestDatasetUtils::test_download_url_retry_http
  /root/project/env/lib/python3.8/site-packages/_pytest/mark/structures.py:552: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
    seen.update(self.parent.keywords)

test/test_io.py::TestVideo::test_read_video_timestamps_corrupted_file
  /root/project/torchvision/io/video.py:406: RuntimeWarning: Failed to open container for /tmp/tmpv9k6hwex.mp4; Caught error: [Errno 1094995529] Invalid data found when processing input: '/tmp/tmpv9k6hwex.mp4'; last error log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found
    warnings.warn(msg, RuntimeWarning)

test/test_models.py::test_memory_efficient_densenet[densenet121]
test/test_models.py::test_memory_efficient_densenet[densenet169]
test/test_models.py::test_memory_efficient_densenet[densenet201]
test/test_models.py::test_memory_efficient_densenet[densenet161]
  /root/project/env/lib/python3.8/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
    warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

test/test_models.py::test_inception_v3_eval

unittest_linux_cpu_py3.9 (3/4)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

/root/project/torchvision/io/video.py:406: Runt...log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found


test/test_internet.py::TestDatasetUtils::test_download_url_retry_http
  <attrs generated init _pytest.fixtures.PseudoFixtureDef>:2: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
    _setattr = _cached_setattr.__get__(self, self.__class__)

test/test_internet.py::TestDatasetUtils::test_download_url_dispatch_download_from_google_drive
  /root/project/env/lib/python3.9/unittest/mock.py:2059: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=11, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
    setattr(_type, entry, MagicProxy(entry, self))

test/test_io.py::TestVideo::test_read_video_timestamps_corrupted_file
  /root/project/torchvision/io/video.py:406: RuntimeWarning: Failed to open container for /tmp/tmpxhpkn6zo.mp4; Caught error: [Errno 1094995529] Invalid data found when processing input: '/tmp/tmpxhpkn6zo.mp4'; last error log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found
    warnings.warn(msg, RuntimeWarning)

test/test_models.py::test_memory_efficient_densenet[densenet121]
test/test_models.py::test_memory_efficient_densenet[densenet169]
test/test_models.py::test_memory_efficient_densenet[densenet201]
test/test_models.py::test_memory_efficient_densenet[densenet161]
  /root/project/env/lib/python3.9/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
    warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

test/test_models.py::test_inception_v3_eval

unittest_macos_cpu_py3.7 (4/4)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

/Users/distiller/project/torchvision/io/video.p...log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found


test/test_internet.py::TestDatasetUtils::test_download_url_dont_exist
  /Users/distiller/project/env/lib/python3.7/site-packages/_pytest/nodes.py:288: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=14, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>
    return (x[1] for x in self.iter_markers_with_node(name=name))

test/test_internet.py::TestDatasetUtils::test_download_url_dont_exist
  /Users/distiller/project/env/lib/python3.7/site-packages/_pytest/nodes.py:288: ResourceWarning: unclosed <ssl.SSLSocket [closed] fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>
    return (x[1] for x in self.iter_markers_with_node(name=name))

test/test_io.py::TestVideo::test_read_video_timestamps_corrupted_file
  /Users/distiller/project/torchvision/io/video.py:406: RuntimeWarning: Failed to open container for /var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/tmpybesgtyz.mp4; Caught error: [Errno 1094995529] Invalid data found when processing input: '/var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/tmpybesgtyz.mp4'; last error log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found
    warnings.warn(msg, RuntimeWarning)

test/test_models.py::test_memory_efficient_densenet[densenet121]
test/test_models.py::test_memory_efficient_densenet[densenet169]
test/test_models.py::test_memory_efficient_densenet[densenet201]
test/test_models.py::test_memory_efficient_densenet[densenet161]
  /Users/distiller/project/env/lib/python3.7/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
    warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

test/test_models.py::test_inception_v3_eval

1 failure not recognized by patterns:

Job	Step	Action
^{unittest_macos_cpu_py3.9}	^{Run tests}	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

datumbox

LGTM overall, just a couple of potential typos marked below:

torchvision/prototype/datasets/_builtin/README.md

NicolasHug

Thanks @pmeier , I gave it a quick look only, but I'll look at it more in details once I implement a prototype dataset.

LGTM, but I'd suggest keeping the line length to either 80 or 120, as we do for the code. Most editor wrap long lines properly, but GitHub doesn't, so we can only make comments on a per-line basis (and when lines are long, this gets inconvenient)

pmeier · 2022-01-04T10:53:20Z

@NicolasHug Sorry, I didn't see your comment before merge. I'll fix the line length the next time I push changes to the file.

Summary: * add contribution instructions for prototype datasets * cleanup * fix links * Update torchvision/prototype/datasets/_builtin/README.md Reviewed By: datumbox, prabhat00155 Differential Revision: D33405355 fbshipit-source-id: 864ae4642a9355abab0e2e760cb36925058f07aa

…e datasets (#5133) Differential Revision: D33405355 Original commit changeset: 864ae4642a93 Original Phabricator Diff: D33405355 fbshipit-source-id: 0127385422452c5ab77d43b1605557484bb6a30b

Summary: * add contribution instructions for prototype datasets * cleanup * fix links * Update torchvision/prototype/datasets/_builtin/README.md Reviewed By: sallysyw Differential Revision: D33479269 fbshipit-source-id: 539e5d6f1c29553296efc83551415e037829c77e

add contribution instructions for prototype datasets

d22cf77

pmeier added module: datasets module: documentation prototype labels Dec 28, 2021

pmeier requested a review from NicolasHug December 28, 2021 08:48

pytorch-probot bot added the ciflow/default label Dec 28, 2021

facebook-github-bot added the cla signed label Dec 28, 2021

pmeier added 2 commits December 28, 2021 09:51

cleanup

0d4a095

Merge branch 'main' into datasets/readme

5cafbc4

This was referenced Dec 28, 2021

Documentation and examples for prototype torchvision #4992

Open

New classification datasets support for FLAVA #5108

Closed

fix links

f6301f3

datumbox approved these changes Jan 4, 2022

View reviewed changes

torchvision/prototype/datasets/_builtin/README.md Outdated Show resolved Hide resolved

pmeier added 2 commits January 4, 2022 11:49

Update torchvision/prototype/datasets/_builtin/README.md

7d4ec3b

Merge branch 'main' into datasets/readme

55efdbb

NicolasHug approved these changes Jan 4, 2022

View reviewed changes

pmeier merged commit cc7e856 into pytorch:main Jan 4, 2022

pmeier deleted the datasets/readme branch January 4, 2022 10:52

pmeier mentioned this pull request Jan 4, 2022

add prettier as non-code auto formatter #5158

Open

pmeier mentioned this pull request Jan 11, 2022

fix category file generation #5188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add contribution instructions for prototype datasets #5133

add contribution instructions for prototype datasets #5133

pmeier commented Dec 28, 2021 •

edited by pytorch-probot bot

Loading

facebook-github-bot commented Dec 28, 2021 •

edited

Loading

datumbox left a comment

NicolasHug left a comment

pmeier commented Jan 4, 2022

add contribution instructions for prototype datasets #5133

add contribution instructions for prototype datasets #5133

Conversation

pmeier commented Dec 28, 2021 • edited by pytorch-probot bot Loading

facebook-github-bot commented Dec 28, 2021 • edited Loading

💊 CI failures summary and remediations

🕵️ 4 new failures recognized by patterns

unittest_linux_cpu_py3.7 (1/4)

unittest_linux_cpu_py3.8 (2/4)

unittest_linux_cpu_py3.9 (3/4)

unittest_macos_cpu_py3.7 (4/4)

1 failure not recognized by patterns:

datumbox left a comment

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

pmeier commented Jan 4, 2022

pmeier commented Dec 28, 2021 •

edited by pytorch-probot bot

Loading

facebook-github-bot commented Dec 28, 2021 •

edited

Loading