Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding option to export per-sample JSON in FiftyOneDataset format #2028

Merged
merged 26 commits into from
Aug 30, 2022
Merged

Adding option to export per-sample JSON in FiftyOneDataset format #2028

merged 26 commits into from
Aug 30, 2022

Conversation

brimoor
Copy link
Contributor

@brimoor brimoor commented Aug 26, 2022

Adds an optional use_dirs=True parameter when exporting in FiftyOneDataset format that causes metadata to be exported in per-sample/frame JSON files rather than the default behavior of writing single samples.json/frames.json files.

This option is useful for large datasets, as well as use cases where one wants to track changes to each sample's labels individually.

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("cifar10", split="test")

dataset.export(
    export_dir="/tmp/cifar10-test",
    dataset_type=fo.types.FiftyOneDataset,
    export_media=False,
    use_dirs=True,
    ordered=False,
)

dataset2 = fo.Dataset.from_dir(
    dataset_dir="/tmp/cifar10-test",
    dataset_type=fo.types.FiftyOneDataset,
)
$ ls /tmp/cifar10-test
metadata.json	samples

$ ls /tmp/cifar10-test/samples | head -n 10
630851edf7b9b41bff2abdea.json
630851edf7b9b41bff2abdee.json
630851edf7b9b41bff2abdef.json
630851edf7b9b41bff2abdf6.json
630851edf7b9b41bff2abdf7.json
630851edf7b9b41bff2abdf8.json
630851edf7b9b41bff2abdf9.json
630851edf7b9b41bff2abe06.json
630851edf7b9b41bff2abe07.json
630851edf7b9b41bff2abe08.json

benjaminpkane and others added 26 commits August 19, 2022 13:16
@benjaminpkane
bumps, release notes
921dff7
@benjaminpkane
add to release notes
628f012
@benjaminpkane
linting
6363b62
@brimoor
Merge branch 'develop' into release/v0.16.6
ad9c90f
@brimoor
updating pkg versions
3d9d59a
@brimoor
removing unnecessary pull
87aefe1
@brimoor
Merge branch 'develop' into release/v0.16.6
07a7941
@brimoor
move App build to end
4e4a30b
@brimoor
updating release notes
ae022c3
@brimoor
always migrate when user is admin
38a9204
@brimoor
moving legacy troubleshooting to another page
f8efbaa
@brimoor
adding docs on coordinating migrations
6bedc9c
@brimoor
adding helpful error when a dataset fails to load
ef536b1
@brimoor
initial work
5eb1cf3
@brimoor
pinning max requirement
ff8760a
@brimoor
using more cloud-friendly layout
f5f671b
@brimoor
adding upgrade note
c6b5387
@brimoor
Merge branch 'release/v0.16.6' into feature/fiftyone-dataset-update
d648803
@brimoor
adding support for rendering progress bars in DynamicBatcher
aa40130
@brimoor
using DynamicBatcher's progress feature
dae6ce6
@brimoor
adding support for importing/exporting directories of docs
613007f
@brimoor
reduce max limit
9e58330
@brimoor
adding support for importing/exporting per-frame directories in Fifty…
7e4829f 
…OneDataset format
@brimoor
adding use_dirs unit test
44b4f9c
@brimoor
documenting use_dirs option
8b98239
@brimoor
Merge branch 'develop' into feature/fiftyone-dataset-update
04109aa
@brimoor brimoor added the enhancement Code enhancement label Aug 26, 2022
@brimoor brimoor requested a review from a team August 26, 2022 05:04
@brimoor brimoor self-assigned this Aug 26, 2022
benjaminpkane
benjaminpkane approved these changes Aug 30, 2022
View reviewed changes
Copy link
Contributor

@benjaminpkane benjaminpkane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brimoor brimoor merged commit d1d21f2 into develop Aug 30, 2022
@brimoor brimoor deleted the feature/fiftyone-dataset-update branch August 30, 2022 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benjaminpkane benjaminpkane benjaminpkane approved these changes

Assignees

@brimoor brimoor

Labels
enhancement Code enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@brimoor @benjaminpkane