Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving to_frames() implementation #1578

Merged
merged 15 commits into from
Feb 6, 2022
Merged

Improving to_frames() implementation #1578

merged 15 commits into from
Feb 6, 2022

Conversation

brimoor
Copy link
Contributor

@brimoor brimoor commented Feb 2, 2022

Previously, the default syntax for frame views dataset.to_frames() would check for parallel directories of frames on disk for each video and synchronously use ffmpeg to sample any non-existent frames when creating the view.

For example, videos with the following paths

/path/to/video1.mp4
/path/to/video2.mp4
...

would be sampled (if necessary) as follows:

/path/to/video1/
    000001.jpg
    000002.jpg
    ...
/path/to/video2/
    000001.jpg
    000002.jpg
    ...

There were numerous problems with this approach:

  • Sampling frames is expensive, which violates the notion that DatasetViews should defer computation until sample access-time.
  • In practice, sampling frames rarely goes as expected for non-H.264/5 streams. For example, ffmpeg may fail to extract the last X% of frames of a video. The previous implementation lacked a good mechanism for gracefully continuing upon failures in such a way that running dataset.to_frames() would not always retry and re-fail to sample these uncomputable frames.

This PR modifies the default behavior of dataset.to_frames() to instead assume that the user has already sampled the frames offline and stored their locations in a filepath field of each Frame of their video dataset. Frames that do not have a filepath populated (eg uncomputable ones) are omitted from the returned frames view.

One can still ask FiftyOne to do the work of sampling frames via dataset.to_frames(sample_frames=True). Moreover, when sampling frames via this syntax:

  • The frame filepath fields will be automatically populated on the input dataset/collection so that the default syntax dataset.to_frames() can subsequently be used
  • Sampling failures are now (by default) gracefully logged rather than raising a fatal exception

Example usages:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# This syntax is currently undocumented
# Use `(video_path, frame_number)` rather than sampling frames
frames = dataset.to_frames(sample_frames="dynamic")
print(dataset.count("frames"))
print(len(frames))

# Sample some frames
# This will store the sampled frame image paths in `filepath` sample fields
frames = dataset.to_frames(sample_frames=True, fps=1, verbose=True)
print(dataset.exists("frames.filepath").count("frames"))
print(len(frames))

# Now sample some different frames
people = dataset.filter_labels("frames.detections", F("label") == "person")
clips = people.to_clips("frames.detections")
frames = clips.to_frames(sample_frames=True, fps=1, verbose=True)
print(len(frames))

# Sample all remaining frames
frames = dataset.to_frames(sample_frames=True, verbose=True)
print(dataset.exists("frames.filepath").count("frames"))
print(dataset.count("frames"))
print(len(frames))

# Now let's actually use the new default behavior, which assumes `filepath` is
# already populated on each frame document
frames = dataset.to_frames()
print(dataset.count("frames"))
print(len(frames))

# Sub-sampling works with the default syntax too
frames = dataset.to_frames(fps=1)
print(len(frames))

# Frames without filepaths are not included in frames views
# To simulate this, we'll randomly clear some filepaths
dataset.match_frames(F.rand() < 0.1).clear_frame_field("filepath")
frames = dataset.to_frames()
print(dataset.count("frames"))
print(dataset.exists("frames.filepath").count("frames"))
print(len(frames))

# If frame documents are missing, these frames are not included in frames views
# by default because no filepaths are available
dataset.match_frames(F("detections.detections").length() > 10).keep_frames()
frames = dataset.to_frames()
print(dataset.count("frames"))
print(len(frames))

# No sampling will happen here because we already sampled all frames
frames = dataset.to_frames(sample_frames=True, sparse=True, verbose=True)
print(dataset.count("frames"))
print(len(frames))

# All frames exist on disk, but frame documents need to be created for the
# frames that we deleted previously so that filepaths can be stored again
frames = dataset.to_frames(sample_frames=True, verbose=True)
print(dataset.count("frames"))
print(len(frames))

# All frames have filepaths stored again, so the default syntax includes all
# frames in the frames view
frames = dataset.to_frames()
print(dataset.count("frames"))
print(len(frames))

# `to_frames()` is very graceful by default
dataset = fo.Dataset()
dataset.add_samples(
    [
        fo.Sample(filepath="non-existent1.mp4"),
        fo.Sample(filepath="non-existent2.mp4"),
        fo.Sample(filepath="non-existent3.mp4"),
        fo.Sample(filepath="non-existent4.mp4"),
        fo.Sample(filepath="non-existent5.mp4"),
    ]
)
view = dataset.to_frames()
view = dataset.to_frames(sample_frames=True, verbose=True)

@brimoor brimoor added the enhancement Code enhancement label Feb 2, 2022
@brimoor brimoor requested a review from a team February 2, 2022 00:33
@brimoor brimoor self-assigned this Feb 2, 2022
Copy link
Member

@ehofesmann ehofesmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Big fan of this new default behavior.

@brimoor brimoor merged commit 33d752c into develop Feb 6, 2022
@brimoor brimoor deleted the to-frames branch February 6, 2022 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Code enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants