Streaming is a PyTorch compatible dataset that enables users to stream training data from cloud-based object stores. Streaming can read files from local disk or from cloud-based object stores. As a drop-in replacement for your PyTorch IterableDataset class, it’s easy to get streaming:

dataloader = torch.utils.data.DataLoader(dataset=ImageStreamingDataset(remote='s3://...'))

Please check the quick start guide and user guide on how to use the Streaming Dataset.

• High performance, accurate streaming of training data from cloud storage
• Efficiently train anywhere, independent of training data location
• Cloud-native, no persistent storage required
• Enhanced data security—data exists ephemerally on training cluster

Streaming is available with Pip:

pip install mosaicml-streaming

Please check our Examples section for the end-to-end model training workflow using Streaming datasets.

Getting started guides, examples, API reference, and other useful information can be found in our docs.

We welcome any contributions, pull requests, or issues!

To start contributing, see our Contributing page.

P.S.: We're hiring!

@misc{mosaicml2022streaming,
    author = {The Mosaic ML Team},
    title = {streaming},
    year = {2022},
    howpublished = {\url{https://github.com/mosaicml/streaming/}},
}