As the number of Hub users grew, it seemed wise to verify one of the key advantages of Hub: its performance. A standard way to measure the performance of a framework is to provide a process for comparisons to discover the industry winner under the same conditions and metrics. Hub claims to be:

Fastest unstructured dataset management for TensorFlow/PyTorch.

The goal of the benchmarks is to show what areas of performance this claim applies to and to guide Hub's team towards in which Hub has still some room for improvement. The benchmarks are split into internal and external ones. The former suggest the relative conditions which are optimal for Hub to maximize its performance. The latter are to determine Hub's place on the ML scene among other actors like PyTorch, Tensorflow, zarr or TileDB.

All of the benchmarks were conducted on the same machine unless stated otherwise in a section related to a particular benchmark. The specification of the resources used for the benchmarks can be found below:

In all of the benchmarks caching (including storage caching) is disabled.

Some benchmarks are parametrized by a variety of arguments, such as:

• dataset
• batch size
• prefetch factor
• number of workers

The time measured is shown in seconds rounded to 4 decimal places unless specified otherwise. Relevant configuration details for the parametrized benchmarks are noted in respective sections.

Presented benchmarks are intended to be reproducible and easy to replicate manually or through automation.

1. Launch the AWS EC2 instance according to the specification in the Method section.
2. Install Hub in the edit mode along with the necessary packages found in all of the requirements files or run sh benchmark_setup.sh (if Hub is not installed) and source into the virtual environment with source ./hub-env/bin/activate.
3. Sequentially run all of the Python files in the benchmarks folder or run sh benchmark_run.sh. If you use benchmark_run, the results will be combined in the results.log file. Otherwise, the results for the benchmarks should be released to the standard output. For the external dataset iteration benchmark only, you may collect the results with grep 'BENCHMARK'.

Note that access to the datasets stored in the S3 bucket is limited. However, you might replicate this set-up by creating a bucket which contains the data in Hub format. For instance, you may upload the dataset with .store using the S3 path as the first argument.

How does Hub compare to zarr and tiledb in terms of read / write sequential access to the dataset?

Remote Hub already performs ~1.14x better than TileDB (which offers local storage only) whereas Hub used locally is over 26x better than TileDB on the access to the entire dataset. The results are even more explicit in batched access.

Read is conducted on the original MNIST dataset (as specified in Method/Datasets section). However, the write test is conducted on a MNIST-like dataset which retains its shape and schema but is given pseudorandomly generated data to write.

MNIST: entire dataset (70000 label and image pairs)

MNIST: in batches of 7000

Hub performs better than zarr despite being based on the framework. TileDB is an outlier among all frameworks.

Remote access to Hub is 8-24x times slower than local.

Write is ~3-5.5x slower than read for all locally stored frameworks. For remote Hub write is 1.4x faster than read.

Is Hub faster in iterating over a dataset than PyTorch DataLoader and Tensorflow Dataset?

Yes, Hub fetching data remotely outperforms both Pytorch and Tensorflow on MNIST dataset. It is 1.12x better than PyTorch and 1.004x better than Tensorflow.

1. Datasets: MNIST & Places365
2. Batch size: 16
3. Prefetch factor: 4
4. Number of workers: 1

Except for the relatively slow performance of Hub's to_pytorch in the local environment, the results of all loaders on MNIST are comparable.

Places365, a significantly larger dataset, sheds light on the real differences among the frameworks. Not surprisingly, local storage surpasses the remote ones - S3 followed by Wasabi, heavily affected by the network latency. The best performing framework turns out to be Tensorflow, closely followed by Hub's to_tensorflow implementation. The biggest outlier is Hub's local to_pytorch which could not be measured on time as it is over 10x slower than other loaders.

PyTorch's native DataLoader as well as Hub's to_pytorch function are generally slower than Tensorflow.

We measure the time to compress (PNG) a sample image using PIL and Hub.

The results below measure compression time of the sample image at a batch size of 100.

There are no drops of performance of Hub in relation to the Python Imaging Library while compressing images. In fact, Hub performs slightly better than PIL library.

We measure the time to fetch an uncached random sample from a dataset, varying over several standard datasets and further at several batch sizes.

Random offsets are also used to ensure that no caching is being taken advantage of externally.

Hub performs relatively uniformly over the various batch sizes with the notable exception of Omniglot test dataset. It can be speculated that a few times lower number of images in the dataset compared to others allow Hub to perform much better than in the case of other datasets. Reading single element batches is slower than of batches containing multiple elements.

We measure the time to iterate over a full dataset (MNIST) in both pytorch and tensorflow (separately). Benchmarks also vary over multiple preset batch sizes and prefetch factors.

Increasing the batch size leads to a better performance. The transition from the size of 1 to 16 leads to a decrease in iteration time by over 85%. Tensorflow's performance seems not to be drastically improved by prefetching. For PyTorch, however, in smaller batches, an appropriate prefetch factor can elicit a 5-20% improvement. For both Tensorflow and PyTorch a relatively optimal balance is achieved at the prefetch factor equal to 4 and the batch size of length 16. These parameters are used in the external dataset iteration section described below.

This section is incomplete.

It has been shown that the Hub framework is the fastest among its competitors for the most common dataset operations (read and write). Hub team needs to continue improving to_pytorch and to_tensorflow functions to increase its dataset iteration scores. Benchmarks should be re-calculated every time new features are added to Hub. Further plans with regards to the benchmarks are outlined here.