From 8cd8d97c066c11ec5a119ee1a5aa46f64c4e09c9 Mon Sep 17 00:00:00 2001 From: kristina-arezina Date: Wed, 26 May 2021 15:51:59 -0400 Subject: [PATCH] changed read me --- README.md | 148 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 78 insertions(+), 70 deletions(-) diff --git a/README.md b/README.md index 76ac030c6e..9657128359 100644 --- a/README.md +++ b/README.md @@ -40,20 +40,20 @@ ### What is Hub for? -Software 2.0 needs Data 2.0, and Hub delivers it. Most of the time Data Scientists/ML researchers work on data management and preprocessing instead of training models. With Hub, we are fixing this. We store your (even petabyte-scale) datasets as single numpy-like array on the cloud, so you can seamlessly access and work with it from any machine. Hub makes any data type (images, text files, audio, or video) stored in cloud usable as fast as if it were stored on premise. With same dataset view, your team can always be in sync. +Software 2.0 needs Data 2.0, and Hub delivers it. Most of the time Data Scientists/ML researchers work on data management and preprocessing instead of training models. With Hub, we are fixing this. We store your (even petabyte-scale) datasets as single numpy-like array on the cloud, so you can seamlessly access and work with it from any machine. Hub makes any data type (images, text files, audio, or video) stored in cloud usable as fast as if it were stored on premise. With same dataset view, your team can always be in sync. Hub is being used by Waymo, Red Cross, World Resources Institute, Omdena, and others. -### Features +### Features -* Store and retrieve large datasets with version-control -* Collaborate as in Google Docs: Multiple data scientists working on the same data in sync with no interruptions -* Access from multiple machines simultaneously -* Deploy anywhere - locally, on Google Cloud, S3, Azure as well as Activeloop (by default - and for free!) -* Integrate with your ML tools like Numpy, Dask, Ray, [PyTorch](https://docs.activeloop.ai/en/latest/integrations/pytorch.html), or [TensorFlow](https://docs.activeloop.ai/en/latest/integrations/tensorflow.html) -* Create arrays as big as you want. You can store images as big as 100k by 100k! -* Keep shape of each sample dynamic. This way you can store small and big arrays as 1 array. -* [Visualize](http://app.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme) any slice of the data in a matter of seconds without redundant manipulations +- Store and retrieve large datasets with version-control +- Collaborate as in Google Docs: Multiple data scientists working on the same data in sync with no interruptions +- Access from multiple machines simultaneously +- Deploy anywhere - locally, on Google Cloud, S3, Azure as well as Activeloop (by default - and for free!) +- Integrate with your ML tools like Numpy, Dask, Ray, [PyTorch](https://docs.activeloop.ai/en/latest/integrations/pytorch.html), or [TensorFlow](https://docs.activeloop.ai/en/latest/integrations/tensorflow.html) +- Create arrays as big as you want. You can store images as big as 100k by 100k! +- Keep shape of each sample dynamic. This way you can store small and big arrays as 1 array. +- [Visualize](http://app.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme) any slice of the data in a matter of seconds without redundant manipulations


@@ -63,8 +63,8 @@ Visualization of a dataset uploaded to Hub via app.activeloop.ai (free tool).

- ## Getting Started + Work with public or your own data, locally or on any cloud. ### Access public data. Fast. @@ -76,6 +76,7 @@ pip3 install hub ``` Access public datasets in Hub by following a straight-forward convention which merely requires a few lines of simple code. Run this excerpt to get the first thousand images in the [MNIST database](https://app.activeloop.ai/dataset/activeloop/mnist/?utm_source=github&utm_medium=repo&utm_campaign=readme) in the numpy array format: + ```python from hub import Dataset @@ -83,6 +84,7 @@ mnist = Dataset("activeloop/mnist") # loading the MNIST data lazily # saving time with *compute* to retrieve just the necessary data mnist["image"][0:1000].compute() ``` + You can find all the other popular datasets on [app.activeloop.ai](https://app.activeloop.ai/datasets/popular/?utm_source=github&utm_medium=repo&utm_campaign=readme). ### Train a model @@ -100,11 +102,13 @@ mnist = mnist.to_pytorch(lambda x: (x["image"], x["label"])) train_loader = torch.utils.data.DataLoader(mnist, batch_size=1, num_workers=0) for image, label in train_loader: - # Training loop here + # Define your training loop here ``` -### Create a local dataset +### Create a local dataset + If you want to work on your own data locally, you can start by creating a dataset: + ```python from hub import Dataset, schema import numpy as np @@ -133,78 +137,80 @@ Also, if you need a publicly available dataset that you cannot find in the Hub, 1. Register a free account at [Activeloop](https://app.activeloop.ai/register/?utm_source=github&utm_medium=repo&utm_campaign=readme) and authenticate locally: - ```sh - activeloop register - activeloop login + ```sh + activeloop register + activeloop login - # alternatively, add username and password as arguments (use on platforms like Kaggle) - activeloop login -u username -p password - ``` + # alternatively, add username and password as arguments (use on platforms like Kaggle) + activeloop login -u username -p password + ``` 2. Then create a dataset, specifying its name and upload it to your account. For instance: - ```python - from hub import Dataset, schema - import numpy as np - - ds = Dataset( - "username/dataset_name", - shape = (4,), - mode = "w+", - schema = { - "image": schema.Tensor((512, 512), dtype="float"), - "label": schema.Tensor((512, 512), dtype="float"), - } - ) - - ds["image"][:] = np.zeros((4, 512, 512)) - ds["label"][:] = np.zeros((4, 512, 512)) - ds.flush() - ``` + + ```python + from hub import Dataset, schema + import numpy as np + + ds = Dataset( + "username/dataset_name", + shape = (4,), + mode = "w+", + schema = { + "image": schema.Tensor((512, 512), dtype="float"), + "label": schema.Tensor((512, 512), dtype="float"), + } + ) + + ds["image"][:] = np.zeros((4, 512, 512)) + ds["label"][:] = np.zeros((4, 512, 512)) + ds.flush() + ``` 3. Access it from anywhere else in the world, on any device having a command line: - ```python - from hub import Dataset - ds = Dataset("username/dataset_name") - ``` + ```python + from hub import Dataset + ds = Dataset("username/dataset_name") + ``` ## Documentation For more advanced data pipelines like uploading large datasets or applying many transformations, please refer to our [documentation](http://docs.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme). ## Tutorial Notebooks -The [examples](https://github.com/activeloopai/Hub/tree/master/examples) directory has a series of examples and the [notebooks](https://github.com/activeloopai/Hub/tree/master/examples/notebooks) has some notebooks with use cases. Some of the notebooks are listed of below. -| Notebook | Description | | -|:--- |:--- |---: | -| [Uploading Images](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201a%20-%20Uploading%20Images.ipynb) | Overview on how to upload and store images on Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201a%20-%20Uploading%20Images.ipynb) | -| [Uploading Dataframes](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201b%20-%20Uploading%20Dataframes.ipynb) | Overview on how to upload Dataframes on Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201b%20-%20Uploading%20Dataframes.ipynb) | -| [Uploading Audio](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201c%20-%20Uploading%20Audio.ipynb) | Explains how to handle audio data in Hub|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201c%20-%20Uploading%20Audio.ipynb) | -| [Retrieving Remote Data](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%202%20-%20Retrieving%20Remote%20Data.ipynb) | Explains how to retrieve Data| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/tutorial/Tutorial%202%20-%20Retrieving%20Remote%20Data.ipynb) | -| [Transforming Data](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%203%20-%20Transforming%20Data.ipynb) | Briefs on how data transformation with Hub|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%203%20-%20Transforming%20Data.ipynb) | -| [Dynamic Tensors](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%204%20-%20What%20are%20Dynamic%20Tensors.ipynb) | Handling data with variable shape and sizes|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%204%20-%20What%20are%20Dynamic%20Tensors.ipynb) | -| [NLP using Hub](https://github.com/activeloopai/Hub/blob/master/examples/notebooks/nlp_using_hub.ipynb) | Fine Tuning Bert for CoLA|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/notebooks/nlp_using_hub.ipynb) | -| [Getting Started with Text on Hub](https://github.com/activeloopai/Hub/blob/master/examples/notebooks/Getting_Started_with_Text_on_Hub.ipynb) | Overview on using Text datasets in Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/notebooks/Getting_Started_with_Text_on_Hub.ipynb) | +The [examples](https://github.com/activeloopai/Hub/tree/master/examples) directory has a series of examples and the [notebooks](https://github.com/activeloopai/Hub/tree/master/examples/notebooks) has some notebooks with use cases. Some of the notebooks are listed of below. +| Notebook | Description | | +| :---------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | +| [Uploading Images](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201a%20-%20Uploading%20Images.ipynb) | Overview on how to upload and store images on Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201a%20-%20Uploading%20Images.ipynb) | +| [Uploading Dataframes](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201b%20-%20Uploading%20Dataframes.ipynb) | Overview on how to upload Dataframes on Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201b%20-%20Uploading%20Dataframes.ipynb) | +| [Uploading Audio](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201c%20-%20Uploading%20Audio.ipynb) | Explains how to handle audio data in Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%201c%20-%20Uploading%20Audio.ipynb) | +| [Retrieving Remote Data](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%202%20-%20Retrieving%20Remote%20Data.ipynb) | Explains how to retrieve Data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/tutorial/Tutorial%202%20-%20Retrieving%20Remote%20Data.ipynb) | +| [Transforming Data](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%203%20-%20Transforming%20Data.ipynb) | Briefs on how data transformation with Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%203%20-%20Transforming%20Data.ipynb) | +| [Dynamic Tensors](https://github.com/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%204%20-%20What%20are%20Dynamic%20Tensors.ipynb) | Handling data with variable shape and sizes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/tutorial/Tutorial%204%20-%20What%20are%20Dynamic%20Tensors.ipynb) | +| [NLP using Hub](https://github.com/activeloopai/Hub/blob/master/examples/notebooks/nlp_using_hub.ipynb) | Fine Tuning Bert for CoLA | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/notebooks/nlp_using_hub.ipynb) | +| [Getting Started with Text on Hub](https://github.com/activeloopai/Hub/blob/master/examples/notebooks/Getting_Started_with_Text_on_Hub.ipynb) | Overview on using Text datasets in Hub | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/activeloopai/Hub/blob/master/examples/notebooks/Getting_Started_with_Text_on_Hub.ipynb) | ## Use Cases -* **Satellite and drone imagery**: [Smarter farming with scalable aerial pipelines](https://activeloop.ai/usecase/intelinair?utm_source=github&utm_medium=repo&utm_campaign=readme), [Mapping Economic Well-being in India](https://towardsdatascience.com/faster-machine-learning-using-hub-by-activeloop-4ffb3420c005), [Fighting desert Locust in Kenya with Red Cross](https://omdena.com/projects/ai-desert-locust/) -* **Medical Images**: Volumetric images such as MRI or Xray -* **Self-Driving Cars**: [Radar, 3D LIDAR, Point Cloud, Semantic Segmentation, Video Objects](https://medium.com/snarkhub/extending-snark-hub-capabilities-to-handle-waymo-open-dataset-4dc7b7d8ab35) -* **Retail**: Self-checkout datasets -* **Media**: Images, Video, Audio storage + +- **Satellite and drone imagery**: [Smarter farming with scalable aerial pipelines](https://activeloop.ai/usecase/intelinair?utm_source=github&utm_medium=repo&utm_campaign=readme), [Mapping Economic Well-being in India](https://towardsdatascience.com/faster-machine-learning-using-hub-by-activeloop-4ffb3420c005), [Fighting desert Locust in Kenya with Red Cross](https://omdena.com/projects/ai-desert-locust/) +- **Medical Images**: Volumetric images such as MRI or Xray +- **Self-Driving Cars**: [Radar, 3D LIDAR, Point Cloud, Semantic Segmentation, Video Objects](https://medium.com/snarkhub/extending-snark-hub-capabilities-to-handle-waymo-open-dataset-4dc7b7d8ab35) +- **Retail**: Self-checkout datasets +- **Media**: Images, Video, Audio storage ## Why Hub specifically? There are quite a few dataset management libraries which offer functionality that might seem similar to Hub. In fact, quite a few users migrate data from PyTorch or Tensorflow Datasets to Hub. Here are a few startling differences you will encounter after switching to Hub: -* the data is provided in chunks, which you may stream from a remote location, instead of downloading all of it at once -* as only the necessary portion of the dataset is evaluated, you are able to work on the data immediately -* you are able to store the data that would not fit in your memory in its entirety -* you may version control and collaborate with multiple users on your datasets across different machines -* you are equipped with tools that enhance your understanding of the data in a manner of seconds, such as our visualization tool -* you can easily prepare your data for multiple training libraries at ones (e.g. you can use the same dataset for training with PyTorch and Tensorflow) +- the data is provided in chunks, which you may stream from a remote location, instead of downloading all of it at once +- as only the necessary portion of the dataset is evaluated, you are able to work on the data immediately +- you are able to store the data that would not fit in your memory in its entirety +- you may version control and collaborate with multiple users on your datasets across different machines +- you are equipped with tools that enhance your understanding of the data in a manner of seconds, such as our visualization tool +- you can easily prepare your data for multiple training libraries at ones (e.g. you can use the same dataset for training with PyTorch and Tensorflow) ## Community @@ -214,7 +220,7 @@ We'd love your feedback by completing our 3-minute [**survey**](https://forms.gl tweet on Twitter. -As always, thanks to our amazing contributors! +As always, thanks to our amazing contributors! @@ -225,13 +231,15 @@ Made with [contributors-img](https://contrib.rocks). Please read [CONTRIBUTING.md](CONTRIBUTING.md) to know how to get started with making contributions to Hub. ## Examples + Activeloop's Hub format lets you achieve faster inference at a lower cost. We have 30+ popular datasets already on our platform. These include: + - COCO - CIFAR-10 - PASCAL VOC - Cars196 - KITTI -- EuroSAT +- EuroSAT - Caltech-UCSD Birds 200 - Food101 @@ -239,26 +247,26 @@ Check these and many more popular datasets on our [visualizer web app](https://a ## README Badge -Using Hub? Add a README badge to let everyone know: - +Using Hub? Add a README badge to let everyone know: [![hub](https://img.shields.io/badge/powered%20by-hub%20-ff5a1f.svg)](https://github.com/activeloopai/Hub) ``` [![hub](https://img.shields.io/badge/powered%20by-hub%20-ff5a1f.svg)](https://github.com/activeloopai/Hub) ``` + ## Usage Tracking + By default, we collect anonymous usage data using Bugout (here's the [code](https://github.com/activeloopai/Hub/blob/853456a314b4fb5623c936c825601097b0685119/hub/__init__.py#L24) that does it). It only logs Hub library's own actions and parameters, and no user/ model data is collected. This helps the Activeloop team to understand how the tool is used and how to deliver maximum value to the community by building features that matter to you. You can easily opt-out of usage tracking during login. - ## Disclaimers Similarly to other dataset management packages, `Hub` is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license. If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a [GitHub issue](https://github.com/activeloopai/Hub/issues/new). Thanks for your contribution to the ML community! - ## Acknowledgement - This technology was inspired from our experience at Princeton University and would like to thank William Silversmith @SeungLab with his awesome [cloud-volume](https://github.com/seung-lab/cloud-volume) tool. We are heavy users of [Zarr](https://zarr.readthedocs.io/en/stable/) and would like to thank their community for building such a great fundamental block. + +This technology was inspired from our experience at Princeton University and would like to thank William Silversmith @SeungLab with his awesome [cloud-volume](https://github.com/seung-lab/cloud-volume) tool. We are heavy users of [Zarr](https://zarr.readthedocs.io/en/stable/) and would like to thank their community for building such a great fundamental block.