Merge branch 'develop' into groups-update

d-demirci · Aug 26, 2022 · b1f266c · b1f266c
2 parents c0c08fe + 43298f6
commit b1f266c
Show file tree

Hide file tree

Showing 49 changed files with 1,346 additions and 347 deletions.
diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml
@@ -61,7 +61,7 @@ jobs:
           pip install -r requirements/dev.txt
           pip install fiftyone-brain fiftyone-db
           pip install pycocotools tensorflow torch torchvision
-          pip install -e .
+          pip install .
       - name: Build docs
         run: |
           ./docs/generate_docs.bash
@@ -95,7 +95,7 @@ jobs:
           cp -r docs-download/ source/docs/fiftyone/
           rm -r docs-download/
       - name: Create website PR
-        uses: peter-evans/create-pull-request@v2
+        uses: peter-evans/create-pull-request@v4
         with:
           branch: fiftyone-docs
           commit-message: |

diff --git a/docs/source/cli/index.rst b/docs/source/cli/index.rst
@@ -754,7 +754,7 @@ FiftyOne deployments.
 
 .. code-block:: shell
 
-    # Migrate the database and all datasets to the current package version
+    # Migrate the database and all datasets to the current client version
     fiftyone migrate --all
 
 .. code-block:: shell

diff --git a/docs/source/getting_started/install.rst b/docs/source/getting_started/install.rst
@@ -267,7 +267,7 @@ Here's the workflow for downgrading to an older version of FiftyOne:
 .. code-block:: shell
 
     # The version that you wish to downgrade to
-    VERSION=0.14.4  # for example
+    VERSION=0.15.1
 
     # Migrate the database
     fiftyone migrate --all -v $VERSION
@@ -283,29 +283,16 @@ your ``fiftyone`` package without first running
 :ref:`fiftyone migrate <cli-fiftyone-migrate>`, don't worry, you simply need to
 reinstall the newer version of FiftyOne and then follow these instructions.
 
+See :ref:`this page <troubleshooting-downgrades>` if you need to install
+FiftyOne v0.7.3 or earlier.
+
 .. note::
 
   If you are working with a
   :ref:`custom/shared MongoDB database <configuring-mongodb-connection>`, you
   can use :ref:`database admin privileges <database-migrations>` to control
   which clients are allowed to downgrade your FiftyOne deployment.
 
-.. note::
-
-    The :ref:`fiftyone migrate <cli-fiftyone-migrate>` command was introduced
-    in FiftyOne v0.7.3. If you would like to downgrade from a FiftyOne version
-    prior to v0.7.3 (to a yet older version), then you will first need to
-    *upgrade* to v0.7.3 or later and then follow the instructions above.
-
-.. note::
-
-    To install a FiftyOne version **prior to v0.7.0**, you must add an
-    ``--index`` option to ``pip install``:
-
-    .. code-block:: shell
-
-        pip install --index https://pypi.voxel51.com fiftyone==<version>
-
 .. _uninstalling-fiftyone:
 
 Uninstalling FiftyOne

diff --git a/docs/source/getting_started/troubleshooting.rst b/docs/source/getting_started/troubleshooting.rst
@@ -60,8 +60,8 @@ See the
 "Package 'fiftyone' requires a different Python"
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-This error occurs when attempting to install FiftyOne with an unsupported Python
-version (either too old or too new). See the
+This error occurs when attempting to install FiftyOne with an unsupported
+Python version (either too old or too new). See the
 :ref:`installation guide <install-prereqs>` for details on which versions of
 Python are supported by FiftyOne.
 
@@ -165,7 +165,33 @@ then FiftyOne's database service will attempt to start up on import using the
 MongoDB distribution provided by `fiftyone-db`. If the database fails to start,
 importing `fiftyone` will result in exceptions being raised.
 
-.. _troubleshooting-mongodb-exits
+.. _troubleshooting-downgrades:
+
+Downgrading to old versions
+---------------------------
+
+The :ref:`fiftyone migrate <cli-fiftyone-migrate>` command was introduced in
+FiftyOne v0.7.3. If you would like to downgrade from a FiftyOne version
+prior to v0.7.3 (to a yet older version), then you will first need to
+:ref:`upgrade <upgrading-fiftyone>` to v0.7.3 or later and then
+:ref:`downgrade <downgrading-fiftyone>`:
+
+.. code-block:: shell
+
+  # The version that you wish to downgrade to
+  VERSION=0.7.0
+
+  pip install fiftyone==0.7.3
+  fiftyone migrate --all -v $VERSION
+  pip install fiftyone==$VERSION
+
+To install a FiftyOne version prior to v0.7.0, you must add ``--index``:
+
+.. code-block:: shell
+
+    pip install --index https://pypi.voxel51.com fiftyone==<version>
+
+.. _troubleshooting-mongodb-exits:
 
 Database exits
 --------------
@@ -176,7 +202,6 @@ Running `ulimit -n 64000` should resolve the issue. 64,000 is the recommended
 open file limit.  MongoDB has full documentation on the issue
 `here <https://docs.mongodb.com/manual/reference/ulimit/>`_. 
 
-
 .. _troubleshooting-mongodb-linux:
 
 Troubleshooting Linux imports

diff --git a/docs/source/release-notes.rst b/docs/source/release-notes.rst
@@ -3,6 +3,101 @@ FiftyOne Release Notes
 
 .. default-role:: code
 
+.. _release-notes-v0.16.6:
+
+FiftyOne 0.16.6
+---------------
+*Released August 25, 2022*
+
+App
+
+- Fixed a bug that caused the App to break when sample tags contained `.`
+  `#1924 <https://github.com/voxel51/fiftyone/pull/1924>`_
+- Fixed search results alignment
+  `#1930 <https://github.com/voxel51/fiftyone/pull/1930>`_
+- Fixed App refreshes after view changes had occurred from the view bar
+  `#1931 <https://github.com/voxel51/fiftyone/pull/1931>`_
+- Fixed mask targets rendering in the tooltip
+  `#1943 <https://github.com/voxel51/fiftyone/pull/1943>`_
+  `#1949 <https://github.com/voxel51/fiftyone/pull/1949>`_
+- Fixed classification confusion matrix connections
+  `#1967 <https://github.com/voxel51/fiftyone/pull/1967>`_
+
+Core
+
+- Added a save context that enables
+  :ref:`efficient batch edits <efficient-batch-edits>` of datasets and views
+  `#1727 <https://github.com/voxel51/fiftyone/pull/1727>`_
+- Added Plotly v5 support
+  `#1981 <https://github.com/voxel51/fiftyone/pull/1981>`_
+- Added a :ref:`quantiles aggregation <aggregations-quantiles>`
+  `#1937 <https://github.com/voxel51/fiftyone/pull/1937>`_
+- Added support for writing transformed images/videos to new locations in the
+  :func:`transform_images() <fiftyone.utils.image.transform_images>` and
+  :func:`transform_videos() <fiftyone.utils.video.transform_videos>` functions
+  `#2007 <https://github.com/voxel51/fiftyone/pull/2007>`_
+- Added support for configuring the
+  :ref:`package-wide logging level <configuring-fiftyone>`
+  `#2009 <https://github.com/voxel51/fiftyone/pull/2009>`_
+- Added more full-featured support for serializing and deserializing datasets,
+  views, and samples via `to_dict()` and `from_dict()`
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Added support for dynamic attributes when performing coerced exports
+  `#1993 <https://github.com/voxel51/fiftyone/pull/1993>`_
+- Introduced the notion of client compatability versions
+  `#2017 <https://github.com/voxel51/fiftyone/pull/2017>`_
+- Extended :meth:`stats() <fiftyone.core.collections.SampleCollection>` to all
+  sample collections `#1940 <https://github.com/voxel51/fiftyone/pull/1940>`_
+- Added support for serializing aggregations
+  `#1911 <https://github.com/voxel51/fiftyone/pull/1911>`_
+- Added :func:`weighted_sample() <fiftyone.utils.random.weighted_sample>`
+  and :func:`balanced_sample() <fiftyone.utils.random.balanced_sample>`
+  utility methods `#1925 <https://github.com/voxel51/fiftyone/pull/1925>`_
+- Added an optional ``new_ids=True`` option to
+  :meth:`Dataset.add_collection() <fiftyone.core.dataset.Dataset.add_collection>`
+  that generates new sample/frame IDs when adding the samples
+  `#1927 <https://github.com/voxel51/fiftyone/pull/1927>`_
+- Added support for the `path` variable in `dataset.yaml` of
+  :ref:`YOLOv5 datasets <YOLOv5Dataset-import>`
+  `#1903 <https://github.com/voxel51/fiftyone/issues/1903>`_
+- Fixed a bug that prevented using 
+  :meth:`set_values() <fiftyone.core.collections.SampleCollection.set_values>`
+  to set frame-level label fields
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed automatic declaration of frame fields when computing embeddings on a
+  frame view `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a regression that caused label ID fields to be returned as
+  `ObjectID` `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that allowed default frame fields to be excluded
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- :class:`ClipsView <fiftyone.core.clips.ClipsView>` instances will now report
+  their `metadata` type as |VideoMetadata|
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed
+  :meth:`load_evaluation_view() <fiftyone.core.dataset.Dataset.load_evaluation_view>`
+  when ``select_fields`` is ``True``
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed boolean field parsing when declaring fields
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that caused nested embedded documents to corrupt datasets
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that prevented assignment of array-valued dynamic attributes
+  to labels `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+
+Annotation
+
+- Added a new :ref:`Label Studio integration! <label-studio-integration>`
+  `#1848 <https://github.com/voxel51/fiftyone/pull/1848>`_
+- Optimized loading CVAT annotations and performing operations on
+  :class:`CVATAnnotationResults <fiftyone.utils.cvat.CVATAnnotationResults>`
+  `#1944 <https://github.com/voxel51/fiftyone/pull/1944>`_
+- Upgraded the :class:`AnnotationAPI <fiftyone.utils.annotations.AnnotationAPI>`
+  interface `#1997 <https://github.com/voxel51/fiftyone/pull/1997>`_
+- Fixed loading group IDs in CVAT video tasks
+  `#1917 <https://github.com/voxel51/fiftyone/pull/1917>`_
+- Fixed uploading to a CVAT project when no label schema is provided
+  `#1926 <https://github.com/voxel51/fiftyone/pull/1926>`_
+
 .. _release-notes-v0.16.5:
 
 FiftyOne 0.16.5

diff --git a/docs/source/user_guide/config.rst b/docs/source/user_guide/config.rst
@@ -76,8 +76,8 @@ FiftyOne supports the configuration options described below:
 +-------------------------------+-------------------------------------+-------------------------------+----------------------------------------------------------------------------------------+
 | `do_not_track`                | `FIFTYONE_DO_NOT_TRACK`             | `False`                       | Controls whether UUID based import and App usage events are tracked.                   |
 +-------------------------------+-------------------------------------+-------------------------------+----------------------------------------------------------------------------------------+
-| `logging_level`               | `FIFTYONE_LOGGING_LEVEL`            | `INFO`                        | Controls FiftyOne's package-wide logging level. Can be any valid ``logging`` level,    |
-|                               |                                     |                               | such as ``logging.INFO``.                                                              |
+| `logging_level`               | `FIFTYONE_LOGGING_LEVEL`            | `INFO`                        | Controls FiftyOne's package-wide logging level. Can be any valid ``logging`` level as  |
+|                               |                                     |                               | a string: ``DEBUG, INFO, WARNING, ERROR, CRITICAL``.                                   |
 +-------------------------------+-------------------------------------+-------------------------------+----------------------------------------------------------------------------------------+
 | `model_zoo_dir`               | `FIFTYONE_MODEL_ZOO_DIR`            | `~/fiftyone/__models__`       | The default directory in which to store models that are downloaded from the            |
 |                               |                                     |                               | :ref:`FiftyOne Model Zoo <model-zoo>`.                                                 |
@@ -417,7 +417,7 @@ New FiftyOne versions occasionally introduce data model changes that require
 database migrations when you :ref:`upgrade <upgrading-fiftyone>` or
 :ref:`downgrade <downgrading-fiftyone>`.
 
-Database upgrades happen automatically in two steps:
+By default, database upgrades happen automatically in two steps:
 
 -   **Database**: when you import FiftyOne for the first time using a newer
     version of the Python package, the database's version is automatically
@@ -430,16 +430,26 @@ Database downgrades must be manually performed. See
 :ref:`this page <downgrading-fiftyone>` for instructions.
 
 You can use the :ref:`fiftyone migrate <cli-fiftyone-migrate>` command to view
-the current versions of your database and datasets:
+the current versions of your client, database, and datasets:
 
 .. code-block:: shell
 
-    # View your client version
-    fiftyone --version
-
-    # View your database and dataset versions
+    # View your client, database, and dataset versions
     fiftyone migrate --info
 
+.. code-block:: text
+
+    Client version: 0.16.6
+    Compatible versions: >=0.16.3,<0.17
+
+    Database version: 0.16.6
+
+    dataset                      version
+    ---------------------------  ---------
+    bdd100k-validation           0.16.5
+    quickstart                   0.16.5
+    ...
+
 Restricting migrations
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -448,10 +458,14 @@ allowed to upgrade/downgrade your FiftyOne database. The default is `True`,
 which means that upgrades are automatically peformed when you connect to your
 database with newer Python client versions.
 
-If you set `database_admin` to `False`, your database will refuse connections
-from your FiftyOne client if its version does not match the database's current
-version, and datasets will refuse migrations to versions other than the
-database's current version.
+If you set `database_admin` to `False`, your client will **never** cause the
+database to be migrated to a new version. Instead, you'll see the following
+behavior:
+
+-   If your client is compatible with the current database version, you will be
+    allowed to connect to the database and use FiftyOne
+-   If your client is not compatible with the current database version, you
+    will see an informative error message when you import the library
 
 You can restrict migrations by adding the following entry to your
 `~/.fiftyone/config.json` file:
@@ -477,6 +491,70 @@ or by setting the following environment variable:
     trigger automatic database upgrades by connecting to the database with
     newer Python client versions.
 
+Coordinating a migration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are working in an environment where multiple services are connecting to
+your MongoDB database at any given time, use this strategy to upgrade your
+deployment:
+
+1.  Ensure that all clients are running without database admin privileges,
+    e.g., by adding this to their `~/.fiftyone/config.json`:
+
+.. code-block:: json
+
+    {
+        "database_admin": false
+    }
+
+2.  Perform a test upgrade of one client and ensure that it is compatible with
+    your current database version:
+
+.. code-block:: shell
+
+    # In a test environment
+    pip install --upgrade fiftyone
+
+    # View client's compatibility info
+    fiftyone migrate --info
+
+.. code-block:: python
+
+    import fiftyone as fo
+
+    # Convince yourself that the new client can load a dataset
+    dataset = fo.load_dataset(...)
+
+3.  Now upgrade the client version used by all services:
+
+.. code-block:: shell
+
+    # In all client environments
+    pip install --upgrade fiftyone
+
+4.  Once all services are running the new client version, upgrade the database
+    with admin privileges:
+
+.. code-block:: shell
+
+    export FIFTYONE_DATABASE_ADMIN=true
+
+    pip install --upgrade fiftyone
+    fiftyone migrate --all
+
+.. note::
+
+    Newly created datasets will always bear the
+    :meth:`version <fiftyone.core.dataset.Dataset.version>` of the Python
+    client that created them, which may differ from your database's version
+    if you are undergoing a migration.
+
+    If the new client's version is not in the compatibility range for the old
+    clients that are still in use, the old clients will not be able to load
+    the new datasets.
+
+    Therefore, it is recommended to upgrade all clients as soon as possible!
+
 .. _configuring-timezone:
 
 Configuring a timezone

diff --git a/docs/source/user_guide/dataset_creation/datasets.rst b/docs/source/user_guide/dataset_creation/datasets.rst
@@ -2225,6 +2225,7 @@ where `dataset.yaml` contains the following information:
 
 .. code-block:: text
 
+    path: <dataset_dir>  # optional
     train: ./images/train/
     val: ./images/val/
 
@@ -2238,7 +2239,9 @@ See `this page <https://docs.ultralytics.com/tutorials/train-custom-datasets>`_
 for a full description of the possible format of `dataset.yaml`. In particular,
 the dataset may contain one or more splits with arbitrary names, as the
 specific split being imported or exported is specified by the `split` argument
-to :class:`fiftyone.utils.yolo.YOLOv5DatasetImporter`.
+to :class:`fiftyone.utils.yolo.YOLOv5DatasetImporter`. Also, `dataset.yaml` can
+be located outside of `<dataset_dir>` as long as the optional `path` is
+provided.
 
 .. note::
 
@@ -2259,7 +2262,9 @@ where `<target>` is the zero-based integer index of the object class label from
 `[0, 1] x [0, 1]`, and `<confidence>` is an optional detection confidence in
 `[0, 1]`.
 
-Unlabeled images have no corresponding TXT file in `labels/`.
+Unlabeled images have no corresponding TXT file in `labels/`. The label file
+path for each image is obtained by replacing `images/` with `labels/` in the
+respective image path.
 
 The image and labels directories for a given split may contain nested
 subfolders of parallelly organized images and labels.