Release v0.16.6 (voxel51#2008)

* bumps, release notes * add to release notes * linting * updating pkg versions * removing unnecessary pull * move App build to end * updating release notes * always migrate when user is admin * moving legacy troubleshooting to another page * adding docs on coordinating migrations * adding helpful error when a dataset fails to load * pinning max requirement * using more cloud-friendly layout * adding upgrade note * update create docs pr action Co-authored-by: brimoor <brimoor@umich.edu>
shalevy1 · Aug 25, 2022 · 43298f6 · 43298f6
1 parent c11811a
commit 43298f6
Show file tree

Hide file tree

Showing 12 changed files with 287 additions and 76 deletions.
diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml
@@ -95,7 +95,7 @@ jobs:
           cp -r docs-download/ source/docs/fiftyone/
           rm -r docs-download/
       - name: Create website PR
-        uses: peter-evans/create-pull-request@v2
+        uses: peter-evans/create-pull-request@v4
         with:
           branch: fiftyone-docs
           commit-message: |

diff --git a/docs/source/getting_started/install.rst b/docs/source/getting_started/install.rst
@@ -267,7 +267,7 @@ Here's the workflow for downgrading to an older version of FiftyOne:
 .. code-block:: shell
 
     # The version that you wish to downgrade to
-    VERSION=0.14.4  # for example
+    VERSION=0.15.1
 
     # Migrate the database
     fiftyone migrate --all -v $VERSION
@@ -283,29 +283,16 @@ your ``fiftyone`` package without first running
 :ref:`fiftyone migrate <cli-fiftyone-migrate>`, don't worry, you simply need to
 reinstall the newer version of FiftyOne and then follow these instructions.
 
+See :ref:`this page <troubleshooting-downgrades>` if you need to install
+FiftyOne v0.7.3 or earlier.
+
 .. note::
 
   If you are working with a
   :ref:`custom/shared MongoDB database <configuring-mongodb-connection>`, you
   can use :ref:`database admin privileges <database-migrations>` to control
   which clients are allowed to downgrade your FiftyOne deployment.
 
-.. note::
-
-    The :ref:`fiftyone migrate <cli-fiftyone-migrate>` command was introduced
-    in FiftyOne v0.7.3. If you would like to downgrade from a FiftyOne version
-    prior to v0.7.3 (to a yet older version), then you will first need to
-    *upgrade* to v0.7.3 or later and then follow the instructions above.
-
-.. note::
-
-    To install a FiftyOne version **prior to v0.7.0**, you must add an
-    ``--index`` option to ``pip install``:
-
-    .. code-block:: shell
-
-        pip install --index https://pypi.voxel51.com fiftyone==<version>
-
 .. _uninstalling-fiftyone:
 
 Uninstalling FiftyOne

diff --git a/docs/source/getting_started/troubleshooting.rst b/docs/source/getting_started/troubleshooting.rst
@@ -165,6 +165,32 @@ then FiftyOne's database service will attempt to start up on import using the
 MongoDB distribution provided by `fiftyone-db`. If the database fails to start,
 importing `fiftyone` will result in exceptions being raised.
 
+.. _troubleshooting-downgrades:
+
+Downgrading to old versions
+---------------------------
+
+The :ref:`fiftyone migrate <cli-fiftyone-migrate>` command was introduced in
+FiftyOne v0.7.3. If you would like to downgrade from a FiftyOne version
+prior to v0.7.3 (to a yet older version), then you will first need to
+:ref:`upgrade <upgrading-fiftyone>` to v0.7.3 or later and then
+:ref:`downgrade <downgrading-fiftyone>`:
+
+.. code-block:: shell
+
+  # The version that you wish to downgrade to
+  VERSION=0.7.0
+
+  pip install fiftyone==0.7.3
+  fiftyone migrate --all -v $VERSION
+  pip install fiftyone==$VERSION
+
+To install a FiftyOne version prior to v0.7.0, you must add ``--index``:
+
+.. code-block:: shell
+
+    pip install --index https://pypi.voxel51.com fiftyone==<version>
+
 .. _troubleshooting-mongodb-exits:
 
 Database exits
@@ -176,7 +202,6 @@ Running `ulimit -n 64000` should resolve the issue. 64,000 is the recommended
 open file limit.  MongoDB has full documentation on the issue
 `here <https://docs.mongodb.com/manual/reference/ulimit/>`_. 
 
-
 .. _troubleshooting-mongodb-linux:
 
 Troubleshooting Linux imports

diff --git a/docs/source/release-notes.rst b/docs/source/release-notes.rst
@@ -3,6 +3,101 @@ FiftyOne Release Notes
 
 .. default-role:: code
 
+.. _release-notes-v0.16.6:
+
+FiftyOne 0.16.6
+---------------
+*Released August 25, 2022*
+
+App
+
+- Fixed a bug that caused the App to break when sample tags contained `.`
+  `#1924 <https://github.com/voxel51/fiftyone/pull/1924>`_
+- Fixed search results alignment
+  `#1930 <https://github.com/voxel51/fiftyone/pull/1930>`_
+- Fixed App refreshes after view changes had occurred from the view bar
+  `#1931 <https://github.com/voxel51/fiftyone/pull/1931>`_
+- Fixed mask targets rendering in the tooltip
+  `#1943 <https://github.com/voxel51/fiftyone/pull/1943>`_
+  `#1949 <https://github.com/voxel51/fiftyone/pull/1949>`_
+- Fixed classification confusion matrix connections
+  `#1967 <https://github.com/voxel51/fiftyone/pull/1967>`_
+
+Core
+
+- Added a save context that enables
+  :ref:`efficient batch edits <efficient-batch-edits>` of datasets and views
+  `#1727 <https://github.com/voxel51/fiftyone/pull/1727>`_
+- Added Plotly v5 support
+  `#1981 <https://github.com/voxel51/fiftyone/pull/1981>`_
+- Added a :ref:`quantiles aggregation <aggregations-quantiles>`
+  `#1937 <https://github.com/voxel51/fiftyone/pull/1937>`_
+- Added support for writing transformed images/videos to new locations in the
+  :func:`transform_images() <fiftyone.utils.image.transform_images>` and
+  :func:`transform_videos() <fiftyone.utils.video.transform_videos>` functions
+  `#2007 <https://github.com/voxel51/fiftyone/pull/2007>`_
+- Added support for configuring the
+  :ref:`package-wide logging level <configuring-fiftyone>`
+  `#2009 <https://github.com/voxel51/fiftyone/pull/2009>`_
+- Added more full-featured support for serializing and deserializing datasets,
+  views, and samples via `to_dict()` and `from_dict()`
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Added support for dynamic attributes when performing coerced exports
+  `#1993 <https://github.com/voxel51/fiftyone/pull/1993>`_
+- Introduced the notion of client compatability versions
+  `#2017 <https://github.com/voxel51/fiftyone/pull/2017>`_
+- Extended :meth:`stats() <fiftyone.core.collections.SampleCollection>` to all
+  sample collections `#1940 <https://github.com/voxel51/fiftyone/pull/1940>`_
+- Added support for serializing aggregations
+  `#1911 <https://github.com/voxel51/fiftyone/pull/1911>`_
+- Added :func:`weighted_sample() <fiftyone.utils.random.weighted_sample>`
+  and :func:`balanced_sample() <fiftyone.utils.random.balanced_sample>`
+  utility methods `#1925 <https://github.com/voxel51/fiftyone/pull/1925>`_
+- Added an optional ``new_ids=True`` option to
+  :meth:`Dataset.add_collection() <fiftyone.core.dataset.Dataset.add_collection>`
+  that generates new sample/frame IDs when adding the samples
+  `#1927 <https://github.com/voxel51/fiftyone/pull/1927>`_
+- Added support for the `path` variable in `dataset.yaml` of
+  :ref:`YOLOv5 datasets <YOLOv5Dataset-import>`
+  `#1903 <https://github.com/voxel51/fiftyone/issues/1903>`_
+- Fixed a bug that prevented using 
+  :meth:`set_values() <fiftyone.core.collections.SampleCollection.set_values>`
+  to set frame-level label fields
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed automatic declaration of frame fields when computing embeddings on a
+  frame view `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a regression that caused label ID fields to be returned as
+  `ObjectID` `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that allowed default frame fields to be excluded
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- :class:`ClipsView <fiftyone.core.clips.ClipsView>` instances will now report
+  their `metadata` type as |VideoMetadata|
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed
+  :meth:`load_evaluation_view() <fiftyone.core.dataset.Dataset.load_evaluation_view>`
+  when ``select_fields`` is ``True``
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed boolean field parsing when declaring fields
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that caused nested embedded documents to corrupt datasets
+  `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+- Fixed a bug that prevented assignment of array-valued dynamic attributes
+  to labels `#1922 <https://github.com/voxel51/fiftyone/pull/1922>`_
+
+Annotation
+
+- Added a new :ref:`Label Studio integration! <label-studio-integration>`
+  `#1848 <https://github.com/voxel51/fiftyone/pull/1848>`_
+- Optimized loading CVAT annotations and performing operations on
+  :class:`CVATAnnotationResults <fiftyone.utils.cvat.CVATAnnotationResults>`
+  `#1944 <https://github.com/voxel51/fiftyone/pull/1944>`_
+- Upgraded the :class:`AnnotationAPI <fiftyone.utils.annotations.AnnotationAPI>`
+  interface `#1997 <https://github.com/voxel51/fiftyone/pull/1997>`_
+- Fixed loading group IDs in CVAT video tasks
+  `#1917 <https://github.com/voxel51/fiftyone/pull/1917>`_
+- Fixed uploading to a CVAT project when no label schema is provided
+  `#1926 <https://github.com/voxel51/fiftyone/pull/1926>`_
+
 .. _release-notes-v0.16.5:
 
 FiftyOne 0.16.5

diff --git a/docs/source/user_guide/config.rst b/docs/source/user_guide/config.rst
@@ -417,7 +417,7 @@ New FiftyOne versions occasionally introduce data model changes that require
 database migrations when you :ref:`upgrade <upgrading-fiftyone>` or
 :ref:`downgrade <downgrading-fiftyone>`.
 
-Database upgrades happen automatically in two steps:
+By default, database upgrades happen automatically in two steps:
 
 -   **Database**: when you import FiftyOne for the first time using a newer
     version of the Python package, the database's version is automatically
@@ -430,16 +430,26 @@ Database downgrades must be manually performed. See
 :ref:`this page <downgrading-fiftyone>` for instructions.
 
 You can use the :ref:`fiftyone migrate <cli-fiftyone-migrate>` command to view
-the current versions of your database and datasets:
+the current versions of your client, database, and datasets:
 
 .. code-block:: shell
 
-    # View your client version
-    fiftyone --version
-
-    # View your database and dataset versions
+    # View your client, database, and dataset versions
     fiftyone migrate --info
 
+.. code-block:: text
+
+    Client version: 0.16.6
+    Compatible versions: >=0.16.3,<0.17
+
+    Database version: 0.16.6
+
+    dataset                      version
+    ---------------------------  ---------
+    bdd100k-validation           0.16.5
+    quickstart                   0.16.5
+    ...
+
 Restricting migrations
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -448,10 +458,14 @@ allowed to upgrade/downgrade your FiftyOne database. The default is `True`,
 which means that upgrades are automatically peformed when you connect to your
 database with newer Python client versions.
 
-If you set `database_admin` to `False`, your database will refuse connections
-from your FiftyOne client if its version does not match the database's current
-version, and datasets will refuse migrations to versions other than the
-database's current version.
+If you set `database_admin` to `False`, your client will **never** cause the
+database to be migrated to a new version. Instead, you'll see the following
+behavior:
+
+-   If your client is compatible with the current database version, you will be
+    allowed to connect to the database and use FiftyOne
+-   If your client is not compatible with the current database version, you
+    will see an informative error message when you import the library
 
 You can restrict migrations by adding the following entry to your
 `~/.fiftyone/config.json` file:
@@ -477,6 +491,70 @@ or by setting the following environment variable:
     trigger automatic database upgrades by connecting to the database with
     newer Python client versions.
 
+Coordinating a migration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are working in an environment where multiple services are connecting to
+your MongoDB database at any given time, use this strategy to upgrade your
+deployment:
+
+1.  Ensure that all clients are running without database admin privileges,
+    e.g., by adding this to their `~/.fiftyone/config.json`:
+
+.. code-block:: json
+
+    {
+        "database_admin": false
+    }
+
+2.  Perform a test upgrade of one client and ensure that it is compatible with
+    your current database version:
+
+.. code-block:: shell
+
+    # In a test environment
+    pip install --upgrade fiftyone
+
+    # View client's compatibility info
+    fiftyone migrate --info
+
+.. code-block:: python
+
+    import fiftyone as fo
+
+    # Convince yourself that the new client can load a dataset
+    dataset = fo.load_dataset(...)
+
+3.  Now upgrade the client version used by all services:
+
+.. code-block:: shell
+
+    # In all client environments
+    pip install --upgrade fiftyone
+
+4.  Once all services are running the new client version, upgrade the database
+    with admin privileges:
+
+.. code-block:: shell
+
+    export FIFTYONE_DATABASE_ADMIN=true
+
+    pip install --upgrade fiftyone
+    fiftyone migrate --all
+
+.. note::
+
+    Newly created datasets will always bear the
+    :meth:`version <fiftyone.core.dataset.Dataset.version>` of the Python
+    client that created them, which may differ from your database's version
+    if you are undergoing a migration.
+
+    If the new client's version is not in the compatibility range for the old
+    clients that are still in use, the old clients will not be able to load
+    the new datasets.
+
+    Therefore, it is recommended to upgrade all clients as soon as possible!
+
 .. _configuring-timezone:
 
 Configuring a timezone

diff --git a/fiftyone/core/dataset.py b/fiftyone/core/dataset.py
@@ -4910,6 +4910,25 @@ def _load_dataset(name, virtual=False):
     if not virtual:
         fomi.migrate_dataset_if_necessary(name)
 
+    try:
+        return _do_load_dataset(name, virtual=virtual)
+    except Exception as e:
+        try:
+            version = fomi.get_dataset_revision(name)
+        except:
+            raise e
+
+        if version != focn.VERSION:
+            raise ValueError(
+                "Failed to load dataset '%s' from v%s using client v%s. "
+                "You may need to upgrade your client"
+                % (name, version, focn.VERSION)
+            ) from e
+
+        raise e
+
+
+def _do_load_dataset(name, virtual=False):
     try:
         # pylint: disable=no-member
         dataset_doc = foo.DatasetDocument.objects.get(name=name)
@@ -4968,6 +4987,7 @@ def _load_dataset(name, virtual=False):
             ]
 
     dataset_doc.save()
+
     return dataset_doc, sample_doc_cls, frame_doc_cls