Skip to content

Commit

Permalink
Document PEP 585, 563, 604 and more (python#9763)
Browse files Browse the repository at this point in the history 
Fixes python#8629, fixes python#8523.

This creates a new page to document issues arising from discrepancies
between the runtime and annotations. I felt this was better, rather than
force-fitting things into existing pages and "common issues", for
instance, it prevents us from having to explain PEP 563 in several
different places.

I do still list the runtime errors you'd get in the "common issues" page
to preserve SEO :-)

"String literal types", "Class name forward references", and "Import
cycles" are basically the same as where they were copied over from.

This also factors out the documentation of PEP 604 that I promised when
merging that PR (it seemed pretty verbose, particularly for the "kinds
of types" page). It's also a good place to document PEP 613, when we get
around to supporting that.

Resolves python#9856.

Co-authored-by: hauntsaninja <>
  • Loading branch information
@hauntsaninja
hauntsaninja authored Jan 6, 2021
1 parent 28f92ac commit 6e8c0cd
Show file tree
Hide file tree
Showing 5 changed files with 361 additions and 259 deletions.
5 changes: 5 additions & 0 deletions docs/source/builtin_types.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,8 @@ though they are similar to abstract base classes defined in
:py:mod:`collections.abc` (formerly ``collections``), they are not identical. In
particular, prior to Python 3.9, the built-in collection type objects do not
support indexing.

In Python 3.9 and later, built-in collection type objects support indexing. This
means that you can use built-in classes or those from :py:mod:`collections.abc`
instead of importing from :py:mod:`typing`. See :ref:`generic-builtins` for more
details.
121 changes: 15 additions & 106 deletions docs/source/common_issues.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,21 @@ checking would require a large number of ``assert foo is not None``
checks to be inserted, and you want to minimize the number
of code changes required to get a clean mypy run.

Issues with code at runtime
---------------------------

Idiomatic use of type annotations can sometimes run up against what a given
version of Python considers legal code. These can result in some of the
following errors when trying to run your code:

* ``ImportError`` from circular imports
* ``NameError: name 'X' is not defined`` from forward references
* ``TypeError: 'type' object is not subscriptable`` from types that are not generic at runtime
* ``ImportError`` or ``ModuleNotFoundError`` from use of stub definitions not available at runtime
* ``TypeError: unsupported operand type(s) for |: 'type' and 'type'`` from use of new syntax

For dealing with these, see :ref:`runtime_troubles`.

Mypy runs are slow
------------------

Expand Down Expand Up @@ -499,112 +514,6 @@ to see the types of all local variables at once. Example:
run your code. Both are always available and you don't need to import
them.


.. _import-cycles:

Import cycles
-------------

An import cycle occurs where module A imports module B and module B
imports module A (perhaps indirectly, e.g. ``A -> B -> C -> A``).
Sometimes in order to add type annotations you have to add extra
imports to a module and those imports cause cycles that didn't exist
before. If those cycles become a problem when running your program,
there's a trick: if the import is only needed for type annotations in
forward references (string literals) or comments, you can write the
imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime.
Example:

File ``foo.py``:

.. code-block:: python
from typing import List, TYPE_CHECKING
if TYPE_CHECKING:
import bar
def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]':
return [arg]
File ``bar.py``:

.. code-block:: python
from typing import List
from foo import listify
class BarClass:
def listifyme(self) -> 'List[BarClass]':
return listify(self)
.. note::

The :py:data:`~typing.TYPE_CHECKING` constant defined by the :py:mod:`typing` module
is ``False`` at runtime but ``True`` while type checking.

Python 3.5.1 doesn't have :py:data:`~typing.TYPE_CHECKING`. An alternative is
to define a constant named ``MYPY`` that has the value ``False``
at runtime. Mypy considers it to be ``True`` when type checking.
Here's the above example modified to use ``MYPY``:

.. code-block:: python
from typing import List
MYPY = False
if MYPY:
import bar
def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]':
return [arg]
.. _not-generic-runtime:

Using classes that are generic in stubs but not at runtime
----------------------------------------------------------

Some classes are declared as generic in stubs, but not at runtime. Examples
in the standard library include :py:class:`os.PathLike` and :py:class:`queue.Queue`.
Subscripting such a class will result in a runtime error:

.. code-block:: python
from queue import Queue
class Tasks(Queue[str]): # TypeError: 'type' object is not subscriptable
...
results: Queue[int] = Queue() # TypeError: 'type' object is not subscriptable
To avoid these errors while still having precise types you can either use
string literal types or :py:data:`~typing.TYPE_CHECKING`:

.. code-block:: python
from queue import Queue
from typing import TYPE_CHECKING
if TYPE_CHECKING:
BaseQueue = Queue[str] # this is only processed by mypy
else:
BaseQueue = Queue # this is not seen by mypy but will be executed at runtime.
class Tasks(BaseQueue): # OK
...
results: 'Queue[int]' = Queue() # OK
If you are running Python 3.7+ you can use ``from __future__ import annotations``
as a (nicer) alternative to string quotes, read more in :pep:`563`. For example:

.. code-block:: python
from __future__ import annotations
from queue import Queue
results: Queue[int] = Queue() # This works at runtime
.. _silencing-linters:

Silencing linters
Expand Down
1 change: 1 addition & 0 deletions docs/source/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Mypy is a static type checker for Python 3 and Python 2.7.
type_inference_and_annotations
kinds_of_types
class_basics
runtime_troubles
protocols
python2
dynamic_typing
Expand Down
161 changes: 8 additions & 153 deletions docs/source/kinds_of_types.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -243,93 +243,24 @@ more specific type:

.. _alternative_union_syntax:

Alternative union syntax
------------------------
X | Y syntax for Unions
-----------------------

`PEP 604 <https://www.python.org/dev/peps/pep-0604/>`_ introduced an alternative way
for writing union types. Starting with **Python 3.10** it is possible to write
``Union[int, str]`` as ``int | str``. Any of the following options is possible
:pep:`604` introduced an alternative way for spelling union types. In Python
3.10 and later, you can write ``Union[int, str]`` as ``int | str``. It is
possible to use this syntax in versions of Python where it isn't supported by
the runtime with some limitations, see :ref:`runtime_troubles`.

.. code-block:: python
from typing import List
# Use as Union
t1: int | str # equivalent to Union[int, str]
# Use as Optional
t2: int | None # equivalent to Optional[int]
# Use in generics
t3: List[int | str] # equivalent to List[Union[int, str]]
# Use in type aliases
T4 = int | None
x: T4
# Quoted variable annotations
t5: "int | str"
# Quoted function annotations
def f(t6: "int | str") -> None: ...
# Type comments
t6 = 42 # type: int | str
It is possible to use most of these even for earlier versions. However there are some
limitations to be aware of.

.. _alternative_union_syntax_stub_files:

Stub files
""""""""""

All options are supported, regardless of the Python version the project uses.

.. _alternative_union_syntax_37:

Python 3.7 - 3.9
""""""""""""""""

It is necessary to add ``from __future__ import annotations`` to delay the evaluation
of type annotations. Not using it would result in a ``TypeError``.
This does not apply for **type comments**, **quoted function** and **quoted variable** annotations,
as those also work for earlier versions, see :ref:`below <alternative_union_syntax_older_version>`.

.. warning::

Type aliases are **NOT** supported! Those result in a ``TypeError`` regardless
if the evaluation of type annotations is delayed.

Dynamic evaluation of annotations is **NOT** possible (e.g. ``typing.get_type_hints`` and ``eval``).
See `note PEP 604 <https://www.python.org/dev/peps/pep-0604/#change-only-pep-484-type-hints-to-accept-the-syntax-type1-type2>`_.
Use ``typing.Union`` or **Python 3.10** instead if you need those!

.. code-block:: python
from __future__ import annotations
t1: int | None
# Type aliases
T2 = int | None # TypeError!
.. _alternative_union_syntax_older_version:

Older versions
""""""""""""""

+------------------------------------------+-----------+-----------+-----------+
| Python Version | 3.6 | 3.0 - 3.5 | 2.7 |
+==========================================+===========+===========+===========+
| Type comments | yes | yes | yes |
+------------------------------------------+-----------+-----------+-----------+
| Quoted function annotations | yes | yes | |
+------------------------------------------+-----------+-----------+-----------+
| Quoted variable annotations | yes | | |
+------------------------------------------+-----------+-----------+-----------+
| Everything else | | | |
+------------------------------------------+-----------+-----------+-----------+
# Usable in type comments
t3 = 42 # type: int | str
.. _strict_optional:

Expand Down Expand Up @@ -565,82 +496,6 @@ valid for any type, but it's much more
useful for a programmer who is reading the code. This also makes
it easier to migrate to strict ``None`` checking in the future.

Class name forward references
*****************************

Python does not allow references to a class object before the class is
defined. Thus this code does not work as expected:

.. code-block:: python
def f(x: A) -> None: # Error: Name A not defined
...
class A:
...
In cases like these you can enter the type as a string literal — this
is a *forward reference*:

.. code-block:: python
def f(x: 'A') -> None: # OK
...
class A:
...
Starting from Python 3.7 (:pep:`563`), you can add the special import ``from __future__ import annotations``,
which makes the use of string literals in annotations unnecessary:

.. code-block:: python
from __future__ import annotations
def f(x: A) -> None: # OK
...
class A:
...
.. note::

Even with the ``__future__`` import, there are some scenarios that could still
require string literals, typically involving use of forward references or generics in:

* :ref:`type aliases <type-aliases>`;
* :ref:`casts <casts>`;
* type definitions (see :py:class:`~typing.TypeVar`, :py:func:`~typing.NewType`, :py:class:`~typing.NamedTuple`);
* base classes.

.. code-block:: python
# base class example
class A(Tuple['B', 'C']): ... # OK
class B: ...
class C: ...
Of course, instead of using a string literal type or special import, you could move the
function definition after the class definition. This is not always
desirable or even possible, though.

Any type can be entered as a string literal, and you can combine
string-literal types with non-string-literal types freely:

.. code-block:: python
def f(a: List['A']) -> None: ... # OK
def g(n: 'int') -> None: ... # OK, though not useful
class A: pass
String literal types are never needed in ``# type:`` comments and :ref:`stub files <stub-files>`.

String literal types must be defined (or imported) later *in the same
module*. They cannot be used to leave cross-module references
unresolved. (For dealing with import cycles, see
:ref:`import-cycles`.)

.. _type-aliases:

Type aliases
Expand Down
Loading

0 comments on commit 6e8c0cd

Please sign in to comment.