Merge branch 'trunk' into 8079-no-gmpy

cwaldbieser · Oct 20, 2016 · 2208981 · 2208981
2 parents df1589b + fdf4270
commit 2208981
Show file tree

Hide file tree

Showing 154 changed files with 3,367 additions and 2,745 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,5 @@
 *.egg-info/
+.eggs/
 *.o
 *.py[co]
 *.so

diff --git a/.travis.yml b/.travis.yml
@@ -55,14 +55,30 @@ matrix:
       # see: https://docs.travis-ci.com/user/languages/objective-c/#Supported-OS-X-iOS-SDK-versions
       osx_image: xcode7.1
       python: 2.7
-      env: TOXENV=py27-alldeps-withcov-posix,codecov-publish
+      env: TOXENV=py27-alldeps-withcov-macos1010,codecov-publish
+    #
+    # OSX 10.11
+    #
+    - language: generic
+      os: osx
+      # 8 is OS X 10.11.x
+      # see: https://docs.travis-ci.com/user/languages/objective-c/#Supported-OS-X-iOS-SDK-versions
+      osx_image: xcode8
+      env: TOXENV=py35-alldeps-withcov-macos,codecov-publish
+    - language: generic
+      os: osx
+      osx_image: xcode8
+      # Test cfreactor explicitly
+      env:
+        - TOXENV=py35-alldeps-withcov-macos,codecov-publish
+        - TWISTED_REACTOR=cf
+
 
   allow_failures:
-    # For now, OSX run on Travis-CI has a few failures.
-    - os: osx
     # twistedchecker job was introduce as an experimental job.
     # Once it is stable we can enforce it
     - env: TOXENV=txchecker-travis
+    - osx_image: xcode7.1
 
 
 addons:
@@ -75,6 +91,7 @@ addons:
 cache:
   directories:
     - $HOME/.cache/pip
+    - $HOME/.pyenv
 
 
 install:

diff --git a/.travis/install.sh b/.travis/install.sh
@@ -8,6 +8,20 @@ if [[ "$(uname -s)" == 'Darwin' ]]; then
     python -m pip install --user virtualenv
     python -m virtualenv ~/.venv
     source ~/.venv/bin/activate
+
+    if [[ "${TOXENV}" == "py35-alldeps-withcov-macos,codecov-publish" ]]; then
+
+        brew update;
+        brew upgrade openssl;
+        brew install pyenv;
+        PYENV_ROOT="$HOME/.pyenv";
+        PATH="$PYENV_ROOT/bin:$PATH";
+        eval "$(pyenv init -)";
+        pyenv install -s 3.5.2;
+        pyenv global system 3.5.2;
+        pyenv rehash;
+
+    fi
 fi
 
 # Temporary workaround for https://github.com/pypa/setuptools/issues/776;

diff --git a/.travis/run.sh b/.travis/run.sh
@@ -5,6 +5,13 @@ set -x
 if [[ "$(uname -s)" == "Darwin" ]]; then
     # Initialize the virtualenv created at install time.
     source ~/.venv/bin/activate
+
+    if [[ "${TOXENV}" == "py35-alldeps-withcov-macos,codecov-publish" ]]; then
+        # Add pyenv path
+        PYENV_ROOT="$HOME/.pyenv";
+        PATH="$PYENV_ROOT/bin:$PATH";
+        eval "$(pyenv init -)";
+    fi
 fi
 
 tox -- $TOX_FLAGS
diff --git a/.travis/twistedchecker-trunk-diff.sh b/.travis/twistedchecker-trunk-diff.sh
@@ -25,7 +25,7 @@ if [ ! -d "$target" ]; then
 fi
 
 # Make sure we have trunk on the local repo.
-git fetch origin trunk:refs/remotes/origin/trunk
+git fetch origin +refs/heads/trunk:refs/remotes/origin/trunk
 
 mkdir -p build/
 twistedchecker -f parseable $target > build/twistedchecker-branch.report

diff --git a/NEWS b/NEWS
@@ -1677,7 +1677,7 @@ Improved Documentation
  - Twisted Trial's how-to documentation now has a link to Twisted's
    contribution guidelines and has been reformatted. (#7475)
  - Fixed a path error in the make.bat file for building Sphinx
-   documentation, so that it is now possible to build the documenation
+   documentation, so that it is now possible to build the documentation
    using make.bat on Windows. (#7542)
 
 Deprecations and Removals
@@ -4095,7 +4095,7 @@ Bugfixes
    --inplace' will compile the extension and cause the test to both
    run and pass. (#4331)
  - twisted.python.logfile.LogFile now raises a descriptive exception
-   when passed a log  directoy which does not exist. (#4701)
+   when passed a log  directory which does not exist. (#4701)
  - Fixed a bug where Inotify will fail to add a filepatch to watchlist
    after it has been added/ignored previously. (#4708)
  - IPv4Address and UNIXAddress object comparison operators fixed
@@ -4189,7 +4189,7 @@ Bugfixes
 Improved Documentation
 ----------------------
  - Changed the man page for ckeygen to accurately reflect what it
-   does, and corrected its synposis so that a second "ckeygen" is not
+   does, and corrected its synopsis so that a second "ckeygen" is not
    a required part of the ckeygen command line.  (#4738)
 
 Other
@@ -4824,7 +4824,7 @@ Twisted Conch 10.1.0 (2010-06-27)
 Features
 --------
  - twisted.conch.ssh.transport.SSHTransportBase now allows supported
-   ssh protocol versions to be overriden. (#4428)
+   ssh protocol versions to be overridden. (#4428)
 
 Bugfixes
 --------
@@ -4998,7 +4998,7 @@ Features
    zombie processes in this case. (#2078)
 
  - High-throughput applications based on Perspective Broker should now
-   run noticably faster thanks to the use of a more efficient decoding
+   run noticeably faster thanks to the use of a more efficient decoding
    function in Twisted Spread. (#2310)
 
  - Documentation for trac-post-commit-hook functionality in svn-dev
@@ -6368,9 +6368,9 @@ Features
 Fixes
 -----
  - Handle empty passwords in SMTP auth. (#2521)
- - Fix IMAP4Client's parsing of literals which are not preceeded by whitespace.
+ - Fix IMAP4Client's parsing of literals which are not preceded by whitespace.
    (#2700)
- - Handle MX lookup suceeding without answers. (#2807)
+ - Handle MX lookup succeeding without answers. (#2807)
  - Fix issues with aliases(5) process support. (#2729)
 
 Misc

diff --git a/bin/admin/change-versions b/bin/admin/change-versions
diff --git a/bin/trial b/bin/trial
diff --git a/docs/core/development/policy/compatibility-policy.rst b/docs/core/development/policy/compatibility-policy.rst
@@ -130,8 +130,10 @@ Test Changes
 No code or data in a test package should be imported or used by a non-test package within Twisted.
 By doing so, there's no chance anything could access these objects by going through the public API.
 
-Test code and test helpers are considered private API and it should be imported outside
+Test code and test helpers are considered private API and should not be imported outside
 of the Twisted testing infrastructure.
+As an exception to this, :api:`twisted.test.proto_helpers` is considered a public API
+(see `#6435 <https://twistedmatrix.com/trac/ticket/6435>`_ for more discussion).
 
 
 Private Changes

diff --git a/docs/core/development/policy/release-process.rst b/docs/core/development/policy/release-process.rst
@@ -107,9 +107,9 @@ How to do a pre-release
 
 1. Check buildbot to make sure all supported platforms are green (wait for pending builds if necessary).
 2. If a previously supported platform does not currently have a buildbot, move from supported platforms to "expected to work" in ``INSTALL.rst``.
-3. In your Git-SVN-enabled Git repo, fetch and check out the new release branch.
-4. Run ``./bin/admin/change-versions --prerelease``
-5. Commit the changes made by change-versions
+3. In your Git repo, fetch and check out the new release branch.
+4. Run ``python -m incremental.update Twisted --newversion $RELEASErc1``
+5. Commit the changes made by Incremental.
 6. Run ``./bin/admin/build-news .``
 7. Commit the changes made by build-news - this automatically removes the NEWS topfiles (see #4315)
 8. Bump copyright dates in ``LICENSE``, ``twisted/copyright.py``, and ``README.rst`` if required
@@ -138,7 +138,7 @@ Pre-release announcement
 
 The pre-release announcement should mention the important changes since the last release, and exhort readers to test this pre-release.
 
-Here's what the $RELEASEpre1 release announcement might look like::
+Here's what the $RELEASErc1 release announcement might look like::
 
     Live from PyCon Atlanta, I'm pleased to herald the approaching
     footsteps of the $API release.
@@ -175,16 +175,18 @@ Prepare the branch
 ~~~~~~~~~~~~~~~~~~
 
 1. Have the release branch, previously used to generate a pre-release, checked out
-2. Run ``./bin/admin/change-versions``
-3. Add the quote of the release to the ``README.rst``
-4. Make a new quote file for the next version
+2. Run ``python -m incremental.update Twisted``.
+3. Revert the prerelease newsfile changes, in order.
+4. Run ``./bin/admin/build-news .`` to make the final newsfile.
+5. Add the quote of the release to the ``README.rst``
+6. Make a new quote file for the next version
 
    - ``git mv docs/fun/Twisted.Quotes docs/historic/Quotes/Twisted-$API; echo '' > docs/fun/Twisted.Quotes; git add docs/fun/Twisted.Quotes``
 
-5. Commit the version and ``README.rst`` changes.
-6. Submit the ticket for review
-7. Pause until the ticket is reviewed and accepted.
-8. Tag the release.
+7. Commit the version and ``README.rst`` changes.
+8. Submit the ticket for review
+9. Pause until the ticket is reviewed and accepted.
+10. Tag the release.
 
    - ``git tag -s twisted-$RELEASE -m "Tag $RELEASE release"``
    - ``git push --tags``
@@ -214,8 +216,7 @@ Update documentation
 
 1. Get the dependencies
 
-  - Pydoctor (use the branch "twisted" from https://github.com/twisted/pydoctor)
-  - Epydoc
+  - PyDoctor (from PyPI)
 
 2. Build the documentation
 
@@ -281,9 +282,11 @@ Announce
   - Twitter, if you feel like it
   - ``#twisted`` topic on IRC (you'll need ops)
 
-5. Merge the release branch into trunk, closing the release ticket at the same time.
-6. Close the release milestone (which should have no tickets in it).
-7. Open a milestone for the next release.
+5. Run ``python -m incremental Twisted --dev`` to add a `dev0` postfix.
+6. Commit the dev0 update change.
+7. Merge the release branch into trunk, closing the release ticket at the same time.
+8. Close the release milestone (which should have no tickets in it).
+9. Open a milestone for the next release.
 
 
 Release announcement
@@ -365,7 +368,7 @@ This section goes over doing these "point" releases.
 3. Cherry-pick the merge commits that merge the bugfixes into trunk, onto the new release branch.
 4. Go through the rest of the process for a full release from "How to do a pre-release", merging the release branch into trunk as normal as the end of the process.
 
-  - Instead of just ``--prerelease`` when running the change-versions script, add the patch flag, making it ``--patch --prerelease``.
+  - Instead of just ``--rc`` when running the change-versions script, add the patch flag, making it ``--patch --rc``.
   - Instead of waiting a week, a shorter pause is acceptable for a patch release.
 
 

diff --git a/docs/core/howto/defer.rst b/docs/core/howto/defer.rst
@@ -332,6 +332,7 @@ Now ``isValidUser`` could be either ``synchronousIsValidUser`` or ``asynchronous
 
 It is also possible to modify ``synchronousIsValidUser`` to return a Deferred, see :doc:`Generating Deferreds <gendefer>` for more information.
 
+.. _core-howto-defer-deferreds-cancellation:
 
 Cancellation
 ------------
@@ -456,8 +457,75 @@ Now if someone calls ``cancel()`` on the ``Deferred`` returned from ``HTTPClient
 Care should be taken not to ``callback()`` a Deferred that has already been cancelled.
 
 
+.. _core-howto-defer-deferreds-timeouts:
+
+Timeouts
+--------
+
+Timeouts are a special case of :ref:`Cancellation <core-howto-defer-deferreds-cancellation>`.
+Let's say we have a :api:`twisted.internet.defer.Deferred <Deferred>` representing a task that may take a long time.
+We want to put an upper bound on that task, so we want the :api:`twisted.internet.defer.Deferred <Deferred>` to time
+out X seconds in the future.
+
+A convenient API to do so is :api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>`.
+By default, it will fail with a :api:`twisted.internet.defer.TimeoutError <TimeoutError>` if the :api:`twisted.internet.defer.Deferred <Deferred>` hasn't fired (with either an errback or a callback) within ``timeout`` seconds.
+
+.. code-block:: python
+
+    import random
+    from twisted.internet import task
+
+    def f():
+        return "Hopefully this will be called in 3 seconds or less"
+
+    def main(reactor):
+        delay = random.uniform(1, 5)
+
+        def called(result):
+            print("{0} seconds later:".format(delay), result)
+
+        d = task.deferLater(reactor, delay, f)
+        d.addTimeout(3, reactor).addBoth(called)
+
+        return d
+
+    # f() will be timed out if the random delay is greater than 3 seconds
+    task.react(main)
+
+
+:api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>` uses the :api:`twisted.internet.defer.Deferred.cancel <Deferred.cancel>` function under the hood, but can distinguish between a user's call to :api:`twisted.internet.defer.Deferred.cancel <Deferred.cancel>` and a cancellation due to a timeout.
+By default, :api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>` translates a :api:`twisted.internet.defer.CancelledError <CancelledError>` produced by the timeout into a :api:`twisted.internet.error.TimeoutError <TimeoutError>`.
+
+However, if you provided a custom :ref:`cancellation <core-howto-defer-deferreds-cancellation>` when creating the :api:`twisted.internet.defer.Deferred <Deferred>`, then cancelling it may not produce a :api:`twisted.internet.defer.CancelledError <CancelledError>`.  In this case, the default behavior of :api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>` is to preserve whatever callback or errback value your custom cancellation function produced.  This can be useful if, for instance, a cancellation or timeout should produce a default value instead of an error.
+
+:api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>` also takes an optional callable ``onTimeoutCancel`` which is called immediately after the deferred times out.  ``onTimeoutCancel`` is not called if it the deferred is otherwise cancelled before the timeout. It takes an arbitrary value, which is the value of the deferred at that exact time (probably a :api:`twisted.internet.defer.CancelledError <CancelledError>` :api:`twisted.python.failure.Failure <Failure>`), and the ``timeout``.  This can be useful if, for instance, the cancellation or timeout does not result in an error but you want to log the timeout anyway.  It can also be used to alter the return value.
+
+.. code-block:: python
+
+    from twisted.internet import task, defer
+
+    def logTimeout(result, timeout):
+        print("Got {0!r} but actually timed out after {1} seconds".format(
+            result, timeout))
+        return result + " (timed out)"
+
+    def main(reactor):
+        # generate a deferred with a custom canceller function, and never
+        # never callback or errback it to guarantee it gets timed out
+        d = defer.Deferred(lambda c: c.callback("Everything's ok!"))
+        d.addTimeout(2, reactor, onTimeoutCancel=logTimeout)
+        d.addBoth(print)
+        return d
+
+    task.react(main)
+
+
+Note that the exact place in the callback chain that :api:`twisted.internet.defer.Deferred.addTimeout <Deferred.addTimeout>` is added determines how much of the callback chain should be timed out.  The timeout encompasses all the callbacks and errbacks added to the :api:`twisted.internet.defer.Deferred <Deferred>` before the call to :api:`twisted.internet.defer.Deferred.addTimeout <addTimeout>`, and none of the callbacks and errbacks added after the call.  The timeout also starts counting down as soon as soon as it's invoked.
+
+
 .. _core-howto-defer-deferredlist:
 
+
 DeferredList
 ------------
 

diff --git a/docs/core/howto/python3.rst b/docs/core/howto/python3.rst
@@ -8,7 +8,7 @@ Twisted is currently being ported to work with Python 3.3+.
 This document covers Twisted-specific issues in porting your code to Python 3.
 
 Most, but not all, of Twisted has been ported, and therefore only a subset of modules are installed under Python 3.
-You can see the remaining modules that need to be ported at :api:`twisted.python._dist3 <twisted.python._dist3>`, if it is not listed there, then most of all of that module will be ported.
+You can see the remaining modules that need to be ported at :api:`twisted.python._setup.notPortedModules <twisted.python._setup.notPortedModules>`, if it is not listed there, then most of all of that module will be ported.
 
 
 API Differences