merge trunk.

samantehrani · Aug 6, 2016 · fc75d9c · fc75d9c
2 parents df409ef + 88bae4e
commit fc75d9c
Show file tree

Hide file tree

Showing 50 changed files with 1,094 additions and 223 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -7,6 +7,7 @@ source = twisted
 source=
    twisted
    build/*/lib/python*/site-packages/twisted
+   build/*/Lib/site-packages/twisted
    build/pypy*/site-packages/twisted
 
 [report]

diff --git a/.travis.yml b/.travis.yml
@@ -30,6 +30,8 @@ matrix:
     # on starting separate jobs.
     - python: 2.7
       env: TOXENV=narrativedocs,apidocs,pyflakes,topfile,manifest-checker
+    - python: 3.5
+      env: TOXENV=pyflakes3
     # Twistedchecker is running as a separate job so that we can ignore if it
     # fails.
     - python: 2.7

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -38,6 +38,7 @@ include twisted/internet/test/fake_CAs/*
 include twisted/mail/test/rfc822.message
 
 # Some extras
+recursive-include twisted *.3only
 recursive-include twisted *.glade *.pxi *.h *.c *.bat *.g *.pyx *.zsh *.txt
 
 # Docs

diff --git a/docs/core/howto/defer-intro.rst b/docs/core/howto/defer-intro.rst
@@ -390,6 +390,67 @@ With ``inlineCallbacks``, we can rewrite this as:
 Our exception handling is simplified because we can use Python's familiar ``try`` / ``except`` syntax for handling ``ConnectionError``\ s.
 
 
+Coroutines with async/await
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+    Only available on Python 3.5 and higher.
+
+    .. versionadded:: 16.4
+
+On Python 3.5 and higher, the :pep:`492` ("Coroutines with async and await syntax") "await" functionality can be used with Deferreds by the use of :api:`twisted.internet.defer.ensureDeferred <ensureDeferred>`.
+It is similar to ``inlineCallbacks``, except that it uses the ``await`` keyword instead of ``yield``, the ``return`` keyword instead of ``returnValue``, and is a function rather than a decorator.
+
+Calling a coroutine (that is, the result of a function defined by ``async def funcname():``) with :api:`twisted.internet.defer.ensureDeferred <ensureDeferred>` will allow you to "await" on Deferreds inside it, and will return a standard Deferred.
+You can mix and match code which uses regular Deferreds, ``inlineCallbacks``, and ``ensureDeferred`` freely.
+
+Awaiting on a Deferred which fires with a Failure will raise the exception inside your coroutine as if it were regular Python.
+If your coroutine raises an exception, it will be translated into a Failure fired on the Deferred that ``ensureDeferred`` returns for you.
+Calling ``return`` will cause the Deferred that ``ensureDeferred`` returned for you to fire with a result.
+
+.. code-block:: python3
+
+   import json
+   from twisted.internet.defer import ensureDeferred
+   from twisted.logger import Logger
+   log = Logger()
+
+   async def getUsers():
+       try:
+           return json.loads(await makeRequest("GET", "/users"))
+       except ConnectionError:
+           log.failure("makeRequest failed due to connection error")
+           return []
+
+   def do():
+       d = ensureDeferred(getUsers())
+       d.addCallback(print)
+       return d
+
+
+When writing coroutines, you do not need to use :api:`twisted.internet.defer.ensureDeferred <ensureDeferred>` when you are writing a coroutine which calls other coroutines which await on Deferreds; you can just ``await`` on it directly.
+For example:
+
+.. code-block:: python3
+
+    async def foo():
+        res = await someFunctionThatReturnsADeferred()
+        return res
+
+    async def bar():
+        baz = await someOtherDeferredFunction()
+        fooResult = await foo()
+        return baz + fooResult
+
+    def myDeferredReturningFunction():
+        coro = bar()
+        return ensureDeferred(coro)
+
+
+Even though Deferreds were used in both coroutines, only ``bar`` had to be wrapped in :api:`twisted.internet.defer.ensureDeferred <ensureDeferred>` to return a Deferred.
+
+
 Conclusion
 ----------
 
@@ -403,6 +464,7 @@ You have been introduced to asynchronous code and have seen how to use :api:`twi
 - Wrap multiple asynchronous operations with one error handler
 - Do something after an asynchronous operation, regardless of whether it succeeded or failed
 - Write code without callbacks using ``inlineCallbacks``
+- Write coroutines that interact with Deferreds using ``ensureDeferred``
 
 These are very basic uses of :api:`twisted.internet.defer.Deferred <Deferred>`.
 For detailed information about how they work, how to combine multiple Deferreds, and how to write code that mixes synchronous and asynchronous APIs, see the :doc:`Deferred reference <defer>`.

diff --git a/docs/core/howto/defer.rst b/docs/core/howto/defer.rst
@@ -18,7 +18,7 @@ Deferreds
 ---------
 
 Twisted uses the :api:`twisted.internet.defer.Deferred <Deferred>` object to manage the callback sequence.
-The client application attaches a series of functions to the deferred to be called in order when the results of the asynchronous request are available (this series of functions is known as a series of **callbacks**, or a **callback chain**), together with a series of functions to be called if there is an error in the asynchronous request (known as a series of **errbacks** or an**errback chain**).
+The client application attaches a series of functions to the deferred to be called in order when the results of the asynchronous request are available (this series of functions is known as a series of **callbacks**, or a **callback chain**), together with a series of functions to be called if there is an error in the asynchronous request (known as a series of **errbacks** or an **errback chain**).
 The asynchronous library code calls the first callback when the result is available, or the first errback when an error occurs, and the ``Deferred`` object then hands the results of each callback or errback function to the next function in the chain.
 
 

diff --git a/tox.ini b/tox.ini
@@ -42,8 +42,9 @@ deps =
      codecov-publish: codecov
 
      ; Code quality checkers
-     pyflakes: pyflakes
+     pyflakes{,3}: pyflakes
      manifest-checker: check-manifest
+
      {twistedchecker,txchecker}: twistedchecker>=0.6.0
      txchecker: diff_cover
 
@@ -65,18 +66,12 @@ commands =
     ; Install PyDoctor here so it DOESNT overwrite Twisted
     py27-{tests,coverage}: pip install --no-deps epydoc pydoctor
 
-    {tests,nomodules}: {envbindir}/trial --reactor={env:TWISTED_REACTOR:default} --reporter={env:TRIAL_REPORTER:verbose}  {posargs:twisted}
+    {tests,nomodules}: python -m twisted.trial --reactor={env:TWISTED_REACTOR:default} --reporter={env:TRIAL_REPORTER:verbose}  {posargs:twisted}
 
     coverage: python {toxinidir}/admin/_copy.py {toxinidir}/admin/zz_coverage.pth {envsitepackagesdir}/zz_coverage.pth
     coverage: coverage erase
 
-    # FIXME: https://bitbucket.org/ned/coveragepy/issues/506
-    # Here we have a conditional coverage run as we have bin/trial on posix
-    # and Scripts/trial.exe on Windows.
-    coverage-posix: coverage run -p --rcfile={toxinidir}/.coveragerc {envbindir}/trial --reactor={env:TWISTED_REACTOR:default} --reporter={env:TRIAL_REPORTER:verbose} {posargs:twisted}
-    # FIXME: https://twistedmatrix.com/trac/ticket/8611
-    # {toxinidir}/bin/trial works for Python 2.7 but not for Python 3.4
-    coverage-windows: coverage run -p --rcfile={toxinidir}/.coveragerc {toxinidir}/bin/trial --reactor={env:TWISTED_REACTOR:default} --reporter={env:TRIAL_REPORTER:verbose} {posargs:twisted}
+    coverage: coverage run -p --rcfile={toxinidir}/.coveragerc -m twisted.trial --reactor={env:TWISTED_REACTOR:default} --reporter={env:TRIAL_REPORTER:verbose} {posargs:twisted}
     # Continue with copying the files outside of the ven and into the root dir
     # so that buildbot or other jobs can access them.
 
@@ -86,16 +81,15 @@ commands =
     # remove these lines.
     coverage: python {toxinidir}/admin/_copy.py "{toxworkdir}/.coverage*" {toxinidir}
 
-    codecov-publish: coverage combine {toxworkdir}
+    codecov-publish: coverage combine
     codecov-publish: coverage xml -o coverage.xml -i
     codecov-publish: codecov {env:CODECOV_OPTIONS:}
 
     twistedchecker: twistedchecker {posargs:twisted}
-
-    # Run twistedchecker with a diff base on trunk.
     txchecker-travis: {toxinidir}/.travis/twistedchecker-trunk-diff.sh {posargs:twisted}
-
+    
     pyflakes: pyflakes {posargs:twisted admin bin}
+    pyflakes3: pyflakes {posargs:twisted/internet/test/_awaittests.py.3only}
 
     apidocs: {toxinidir}/bin/admin/build-apidocs {toxinidir} apidocs
     narrativedocs: sphinx-build -aW -b html -d {toxinidir}/docs/_build {toxinidir}/docs {toxinidir}/docs/_build/
@@ -107,9 +101,14 @@ commands =
 [testenv:twistedchecker]
 basepython=python2.7
 changedir={toxinidir}
+[testenv:codecov-publish]
+changedir={toxinidir}
 [testenv:pyflakes]
 basepython=python2.7
 changedir={toxinidir}
+[testenv:pyflakes3]
+basepython=python3.5
+changedir={toxinidir}
 [testenv:apidocs]
 basepython=python2.7
 changedir={toxinidir}

diff --git a/twisted/conch/client/default.py b/twisted/conch/client/default.py
@@ -14,6 +14,8 @@
 from __future__ import print_function
 
 from twisted.python import log
+from twisted.python.compat import (
+    nativeString, raw_input, _PY3, _b64decodebytes as decodebytes)
 from twisted.python.filepath import FilePath
 
 from twisted.conch.error import ConchError
@@ -24,7 +26,10 @@
 
 from twisted.conch.client import agent
 
-import os, sys, base64, getpass
+import os, sys, getpass, contextlib
+
+if _PY3:
+    import io
 
 # The default location of the known hosts file (probably should be parsed out
 # of an ssh config file someday).
@@ -84,9 +89,13 @@ def verifyHostKey(transport, host, pubKey, fingerprint):
     return kh.verifyHostKey(ui, actualHost, host, actualKey)
 
 
+
 def isInKnownHosts(host, pubKey, options):
-    """checks to see if host is in the known_hosts file for the user.
-    returns 0 if it isn't, 1 if it is and is the same, 2 if it's changed.
+    """
+    Checks to see if host is in the known_hosts file for the user.
+
+    @return: 0 if it isn't, 1 if it is and is the same, 2 if it's changed.
+    @rtype: L{int}
     """
     keyType = common.getNS(pubKey)[0]
     retVal = 0
@@ -110,7 +119,7 @@ def isInKnownHosts(host, pubKey, options):
             if hostKeyType != keyType: # incorrect type of key
                 continue
             try:
-                decodedKey = base64.decodestring(encodedKey)
+                decodedKey = decodebytes(encodedKey)
             except:
                 continue
             if decodedKey == pubKey:
@@ -131,6 +140,7 @@ def __init__(self, user, options, *args):
         if not options.identitys:
             options.identitys = ['~/.ssh/id_rsa', '~/.ssh/id_dsa']
 
+
     def serviceStarted(self):
         if 'SSH_AUTH_SOCK' in os.environ and not self.options['noagent']:
             log.msg('using agent')
@@ -141,36 +151,54 @@ def serviceStarted(self):
         else:
             userauth.SSHUserAuthClient.serviceStarted(self)
 
+
     def serviceStopped(self):
         if self.keyAgent:
             self.keyAgent.transport.loseConnection()
             self.keyAgent = None
 
+
     def _setAgent(self, a):
         self.keyAgent = a
         d = self.keyAgent.getPublicKeys()
         d.addBoth(self._ebSetAgent)
         return d
 
+
     def _ebSetAgent(self, f):
         userauth.SSHUserAuthClient.serviceStarted(self)
 
+
     def _getPassword(self, prompt):
-        try:
-            oldout, oldin = sys.stdout, sys.stdin
-            sys.stdin = sys.stdout = open('/dev/tty','r+')
-            p=getpass.getpass(prompt)
-            sys.stdout,sys.stdin=oldout,oldin
-            return p
-        except (KeyboardInterrupt, IOError):
-            print()
-            raise ConchError('PEBKAC')
+        """
+        Prompt for a password using L{getpass.getpass}.
+
+        @param prompt: Written on tty to ask for the input.
+        @type prompt: L{str}
+        @return: The input.
+        @rtype: L{str}
+        """
+        with self._replaceStdoutStdin():
+            try:
+                p = getpass.getpass(prompt)
+                return p
+            except (KeyboardInterrupt, IOError):
+                print()
+                raise ConchError('PEBKAC')
+
 
     def getPassword(self, prompt = None):
-        if not prompt:
-            prompt = "%s@%s's password: " % (self.user, self.transport.transport.getPeer().host)
+        if prompt:
+            prompt = nativeString(prompt)
+        else:
+            prompt = ("%s@%s's password: " %
+                (nativeString(self.user), self.transport.transport.getPeer().host))
         try:
-            p = self._getPassword(prompt)
+            # We don't know the encoding the other side is using,
+            # signaling that is not part of the SSH protocol. But
+            # using our defaultencoding is better than just going for
+            # ASCII.
+            p = self._getPassword(prompt).encode(sys.getdefaultencoding())
             return defer.succeed(p)
         except ConchError:
             return defer.fail()
@@ -209,7 +237,7 @@ def signData(self, publicKey, signData):
         data, if one is available.
 
         @type publicKey: L{Key}
-        @type signData: L{str}
+        @type signData: L{bytes}
         """
         if not self.usedFiles: # agent key
             return self.keyAgent.signData(publicKey.blob(), signData)
@@ -230,10 +258,10 @@ def getPrivateKey(self):
             return defer.succeed(keys.Key.fromFile(file))
         except keys.EncryptedKeyError:
             for i in range(3):
-                prompt = "Enter passphrase for key '%s': " % \
-                    self.usedFiles[-1]
+                prompt = "Enter passphrase for key '%s': " % self.usedFiles[-1]
                 try:
-                    p = self._getPassword(prompt)
+                    p = self._getPassword(prompt).encode(
+                        sys.getfilesystemencoding())
                     return defer.succeed(keys.Key.fromFile(file, passphrase=p))
                 except (keys.BadKeyError, ConchError):
                     pass
@@ -246,18 +274,51 @@ def getPrivateKey(self):
 
     def getGenericAnswers(self, name, instruction, prompts):
         responses = []
-        try:
-            oldout, oldin = sys.stdout, sys.stdin
-            sys.stdin = sys.stdout = open('/dev/tty','r+')
+        with self._replaceStdoutStdin():
             if name:
-                print(name)
+                print(name.decode("utf-8"))
             if instruction:
-                print(instruction)
+                print(instruction.decode("utf-8"))
             for prompt, echo in prompts:
+                prompt = prompt.decode("utf-8")
                 if echo:
                     responses.append(raw_input(prompt))
                 else:
                     responses.append(getpass.getpass(prompt))
-        finally:
-            sys.stdout,sys.stdin=oldout,oldin
         return defer.succeed(responses)
+
+
+    @classmethod
+    def _openTty(cls):
+        """
+        Open /dev/tty as two streams one in read, one in write mode,
+        and return them.
+
+        @return: File objects for reading and writing to /dev/tty,
+                 corresponding to standard input and standard output.
+        @rtype: A L{tuple} of L{io.TextIOWrapper} on Python 3.
+                A L{tuple} of binary files on Python 2.
+        """
+        stdin = open("/dev/tty", "rb")
+        stdout = open("/dev/tty", "wb")
+        if _PY3:
+            stdin = io.TextIOWrapper(stdin)
+            stdout = io.TextIOWrapper(stdout)
+        return stdin, stdout
+
+
+    @classmethod
+    @contextlib.contextmanager
+    def _replaceStdoutStdin(cls):
+        """
+        Contextmanager that replaces stdout and stdin with /dev/tty
+        and resets them when it is done.
+        """
+        oldout, oldin = sys.stdout, sys.stdin
+        sys.stdin, sys.stdout = cls._openTty()
+        try:
+            yield
+        finally:
+            sys.stdout.close()
+            sys.stdin.close()
+            sys.stdout, sys.stdin = oldout, oldin