Merge pull request #1828 from finos/starlette-ws

Add python webserver handlers and clients for starlette (fastapi) and aiohttp
finos · Jun 18, 2022 · eae7dce · eae7dce
2 parents 00236dd + f74dd46
commit eae7dce
Show file tree

Hide file tree

Showing 65 changed files with 2,632 additions and 604 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -1350,7 +1350,7 @@ jobs:
     ################
     # Python
     - name: Install python dependencies
-      run: python -m pip install --upgrade pip wheel setuptools "jupyterlab>=3.2" numpy "pyarrow>=5" pytest pytest-cov mock Faker psutil pytest-tornado pytz
+      run: python -m pip install --upgrade aiohttp Faker fastapi "jupyterlab>=3.2" mock numpy pip psutil "pyarrow>=5" pytest pytest-cov pytest-aiohttp pytest-asyncio pytest-tornado pytz setuptools wheel
 
     #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
     #~~~~~~~~~ Build Pipelines ~~~~~~~~~#

diff --git a/docs/md/python.md b/docs/md/python.md
@@ -12,9 +12,13 @@ well as Python-specific data loading support for [NumPy](https://numpy.org/),
 
 Additionally, `perspective-python` provides a session manager suitable for
 integration into server systems such as
-[Tornado websockets](https://www.tornadoweb.org/en/stable/websocket.html), which
-allows fully _virtual_ Perspective tables to be interacted with by multiple
-`<perspective-viewer>` in a web browser.
+[Tornado websockets](https://www.tornadoweb.org/en/stable/websocket.html),
+[AIOHTTP](https://docs.aiohttp.org/en/stable/web_quickstart.html#websockets),
+or [Starlette](https://www.starlette.io/websockets/)/[FastAPI](https://fastapi.tiangolo.com/advanced/websockets/),
+which allows fully _virtual_ Perspective tables to be interacted with by multiple
+`<perspective-viewer>` in a web browser. You can also interact with a Perspective
+table from python clients, and to that end client libraries are implemented for
+both Tornado and AIOHTTP.
 
 As `<perspective-viewer>` will only consume the data necessary to render the
 current screen, this runtime mode allows _ludicrously-sized_ datasets with
@@ -31,9 +35,13 @@ The `perspective` module exports several tools:
 - `Table`, the table constructor for Perspective, which implements the `table`
   and `view` API in the same manner as the JavaScript library.
 - `PerspectiveWidget` the JupyterLab widget for interactive visualization.
-- `PerspectiveTornadoHandler`, an integration with
-  [Tornado](https://www.tornadoweb.org/) that interfaces seamlessly with
+- Perspective webserver handlers  that interfaces seamlessly with
   `<perspective-viewer>` in JavaScript.
+  - `PerspectiveTornadoHandler` for [Tornado](https://www.tornadoweb.org/)
+  - `PerspectiveStarletteHandler` for [Starlette](https://www.starlette.io/) and [FastAPI](https://fastapi.tiangolo.com)
+  - `PerspectiveAIOHTTPHandler` for [AIOHTTP](https://docs.aiohttp.org),
+  - `tornado_websocket`, a Tornado-based websocket client
+  - `aiohttp_websocket` an AIOHTTP-based websocket client
 - `PerspectiveManager` the session manager for a shared server deployment of
   `perspective-python`.
 
@@ -342,7 +350,7 @@ Using Tornado and
 as well as `Perspective`'s JavaScript library, we can set up "distributed"
 Perspective instances that allows multiple browser `perspective-viewer` clients
 to read from a common `perspective-python` server, as in the
-[Tornado Example Project](https://github.com/finos/perspective/tree/master/examples/tornado-python).
+[Tornado Example Project](https://github.com/finos/perspective/tree/master/examples/python-tornado).
 
 This architecture works by maintaining two `Tables`—one on the server, and one
 on the client that mirrors the server's `Table` automatically using `on_update`.
@@ -409,7 +417,11 @@ _*index.html*_
 
 For a more complex example that offers distributed editing of the server
 dataset, see
-[client_server_editing.html](https://github.com/finos/perspective/blob/master/examples/tornado-python/client_server_editing.html).
+[client_server_editing.html](https://github.com/finos/perspective/blob/master/examples/python-tornado/client_server_editing.html).
+
+We also provide examples for Starlette/FastAPI and AIOHTTP:
+- [Starlette Example Project](https://github.com/finos/perspective/tree/master/examples/python-starlette).
+- [AIOHTTP Example Project](https://github.com/finos/perspective/tree/master/examples/python-aiohttp).
 
 ### Server-only Mode
 

diff --git a/examples/python-aiohttp/README.md b/examples/python-aiohttp/README.md
@@ -0,0 +1,7 @@
+# python-aiohttp
+
+This example contains a simple `perspective-python` folder that uses AIOHTTP to serve a static dataset to the user through various [data bindings](https://perspective.finos.org/docs/md/server.html):
+
+- `index.html`: a [client/server replicated](https://perspective.finos.org/docs/md/server.html#clientserver-replicated) setup that synchronizes the client and server data using Apache Arrow.
+- `server_mode.html`: a [server-only](https://perspective.finos.org/docs/md/server.html#server-only) setup that reads data and performs operations directly on the server using commands sent through the Websocket.
+- `client_server_editing`: a client-server replicated setup that also enables editing, where edits from multiple clients are applied properly to the server, and then synchronized back to the clients.
diff --git a/examples/tornado-python/package.json → examples/python-aiohttp/package.json b/examples/tornado-python/package.json → examples/python-aiohttp/package.json
@@ -1,5 +1,5 @@
 {
-    "name": "tornado-python",
+    "name": "python-aiohttp",
     "private": true,
     "version": "1.4.0",
     "description": "An example of editing a `perspective-python` server from the browser.",

diff --git a/examples/python-aiohttp/server.py b/examples/python-aiohttp/server.py
@@ -0,0 +1,63 @@
+################################################################################
+#
+# Copyright (c) 2019, the Perspective Authors.
+#
+# This file is part of the Perspective library, distributed under the terms of
+# the Apache License 2.0.  The full license can be found in the LICENSE file.
+#
+import asyncio
+import os
+import os.path
+import logging
+import threading
+
+from aiohttp import web
+
+from perspective import Table, PerspectiveManager, PerspectiveAIOHTTPHandler
+
+
+here = os.path.abspath(os.path.dirname(__file__))
+file_path = os.path.join(
+    here, "..", "..", "node_modules", "superstore-arrow", "superstore.arrow"
+)
+
+
+def perspective_thread(manager):
+    """Perspective application thread starts its own event loop, and
+    adds the table with the name "data_source_one", which will be used
+    in the front-end."""
+    psp_loop = asyncio.new_event_loop()
+    manager.set_loop_callback(psp_loop.call_soon_threadsafe)
+    with open(file_path, mode="rb") as file:
+        table = Table(file.read(), index="Row ID")
+        manager.host_table("data_source_one", table)
+    psp_loop.run_forever()
+
+
+def make_app():
+    manager = PerspectiveManager()
+
+    thread = threading.Thread(target=perspective_thread, args=(manager,))
+    thread.daemon = True
+    thread.start()
+
+    async def websocket_handler(request):
+        handler = PerspectiveAIOHTTPHandler(manager=manager, request=request)
+        await handler.run()
+
+    app = web.Application()
+    app.router.add_get("/websocket", websocket_handler)
+    app.router.add_static(
+        "/node_modules/@finos", "../../node_modules/@finos", follow_symlinks=True
+    )
+    app.router.add_static(
+        "/node_modules", "../../node_modules/@finos", follow_symlinks=True
+    )
+    app.router.add_static("/", "../python-tornado", show_index=True)
+    return app
+
+
+if __name__ == "__main__":
+    app = make_app()
+    logging.critical("Listening on http://localhost:8080")
+    web.run_app(app, host="0.0.0.0", port=8080)
diff --git a/examples/python-starlette/README.md b/examples/python-starlette/README.md
@@ -0,0 +1,7 @@
+# python-starlette
+
+This example contains a simple `perspective-python` folder that uses Starlette/FastAPI to serve a static dataset to the user through various [data bindings](https://perspective.finos.org/docs/md/server.html):
+
+- `index.html`: a [client/server replicated](https://perspective.finos.org/docs/md/server.html#clientserver-replicated) setup that synchronizes the client and server data using Apache Arrow.
+- `server_mode.html`: a [server-only](https://perspective.finos.org/docs/md/server.html#server-only) setup that reads data and performs operations directly on the server using commands sent through the Websocket.
+- `client_server_editing`: a client-server replicated setup that also enables editing, where edits from multiple clients are applied properly to the server, and then synchronized back to the clients.
diff --git a/examples/python-starlette/package.json b/examples/python-starlette/package.json
@@ -0,0 +1,24 @@
+{
+    "name": "python-starlette",
+    "private": true,
+    "version": "1.4.0",
+    "description": "An example of editing a `perspective-python` server from the browser.",
+    "scripts": {
+        "start": "PYTHONPATH=../../python/perspective python3 server.py"
+    },
+    "keywords": [],
+    "license": "Apache-2.0",
+    "dependencies": {
+        "@finos/perspective": "^1.4.0",
+        "@finos/perspective-viewer": "^1.4.0",
+        "@finos/perspective-viewer-d3fc": "^1.4.0",
+        "@finos/perspective-viewer-datagrid": "^1.4.0",
+        "@finos/perspective-workspace": "^1.4.0",
+        "superstore-arrow": "^1.0.0"
+    },
+    "devDependencies": {
+        "@finos/perspective-webpack-plugin": "^1.4.0",
+        "npm-run-all": "^4.1.3",
+        "rimraf": "^2.5.2"
+    }
+}
diff --git a/examples/python-starlette/server.py b/examples/python-starlette/server.py
@@ -0,0 +1,79 @@
+################################################################################
+#
+# Copyright (c) 2019, the Perspective Authors.
+#
+# This file is part of the Perspective library, distributed under the terms of
+# the Apache License 2.0.  The full license can be found in the LICENSE file.
+#
+import asyncio
+import os
+import os.path
+import logging
+import threading
+import uvicorn
+
+from fastapi import FastAPI, WebSocket
+from fastapi.middleware.cors import CORSMiddleware
+from starlette.responses import FileResponse
+from starlette.staticfiles import StaticFiles
+
+from perspective import Table, PerspectiveManager, PerspectiveStarletteHandler
+
+
+here = os.path.abspath(os.path.dirname(__file__))
+file_path = os.path.join(
+    here, "..", "..", "node_modules", "superstore-arrow", "superstore.arrow"
+)
+
+
+def static_nodemodules_handler(rest_of_path):
+    if rest_of_path.startswith("@finos"):
+        return FileResponse("../../node_modules/{}".format(rest_of_path))
+    return FileResponse("../../node_modules/@finos/{}".format(rest_of_path))
+
+
+def perspective_thread(manager):
+    """Perspective application thread starts its own event loop, and
+    adds the table with the name "data_source_one", which will be used
+    in the front-end."""
+    psp_loop = asyncio.new_event_loop()
+    manager.set_loop_callback(psp_loop.call_soon_threadsafe)
+    with open(file_path, mode="rb") as file:
+        table = Table(file.read(), index="Row ID")
+        manager.host_table("data_source_one", table)
+    psp_loop.run_forever()
+
+
+def make_app():
+    manager = PerspectiveManager()
+
+    thread = threading.Thread(target=perspective_thread, args=(manager,))
+    thread.daemon = True
+    thread.start()
+
+    async def websocket_handler(websocket: WebSocket):
+        handler = PerspectiveStarletteHandler(manager=manager, websocket=websocket)
+        await handler.run()
+
+    # static_html_files = StaticFiles(directory="../python-tornado", html=True)
+    static_html_files = StaticFiles(directory="../python-tornado", html=True)
+
+    app = FastAPI()
+    app.add_api_websocket_route("/websocket", websocket_handler)
+    app.get("/node_modules/{rest_of_path:path}")(static_nodemodules_handler)
+    app.mount("/", static_html_files)
+
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+    return app
+
+
+if __name__ == "__main__":
+    app = make_app()
+    logging.critical("Listening on http://localhost:8080")
+    uvicorn.run(app, host="0.0.0.0", port=8080)
diff --git a/examples/tornado-streaming-python/index.html → examples/python-tornado-streaming/index.html b/examples/tornado-streaming-python/index.html → examples/python-tornado-streaming/index.html
diff --git a/...les/tornado-streaming-python/package.json → ...les/python-tornado-streaming/package.json b/...les/tornado-streaming-python/package.json → ...les/python-tornado-streaming/package.json
@@ -1,5 +1,5 @@
 {
-    "name": "tornado-streaming-python",
+    "name": "python-tornado-streaming",
     "private": true,
     "version": "1.4.0",
     "description": "An example of streaming a `perspective-python` server to the browser.",

diff --git a/examples/tornado-streaming-python/server.py → examples/python-tornado-streaming/server.py b/examples/tornado-streaming-python/server.py → examples/python-tornado-streaming/server.py
diff --git a/examples/tornado-python/README.md → examples/python-tornado/README.md b/examples/tornado-python/README.md → examples/python-tornado/README.md
@@ -1,4 +1,4 @@
-# tornado-python
+# python-tornado
 
 This example contains a simple `perspective-python` folder that uses Tornado to serve a static dataset to the user through various [data bindings](https://perspective.finos.org/docs/md/server.html):
 

diff --git a/...tornado-python/client_server_editing.html → ...python-tornado/client_server_editing.html b/...tornado-python/client_server_editing.html → ...python-tornado/client_server_editing.html
diff --git a/examples/tornado-python/index.html → examples/python-tornado/index.html b/examples/tornado-python/index.html → examples/python-tornado/index.html
diff --git a/examples/python-tornado/package.json b/examples/python-tornado/package.json
@@ -0,0 +1,24 @@
+{
+    "name": "python-tornado",
+    "private": true,
+    "version": "1.4.0",
+    "description": "An example of editing a `perspective-python` server from the browser.",
+    "scripts": {
+        "start": "PYTHONPATH=../../python/perspective python3 server.py"
+    },
+    "keywords": [],
+    "license": "Apache-2.0",
+    "dependencies": {
+        "@finos/perspective": "^1.4.0",
+        "@finos/perspective-viewer": "^1.4.0",
+        "@finos/perspective-viewer-d3fc": "^1.4.0",
+        "@finos/perspective-viewer-datagrid": "^1.4.0",
+        "@finos/perspective-workspace": "^1.4.0",
+        "superstore-arrow": "^1.0.0"
+    },
+    "devDependencies": {
+        "@finos/perspective-webpack-plugin": "^1.4.0",
+        "npm-run-all": "^4.1.3",
+        "rimraf": "^2.5.2"
+    }
+}
diff --git a/examples/tornado-python/server.py → examples/python-tornado/server.py b/examples/tornado-python/server.py → examples/python-tornado/server.py
diff --git a/examples/tornado-python/server_mode.html → examples/python-tornado/server_mode.html b/examples/tornado-python/server_mode.html → examples/python-tornado/server_mode.html
diff --git a/python/perspective/bench/tornado/async_server.py b/python/perspective/bench/tornado/async_server.py
@@ -77,7 +77,7 @@ def make_app(manager):
         [
             (
                 r"/",
-                perspective.tornado_handler.PerspectiveTornadoHandler,
+                perspective.handlers.tornado.PerspectiveTornadoHandler,
                 {"manager": manager},
             )
         ]

diff --git a/python/perspective/bench/tornado/bench.py b/python/perspective/bench/tornado/bench.py
@@ -194,7 +194,7 @@ async def client(self, client_id):
             )
             await asyncio.sleep(delay)
 
-        psp_client = await perspective.tornado_handler.websocket(self.url)
+        psp_client = await perspective.client.tornado.websocket(self.url)
         results = []
 
         for i in range(self.num_runs):

diff --git a/python/perspective/docs/perspective.core.rst b/python/perspective/docs/perspective.core.rst
@@ -1,9 +1,9 @@
 ``perspective.core`` contains modules that implements ``perspective-python`` in various environments,
-most notably ``PerspectiveWidget`` and ``PerspectiveTornadoHandler``.
+most notably ``PerspectiveWidget`` and the various Perspective web server handlers.
 
 Additionally, ``perspective.core`` defines several enums that provide easy access to aggregate options, different plugins, sort directions etc.
 
-For usage of ``PerspectiveWidget`` and ``PerspectiveTornadoHandler``, see the User Guide in the sidebar.
+For usage of ``PerspectiveWidget`` and the Perspective web server handlers, see the User Guide in the sidebar.
 
 .. autofunction:: perspective.set_threadpool_size
 
@@ -26,16 +26,37 @@ PerspectiveWidget
    :show-inheritance:
    :exclude-members: random
 
-PerspectiveTornadoHandler
-=========================
+Perspective Webserver Handlers
+=================================
 
-``PerspectiveTornadoHandler`` is a ready-made Perspective server that interfaces seamlessly with
+Perspective provides several ready-made integrations with webserver libraries that interfaces seamlessly with
 ``@finos/perspective-viewer`` in Javascript.
 
-.. automodule:: perspective.tornado_handler.tornado_handler
+.. automodule:: perspective.handlers.tornado
    :members:
    :show-inheritance:
 
+.. automodule:: perspective.handlers.starlette
+   :members:
+   :show-inheritance:
+
+.. automodule:: perspective.handlers.aiohttp
+   :members:
+   :show-inheritance:
+
+Perspective Websocket Clients
+==============================
+Perspective also provides several client interfaces to integrate with the above Perspective webserver handlers.
+
+.. automodule:: perspective.client.tornado
+   :members:
+   :show-inheritance:
+
+.. automodule:: perspective.client.aiohttp
+   :members:
+   :show-inheritance:
+
+
 PerspectiveManager
 ==================
 

diff --git a/python/perspective/perspective/__init__.py b/python/perspective/perspective/__init__.py
@@ -6,7 +6,9 @@
 # the Apache License 2.0.  The full license can be found in the LICENSE file.
 #
 
-from .libpsp import *  # noqa: F401, F403
-from .core import *  # noqa: F401, F403
-from .core._version import __version__  # noqa: F401
-from .widget import *  # noqa: F401, F403
+from .libpsp import *
+from .core import *
+from .core._version import __version__
+from .client import *
+from .handlers import *
+from .widget import *
diff --git a/python/perspective/perspective/client/__init__.py b/python/perspective/perspective/client/__init__.py
@@ -6,6 +6,14 @@
 # the Apache License 2.0.  The full license can be found in the LICENSE file.
 #
 
-from .client import PerspectiveClient  # noqa: F401
+from .client import PerspectiveClient
 
-__all__ = ["PerspectiveClient"]
+try:
+    from .aiohttp import PerspectiveAIOHTTPClient, websocket as aiohttp_websocket
+except ImportError:
+    ...
+
+try:
+    from .tornado import PerspectiveTornadoClient, websocket as tornado_websocket
+except ImportError:
+    ...
-Original file line number
+Diff line change
@@ Expand Up / @@ -77,7 +77,7 @@ def make_app(manager): @@
             [
                 (
                     r"/",
-                    perspective.tornado_handler.PerspectiveTornadoHandler,
+                    perspective.handlers.tornado.PerspectiveTornadoHandler,
                     {"manager": manager},
                 )
             ]
@@ Expand Down @@