How to integrate with IPython, particularly the notebook.

Integration with normal IPython is relatively easy, you only need to be able to take into account IPython's interactive mode, and select the right backend. Visvis does this all automatically. For integration in Ipython's Qt app, a similar story applies. But might need some extra code to interface with IPython in the right way.

For the notebook it's much more complex, since now we're dealing with HTML and javascript. However, we cannot ignore the raising popularity of IPython's notebook. We can also expect other web-based computing solutions to grow in the coming years, e.g Continuum's Wakari.

I looked at Galry a (tiny) bit, since it is also in the progress of developing such a feature. The approach taken there is to describe a scene in a way that can be serialized (via JSON) and send to Javascript, where it is rendered using WebGl. Although this approach seems reasonable, it has some limitations. Most importantly, this looks a lot like one-way communication. For instance any interaction much be implemented in Javascript as well as in Python+GUI.

Would it be possible to define an interface (similar to the backends in visvis) that defines key presses and mouse moves, etc. and simply push bitmaps to the canvas. In other words, the visualization toolkit does all the rendering off-line in OpenGl. Comminication with HTML5/javascript consists of user interaction one way, and image data the other. I think such an approach would result in much thinner backend and allow non-opengl backends too. In a remote computing scenario, this means that the server does the visualizations and the client is much thinner.

If this works, OpenGl 3.x compatibility would not even be necessary (but would still probably be a good idea to keep the code clean).

Michael Droettboom is working on a backend for MPL that should work in the notebook. He has tried several things now seems to be taking a similar route: http://mdboom.github.com/blog/2012/10/11/matplotlib-in-the-browser-its-coming/