@exarkun commented

Building the Lore-format documentation is currently part of the continuous integration employed in Twisted's development process.

At some point we need to set up something that does the equivalent of the [http://buildbot.twistedmatrix.com/builders/documentation/builds/512/steps/process-docs/logs/stdio Lore process docs step. I'd like to do this even before we've actually switched over, so we can use it to tell us about any problems in branches related to the conversion.

	@thijstriemstra commented

Replying to exarkun:

At some point we need to set up something that does the equivalent of the [http://buildbot.twistedmatrix.com/builders/documentation/builds/512/steps/process-docs/logs/stdio Lore process docs step. I'd like to do this even before we've actually switched over, so we can use it to tell us about any problems in branches related to the conversion.

See #4225 that seems related.

	@exarkun commented

One of the steps for lore2sphinx is to run sphinx-quickstart. This prompts for a lot of information, some of which I don't have. I guessed and used defaults for a lot and got far enough so that I could run "make html" which threw out hundreds of warnings and errors, for example:

twisted-sphinx/source/projects/core/index.rst:: WARNING: document isn't included in any toctree
twisted-sphinx/source/projects/words/howto/im.rst:61: (ERROR/3) Unknown interpreted text role "api".
twisted-sphinx/source/projects/web2/examples/index.rst:66: WARNING: download file not readable: projects/web2/examples/auth/credsetup.py

So I think I must have done something wrong. Some more instructions would be helpful.

	@khorn commented

@exarkun:

The lore2sphinx repo includes an already-quickstarted sphinx project in the profiles/twisted directory. While this will need to be modified a bit when doing the "real" conversion, the intention was to provide a common baseline for people to test it against during development, though I don't think anyone but thijs ever did.

Also, keep in mind lore2sphinx was intended to be a "run it once for real" tool. Even then it's not perfect, and making it so would likely consume the rest of my useful programming career accounting for special cases, etc. When I run it and then use sphinx-build from either Sphinx 0.6.5 or Sphinx 1.0b2, I get just 13 errors (as of last week). My feeling is that those can be fixed manually.

	@khorn commented

#4553 was a duplicate of this ticket

	@exarkun commented

Also, keep in mind lore2sphinx was intended to be a "run it once for real" tool.

I know. You've mentioned this too many times for me to ever be able to forget it. :) Despite that, I don't see why we shouldn't be able to run it as many times as we want leading up to the actual conversion. And we need to deal with outstanding branches with changes to the lore source files somehow. It's possible that manually rewriting the docs is the easiest way to go here, but I don't think we should completely throw away the idea of being able to automate at least part of the conversion of such changes.

Even then it's not perfect, and making it so would likely consume the rest of my useful programming career accounting for special cases, etc.

I'm definitely not pushing for this.

When I run it and then use sphinx-build from either Sphinx 0.6.5 or Sphinx 1.0b2, I get just 13 errors (as of last week).

This is more interesting. I used Sphinx 1.0b2. Why did I get hundreds of errors? I suppose it must have had something to do with some of those arguments I gave to sphinx-quickstart. I'll try again with the pre-started project in the lore2sphinx repository.

	@khorn commented

I think using the pre-quickstarted project will help. What kinds of errors were you seeing?

The most likely possibility I can think of is that the docs are full of links to example files, which the fabfile and/or the l2sbuilder.py script copy as an intermediate step in the overall conversion process. If you skip this step, you'll see a whole lot of "I can't find this file" type warnings from Sphinx.

	@exarkun commented

the fabfile and/or the l2sbuilder.py script copy as an intermediate step

What step is this? I don't see it documented anywhere.

	@khorn commented

Well, it's documented in the fabfile and l2s_builder code :)

But I'm afraid that's about it.

	@khorn commented

okay, folks, the plan has changed slightly...

exarkun has created a buildbot which runs lore2sphinx on the current lore sources and creates Sphinx output.

quote from exarkun's email:

At last I've got a buildbot set up generating the sphinx docs.  The
build results can be seen here:

http://buildbot.twistedmatrix.com/builds/sphinx-html-15615/

(or with a different revision number for different revisions; the
containing directory is browseable).

In order to minimize the pain of the final conversion, we want to see if we can get the results of the buildbot to look as good as possible before "pulling the trigger".

So we'll be creating "chunk tickets" for that...I'll post the list here once I've created them.

	@khorn commented

Here's a list of the chunk tickets:

• projects/conch - #4566
• projects/core/development - #4567
• projects/core/howto (except ‘tutorial/’) - #4568
• projects/core/howto/tutorial - #4569
• projects/historic - #4570
• projects/lore - #4571
• projects/mail - #4572
• projects/names - #4573
• projects/pair - #4574
• projects/vfs - #4575
• projects/web (everything except web-in-60) - #4576
• projects/web/web-in-60 - #4577
• projects/web2 - #4578
• projects/words - #4579

	@khorn commented

NOTE: it needs to be possible to build the docs without the docs for vfs and web2, since they are not currently (and should not be) published to the website

It should be possible to handle this using one of the "conditional content" mechanisms in Sphinx.

	gtaylor commented

Sorry if this isn't the right place to ask, but where can we go to learn more about helping with this?

	@khorn commented

@gtaylor:

Basically, the plan has remained the same. I think everyone just got busy (I certainly did). We finish reviewing and committing the "chunk" tickets listed in the comments above to make the Lore source convert as close to correctly as possible. Then we convert it to Sphinx using lore2sphiunx and get it into SVN. Then we set up a new set of "chunk" tickets for reviewing and fixing the RestructuredText source.

One possible obstacle is that the lore2sphinx buildbot seems to have broken sometime in the last couple of months. We need to see if we can get that fixed.

As for helping, you can try reviewing and fixing the tickets above, or perhaps post to the Twisted mailing list, which is probably an easier forum for discussions like this.

	@thijstriemstra commented

Another ticket: #4621 - lore2sphinx should make pretty links to included files.