Wednesday, March 6, 2019

I finally continued to work for Lino Presto! But only for half an hour because then I tried to build the German docs. And this gave a failure.

I opened #2866 (failed to reach any of the inventories with the following issues).

I had a published Sphinx doctree at http://de.presto.lino-framework.org with an invalid objects.inv file so that intersphinx was not able to read it. It said:

ValueError: unknown or unsupported inventory version: ValueError(u'invalid inventory header: ',)

This message is not the problem (I guess it was because that site was generated with another Sphinx version, but that’s unrelevant). My problem is that this error was never reported. It was never reported because sphinx.ext.intersphinx.fetch_inventories() does something forbidden:

try:
   ...
except Exception as err:
    err.args = ('intersphinx inventory %r not fetchable due to %s: %s',
                inv, err.__class__, err)  # LS20190306
    raise
try:
    ...
except Exception as err:
    err.args = ('intersphinx inventory %r not readable due to %s: %s',
                inv, err.__class__.__name__, err)  # LS20190306

To fix my problem, I replaced both lines:

err.args = (msgtpl, inv, err.__class__, err)

by:

err.args = (msgtpl, inv, err.__class__, str(err))

The problem might occur only when there is an additional exception “intersphinx inventory has moved” involved. Here is the full output:

loading intersphinx inventory from http://de.presto.lino-framework.org...
intersphinx inventory has moved: http://de.presto.lino-framework.org -> http://de.presto.lino-framework.org/
WARNING: failed to reach any of the inventories with the following issues:
intersphinx inventory 'http://de.presto.lino-framework.org' not readable due to ValueError: ('intersphinx inventory %r not readable due to %s: %s', 'http://de.presto.lino-framework.org', 'ValueError', ValueError(...))

Another problem was that intersphinx later reports every failure as a separate call to warning() and when tolerate_warnings is False, we see only the first warning.

I created a pull request for Sphinx at https://github.com/sphinx-doc/sphinx/pull/6139

Now that reporting works (at least on my machine), I see that the “unrelated” problem touches me more than I thought. Sphinx seems no longer able to read the inventories created by my sphinx:

intersphinx inventory 'http://atelier.lino-framework.org' not readable due to ValueError: invalid inventory header ''

This one works:

$ python -m sphinx.ext.intersphinx http://de.presto.lino-framework.org/objects.inv

I did some tests:

>>> import requests
>>> BUFSIZE = 16 * 1024
>>> url = "http://atelier.lino-framework.org/objects.inv"
>>> r = requests.get(url, stream=True, timeout=None)
>>> chunk = r.raw.read(BUFSIZE)
>>> print(chunk.splitlines()[0])
# Sphinx inventory version 2

Until I realized that it was simply caused by my recent change in interproject. I still didn’t understand the documentation for intersphinx_mapping.

The mappings generated by interproject are new format. Every value of the dict must be a tuple with two items. The first item is the base url (the base URI of the links to generate) and may not be empty (at least not for interproject). The second item is usually None. Except when ATELIER_USE_LOCAL_BUILDS is not empty, and otherwise the full path of the objects.inv file.

There is some problem with the doctree dependencies. I am still fiddling with understanding them.

  • we have a lot of projects, and most of these have a doctree. Some have no doctree (e.g. commondata), some others have several doctrees (e.g. “docs” and “docs_fr”)

  • we want certain doctrees to refer to some doctrees of other projects

  • if doctree A refers to doctree B, we want the build to fail if intersphinx was unable to load that doctree B

  • intersphinx can load the objects.inv file of a doctree either via Internet or from the local build.

  • Sphinx itself does not “know” the URI where the result is going to be published.

  • When running on Travis, intersphinx always loads its doctrees via internet (since the project being built is the only one

  • intersphinx can also load doctrees from a local objects.inv file.

  • I sometimes want to to build all my doctrees when I am offline and before publishing them.

  • the book imports the lino and generates its API docs, but the doctree of lino wants to refer to the book doctree.

I fixed one important problem (“important” at least for me because i want to do above things) : when specifying an explicit prjlist to interproject.configure the root_dir of the project descriptors werde not correctly set.

The ordering cannot be handled automatically because e.g. the lino docs refer to the book while the book depends on lino.

IOW when you have no internet connection and then run pp inv clean -b, I will have some troubles to get all my doctrees built again. A simple pp inv bd will fail because pp does not loop over the project in the required order.

For example pp inv prep test should run on lino, then xl, …. and finally in book (in that order). But pp inv bd needs another ordering. And we cannot simply the order (as I believed until today).

TODO: We even have a circular dependency : book requires cosi and cosi requires book. This can be resolved only when you have internet. This circular dependency should be removed by moving content from one doctree to the other.

I moved the noi docs from man to noi. Note that these docs are just the “user docs” but not the specs. The specs of noi remain in the book because noi is the “first” (or “pilot”) application for some plugins of the xl (e.g. tickets and working). That’s also why the specs for weleup should remain in the book: weleup is needed to explain plugins like reception or art61 which are defined in lino_welfare.

Uuh, documentation is such a complex thing! But we are advancing. It is getting better each time!