Monday, January 21, 2019

Importing data from TIM to Lino Così

I imported data from a Danny’s TIM database to cosi, and of course there were some minor problems since this project has been sleeping for 3 years and the code is not covered by any test (it is to be run only once per TIM customer).

I have a local copy of their TIM data on my machine (not on the server), and a local copy of their project directory with a modified settings.py to add tim2lino:

$ python manage.py prep --traceback --noinput $*
$ dm dump2py 20190121
$ zip 20190121.zip 20190121/*

Note that some of the following changes should not (but might unexpectedly) cause problems in lino_xl.lib.tim2lino.spzloader2 which is also still being used. We’ll see.

Changes

Products in Noi now inherit from Referrable. Because at least previous TIM users want a ref field per product. And I imagine that other users might do so as well.

One failure was:

Failed to save InvoiceItem(seqno=1,vat_class=<VatClasses.normal:2>,qty='04.02.',voucher=10,title='Neues Dach angefertigt'):
Traceback (most recent call last):
  ...
  File "/work/lino/lino/utils/dpy.py", line 138, in save
    self.try_save(*args, **kw)
  File "/work/lino/lino/utils/dpy.py", line 158, in try_save
    obj.full_clean()
  File "/work/xl/lino_xl/lib/trading/mixins.py", line 128, in full_clean
    super(ProductDocItem, self).full_clean()
  File "/work/lino/lino/mixins/sequenced.py", line 257, in full_clean
    super(Sequenced, self).full_clean(*args, **kw)
  File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/base.py", line 1228, in full_clean
    self.clean_fields(exclude=exclude)
  File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/base.py", line 1270, in clean_fields
    setattr(self, f.attname, f.clean(raw_value, self))
  File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 605, in clean
    value = self.to_python(value)
  File "/work/lino/lino/core/fields.py", line 793, in to_python
    return quantities.parse(value)
  File "/work/lino/lino/utils/quantities.py", line 219, in parse
    return parse_decimal(s)
  File "/work/lino/lino/utils/quantities.py", line 226, in parse_decimal
    return Decimal(s)
  File "/usr/lib/python2.7/decimal.py", line 547, in __new__
    "Invalid literal for Decimal: %r" % value)
  File "/usr/lib/python2.7/decimal.py", line 3872, in _raise_error
    raise error(explanation)
decimal.InvalidOperation: Problem installing fixture '/work/xl/lino_xl/lib/tim2lino/fixtures/tim2lino.py': Invalid literal for Decimal: u'04.02.'
(py27) luc@doll:~/mypy/lino_local/fijal$ dm prep

So there is an invalid value ‘04.02.’ in a qty field. The question was how to handle this. In above case we just want a warning during the import and have the qty field for this row empty in Lino.

Another problem was:

/xl/lino_xl/lib/tim2lino/timloader1.py:506: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if row.etat == self.etat_registered:

New plugin attribute lino_xl.lib.tim2lino.Plugin.timloader_kwargs to be used e.g. as follows:

def setup_plugins(self):
    self.plugins.tim2lino.configure(
        languages='de fr',
        timloader_kwargs=dict(etat_registered="C"))
    super(Site, self).setup_plugins()

This new feature is not tested because the reason was just a missing “u” prefix.

I also had:

Cannot assign "<CommonItems.expenses:6>": "Account.sheet_item" must be a "Item" instance.

and ATM decided to simply not try to allocate sheet items.

Another bug in TIM

Marc reported and I fixed a bug in TIM which caused the dialog window of a printer job to not open when SetAsPdf() was .t. (e.g. from a previous print run or globally). This was probably because the functions GraPrint(), ArtPrint(), PrjPrint(), GenPrint() and TptPrint() hadn’t been adapted when (more than 10 years ago) the argument 8# (lDlg) to OpenPrinter had been inserted to the signature. So these functions actually called OpenPrinter with lDlg having the value of SetAsPdf().

I had observed this problem for SPZ earlier but did not have time (nor need) to investigate. This was the second long-living bug in TIM reported by Marc this year. Congratulations.

While restoring a snapshot to a prototype on mylino.net I had a MemoryError. This time the restore.py was not cruelly “Killed” but reported a MemoryError exception and a traceback.

The memory situation on this machine is typical for a Lino production server:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.1G        724M         20M        133M        707M
Swap:            0B          0B          0B

I tried pm dump2py with --max-row-count 20000 and now it works. Aha, so 50000 is just a bit too much. Indeed, this default value is just “clinically tested”.

More thoughts about documentation dependency

I am afraid that the current structure of our documentation is still not optimal.

(1) Docs should be together with their code in a same directory. Every repository should have a xfile:docs directory with an overview and at least a docs/changes page, so that we can require that every commit with code changes should also commit the corresponding change note. A change note* is a dated entry with at least one sentence of English text that explains the change.

You might ask “What’s so difficult with this?” and “Why are we trying such a complex construct as the book repository?” It’s because we have a second requirement:

2. Change notes should be able to refer to the documentation.

At the code level we have a clear dependency lineage “atelier –> lino –> lino_xl –> lino_xxx –> lino_book” (where xxx are the different apps maintained by the Lino team). There is a graphic of this in Repositories of the Lino framework.

The pp command loops though our projects in a given order. This order is important. It is specified in the atelier config (see The config.py file).

Commands like inv test and inv ci rely on this order. They must should run in down the dependency line. The tests in Lino should pass before we can expect the tests in XL to pass, Travis should not start testing XL before lino has been committed, etc.

For commands like inv prep there is currently no dependency. And I don’t think that one day we will have demo projects that depend on data in a demo project of another repository.

But things are different for commands like inv bd and inv pd.

A change note in the docs of XL should be able to refer to things like lino.core.actors (something in the Lino docs) lino_cosi.lib.products (something in the Così docs, e.g. some application that might be a use case).

We have a kind of circular dependency.

All these thoughts caused the following avalanche of changes:

• I added a --reverse option to pp. We will use this option for commands like inv bd and inv pd. A typical run for testing and building everything will be:

  $ pp inv prep test
  $ pp -r inv clean -b bd pd
  

• The lino and xl now have their own docs directories published at http://core.lino-framework.org and http://xl.lino-framework.org

  The main reason for these doctrees are the change notes, which are now finally separate for Lino and XL.

• I moved the lino part of the manuals repo back to book/docs, IOW I reconciled “non-technical” and “technical” docs about Lino. This separation had become useless and BTW it was incomplete (there were still quite some duplicate rst files in both repos).

  Everything is now back below www.lino-framework.org and the docs subdomain will be removed (currently it serves the same content as www).

• The door seems open now to grow the docs of every application code repository and move more specs out of the Lino book. Especially accounting stuff should move to the Cosi docs.

• Note that you still need to build the book in order to update the help_texts in code source. Which means that a change in some help text of a field or action will become visible only after a full build. And some tests cover the translation of a help text:

  $ inv clean -b bd mm
  

• More and more doc pages now use literalinclude to include the /shared/include/defs.rst file. Building these docs requires a clone of the book repository with local name “book”.

There are some other rules for our documentation:

• Docs for applications should be in the language of the product owner

• docstrings in code should be relatively short and rather refer to some prosa docs

The change notes for Lino Extensions Library are now separated from those of The lino package: Changelog Changelog