Monday, January 21, 2019¶
Importing data from TIM to Lino Così¶
I imported data from a Danny’s TIM database to cosi, and of course there were some minor problems since this project has been sleeping for 3 years and the code is not covered by any test (it is to be run only once per TIM customer).
I have a local copy of their TIM data on my machine (not on the server), and a
local copy of their project directory with a modified
$ python manage.py prep --traceback --noinput $* $ dm dump2py 20190121 $ zip 20190121.zip 20190121/*
Note that some of the following changes should not (but might unexpectedly)
cause problems in
lino_xl.lib.tim2lino.spzloader2 which is also still
being used. We’ll see.
Products in Noi now inherit from Referrable. Because at least previous TIM users want a ref field per product. And I imagine that other users might do so as well.
One failure was:
Failed to save InvoiceItem(seqno=1,vat_class=<VatClasses.normal:2>,qty='04.02.',voucher=10,title='Neues Dach angefertigt'): Traceback (most recent call last): ... File "/work/lino/lino/utils/dpy.py", line 138, in save self.try_save(*args, **kw) File "/work/lino/lino/utils/dpy.py", line 158, in try_save obj.full_clean() File "/work/xl/lino_xl/lib/sales/mixins.py", line 128, in full_clean super(ProductDocItem, self).full_clean() File "/work/lino/lino/mixins/sequenced.py", line 257, in full_clean super(Sequenced, self).full_clean(*args, **kw) File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/base.py", line 1228, in full_clean self.clean_fields(exclude=exclude) File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/base.py", line 1270, in clean_fields setattr(self, f.attname, f.clean(raw_value, self)) File "/virtualenvs/py27/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 605, in clean value = self.to_python(value) File "/work/lino/lino/core/fields.py", line 793, in to_python return quantities.parse(value) File "/work/lino/lino/utils/quantities.py", line 219, in parse return parse_decimal(s) File "/work/lino/lino/utils/quantities.py", line 226, in parse_decimal return Decimal(s) File "/usr/lib/python2.7/decimal.py", line 547, in __new__ "Invalid literal for Decimal: %r" % value) File "/usr/lib/python2.7/decimal.py", line 3872, in _raise_error raise error(explanation) decimal.InvalidOperation: Problem installing fixture '/work/xl/lino_xl/lib/tim2lino/fixtures/tim2lino.py': Invalid literal for Decimal: u'04.02.' (py27) luc@doll:~/mypy/lino_local/fijal$ dm prep
So there is an invalid value ‘04.02.’ in a qty field. The question was how to handle this. In above case we just want a warning during the import and have the qty field for this row empty in Lino.
Another problem was:
/xl/lino_xl/lib/tim2lino/timloader1.py:506: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if row.etat == self.etat_registered:
New plugin attribute
lino_xl.lib.tim2lino.Plugin.timloader_kwargs to be
used e.g. as follows:
def setup_plugins(self): self.plugins.tim2lino.configure( languages='de fr', timloader_kwargs=dict(etat_registered="C")) super(Site, self).setup_plugins()
This new feature is not tested because the reason was just a missing “u” prefix.
I also had:
Cannot assign "<CommonItems.expenses:6>": "Account.sheet_item" must be a "Item" instance.
and ATM decided to simply not try to allocate sheet items.
Another bug in TIM¶
Marc reported and I fixed a bug in TIM which caused the dialog window of a printer job to not open when SetAsPdf() was .t. (e.g. from a previous print run or globally). This was probably because the functions GraPrint(), ArtPrint(), PrjPrint(), GenPrint() and TptPrint() hadn’t been adapted when (more than 10 years ago) the argument 8# (lDlg) to OpenPrinter had been inserted to the signature. So these functions actually called OpenPrinter with lDlg having the value of SetAsPdf().
I had observed this problem for SPZ earlier but did not have time (nor need) to investigate. This was the second long-living bug in TIM reported by Marc this year. Congratulations.
The memory situation on this machine is typical for a Lino production server:
$ free -h total used free shared buff/cache available Mem: 1.9G 1.1G 724M 20M 133M 707M Swap: 0B 0B 0B
--max-row-count 20000 and now it works. Aha,
so 50000 is just a bit too much. Indeed, this default value is just “clinically
More thoughts about documentation dependency¶
I am afraid that the current structure of our documentation is still not optimal.
(1) Docs should be together with their code in a same directory. Every
repository should have a xfile:docs directory with an overview and at least a
docs/changes page, so that we can require that every commit with code
changes should also commit the corresponding change note. A change note* is
a dated entry with at least one sentence of English text that explains the
You might ask “What’s so difficult with this?” and “Why are we trying such a complex construct as the book repository?” It’s because we have a second requirement:
Change notes should be able to refer to the documentation.
At the code level we have a clear dependency lineage “atelier –> lino –> lino_xl –> lino_xxx –> lino_book” (where xxx are the different apps maintained by the Lino team). There is a graphic of this in Components of the Lino framework.
The pp command loops though our projects in a given order. This order is important. It is specified in the atelier config (see The config.py file).
Commands like inv test and inv ci rely on this order. They must should run in down the dependency line. The tests in Lino should pass before we can expect the tests in XL to pass, Travis should not start testing XL before lino has been committed, etc.
For commands like inv prep there is currently no dependency. And I don’t think that one day we will have demo projects that depend on data in a demo project of another repository.
But things are different for commands like inv bd and inv pd.
A change note in the docs of XL should be able to refer to things like
lino.core.actors (something in the Lino docs)
lino_cosi.lib.products (something in the Così docs, e.g. some
application that might be a use case).
We have a kind of circular dependency.
All these thoughts caused the following avalanche of changes:
I added a
--reverseoption to pp. We will use this option for commands like inv bd and inv pd. A typical run for testing and building everything will be:
$ pp inv prep test $ pp -r inv clean -b bd pd
The main reason for these doctrees are the change notes, which are now finally separate for Lino and XL.
I moved the
linopart of the
manualsrepo back to book/docs, IOW I reconciled “non-technical” and “technical” docs about Lino. This separation had become useless and BTW it was incomplete (there were still quite some duplicate rst files in both repos).
Everything is now back below www.lino-framework.org and the docs subdomain will be removed (currently it serves the same content as www).
The door seems open now to grow the
docsof every application code repository and move more specs out of the Lino book. Especially accounting stuff should move to the Cosi docs.
Note that you still need to build the book in order to update the help_texts in code source. Which means that a change in some help text of a field or action will become visible only after a full build. And some tests cover the translation of a help text:
$ inv clean -b bd mm
More and more doc pages now use
literalincludeto include the
/shared/include/defs.rstfile. Building these docs requires a clone of the book repository with local name “book”.
There are some other rules for our documentation:
Docs for applications should be in the language of the product owner
docstrings in code should be relatively short and rather refer to some prosa docs