Friday, October 5, 2018

More about help texts

I tried to cover #2571 by the test suite.

In cal : Calendar functionality I added a test to cover whether the help text of the update_guests action has been installed and translated correctly:

>>> with translation.override('de'):
...     print(cal.Event.update_guests.help_text)
... 
Teilnehmerliste für diesen Kalendereintrag füllen entsprechend der Vorschläge.

The test failed saying:

File "...docs/specs/cal.rst", line 426, in cal.rst
Failed example:
    with translation.override('de'):
        print(cal.Event.update_guests.help_text)
    #doctest: +NORMALIZE_WHITESPACE +REPORT_CDIFF -SKIP
Expected:
    Teilnehmerliste für diesen Kalendereintrag füllen entsprechend der Vorschläge.
Got:
    Teilnehmerliste für diesen Kalendereintrag füllen entsprechend der Vorschläge.

And I wanted to understand why. As a hint I had a warning being issued by the doctest module:

/usr/lib/python2.7/doctest.py:1556: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

The solution was to explicitly call str() on the translatable text:

print(str(cal.Event.update_guests.help_text))

That’s needed because help_text is lazy (it becomes a string only when you use it), and because doctest cannot guess this.

User roles and their usage

I wrote a new virtual table lino.modlib.users.UserRoles which should help users to understand how the permission system is structured. The table is especially impressive in The Lino Welfare Standard User Types. It needs more work, though: Write docstrings for the different roles. And when printed as pdf there is a layout problem which is obviously caused because there are many columns. To reproduce, run runserver in lino_welfare.projects.eupen, go to Explorer ‣ System ‣ User roles and click the print to pdf button.

Importing invoices from TIM to Lino Tera

For invoices with a different invoice recipient we don’t need to import the project.

Some messages I had today:

Failed to load record OrderedDict([(u'IDJNL', u'VKE'), (u'IDDOC', u'180126'), (u'IDPAR', u'0000224'), (u'NB1', u'01.05.2018-30.06.2018'), (u'MONT', u'     15.00'), (u'ETAT', u'C'), (u'DATE', datetime.date(2018, 6, 30)), (u'DATECH', datetime.date(2018, 6, 30)), (u'NB2', u''), (u'AUTEUR', u'VW'), (u'MATCH', u''), (u'ATTRIB', u'DOP'), (u'IDMFC', u''), (u'IDPAR2', u'E930092'), (u'PERIODE', u'1806'), (u'MEMO', None), (u'DC', u'D'), (u'IDDEV', u'EUR'), (u'COURS', u'         1')]) from VEN : No Therapy with reference u'0000224'

Here is the final one:

Started manage.py run tl3.py (using prod_sites.abtz.settings) --> PID 10360
Loading readonly /mnt/tim/spz/VEN.FOX...
16934 rows have been loaded from /mnt/tim/spz/VEN.FOX.
Loading readonly /mnt/tim/spz/VNL.FOX...
75216 rows have been loaded from /mnt/tim/spz/VNL.FOX.
Deleting 0 obsolete partners
Register 491 vouchers
<class 'lino_tera.lib.trading.models.InvoiceItem'> : 4686 success, 0 failed.
<class 'lino_xl.lib.trading.models.VatProductInvoice'> : 492 success, 0 failed.
Done manage.py run tl3.py (PID 10360)

Vera will love to hear this. Our initial plan was that she must enter the totals of these invoices by hand, since importing them was estimated to be more work. I am sure she will prefer verifying with me whether the 491 documents were correctly imported.

Migration tests

Hamza and I fixed #2522 and did some huge progress with a new type of tests which we call “database migration tests”. We added database migration tests to two demo projects lino_book.projects.tera1 and lino_welfare.projects.eupen. These two applications are the first applications with “stable migration support”.

We have a new class RestoreTestCase to be used simply as follows:

from lino.utils.djangotest import RestoreTestCase

class TestCase(RestoreTestCase):
    tested_versions = ['18.8.0']

By convention this code should be in a file named test_restore.py in the tests directory of the demo project. And of course the version numbers will change with every release of that application.

We have a new admin command makemigdump. This command does not yet work, Hamza will write this according to what we planned.

We started documentation in a new page Migration tests and reorganized related documents.

Why we modify sys.argv

The implementation of the RestoreTestCase is rather short:

def test_restore(self):
    for v in self.tested_versions:
        run_args = ["tests/dumps/{}/restore.py".format(v),
                    "--noinput"]
        sys.argv = ["manage.py", "run"] + run_args
        call_command("run", *run_args)

As you see, it contains a hack: we modify sys.argv. That’s not common practive, so here is why we did this.

#2522 was because the test_restore.py in tera1 did:

from django.core.management import call_command
call_command("run", "tests/dumps/18.8.0/restore.py", "--noinput")

This Python process had been invoked as part of the test suite using:

$ python manage.py test --noinput --failfast

so the value of sys.argv was:

['manage.py', 'test', '--noinput', '--failfast']

But django-admin commands are not supposed to look at sys.argv, they should rely on the args and options passed to their handle() method. That’s intended by design. Django’s call_command() does not modify sys.argv.

But the restore.py does use argparse. It is not a Django admin command. So sees these same command line options. And then complains about the unknown option --failfast.

We tried to avoid manipulating sys.argv by running restore.py in a subprocess. This even worked, but had the disadvantage of importing to the cached demo data, not to the temporary database created by the Django test runner. The difference in speed was considerable: 13 minutes instead of 1 minute:

$ time python manage.py run tests/dumps/18.8.0/restore.py --noinput
real        13m34,279s
user        1m0,018s
sys 0m4,863s

$ time python manage.py test tests.test_restore
real        0m34,735s
user        0m23,253s
sys 0m1,090s

EDIT: Afterwards I realized that we might convert restore.py as a django-admin command, e.g. restore. This would make above hack useless. But caution: don’t forget to adapt restore2preview.py and initdb_preview_from_prod.sh then.