Friday, April 9, 2021

The Mystery of the failing Welfare test suite

I am exploring the failures in Lino Welfare. I picked docs/specs/b2c.rst (the source file for Import bank statements (SEPA BankToCustomer)) as an example. It fails also when I go back to Django 3.1.7 (the version I had before upgrading).

Note that I also have to run pm prep in the gerd demo project after every change. This indicates that the issues are caused by demo data being generated differently because sorting behaviour has changed.

The failure in docs/specs/b2c.rst comes because in lino_welfare.modlib.sepa.fixtures.demo the following line failed with Django 3.2:

qs = Company.objects.filter(sepa_accounts__iban__gt='').distinct()

I changed it to:

qs = Company.objects.exclude(sepa_accounts__iban='').distinct()

This change caused the companies to be sorted differently.

The special sqlite sorting hack in lino_welfare.lib.welfare.models.customize_sqlite() seems also related to these issues. In Django 3.2 the stricmp() function caused a NameError: name ‘cmp’ is not defined and I have no explanation why this error didn’t come already earlier. The built-in cmp() function has been removed in Python 3 but it was still being used, and nobody complained. Which made me suspect that the hack wasn’t active.

Maybe the following documented change is responsible: The django.db.models.Field equality operator now correctly distinguishes inherited field instances across models. Additionally, the ordering of such fields is now defined.

The lino_xl.lib.contacts.Company model had no Meta.ordering specified.

ordering = [‘name’]

The docs about Meta.ordering have a warning “Ordering is not a free operation. Each field you add to the ordering incurs a cost to your database. Each foreign key you add will implicitly include all of its default orderings as well.” Not sure how to understand this. Does it mean that we should rather use only lino.core.tables.AbstractTable.order_by?

I thought that the issue comes because querysets are no longer automatically sorted by id (when Meta.ordering) is not specified. So I replaced, in welfare/**fixtures/*.py, all occurrences of .objects.all() by .objects.order_by(‘id’). And added an .order_by(‘id’) to all calls to .objects.filter(…) that didn’t specify any ordering.

I now at least understand the problem: certain models don’t have a default ordering. And the demo fixtures in welfare happily created coachings and other things based on non-ordered data. That was no problem because unordered data was in some constant way unordered. Until Django 3.2 where this ordering changes.

But that didn’t help, either.

The docs/specs/newcomers.rst fixture helped me to understand more:

$ dt docs/specs/newcomers.rst
Failed example:
    rt.show('newcomers.Competences')
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
Expected:
    ==== ================= ================================ =========
     ID   Benutzer          Fachbereich                      Aufwand
    ---- ----------------- -------------------------------- ---------
     1    Alicia Allmanns   Eingliederungseinkommen (EiEi)   10
     2    Hubert Huppertz   DSBE                             5
     3    Mélanie Mélard    Ausländerbeihilfe                4
     4    Alicia Allmanns   Finanzielle Begleitung           6
     5    Hubert Huppertz   Laufende Beihilfe                2
     6    Mélanie Mélard    Eingliederungseinkommen (EiEi)   10
     7    Alicia Allmanns   DSBE                             5
                                                             **42**
    ==== ================= ================================ =========
    <BLANKLINE>
Got:
    ==== ================= ================================ =========
     ID   Benutzer          Fachbereich                      Aufwand
    ---- ----------------- -------------------------------- ---------
     1    Mélanie Mélard    Eingliederungseinkommen (EiEi)   10
     2    Hubert Huppertz   DSBE                             5
     3    Alicia Allmanns   Ausländerbeihilfe                4
     4    Mélanie Mélard    Finanzielle Begleitung           6
     5    Hubert Huppertz   Laufende Beihilfe                2
     6    Alicia Allmanns   Eingliederungseinkommen (EiEi)   10
     7    Mélanie Mélard    DSBE                             5
                                                             **42**
    ==== ================= ================================ =========
    <BLANKLINE>

Here is the (simplified) code that generates this database content:

FACULTIES = Cycler(newcomers.Faculty.objects.all())
USERS = Cycler(User.objects.all())
for i in range(7):
    yield newcomers.Competence(user=USERS.pop(), faculty=FACULTIES.pop())
>>> users.User.objects.all().ordered
True
>>> newcomers.Faculty.objects.all().ordered
False

Some models have a default ordering:

>>> from lino.api.shell import *
>>> "ORDER BY" in str(contacts.Person.objects.all().query)
True
>>> jobs.Job.objects.all().ordered
True
>>> debts.Account.objects.all().ordered
True
>>> jobs.ContractType.objects.all().ordered
True
>>> art61.ContractType.objects.all().ordered
True
>>> immersion.ContractType.objects.all().ordered
True
>>> immersion.Goal.objects.all().ordered
True

But many models don’t have have it:

>>> from lino.api.shell import *
>>> "ORDER BY" in str(contacts.Company.objects.all().query)
False
>>> pcsw.Client.objects.all().ordered
False
>>> "ORDER BY" in str(clients.ClientContactType.objects.all().query)
False
>>> "ORDER BY" in str(aids.Granting.objects.all().query)
False
>>> "ORDER BY" in str(households.Household.objects.all().query)
False
>>> "ORDER BY" in str(cv.StudyType.objects.all().query)
False
>>> isip.ContractEnding.objects.all().ordered
False
>>> aids.AidType.objects.all().ordered
False
>>> isip.ContractType.objects.all().ordered
False
>>> isip.Contract.objects.all().ordered
False
>>> immersion.Contract.objects.all().ordered
False

Getting notified when long-running command terminates

Since pm prep takes a minute or more to complete (on my computer), I run it as follows to get an acoustic notification when it is done:

$ pm prep --noinput ; espeak done

The –noinput option is useful here because more than once I launched the process and forget to hit ENTER in order to confirm the question:

We are going to flush your database (.../gerd/settings/default.db).
Are you sure (y/n) ? [Y,n]?

But typing “; espeak done” each time is a bit tedious. Can’t we optimize this?

Some surfing: https://gordonlesti.com/linux-audio-notification-after-long-running-command-has-finished/ https://itsfoss.com/notification-terminal-command-completion-ubuntu/

::

$ sudo apt install libnotify-bin