Friday, April 9, 2021¶
The Mystery of the failing Welfare test suite¶
I am exploring the failures in Lino Welfare. I picked
docs/specs/b2c.rst (the source file for b2c : SEPA BankToCustomer) as an
example. It fails also when I go back to Django 3.1.7 (the version I had before
upgrading).
Note that I also have to run pm prep in the gerd demo
project after every change. This indicates that the issues are caused by demo
data being generated differently because sorting behaviour has changed.
The failure in docs/specs/b2c.rst comes because in
lino_welfare.modlib.sepa.fixtures.demo the following line failed with
Django 3.2:
qs = Company.objects.filter(sepa_accounts__iban__gt='').distinct()
I changed it to:
qs = Company.objects.exclude(sepa_accounts__iban='').distinct()
This change caused the companies to be sorted differently.
The special sqlite sorting hack in
lino_welfare.lib.welfare.models.customize_sqlite() seems also related to
these issues. In Django 3.2 the stricmp() function caused a NameError:
name ‘cmp’ is not defined and I have no explanation why this error didn’t come
already earlier. The built-in cmp() function has been removed in Python 3
but it was still being used, and nobody complained. Which made me suspect that
the hack wasn’t active.
Maybe the following documented change is responsible: The django.db.models.Field equality operator now correctly distinguishes inherited field instances across models. Additionally, the ordering of such fields is now defined.
The lino_xl.lib.contacts.Company model had no Meta.ordering specified.
ordering = [‘name’]
The docs about Meta.ordering
have a warning “Ordering is not a free operation. Each field you add to the
ordering incurs a cost to your database. Each foreign key you add will
implicitly include all of its default orderings as well.” Not sure how to
understand this. Does it mean that we should rather use only
lino.core.tables.AbstractTable.order_by?
I thought that the issue comes because querysets are no longer automatically sorted by id (when Meta.ordering) is not specified. So I replaced, in welfare/**fixtures/*.py, all occurrences of .objects.all() by .objects.order_by(‘id’). And added an .order_by(‘id’) to all calls to .objects.filter(…) that didn’t specify any ordering.
I now at least understand the problem: certain models don’t have a default ordering. And the demo fixtures in welfare happily created coachings and other things based on non-ordered data. That was no problem because unordered data was in some constant way unordered. Until Django 3.2 where this ordering changes.
But that didn’t help, either.
The docs/specs/newcomers.rst fixture helped me to understand more:
$ dt docs/specs/newcomers.rst
Failed example:
rt.show('newcomers.Competences')
#doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
Expected:
==== ================= ================================ =========
ID Benutzer Fachbereich Aufwand
---- ----------------- -------------------------------- ---------
1 Alicia Allmanns Eingliederungseinkommen (EiEi) 10
2 Hubert Huppertz DSBE 5
3 Mélanie Mélard Ausländerbeihilfe 4
4 Alicia Allmanns Finanzielle Begleitung 6
5 Hubert Huppertz Laufende Beihilfe 2
6 Mélanie Mélard Eingliederungseinkommen (EiEi) 10
7 Alicia Allmanns DSBE 5
**42**
==== ================= ================================ =========
<BLANKLINE>
Got:
==== ================= ================================ =========
ID Benutzer Fachbereich Aufwand
---- ----------------- -------------------------------- ---------
1 Mélanie Mélard Eingliederungseinkommen (EiEi) 10
2 Hubert Huppertz DSBE 5
3 Alicia Allmanns Ausländerbeihilfe 4
4 Mélanie Mélard Finanzielle Begleitung 6
5 Hubert Huppertz Laufende Beihilfe 2
6 Alicia Allmanns Eingliederungseinkommen (EiEi) 10
7 Mélanie Mélard DSBE 5
**42**
==== ================= ================================ =========
<BLANKLINE>
Here is the (simplified) code that generates this database content:
FACULTIES = Cycler(newcomers.Faculty.objects.all())
USERS = Cycler(User.objects.all())
for i in range(7):
yield newcomers.Competence(user=USERS.pop(), faculty=FACULTIES.pop())
>>> users.User.objects.all().ordered
True
>>> newcomers.Faculty.objects.all().ordered
False
Some models have a default ordering:
>>> from lino.api.shell import *
>>> "ORDER BY" in str(contacts.Person.objects.all().query)
True
>>> jobs.Job.objects.all().ordered
True
>>> debts.Account.objects.all().ordered
True
>>> jobs.ContractType.objects.all().ordered
True
>>> art61.ContractType.objects.all().ordered
True
>>> immersion.ContractType.objects.all().ordered
True
>>> immersion.Goal.objects.all().ordered
True
But many models don’t have have it:
>>> from lino.api.shell import *
>>> "ORDER BY" in str(contacts.Company.objects.all().query)
False
>>> pcsw.Client.objects.all().ordered
False
>>> "ORDER BY" in str(clients.ClientContactType.objects.all().query)
False
>>> "ORDER BY" in str(aids.Granting.objects.all().query)
False
>>> "ORDER BY" in str(households.Household.objects.all().query)
False
>>> "ORDER BY" in str(cv.StudyType.objects.all().query)
False
>>> isip.ContractEnding.objects.all().ordered
False
>>> aids.AidType.objects.all().ordered
False
>>> isip.ContractType.objects.all().ordered
False
>>> isip.Contract.objects.all().ordered
False
>>> immersion.Contract.objects.all().ordered
False
Getting notified when long-running command terminates¶
Since pm prep takes a minute or more to complete (on my computer), I run
it as follows to get an acoustic notification when it is done:
$ pm prep --noinput ; espeak done
The –noinput option is useful here because more than once I launched the process and forget to hit ENTER in order to confirm the question:
We are going to flush your database (.../gerd/settings/default.db).
Are you sure (y/n) ? [Y,n]?
But typing “; espeak done” each time is a bit tedious. Can’t we optimize this?
Some surfing: https://gordonlesti.com/linux-audio-notification-after-long-running-command-has-finished/ https://itsfoss.com/notification-terminal-command-completion-ubuntu/
- ::
$ sudo apt install libnotify-bin