Thursday, February 4, 2016

I tried a release on testlino at cpaseupen for #143. But when importing latest data I got an error similar to the one which happens in the test suite after #706.

Sandeep did not yet find any fix (which is understandable because it is not trivial), so I dived into this and fixed it. The error of the test suite was reproducible as follows:

$ cd docs/tutorials/pisa
$ python manage.py test

Here is the traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 1316, in __run
    compileflags, 1) in test.globs
  File "<doctest index.rst[3]>", line 1, in <module>
    settings.SITE.site_config.update(default_build_method='pisa')
  File "/lino/modlib/system/models.py", line 91, in update
    self.save()
  File "/lino/modlib/system/models.py", line 95, in save
    super(SiteConfig, self).save(*args, **kw)
  File "/django/db/models/base.py", line 700, in save
    force_update=force_update, update_fields=update_fields)
  File "/django/db/models/base.py", line 728, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/django/db/models/base.py", line 793, in _save_table
    forced_update)
  File "/django/db/models/base.py", line 843, in _do_update
    return filtered._update(values) > 0
  File "/django/db/models/query.py", line 645, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/django/db/models/sql/compiler.py", line 1149, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/django/db/models/sql/compiler.py", line 837, in execute_sql
    sql, params = self.as_sql()
  File "/django/db/models/sql/compiler.py", line 1117, in as_sql
    val = field.get_db_prep_save(val, connection=self.connection)
  File "/django/db/models/fields/__init__.py", line 728, in get_db_prep_save
    prepared=False)
  File "/django/db/models/fields/__init__.py", line 720, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/lino/core/choicelists.py", line 711, in get_prep_value
    return value.value
AttributeError: 'str' object has no attribute 'value'

Above error happens at the following line of lino_book.projects.pisa:

settings.SITE.site_config.update(default_build_method='pisa')

Yes, the lino.modlib.system.models.SiteConfig.update() method now should also call full_clean() before saving the object. Done.

Unfortunately this change did not yet solve the error which occured when importing data. This error came because the restore.py script created by pm dump2py also initialized the choicelist fields of database objects using strings instead of their Choice instances.

To be able to load dumps which have been created by running versions, the get_prep_value of a ChoiceListField now calls to_python().

Checkin and test. Problem is fixed.

I also converted pm dump2py so that it now explicitly resolves these strings (by calling the choicelist’s get_by_value).

This revealed a theoretical bug in pm dump2py: the new_content_type_id() function was called for ForeignKey fields only if they were defined on the child model.

I also removed the usage of get_fields_with_model() (which will be removed in Django 1.10) from that module.

After these changes I observed for the first time that the restore.py takes rather long to run. For example in min2:

$ cd lino/projects/min2
$ python manage.py initdb_demo
$ python manage.py dump2py a
$ python manage.py run a/restore.py

While the initdb_demo takes 12s on my machine, the run a/restore.py takes 1m31s.

It is possible that (1) this problem existed already before my changes, and (2) that it occurs only under sqlite. To explore this, I measued the time taken to load the cpaseupen database into their testlino.

Before the dump2py optimizations it took:

real  18m52.292s
user  16m41.148s
sys   0m23.908s

After the optimizations it took:

real 19m6.751s
user 16m55.956s
sys  0m23.784s

Which shows at least that the problem is not caused by today’s optimizations.

In Lino Welfare there was another problem caused by #706:

[2016-02-04 08:46:48.162518] Traceback (most recent call last):
File "/lino_welfare/modlib/cbss/mixins.py", line 256, in execute_request
    retval = self.execute_request_(now, simulate_response)
File "/lino_welfare/modlib/cbss/mixins.py", line 560, in execute_request_
    return self.execute_newstyle(client, info, simulate_response)
File "/lino_welfare/modlib/cbss/models.py", line 857, in execute_newstyle
    si.language = self.language.value
AttributeError: 'str' object has no attribute 'value'"

Note that above traceback was hidden in the database because CBSS requests capture any exception which occurs during the request. The traceback occured during initdb but bekame visible only because the Tx25 object had no result.

>>> obj = cbss.RetrieveTIGroupsRequest.objects.get(pk=1)
>>> print(obj.debug_messages)

The bug was in RetrieveTIGroupsRequest.fill_from_person and in the lino_welfare.modlib.cbss.fixtures.cbss_demo fixture.

This case confirms the general rule: SubfieldBase added some magic to the model so that assigning something to that field would automatically invoke the field’s to_python method. But they removed that magic. So we must find the places which relied on it, and simply have them do the lookup themselves.

While I was there, I removed another usage of SubfieldBase (from lino_xl.lib.sepa.fields.UppercaseTextField).

Also the test suite of Lino Noi had a failure caused by #706. Changes in lino_noi.lib.clocking.models.Session.get_duration and lino.core.fields.QuantityField.

Checkin.