Friday, August 5, 2016

Moved three demo projects

I moved three demo projects from Lino Così to Developer Guide:

• lino_book.projects.apc

• lino_book.projects.pierre

• lino_book.projects.cosi_ee (previously lino_book.projects.ylle)

This is mainly because these projects depend on commondata, a useless dependency for normal production sites. Also they are merely localizations of lino_cosi.projects.std and more useful as examples for local settings.py modifications.

I also moved some tested documents (cosi_ee : A Lino Così for Estonia and Refusing permission to an anonymous request) from Lino Così to Developer Guide because they rely on these demo projects.

And the commondata packages are no longer required by Lino Così. I moved this dependency to Developer Guide.

Missing translations during build on Travis

Travis reports a failing build for Lino Così with problems like this one:

**********************************************************************
File "/home/travis/build/lino-framework/cosi/docs/specs/general.rst", line 140, in general.rst
Failed example:
    rt.login('romain').show_menu()
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -2,7 +2,7 @@
     - Products : Products, Product Categories
     - Comptabilité :
    -  - Sales : Factures vente (SLS), Sales credit notes (SLC)
    -  - Purchases : Factures achat (PRC)
    -  - Financial : Payment Orders (PMO), Caisse (CSH), Bestbank (BNK), Opérations diverses (MSC)
    +  - Sales : Sales invoices (SLS), Sales credit notes (SLC)
    +  - Purchases : Purchase invoices (PRC)
    +  - Financial : Payment Orders (PMO), Cash (CSH), Bestbank (BNK), Miscellaneous Journal Entries (MSC)
       - Create invoices
     - Office : Mes Fichiers téléchargés, Mon courrier sortant, Mes Extraits
**********************************************************************

This looks as if some part of the i18n system is missing. Hmm…

The translation of “Sales invoices” to Factures vente” is in cosi/lino_cosi/lib/cosi/locale/fr/LC_MESSAGES/django.po.

Hamza, I leave this job to you! This is #1071.

Getting Lino to run with Django 1.10

No fixture named ‘few_languages’ found

Django 1.10 complained that No fixture named 'few_languages' found., and Django was right there: When I specify ‘few_languages’ in demo_fixtures and there is no such fixture at all in any of the installed plugins, then the application developer must simply remove ‘few_languages’ from their demo_fixtures.

Same problem with fixtures ‘props’, ‘demo_events’, ‘welfare_std’, ‘mini’, ‘local’.

MyModel._meta.get_all_related_objects() no longer exists

In lino.utils.mti there was a deprecated usage of MyModel._meta.get_all_related_objects(). As they explain in Migrating from the old API, this must become:

[
    f for f in MyModel._meta.get_fields()
    if (f.one_to_many or f.one_to_one)
    and f.auto_created and not f.concrete
]

And since I called MyModel._meta.get_all_related_objects() with three keyword parameters:

include_hidden=True, include_proxy_eq=True, local_only=True

The include_hidden=True goes to get_fields() and the local_only=True becomes an if f.model is model. I did not investigate why we had include_proxy_eq=True but it seems that this is no longer necessary…

Changed Command API

It seems that Django 1.10 has changed something with the handling of positional arguments (they are no longer added to the *args tuple passed to handle(). . I changed the commands pm run, show and pm dump2py.

Query filtering with __isnull

The The lino_xl.lib.statbel.countries plugin specs fails under Django 1.10. It seems that the isnull lookup operator has changed behaviour or maybe even a bug.

I created a script django110_isnull.py in lino_welfare.projects.eupen:

from lino.api.shell import *
Country = rt.models.countries.Country
print Country.objects.all().count()
print Country.objects.filter(actual_country__isnull=True).count()
print Country.objects.filter(actual_country__isnull=False).count()
print Country.objects.exclude(actual_country__isnull=True).count()
be = Country.objects.get(isocode="BE")
print type(be.actual_country)

The output of this script under Django 1.9 is:

270
266
4
4
<type 'NoneType'>

But under 1.10 it is:

270
266
266
4
<type 'NoneType'>

Strange. There are 270 countries, and 4 of them have actual_country set, all the others have a NULL value in that field (as for example Belgium). I currently don’t see another explanation for this than a Django bug (which seems hard to believe). When foo is a nullable ForeignKey, then qs.filter(foo__isnull=False) should not return the same query as qs.filter(foo__isnull=True)!

Anyway, as long as this problem is not fixed, we must leave ‘django<1.10’ in our dependencies.

Hamza, I suggest that you try to write a minimal pure Django project which reproduces the problem. If you can reproduce it, then submit a ticket on Django. For us this is #1115.

AttributeError: ‘module’ object has no attribute ‘FFI’

I am having the following traceback on a production server:

Traceback (most recent call last):
  ...
  File ".../repositories/lino/lino/modlib/weasyprint/models.py", line 10, in <module>
    from .choicelists import *
  File ".../repositories/lino/lino/modlib/weasyprint/choicelists.py", line 25, in <module>
    from weasyprint import HTML
  File ".../virtualenvs/a/lib/python2.7/site-packages/weasyprint/__init__.py", line 338, in <module>
    from .css import PARSER, preprocess_stylesheet  # noqa
  File ".../virtualenvs/a/lib/python2.7/site-packages/weasyprint/css/__init__.py", line 30, in <module>
    from . import computed_values
  File ".../virtualenvs/a/lib/python2.7/site-packages/weasyprint/css/computed_values.py", line 18, in <module>
    from .. import text
  File ".../virtualenvs/a/lib/python2.7/site-packages/weasyprint/text.py", line 18, in <module>
    import cairocffi as cairo
  File ".../virtualenvs/a/lib/python2.7/site-packages/cairocffi/__init__.py", line 20, in <module>
    from ._ffi import ffi
  File ".../virtualenvs/a/lib/python2.7/site-packages/cairocffi/_ffi.py", line 4, in <module>
    ffi = _cffi_backend.FFI('cairocffi._ffi',
AttributeError: 'module' object has no attribute 'FFI'

It happens on the first incoming web request after restarting apache. A second traceback, ending in RuntimeError: populate() isn't reentrant, is just a follow-up of this traceback. I opened ticket #1119 for this.

Yes, there is something wrong with the permissions of cairocffi:

$ python -c 'import _cffi_backend; print type(_cffi_backend.FFI)'
<type 'type'>

$ sudo -u www-data python -c 'import _cffi_backend; print type(_cffi_backend.FFI)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'FFI'

Actually the problem can be reduced to this:

$ python -c "import cairocffi"  # OK

$ sudo -u www-data python -c "import cairocffi"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named cairocffi

That is, as admin used I can import cairocffi, as www-data not. This indicates a permissions problem.

I tried this (now that umask and ownership problems seem fixed):

$ pip install --upgrade --force-reinstall cffi
$ pip install --upgrade --force-reinstall cairocffi
$ pip install --upgrade --force-reinstall CairoSVG

I tried whether there are directories in my virtualenv which are not executable for other group members:

$ find -L env -type d ! -perm /g=x

I made sure that all Python files are owned by group www-data:

$ chown -R admin:www-data env

I visually checked the permissions of all directories:

$ find -L env/repositories -type d | xargs ls -ld | less

Find .pyc files in my site-packages which are not group-writable:

$ find -L env/local/lib/python2.7/site-packages -name '*.pyc' ! -perm /g=w

Find .pyc files in my repositories which are not group-writable:

$ find -L env/repositories/ -name '*.pyc' ! -perm /g=w
env/repositories/lino/lino/modlib/lino_startup/management/commands/show.pyc

Aha, this indicates that the setgid flags are not set. This is not the cause of our problem, but let’s verify the reason. When the show.pyc file was created by user admin, it did not inherit the group of its directory. This came because I hadn’t yet set umask 002 in the activate script of my new environment.

Some background. This problem comes because the site uses lino.modlib.weasyprint which depends on weasyprint, which depends on cairocffi. cairocffi is a CFFI-based drop-in replacement for Pycairo. Pycairo is an object-oriented API for cairo. Cairo is a 2D vector graphics library.

Google found this discussion where SimonSapin suggests:

$ python -c 'print(__import__("_cffi_backend").__file__)'
/home/admin/virtualenvs/a/local/lib/python2.7/site-packages/_cffi_backend.so
$ sudo -u www-data python -c 'print(__import__("_cffi_backend").__file__)'
/usr/lib/python2.7/dist-packages/_cffi_backend.x86_64-linux-gnu.so

And indeed, I have different files.