Monday, December 1, 2014

UnicodeEncodeError

I received another unicode error:

UnicodeEncodeError at /api/sales/InvoicesByJournal
'ascii' codec can't encode character u'\xfc' in position 64: ordinal not in range(128)

It happened when Lino tried to say “As Anonymous you have no permission to run this action” in German (where the internationlized text (u’Als Anonym haben Sie nicht die Berechtigung, diese Aktion auszufxfchren.’) contains non-ascii characters.

I finally wrote a tested document Refusing permission to an anonymous request. which reproduces this situation.

The explanation was that lino.utils.ajax did the following:

(exc_type, exc_info, tb) = sys.exc_info()
...
response += "%s\n\n" % exc_info

which failed when the exc_info of our PermissionDenied exception was a unicode string with non-ascii characters.

One solution would have been to explicitly turn the target string of the % operation into a unicode string:

response += u"%s\n\n" % exc_info

But I preferred to add from __future__ import unicode_literals.

While this fixed our bug, it did not solve another problem in Python’s doctest:

Traceback (most recent call last):
  ...
  File "/usr/lib/python2.7/doctest.py", line 1331, in __run
    if check(example.want, got, self.optionflags):
  File "/usr/lib/python2.7/doctest.py", line 1591, in check_output
    if _ellipsis_match(want, got):
  File "/usr/lib/python2.7/doctest.py", line 289, in _ellipsis_match
    if got.startswith(w):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 81: ordinal not in range(128)

So I removed the translation (the call to _()) on the message.

Remember: The text of an Exception should rather not be internationalized because some error handling code may want to write it to a plain ascii stream.

The new cv module

After quite some discussions with Mathieu I seem finally to see where we need to go. Basically they just want Lino to differentiate between “Studies” and “Trainings”. And basically my only problem is that the CV-related data are currently spread over several modules and I consider it important to tidy up with some legacy problems.

  • Create a new module lino.modlib.cv which unites the CV-related models from lino_welfare.modlib.jobs, lino_welfare.modlib.isip, lino_welfare.modlib.cv, and lino_welfare.chatelet.lib.jobs.

Change in database structure:

  • moved models from jobs to cv : Study, Experience
  • moved models from isip to cv : StudyType, EducationLevel, Status, Regime, Sector
  • renamed field started to start_date and stopped to end_date in: cv.Study, cv.Experience
  • Added a new model cv.Training which is quite similar to cv.Study. So they have a common base class cv.Schoolilng But e.g. their lists of types are different.
  • Added a new model cv.TrainingType
  • Added a new model cv.Duration which is another way to classify employment contracts (i.e. cd.Experience) according their duration.