Saturday, March 18, 2017

Nightly magics with Python dumps

The following is a beautiful example of why I love Python fixtures. It is something I really wouldn’t want to do with an SQL dump.

I am repairing a loss of production data caused by #1354. The whole operation took about 2:30 hours. This ticket was caused by the customer’s internal communication problems, so it is not Lino’s fault.

It is about lino_welfare.modlib.aids. The Confirmation.full_clean() method has a ValidationError “Date range X lies outside of granted period Y”. There are three models which implement Confirmation. This validator caused 320 confirmations to not make it into the new database when I migrated yesterday.

My first question was: how many confirmations did they issue on Friday? I saw this by looking into the corresponding files:

  • SimpleConfirmation ( : 2372 and 2373
  • IncomeConfirmation ( : 2810, 2811 and 2812
  • RefundConfirmation ( : no new rows

I guess that loosing these 5 rows from yesterday wouldn’t be that hard if they only get the 320 older confirmations back. But actually I can easily save these 5 rows as well. I will reload the old files and then add the 5 rows from the new files. Actually I will do it in the other order: first load the 5 rows from the new files, and then reload the old files.

Here is how I did it:

# unpack the last snapshot before the migration
$ unzip /var/backups/lino/ a

# make a snapshot of the current state
$ dump2py b

# use sed to remove almost all lines from two files:
$ sed -e '4,1941d' a/ > b/
$ sed -e '4,1984d' a/ > b/simple_tail

# make a importable:
$ touch a/

$ cp b/ b/
$ nano b/

Find the following three lines of b/


Replace them by:

# load the last lines of evening backup:
# load all other confirmations from morning backup
from a.restore import create_aids_simpleconfirmation, create_aids_incomeconfirmation, create_aids_refundconfirmation

Note how I must import the create_aids_xxxconfirmation functions from the old file because the order of fields has changed.

And then:

$ python run b/

I verified using the web interface (Explorer menu) the number of rows before and after the operation:

model bef aft diff
IncomeConfirmation 2273 2539 266
RefundConfirmation 2586 2616 30
SimpleConfirmation 1984 2008 24
Total 6843 7163 320

Yes, 320 is the number which Lino reported when I migrated the database.

More optimizations in Noi

I explained to Iiris my work on Lino and the Lino Vilma project. She contributed two real tickets. That’s where I discovered #1593.

Other optimizations mainly for #1526 and #1549.

For example the MyCompetences and TicketsByCompetences views.

TODO: Rename the slave_grid_format (of to display_format and make it work even when it is the main element of the window. It might be more intuitive for MyCompetences.

I removed the more_text field from CommentDetail.

I realized that Vote.end_user, Ticket.end_user and faculties.Competence.supplier must be the same model. I renamed faculties.Competence.supplier to faculties.Competence.end_user

OffersByEndUser OffersBySupplier

lino.core.fields.fields_list() now issues a clearer error message when something is not a field. And it is more severe, too. Trigger was a typo bug: I had written an accidental comma behind the field type:

class Foo(dd.Model):
    x = ForeignKey(...),

SuggestedTickets didn’t work. I changed it to SuggestedTicketsByEndUser

The whole thing currently would fail if end_user_model is not a subclass of user_model (which is the case in Noi where a User is just a special kind of Person).

I reimplemented