Saturday, March 18, 2017

Nightly magics with Python dumps

The following is a beautiful example of why I love Python fixtures. It is something I really wouldn’t want to do with an SQL dump.

I am repairing a loss of production data caused by #1354. The whole operation took about 2:30 hours. This ticket was caused by the customer’s internal communication problems, so it is not Lino’s fault.

It is about lino_welfare.modlib.aids. The Confirmation.full_clean() method has a ValidationError “Date range X lies outside of granted period Y”. There are three models which implement Confirmation. This validator caused 320 confirmations to not make it into the new database when I migrated yesterday.

My first question was: how many confirmations did they issue on Friday? I saw this by looking into the corresponding files:

  • SimpleConfirmation (aids_simpleconfirmation.py) : 2372 and 2373

  • IncomeConfirmation (aids_incomeconfirmation.py) : 2810, 2811 and 2812

  • RefundConfirmation (aids_refundconfirmation.py) : no new rows

I guess that loosing these 5 rows from yesterday wouldn’t be that hard if they only get the 320 older confirmations back. But actually I can easily save these 5 rows as well. I will reload the old files and then add the 5 rows from the new files. Actually I will do it in the other order: first load the 5 rows from the new files, and then reload the old files.

Here is how I did it:

# unpack the last snapshot before the migration
$ unzip /var/backups/lino/20170316_1909.zip a

# make a snapshot of the current state
$ dump2py b

# use sed to remove almost all lines from two files:
$ sed -e '4,1941d' a/aids_incomeconfirmation.py > b/income_tail.py
$ sed -e '4,1984d' a/aids_simpleconfirmation.py > b/simple_tail

# make a importable:
$ touch a/__init__.py

$ cp b/restore.py b/restore_new.py
$ nano b/restore_new.py

Find the following three lines of b/restore.py:

execfile("aids_incomeconfirmation.py")
execfile("aids_refundconfirmation.py")
execfile("aids_simpleconfirmation.py")

Replace them by:

# load the last lines of evening backup:
execfile("income_tail.py")
execfile("simple_tail.py")
# load all other confirmations from morning backup
from a.restore import create_aids_simpleconfirmation, create_aids_incomeconfirmation, create_aids_refundconfirmation
execfile("../a/aids_incomeconfirmation.py")
execfile("../a/aids_refundconfirmation.py")
execfile("../a/aids_simpleconfirmation.py")

Note how I must import the create_aids_xxxconfirmation functions from the old restore.py file because the order of fields has changed.

And then:

$ python manage.py run b/restore_new.py

I verified using the web interface (Explorer menu) the number of rows before and after the operation:

model

bef

aft

diff

IncomeConfirmation

2273

2539

266

RefundConfirmation

2586

2616

30

SimpleConfirmation

1984

2008

24

Total

6843

7163

320

Yes, 320 is the number which Lino reported when I migrated the database.

More optimizations in Noi

I explained to Iiris my work on Lino and the Lino Vilma project. She contributed two real tickets. That’s where I discovered #1593.

Other optimizations mainly for #1526 and #1549.

For example the MyCompetences and TicketsByCompetences views.

TODO: Rename the slave_grid_format (of lino.core.actor.Actor) to display_format and make it work even when it is the main element of the window. It might be more intuitive for MyCompetences.

I removed the more_text field from CommentDetail.

I realized that Vote.end_user, Ticket.end_user and faculties.Competence.supplier must be the same model. I renamed faculties.Competence.supplier to faculties.Competence.end_user

OffersByEndUser OffersBySupplier

lino.core.fields.fields_list() now issues a clearer error message when something is not a field. And it is more severe, too. Trigger was a typo bug: I had written an accidental comma behind the field type:

class Foo(dd.Model):
    x = ForeignKey(...),

SuggestedTickets didn’t work. I changed it to SuggestedTicketsByEndUser

The whole thing currently would fail if end_user_model is not a subclass of user_model (which is the case in Noi where a User is just a special kind of Person).

I reimplemented lino_noi.projects.team.settings.fixtures.demo