# Saturday, March 18, 2017¶

## Nightly magics with Python dumps¶

The following is a beautiful example of why I love Python fixtures. It is something I really wouldn’t want to do with an SQL dump.

I am repairing a loss of production data caused by #1354. The whole operation took about 2:30 hours. This ticket was caused by the customer’s internal communication problems, so it is not Lino’s fault.

It is about lino_welfare.modlib.aids. The Confirmation.full_clean() method has a ValidationError “Date range X lies outside of granted period Y”. There are three models which implement Confirmation. This validator caused 320 confirmations to not make it into the new database when I migrated yesterday.

My first question was: how many confirmations did they issue on Friday? I saw this by looking into the corresponding files:

• SimpleConfirmation (aids_simpleconfirmation.py) : 2372 and 2373
• IncomeConfirmation (aids_incomeconfirmation.py) : 2810, 2811 and 2812
• RefundConfirmation (aids_refundconfirmation.py) : no new rows

I guess that loosing these 5 rows from yesterday wouldn’t be that hard if they only get the 320 older confirmations back. But actually I can easily save these 5 rows as well. I will reload the old files and then add the 5 rows from the new files. Actually I will do it in the other order: first load the 5 rows from the new files, and then reload the old files.

Here is how I did it:

# unpack the last snapshot before the migration
$unzip /var/backups/lino/20170316_1909.zip a # make a snapshot of the current state$ dump2py b

# use sed to remove almost all lines from two files:
$sed -e '4,1941d' a/aids_incomeconfirmation.py > b/income_tail.py$ sed -e '4,1984d' a/aids_simpleconfirmation.py > b/simple_tail

# make a importable:
$touch a/__init__.py$ cp b/restore.py b/restore_new.py
$nano b/restore_new.py  Find the following three lines of b/restore.py: execfile("aids_incomeconfirmation.py") execfile("aids_refundconfirmation.py") execfile("aids_simpleconfirmation.py")  Replace them by: # load the last lines of evening backup: execfile("income_tail.py") execfile("simple_tail.py") # load all other confirmations from morning backup from a.restore import create_aids_simpleconfirmation, create_aids_incomeconfirmation, create_aids_refundconfirmation execfile("../a/aids_incomeconfirmation.py") execfile("../a/aids_refundconfirmation.py") execfile("../a/aids_simpleconfirmation.py")  Note how I must import the create_aids_xxxconfirmation functions from the old restore.py file because the order of fields has changed. And then: $ python manage.py run b/restore_new.py


I verified using the web interface (Explorer menu) the number of rows before and after the operation:

model bef aft diff
IncomeConfirmation 2273 2539 266
RefundConfirmation 2586 2616 30
SimpleConfirmation 1984 2008 24
Total 6843 7163 320

Yes, 320 is the number which Lino reported when I migrated the database.

## More optimizations in Noi¶

I explained to Iiris my work on Lino and the Lino Vilma project. She contributed two real tickets. That’s where I discovered #1593.

Other optimizations mainly for #1526 and #1549.

For example the MyCompetences and TicketsByCompetences views.

TODO: Rename the slave_grid_format (of lino.core.actor.Actor) to display_format and make it work even when it is the main element of the window. It might be more intuitive for MyCompetences.

I removed the more_text field from CommentDetail.

I realized that Vote.end_user, Ticket.end_user and faculties.Competence.supplier must be the same model. I renamed faculties.Competence.supplier to faculties.Competence.end_user

OffersByEndUser OffersBySupplier

lino.core.fields.fields_list() now issues a clearer error message when something is not a field. And it is more severe, too. Trigger was a typo bug: I had written an accidental comma behind the field type:

class Foo(dd.Model):
x = ForeignKey(...),


SuggestedTickets didn’t work. I changed it to SuggestedTicketsByEndUser

The whole thing currently would fail if end_user_model is not a subclass of user_model (which is the case in Noi where a User is just a special kind of Person).

I reimplemented lino_noi.projects.team.settings.fixtures.demo