Saturday, September 14, 2019

Migrating data between two servers

I wrote a initdb_from_prod.sh for them. Nice: it is very similar to a standard initdb_from_prod.sh, but OLD is a remote path.

Some of their media files were not readable for group members:

-rw------- 1 www-data www-data 2851086 Jun 12 14:28 /.../media/uploads/2019/06/....jpg

Which caused rsync to exit with an error code:

rsync: send_files failed to open ".../media/uploads/2019/06/....jpg": Permission denied (13)
...
sent 52,279 bytes  received 669,458,491 bytes  5,510,376.71 bytes/sec
total size is 698,343,313  speedup is 1.04
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1677) [generator=3.1.3]

I fixed this by changing the file permissions on the old server:

sudo find ! -perm /g=r -exec chmod g+r '{}' +

Noted that each rsync run asks for my password. I guess this is because ssh-agent is not installed.

En passant I found and fixed another bug in getlino: When running as root, getlino configure now also creates empty directories for --log-base and --backups-base and sets their permissions.

Oops, the restore.py script got killed. One file, cal_guest_1.py (3.7 MB), passed without problems, but changes_change_1.py is obviously too big: 11 MB. Bot files contain 50004 lines (this number comes from the --max-row-count option of pm dump2py). So I set –max-row-count` to 10000 on the production server, make a new snapshot and then run initdb_from_prod.sh again.

And of course I need to do some manual changes for data migration, for example:

TypeError: Comment() got an unexpected keyword argument 'body_preview'

One question is where to store these changes. I cannot use the mirrored restory.py file because it would be overwritten next time. change in make_snapshot.sh : don’t delete the snapshot directory, just run dump2py with –overwrite option.

Last problem for the migration was:

- cal.Event Unresolved value 'pisa' (<class 'str'>) for printing.BuildMethods (set Site.strict_choicelist_values to False to ignore this) (144 object(s) with primary key 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 305, 306, 307, 308, 309, 310, 311, 312, 313, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 751, 752, 753, 754, 755, 757, 758, 759, 760, 762, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 879, 880, 886, 887, 888, 889, 900, 901, 902, 903, 904, 905, 906, 907, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963)

First problem after migration:

TypeError: '>' not supported between instances of 'NoneType' and 'str'

This was a bug in lino_xl.lib.cv.LanguageKnowledge which happened only when CEF Level was empty.

Released XL 19.9.2.