Saturday, July 27, 2019¶

WebDAV support¶

I pushed my work on #3036. Summary of the changes:

I added a new option --webdav which causes the nginx conf file to have a webdav location.

I reviewed the docs (WebDAV, getlino).

The default value for --repos-base is now an empty string.

Running getlino startsite on a server with a shared env (i.e. a non-empty value for --shared-env) must never update existing packages in a shared env, but it should add new repositories. We don’t want one site to potentially break the shared environment. But automatically adding new repositories is handy when setting up a development environment.

If you have a shared-env but an empty repositories-root, Lino will install new repositories below the shared env (root path is built by appending --env-link to the shared env).

You can have an empty shared-env and a non-empty repositories-root. The new site has its own env, but uses repositories from your work directories. Useful on a development machine.

New option --local-prefix which defaults to ‘lino_local’. This is the top-level package name for importable code. On a server with /usr/local/lino/ as --sites-base we will have the following layout:

/usr/local/lino/lino_local/mysite1      # a the mysite1 project
/usr/local/lino/lino_local/settings.py  # server-wide default settings

The projects-root will be added to the PYTHON_PATH when a project runs. The settings.py of a new site says from lino_local.settings import *, and the manage.py sets DJANGO_SETTINGS_MODULE to 'lino_local.mysite1.settings'.

The “project directory” of a project “mysite1” is now /usr/local/lino/lino_local/mysite1 (no longer /usr/local/lino/mysite1). If you want to preserve your existing sites, you must say something like this:

$ cd /usr/local/lino/lino_local
$ mkdir lino_local
$ mv mysite1 lino_local/
$ mv shared/*.py lino_local/

The default for --shared-env is an empty string, which means that new sites will get their own environment. This is the normal case on a production server.

Why do we have an option --repos-base? Can’t we simply say that this is always built from --shared-env by appending --env-link? Because on a development machine you may have e.g.:

projects_root = /usr/local/lino
repositories_root = /home/luc/dell1tb/work
shared_env = /home/luc/dell1tb/virtualenvs/py3

I merged KNOWN_LIBS and KNOWN_APPS.

TODO:

The lino_*.js files generated by nginx for the first request are not group writable. Which means that umask is not set correctly for the nginx service.
rename projects-root to sites-root and project_dir to site_dir (Django calls them “projects” but we call them “sites”). A “Lino project” is definitively not the same as a Lino site.

Community Guide¶

General top-level structure of the cg should probably be:

About this document (free to use in your agreements, maintained by the LSF, …)
About the LSF (actors, contacts, …)
About Lino projects (life cycle, …)
Community rules (agreements, …)

Django migrations¶

I’d like us to start playing with Django migrations. Here are some instructions for getting started with #2322.

Preliminary readings:

https://realpython.com/django-migrations-a-primer/ is a good introduction to the topic.
https://realpython.com/digging-deeper-into-migrations/ important to understand

To start playing, you can now go to cosi demo project (e.g. cosi1 or cosi2) and run makemigrations. You will get:

$ pm makemigrations
No changes detected

No Lino plugin has a migrations directory, which means for Django that no plugin participates in the migrations game.

You can say:

$ pm makemigrations contacts
Migrations for 'contacts':
  /path/to/work/cosi/lino_cosi/lib/contacts/migrations/0001_initial.py
    - Create model CompanyType
    - Create model Partner
    - Create model Role
    - Create model RoleType
    - Create model Company
    - Create model Person
    - Add field type to role
    - Add field city to partner
    - Add field country to partner
    - Add field payment_term to partner
    - Add field purchase_account to partner
    - Add field region to partner
    - Add field company to role
    - Add field person to role

But that won’t help us because it creates the migrations in the wrong place. Django migrations cannot be stored individually per plugin because the database structure of a plugin can vary depending on other plugins of the application.

So please remove the /path/to/work/cosi/lino_cosi/lib/contacts/migrations before going on.

When migration_module is None,

Now uncomment the following line in lino_cosi.lib.cosi.settings:

class Site(Site):
    ...
    migration_module = 'lino_cosi.lib.cosi'

When migration_module is set, Lino automatically sets the MIGRATION_MODULES setting to something like this:

MIGRATION_MODULES = {
   "contacts" : "lino_cosi.lib.cosi`,
   "ledger" : "lino_cosi.lib.cosi`,
   ...
}

IOW for all plugins that have at least one model, if adds an item to that dictionary.

The migration_module specifies the main plugin* of an application. For Lino Così this would be lino_cosi.lib.cosi.

Note that the lino_cosi.lib.cosi plugin has already a central role because it holds the locales directory for all cosi plugins. It would now also hold a migrations directory.

Now try again to run makemigrations. You will get:

django.db.migrations.exceptions.BadMigrationError: Migration user_types in app lino has no Migration class

I guess that this caused by the choicelist fields which do not yet have a serializer defined or because the deconstruct() method is wrong. See here.

Next step for #2322 would be to get this to pass and to generate an initial migration in lino_cosi/lib/cosi/migrations.

more getlino¶

I reviewed and continued my work on #3036.

Hamza, you really don’t need to prefix “sudo” to every command sent to Installer.runcmd() because anyway the whole script must run as root.

Yes it is important to set umask for the nginx server process as well (not just for each uwsgi process in supervisor where it is also required). nginx seems to also create e.g. .pyc or .js files. It is not normal that I need sudo to remove a project directory. The default system umask is 0022, that is, it removes the group write permission on new files.

But how to do this? I found that /etc/init.d/nginx sources another file /etc/default/nginx. So I made getlino add a line “umask 0002” to that file. (I read here that this should no longer be used because Debian now uses systemd. But in /etc/systemd/system I cannot find anything related to nginx.) After doing this, I saw that actually my supervisor conf files were not setting the umask! This was the problem. The nginx main process does not create any files and therefore does not need an umask.

My version of this morning didn’t yet work with --webdav because the client_body_temp_path in nginx.conf pointed to a non-existing path. Now it points to /tmp. Not sure whether this is what we want, but seems to work for me. I did yet actually test a webdav access (Lino Avanti would be good for testing, but currently startsite still fails with an avanti site).

I removed creation of a reminder in :fixture`demo2` of avanti because it failed under mysql. Not yet tested whether this affects the test suite.