Luc's developer blog

Saturday, May 30, 2026

Sat, 30 May 2026 00:00:00 +0000

Saturday, May 30, 2026¶

DaySlave is the only table tha sets use_detail_params_value to True. What does that mean?

DaySlave is the base class for

DailySlaveBase, WeeklySlaveBase and MonthlySlaveBase,

who themselves are base for

DailySlave, WeeklySlave and MonthlySlave.

So these three tables are the concrete views that use this option.

It means that these three tables don’t define their own parameter panel but use the parameter panel of their master. Their master is either DailyView, WeeklyView or MonthlyView. These three masters inherit from their common parent CalendarView. They have another attribute, use_detail_param_panel, set to True.

Maybe the names use_detail_param_panel and use_detail_params_value aren’t good names.

use_detail_param_panel : whether this table has a parameter panel in detail view.

Thursday, May 28, 2026

Thu, 28 May 2026 00:00:00 +0000

Thursday, May 28, 2026¶

In patrols1, when calview.WeeklySlave receives parameter values, it returns an empty queryset.

GET: http://127.0.0.1:8000/api/calview/WeeklySlave?dm=grid&fmt=json&limit=99999&lv=1779921642.6281872&mk=0&rp=weak-key-21&ul=en&wt=t
GET: http://127.0.0.1:8000/api/calview/WeeklySlave?dm=grid&fmt=json&limit=99999&lv=1779921642.6281872&mk=-4&pv=&pv=01.01.2026&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&pv=&rp=weak-key-23&ul=en&wt=t

Thursday, May 14, 2026

Thu, 14 May 2026 00:00:00 +0000

Thursday, May 14, 2026¶

A surprise during data migration.

Loading 11734 objects to table cert_sectionresponse...
Deferred SectionResponse #358 ('Schriftbild') : 20250927
20250927
Traceback (most recent call last):
  File ".../env/repositories/lino/lino/utils/dpy.py", line 142, in try_save
    obj.full_clean()
  File ".../env/repositories/prima/lino_prima/lib/cert/models.py", line 285, in full_clean
    super().full_clean()
  File ".../env/repositories/prima/lino_prima/lib/ratings/mixins.py", line 123, in full_clean
    raise Exception("20250927")
Exception: 20250927
...
Exception: Abandoning with 168 unsaved instances:
- cert.SectionResponse 20250927 (168 object(s) with primary key 358, 1942, 2349, 1053, 3485, 2264, 2435, 325, 3308, 751, 3870, 712, 721, 1156, 2424, 2016, 3906, 1477, 3163, 41, 579, 1535, 2370, 924, 3512, 1813, 763, 946, 3532, 1879, 1088, 1467, 3122, 731, 977, 1910, 3329, 1530, 3176, 1768, 4706, 1892, 2394, 826, 3945, 1469, 3697, 324, 2455, 1217, 2944, 1805, 3645, 1027, 3726, 1146, 3137, 1591, 2988, 1930, 2696, 182, 4012, 1636, 2076, 4066, 16, 3736, 998, 3376, 2633, 4679, 1584, 3225, 2970, 3251, 1136, 2505, 1278, 3058, 1368, 3556, 2006, 2213, 1081, 3400, 1502, 3300, 2008, 83, 1663, 3160, 2705, 4217, 2064, 1979, 2086, 2555, 1868, 2404, 2163, 4257, 1302, 4278, 367, 4299, 1983, 518, 2031, 4702, 658, 2426, 1497, 4244, 1882, 3571, 716, 4228, 413, 2581, 5, 1955, 350, 3516, 766, 3554, 821, 4705, 883, 3575, 39, 1943, 674, 1905, 1664, 3086, 1698, 2620, 1542, 4197, 952, 3584, 606, 3033, 525, 1974, 1343, 4130, 1646, 3016, 3594, 2694, 4208, 114, 3606, 2083, 2590, 1929, 3501, 1556, 3694, 1749, 3084, 154, 3617, 1248, 4482, 3999)

This is because both section.rating_type and score are not None.

nano snapshot/cert_sectionresponse.py

# fields: id, body, body_short_preview, body_full_preview, score, smiley, predicate, certificate, section

loader.save(create_cert_sectionresponse(358,

‘Comment.’,’Comment.’,’Comment.’,: ‘5.0’,None,None,3430,6))

After removing the exception:

lino.core.exceptions.UnresolvedChoice: Unresolved value '5.0' (<class 'str'>) for ratings.Predicates (set Site.strict_choicelist_values to False to ignore this)

After changing the exception into a warning and return:

SectionResponse #358 ('Schriftbild') : non-empty rating_type Prädikate and score is given

358 : 1B (2024/25) » AZEMI, Ajan (1B (2024/25)) 2024/25-1 » Schriftbild 1942 : 1A (2024/25) » BAHA SCHYNS, Andrea (1A (2024/25)) 2024/25-1 » Schriftbild 2349 : 1A (2024/25) » BAHA SCHYNS, Andrea (1A (2024/25)) 2024/25-2 » Schriftbild … 3999 : 2A (2024/25) » Z-GUSHANI, Leyla (2A (2024/25)) 2024/25-2 » Schriftbild

Explanation for mail_queue_enter: create file maildrop/870989.992: Read-only file system

Sun, 10 May 2026 00:00:00 +0000

Explanation for mail_queue_enter: create file maildrop/870989.992: Read-only file system¶

Sunday, May 10, 2026

Here is what happened after mail_queue_enter: create file maildrop/870989.992: Read-only file system.

NB I am still unable to send emails to the debian-user mailinglist. I sent an email to listmaster@lists.debian.org, but they too didn’t respond. That’s another story. EDIT: I thought that I was unable, but then it turned out that my message now passed. Although I had tried multiple times for several days in a row. And AFAIK I didn’t change anything in my DNS configs…

But the members of debian-users are really great: without waiting for further information from me, they started exploring my problem and discussed among themselves: https://lists.debian.org/debian-user/2026/05/threads.html

It’s Andy Smith who found the explanation, less than 24 hours after my post. His second post in the thread gave the correct explanation, which now helped me to find the solution.

The problem came because Systemd runs the monit service with ProtectSystem set to strict and without adding /var/spool/postfix/ to ReadWritePaths.

Here is what I had in my /usr/lib/systemd/system/monit.service:

ProtectSystem=strict
ReadWritePaths=/run/ /var/lib/monit/ /var/log/

I just added /var/spool/postfix/ to the list of read-write paths:

ReadWritePaths=/run/ /var/lib/monit/ /var/log/ /var/spool/postfix/

How did I know that it’s /var/spool/postfix where maildrop tries to write to? Because the error message unfortunately specifies only the relative path. I guessed that this path is specified somewhere in the postfix configs, so I said:

# postconf | grep queue

Which returned a few settings, and one of them was:

queue_directory = /var/spool/postfix

My congratulations to the debian-users list. If only I could tell them all this!

Saturday, May 2, 2026

Sat, 02 May 2026 00:00:00 +0000

Saturday, May 2, 2026¶

Today I have been playing during 2 hours with 7 postfix config parameters (myhostname, myorigin, mydestination, inet_protocols, inet_interfaces, local_transport and smtp_bind_address) on two servers (kuu and lumi). There were moments when I didn’t understand the world and had the feeling that some hidden random error generator is running somewhere…

It seems that the article How to Configure Postfix as a Send-Only SMTP Server on IPv4 is not what we want. A default postfix config is good for us, I just need to set three parameters correctly: myhostname, mydestination and myorigin.

TIL:

postconf key1 key2 shows the current settings, one line key=value for each specified key.
postconf key=value updates the main.cf file from the command line. This is useful when playing around. You can specify multiple key=value pairs. A value can be empty. Know your bash rules.
systemctl restart is more than systemctl reload. When you change inet_interfaces or inet_protocols, reloading won’t do it and postfix even issues a warning… but only in the system journal, not on the command line.
Saying “echo foo | mail -s test root” is not enough. It’s important to specify -r (or --return-address) because mail does not consult /etc/mailname, it will just say username@hostname, and hostname is just our nickname, and most mail servers refuse emails with a non-existing domain name in the From field. Here is a generic command-line for sending a test email:
```
$ echo foo | mail -r `whoami`@`cat /etc/mailname` -s test root
```
(I wonder whether it would be better to set hostname to the the full public name. I set it to the nickname only because I don’t want to see the full public name in the prompt.)
In Postfix you can say “/etc/mailname” as the value for myorigin, but not for myhostname. Seems that Postfix checks for a “/” at the beginning only for myorigin. Don’t ask me why.

Here is my recipe for configuring:

# postconf myhostname=`cat /etc/mailname`
# postconf myorigin=/etc/mailname
# postconf 'mydestination=$myhostname,localhost'
# systemctl restart postfix

To verify these three parameters:

# postconf myhostname mydestination myorigin
myhostname = lumi.mylino.net
mydestination = $myhostname,localhost
myorigin = /etc/mailname

Thursday, April 30, 2026

Thu, 30 Apr 2026 00:00:00 +0000

Thursday, April 30, 2026¶

Today a bug in Lino Prima took me a few hours to hunt and fix.

Look at this code:

    class Challenge(Sequenced)::

def get_siblings(self):
    if self.exam_id:
        return self.__class__.objects.filter(exam=self.exam)
    return self.__class__.objects.filter(project_section=self.project_section)

Above code is bad. Below is the corrected code:

def get_siblings(self):
    if self.exam_id:
        return self.__class__.objects.filter(exam=self.exam)
    if self.project_section:
        return self.__class__.objects.filter(project_section=self.project_section)
    return self.__class__.objects.none()

A lino_prima.lib.ratings.Challenge row has always either exam or project_section set. But on a row having both exam and project_section None (a case that theoretically doesn’t exist, but yesterday it did), the buggy get_siblings() will return all rows with project_section being null, i.e. all rows that have have an non-null exam_id. More exactly there were 3454 such rows. How could this exceptional situation happen? We don’t know. It is not worth to try to find out.

Here is the script I ran in to order to diagnose the problem and to repair the data:

import sys
from lino.api.shell import *

qs = ratings.Challenge.objects.filter(
    project_section__isnull=True).order_by('id')
print(qs.count(), qs)
# 3454 <QuerySet [Challenge #10442 ('Ausdrucksvolles Lesen /10 in Löwen (31.07.)'), Challenge #10443 ('Sprechen /10 in Morgenkreis Experiment (31.07.)'), Challenge #10444 ('Sprechen /10 in Morgenkreis Lesen (31.07.)'), Challenge >

for obj in qs:
    print(obj.seqno, obj)

seqnos = set([obj.seqno for obj in qs])
print("There are", len(seqnos), "different seqnos")
# There are 3454 different seqnos

# sys.exit()

for obj in qs:
  obj.seqno = 0
  # do NOT call full_clean() because that would call set_seqno()
  obj.save()

qs = ratings.Challenge.objects.filter(seqno=0).order_by('id')

for obj in qs:
  obj.full_clean()
  obj.save()

Lesson to learn: Be careful when implementing Sequenced.get_siblings.

mail_queue_enter: create file maildrop/870989.992: Read-only file system

Tue, 28 Apr 2026 00:00:00 +0000

mail_queue_enter: create file maildrop/870989.992: Read-only file system¶

Tuesday, April 28, 2026

I’m on a virtual private server with a freshly installed Debian 13.4. The server is currently accessible via the FQDN lumi.mylino.net. I installed a list of packages like libreoffice, monit and certbot, and created a file /etc/monit/conf.d/lino.conf with the following content:

# generated by getlino
set alert root@localhost with reminder on 2 cycles
check program status with path /usr/local/bin/healthcheck.sh
    if status != 0 then alert
check device ROOT  with path /
    if SPACE usage > 95% then alert
check system $HOST
    if memory usage > 75% for 5 cycles then alert

This file causes the server to collapse after a few hours of operation. When I remove the file and reboot the server, my problem disappears. And when I restore the file and reboot, my problem reappears.

Here is what happens after rebooting when the file exists.

Soon after rebooting, I have the following message in my journal:

apr   28 08:07:30 lumi postfix/postdrop[992]: warning: mail_queue_enter: create file maildrop/870989.992: Read-only file system

This warning reappears every 2 minutes. Which corresponds to the “set daemon 120” in my /etc/monit/monitrc. After two minutes, there are already two of these warnings.

And for each warning, ps aux shows a new group of four processes:

root        1871  0.0  0.1  18012  2368 ?        Ss   22:51   0:00 sudo supervisorctl status
root        1872  0.0  0.3  43956  7044 ?        S    22:51   0:00 sendmail -t
root        1873  0.0  0.3  43820  7068 ?        S    22:51   0:00 /usr/sbin/postdrop -r
postfix     1895  0.0  0.7  53376 14112 ?        S    22:52   0:00 tlsmgr -l -t unix -u -c

And these processes accumulate, and after a few hours the server gets out of memory and starts swapping.

My /etc/monit/conf.d/lino.conf file is obviously the trigger, but on the other hand I’m using the same file on at least one other server with the same Debian version. And in general I’m using this file on other servers at least since 2020, and until now it did not cause such a problem.

More observations:

I read the docs about tlsmgr and yes, everything indicates that some process (probably monit) is trying to send an email and that postfix has some problem, but my postfix configuration is a stand-alone internet site with only the following parameters changed:
```
# postconf myhostname mydestination myorigin
myhostname = lumi.mylino.net
mydestination = $myhostname,localhost
myorigin = /etc/mailname
```
And I can manually send an email to root@localhost by saying:
```
$ echo foo | mail -r luc@lumi.mylino.net root@localhost
```
postfix check doesn’t complain about anything.

My /var/log/monit.log file shows an error:

        No mail servers are defined -- please see the 'set mailserver' statement in the manual

This error goes away when I say ``set mailserver localhost`` in my
:file:`/etc/monit/monitrc` but doesn't make my problem disappear. I'm just a bit
surprised because on another server with the same Debian version the error does
not appear even though I leave :file:`/etc/monit/monitrc` unchanged.

I sent this blog post as a question to the debian-users mailing list. Read here what happened then.

Monday, April 27, 2026

Mon, 27 Apr 2026 00:00:00 +0000

Monday, April 27, 2026¶

Yesterday I finally found the explanation for for #6733 (laudate server not responding and kswapd0 CPU usage at 200%), which has occupied me for more than two weeks. I found the explanation because I had summarized my problem for a Debian consultant. A typical case of “Ask, and it will be given to you; seek, and you will find; knock, and it will be opened to you.” (Mt 7:7). Here is what I told him:

I use Postfix on my servers. Some servers are stand-alone mail servers, others use Zone Media as a smart host. On one of my servers I have the following warning in the journalctl:
apr   25 17:19:21 lumi postfix/postdrop[1000]: warning: mail_queue_enter: create file maildrop/965751.1000: Read-only file system
The first warning comes quickly after rebooting. It comes every 10 seconds (I guess that postdrop is configured to run every 10 seconds). In the beginning, after rebooting, it’s only one warning, After a minute or so there are already 2 of them, and that number continues to grow. ps shows a number of processes I don’t understand, and this list also grows:
postfix     1869  0.0  0.7  53376 14048 ?        S    17:20   0:00 tlsmgr -l -t unix -u -c
root        1877  0.0  0.1  18012  2308 ?        Ss   17:20   0:00 sudo supervisorctl status
root        1878  0.0  0.3  43956  7024 ?        S    17:20   0:00 sendmail -t
root        1879  0.0  0.3  43820  6996 ?        S    17:20   0:00 /usr/sbin/postdrop -r
root        1885  0.0  0.1  18012  2404 ?        Ss   17:22   0:00 sudo supervisorctl status
root        1886  0.0  0.3  43956  6976 ?        S    17:22   0:00 sendmail -t
root        1887  0.0  0.3  43820  7064 ?        S    17:22   0:00 /usr/sbin/postdrop -r
root        1892  0.0  0.0      0     0 ?        I    17:24   0:00 [kworker/0:0-events]
root        1900  0.0  0.1  18012  2336 ?        Ss   17:24   0:00 sudo supervisorctl status
root        1901  0.0  0.3  43956  7064 ?        S    17:24   0:00 sendmail -t
root        1902  0.0  0.3  43820  6880 ?        S    17:24   0:00 /usr/sbin/postdrop -r
root        1914  0.0  0.1  18012  2400 ?        Ss   17:26   0:00 sudo supervisorctl status
root        1915  0.0  0.3  43956  7036 ?        S    17:26   0:00 sendmail -t
root        1916  0.0  0.3  43820  6956 ?        S    17:26   0:00 /usr/sbin/postdrop -r
After a few hours of operation, the server collapses, CPU usage is 100%, and the only thing I can do is to reboot.

The next morning I thought: “Every two minutes… Tries to sudo but fails… Tries to send an email… this sounds like monit”. To verify my suspicion without too much effort, I removed the /etc/monit/conf.d/lino.conf file and then rebooted… and the issue was gone! This file is generated by getlino, so the culprit is getlino! This explains why the problem reappeared even after rebuilding the server (i.e. creating a completely new VPS with a virgin Debian) and reinstalling the Lino sites that were hosted on this server.

A look at the /var/log/monit.log file sheds more light on the issue:

[2026-04-23T12:28:45+0000] info     : 'status' status succeeded (0) -- sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted
sudo: unable to open /etc/sudoers: Operation not permitted
sudo: error initializing audit plugin sudoers_audit
Checking supervisor status...
The supervisor status looks OK
[2026-04-23T12:28:45+0000] error    : Mail: No mail servers are defined -- please see the 'set mailserver' statement in the manual
[2026-04-23T12:28:45+0000] error    : Event queue is full
[2026-04-23T12:28:45+0000] error    : Aborting event - queue over quota
[2026-04-23T12:28:46+0000] error    : 'lumi' mem usage of 100.0% matches resource limit [mem usage > 75.0%]
[2026-04-23T12:28:46+0000] error    : Mail: No mail servers are defined -- please see the 'set mailserver' statement in the manual
[2026-04-23T12:28:47+0000] error    : Event queue is full

Here is the content of the /etc/monit/conf.d/lino.conf file:

# generated by getlino
set alert root@localhost with reminder on 2 cycles
check program status with path /usr/local/bin/healthcheck.sh
    if status != 0 then alert
check device ROOT  with path /
    if SPACE usage > 95% then alert
check system $HOST
    if memory usage > 75% for 5 cycles then alert

Yes, something is going wrong when monit tries to execute “check device ROOT with path / if SPACE usage > 95% then alert”. It’s some silly permission problem. But what exactly? And why did this wreak such a havoc? And how to avoid it in the future?

Observations:

We don’t want getlino to touch /etc/monit/monitrc. We assume monit to be
correctly installed.
I restored the etc/monit/conf.d/lino.conf file except for the line “set alert root@localhost with reminder on 2 cycles” and rebooted. The issue reappears. So this line seems to not be the issue.
“cron wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute. When executing commands, any output is mailed to the owner of the crontab (or to the user named in the MAILTO environment variable in the crontab, if such exists) from the owner of the crontab (or from the email address given in the MAILFROM environment variable in the crontab, if such exists). The children copies of cron running these processes have their name coerced to uppercase, as will be seen in the syslog and ps(1) output.”
In /etc/aliases I have a single entry:
```
luc@lumi:~$ cat /etc/aliases
root: luc
```

And I have a ~/.forward file with my real email address.

But when I send an email to root@localhost, the smart host thinks it is spam:

$ echo "Test" | mail -s "Test 2" root@localhost

Here are the relevant log entries:

apr   27 02:11:37 lumi postfix/pickup[1925]: 8BD7448423: uid=1000 from=<luc@lumi>
apr   27 02:11:37 lumi postfix/cleanup[1932]: 8BD7448423:
        message-id=<20260427021137.8BD7448423@lumi>
apr   27 02:11:37 lumi postfix/qmgr[1926]: 8BD7448423: from=<luc@lumi>,
        size=302, nrcpt=1 (queue active)
apr   27 02:11:38 lumi postfix/smtp[1934]: 8BD7448423: to=<root@localhost>,
        relay=smtp.zone.eu[85.234.244.110]:587, delay=0.55,
        delays=0.02/0.03/0.35/0.16, dsn=5.0.0, status=bounced (host
        smtp.zone.eu[85.234.244.110] said: 550 This message was classified as SPAM and
        may not be delivered (in reply to end of DATA command))
apr   27 02:11:38 lumi postfix/cleanup[1932]: 1EECA48424:
        message-id=<20260427021138.1EECA48424@lumi>
apr   27 02:11:38 lumi postfix/bounce[1936]: 8BD7448423: sender non-delivery
        notification: 1EECA48424
apr   27 02:11:38 lumi postfix/qmgr[1926]: 1EECA48424: from=<>, size=2207,
        nrcpt=1 (queue active)
apr   27 02:11:38 lumi postfix/qmgr[1926]: 8BD7448423: removed
apr   27 02:11:38 lumi postfix/smtp[1934]: 1EECA48424: to=<luc@lumi>,
        relay=smtp.zone.eu[85.234.244.79]:587, delay=0.67, delays=0/0/0.36/0.31,
        dsn=2.0.0, status=sent (250 Message queued as 19dccb4dd94000f038)
apr   27 02:11:38 lumi postfix/qmgr[1926]: 1EECA48424: removed

I did some adjustments in the /etc/postfix/mailn.cf

Wednesday, April 22, 2026

Wed, 22 Apr 2026 00:00:00 +0000

Wednesday, April 22, 2026¶

Working for #6733 (laudate server not responding and kswapd0 CPU usage at 200%).

I ran diff -y over the output of postconf on lumi (left) and lf (right). Most differences seem irrelevant, but the following ones are interesting:

mail_name = Postfix                                           | mail_name = mail.lino-framework.org
mydestination = $myhostname, laudate.ee, lumi, localhost.loca | mydestination = $myhostname, localhost.org, localhost
mydomain = localdomain                                        | mydomain = org
myhostname = lumi                                             | myhostname = lino-framework.org

But even after setting these parameters on lumi, I get only 1 of the three emails I should get.

Here is the journal for postfix on lumi when I send an email to webmaster@laudate.ee:

postfix/cleanup[93831]: AC41A2893A: message-id=<d0b751ab-280e-4f86-9e97-c4cfb19d0a72@saffre-rumma.net>
postfix/local[93832]: 8A1F928936: to=<webmaster@laudate.ee>, relay=local, delay=0.18, delays=0.16/0.01/0/0.01, dsn=2.0.0, status=sent (forwarded as AC41A2893A)
postfix/qmgr[93816]: AC41A2893A: from=<luc@saffre-rumma.net>, size=2153, nrcpt=2 (queue active)
postfix/qmgr[93816]: 8A1F928936: removed
postfix/smtp[93835]: AC41A2893A: to=<luc.saffre@gmail.com>, orig_to=<webmaster@laudate.ee>, relay=gmail-smtp-in.l.google.com[2a00:1450:4010:c07::1b]:25, delay=0.33, delays=0/0.01/0.05/0.26, dsn=2.0.0, status=sent (250 2.0.0 OK  1776852133 2adb3069b0e04-5a4187f1ea2si5982943e87.187 - gsmtp)
postfix/smtp[93836]: AC41A2893A: to=<luc.saffre@gmx.net>, orig_to=<webmaster@laudate.ee>, relay=mx01.emig.gmx.net[212.227.17.5]:25, delay=0.66, delays=0/0.02/0.18/0.46, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=1MkWc2-1vWpNy3rjC-00o0YX)
postfix/qmgr[93816]: AC41A2893A: removed

And somebody on serverfault.com wrote “Please read Postfix’s STANDARD CONFIGURATION README carefully. Postfix has perfect documentation, it’s worth to just dive in it.” Yes, seems that I am too old or too stupid for running my own mail server.

My /etc/dovecot/local.conf contains:

listen = *, ::
auth_mechanisms = plain login

mail_driver = maildir
mail_path = ~/Maildir
mail_inbox_path = ~/Maildir/.INBOX

# unix_listener /var/spool/postfix/private/auth {
#   mode = 0666
# }

ssl = yes
ssl_server_cert_file = /etc/letsencrypt/live/mail.laudate.ee/fullchain.pem
ssl_server_key_file = /etc/letsencrypt/live/mail.laudate.ee/privkey.pem


# Authenticate as system users:
passdb pam {
}

log_debug = category=auth
auth_debug_passwords = yes

Monday, April 13, 2026

Mon, 13 Apr 2026 00:00:00 +0000

Monday, April 13, 2026¶

Working for #6733 (laudate server not responding and kswapd0 CPU usage at 200%).

I’m trying to get Dovecot up and running correctly on laudate. That site was upgraded from Debian 12 to 13, but I told it to ignore my old dovecot configs because I prefer to start a setup from scratch.

Doveconf configuration is split over multiple files, which makes things more maintainable but less readable. Fortunatly for simple minds like me, there is the command doveconf -nP command who turns them into a single-file config.

I used doveconf -nP to output my doveconf compilation of another server, which still runs under Debian 12 and where dovecot works correctly:

auth_mechanisms = plain login
disable_plaintext_auth = no
mail_location = maildir:~/Maildir
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = pam
}
protocols = " pop3"
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
}
service pop3-login {
  inet_listener pop3 {
    port = 110
  }
}
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = </usr/share/dovecot/dh.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
userdb {
  driver = passwd
}

The dovecot upgrader tool converts this to:

# Conversion performed at 2026-04-13 07:03:22.768058
dovecot_config_version = 2.4.2
dovecot_storage_version = 2.4.0
auth_allow_cleartext = yes
auth_mechanisms = plain login
mail_driver = maildir
mail_path = ~/Maildir
mail_privileged_group = mail
protocols = pop3
ssl_client_ca_dir = /etc/ssl/certs

namespace inbox {
  inbox = yes

  mailbox Drafts {
    special_use = \Drafts
  }

  mailbox Junk {
    special_use = \Junk
  }

  mailbox Sent {
    special_use = \Sent
  }

  mailbox "Sent Messages" {
    special_use = \Sent
  }

  mailbox Trash {
    special_use = \Trash
  }
}

passdb pam {
  driver = pam
}

userdb passwd {
  driver = passwd
}

service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
}

service pop3-login {
  inet_listener pop3 {
    port = 110
  }
}

ssl_server {
  cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
  dh_file = /usr/share/dovecot/dh.pem
  key_file = # hidden, use -P to show it
}

# Warnings during conversion:
# - pruned default: inbox prefix = ''

Here is the output of doveconf -nP on laudate after the fresh installation:

dovecot_config_version = 2.4.0
dovecot_storage_version = 2.4.0
fts_autoindex = yes
fts_autoindex_max_recent_msgs = 999
fts_search_add_missing = yes
mail_driver = mbox
mail_home = /home/%{user | username}
mail_inbox_path = /var/mail/%{user}
mail_path = %{home}/mail
mail_privileged_group = mail
protocols {
  pop3 = yes
}
passdb pam {
}
userdb passwd {
}
namespace inbox {
  inbox = yes
  mailbox Drafts {
    special_use = "\\Drafts"
  }
  mailbox Junk {
    special_use = "\\Junk"
  }
  mailbox Trash {
    special_use = "\\Trash"
  }
  mailbox Sent {
    special_use = "\\Sent"
  }
  mailbox "Sent Messages" {
    special_use = "\\Sent"
  }
}
service imap-login {
  inet_listener imap {
  }
  inet_listener imaps {
  }
}
service pop3-login {
  inet_listener pop3 {
  }
  inet_listener pop3s {
  }
}
service submission-login {
  inet_listener submission {
  }
  inet_listener submissions {
  }
}
service lmtp {
  unix_listener lmtp {
  }
}
service imap {
}
service pop3 {
}
service submission {
}
service auth {
  unix_listener auth-userdb {
  }
}
service auth-worker {
}
service dict {
  unix_listener dict {
  }
}
ssl_server {
  cert_file = /etc/dovecot/private/dovecot.pem
  key_file = /etc/dovecot/private/dovecot.key
}

Then I use diff -y to compare these two files:

dovecot_config_version = 2.4.2                                  |  dovecot_config_version = 2.4.0
dovecot_storage_version = 2.4.0                                    dovecot_storage_version = 2.4.0
auth_allow_cleartext = yes                                      |  fts_autoindex = yes
auth_mechanisms = plain login                                   |  fts_autoindex_max_recent_msgs = 999
mail_driver = maildir                                           |  fts_search_add_missing = yes
mail_path = ~/Maildir                                           |  mail_driver = mbox
                                                                >  mail_home = /home/%{user | username}
                                                                >  mail_inbox_path = /var/mail/%{user}
                                                                >  mail_path = %{home}/mail
mail_privileged_group = mail                                       mail_privileged_group = mail
protocols = pop3                                                |  protocols {
ssl_client_ca_dir = /etc/ssl/certs                              |    pop3 = yes
                                                                |  }
                                                                >  passdb pam {
                                                                >  }
                                                                >  userdb passwd {
                                                                >  }
namespace inbox {                                                  namespace inbox {
  inbox = yes                                                        inbox = yes
                                                                <
  mailbox Drafts {                                                   mailbox Drafts {
    special_use = \Drafts                                       |      special_use = "\\Drafts"
  }                                                                  }
                                                                <
  mailbox Junk {                                                     mailbox Junk {
    special_use = \Junk                                         |      special_use = "\\Junk"
                                                                >    }
                                                                >    mailbox Trash {
                                                                >      special_use = "\\Trash"
  }                                                                  }
                                                                <
  mailbox Sent {                                                     mailbox Sent {
    special_use = \Sent                                         |      special_use = "\\Sent"
  }                                                                  }
                                                                <
  mailbox "Sent Messages" {                                          mailbox "Sent Messages" {
    special_use = \Sent                                         |      special_use = "\\Sent"
  }                                                                  }
                                                                |  }
  mailbox Trash {                                               |  service imap-login {
    special_use = \Trash                                        |    inet_listener imap {
                                                                >    }
                                                                >    inet_listener imaps {
  }                                                                  }
}                                                                  }
                                                                |  service pop3-login {
passdb pam {                                                    |    inet_listener pop3 {
  driver = pam                                                  |    }
                                                                >    inet_listener pop3s {
                                                                >    }
}                                                                  }
                                                                |  service submission-login {
userdb passwd {                                                 |    inet_listener submission {
  driver = passwd                                               |    }
                                                                >    inet_listener submissions {
                                                                >    }
                                                                >  }
                                                                >  service lmtp {
                                                                >    unix_listener lmtp {
                                                                >    }
                                                                >  }
                                                                >  service imap {
                                                                >  }
                                                                >  service pop3 {
                                                                >  }
                                                                >  service submission {
}                                                                  }
                                                                <
service auth {                                                     service auth {
  unix_listener /var/spool/postfix/private/auth {               |    unix_listener auth-userdb {
    group = postfix                                             <
    mode = 0666                                                 <
    user = postfix                                              <
  }                                                                  }
}                                                                  }
                                                                |  service auth-worker {
service pop3-login {                                            |  }
  inet_listener pop3 {                                          |  service dict {
    port = 110                                                  |    unix_listener dict {
  }                                                                  }
}                                                                  }
                                                                <
ssl_server {                                                       ssl_server {
  cert_file = /etc/letsencrypt/live/mail.example.com/fullchain. |    cert_file = /etc/dovecot/private/dovecot.pem
  dh_file = /usr/share/dovecot/dh.pem                           |    key_file = /etc/dovecot/private/dovecot.key
  key_file = # hidden, use -P to show it                        |  }
}                                                               <
                                                                <
# Warnings during conversion:                                   <
# - pruned default: inbox prefix = ''                           <

I created a local.conf file where I plan to add my own settings. First attempt:

ssl_server {
  cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
  key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
}

Now Thunderbird reports a new error message:

Unable to communicate securely with peer: requested domain name does not match the server’s certificate.

This site attempts to identify itself with invalid information. Wrong Site. The certificate belongs to a different site, which could mean that someone is trying to impersonate this site.

And when I ask TB for details, it seems to report that it received a certificate for “example.com”, not “mail.example.com”. So the subdomain “mail.” has been lost somewhere.

But certbot certificates on laudate says that it does use mail.example.com:

Certificate Name: mail.example.com
  Serial Number: 6056e09415ac60c5115699b575c5ac3989b
  Key Type: RSA
  Domains: mail.example.com
  Expiry Date: 2026-07-08 06:24:24+00:00 (VALID: 85 days)
  Certificate Path: /etc/letsencrypt/live/mail.example.com/fullchain.pem
  Private Key Path: /etc/letsencrypt/live/mail.example.com/privkey.pem

I gave up here because I can actually live without dovecot and because there are more urgent things to do right now, and last but not least because dovecot is the main suspect culprit for #6733 (laudate server not responding and kswapd0 CPU usage at 200%). I said apt remove dovecot-pop3d to uninstall it. I also deleted two users dovecot and dovenull.

P.S. But the next day, after realizing that dovecot is innocent, I reinstalled it. I had to manually re-create the two users dovecot and dovenull by saying:

# adduser --disabled-login --no-create-home dovecot
# adduser --disabled-login --no-create-home dovenull

Friday, April 10, 2026

Fri, 10 Apr 2026 00:00:00 +0000

Friday, April 10, 2026¶

Working for #6733 (laudate server not responding and kswapd0 CPU usage at 200%).

Today I upgraded laudate from Debian 12 to 13

I made a snapshot of ets, the only production Lino site on laudate.

Here is a list of the websites on laudate:

$ ll /etc/nginx/sites-enabled/
total 16
drwxr-xr-x 2 root root 4096  4. dets  07:15 .
drwxr-xr-x 8 root root 4096 10. apr   20:37 ..
-rw-rw-r-- 1 root root  275  3. dets   2024 default.conf
-rw-rw-r-- 2 root root 2107  7. dets   2023 ets.conf
lrwxrwxrwx 1 root root   36 31. dets   2022 jane.conf -> /etc/nginx/sites-available/jane.conf
lrwxrwxrwx 1 root root   42 18. märts  2021 laudate.ee.conf -> /etc/nginx/sites-available/laudate.ee.conf
lrwxrwxrwx 1 root root   48 21. sept   2021 lists.laudate.ee.conf -> /etc/nginx/sites-available/lists.laudate.ee.conf
lrwxrwxrwx 1 root root   49  2. okt    2021 sinod.katoliku.ee.conf -> /etc/nginx/sites-available/sinod.katoliku.ee.conf
lrwxrwxrwx 1 root root   39  1. jaan   2023 welcht1.conf -> /etc/nginx/sites-available/welcht1.conf
lrwxrwxrwx 1 root root   39  1. jaan   2023 weleup1.conf -> /etc/nginx/sites-available/weleup1.conf

I removed the obsolete sites jane, lists and weleup1.

I then followed the instructions at https://linuxconfig.org/how-to-upgrade-debian-to-latest-version

The apt update gives some warnings:

$ sudo apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://security.debian.org bookworm-security InRelease
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
Hit:4 http://mirror.hetzner.de/debian/packages buster InRelease
Hit:5 http://mirror.hetzner.de/debian/packages buster-updates InRelease
Hit:6 http://mirror.hetzner.de/debian/packages buster-backports InRelease
Ign:7 http://mirror.hetzner.de/debian/security buster/updates InRelease
Hit:8 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg InRelease
Err:9 http://mirror.hetzner.de/debian/security buster/updates Release
  404  Not Found [IP: 2a01:4f8:0:1::1:97 80]
Hit:10 https://packages.groonga.org/debian bullseye InRelease
Reading package lists... Done
E: The repository 'http://mirror.hetzner.de/debian/security buster/updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://apt.postgresql.org/pub/repos/apt/dists/bullseye-pgdg/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
N: Repository 'Debian bookworm' changed its 'non-free component' value from 'non-free' to 'non-free non-free-firmware'
N: More information about this can be found online in the Release notes at: https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.html#non-free-split

I removed the obsolete files for groonga and hetzner from the /etc/apt/sources.d.

I answered to keep current version for the following config files:

/etc/bash.bashrc
/etc/crontab
/etc/monit/monitrc
/etc/nginx/nginx.conf
/etc/default/opendkim
/etc/redis/redis.conf

For /etc/ssh/sshd_config and, of course, for /etc/dovecot/dovecot.conf, I decided to install the package maintainer’s version.

Also for /etc/default/grub I took the new version because AFAIK we never did any important modification to this. Here is the diff before upgrading:

│  GRUB_DEFAULT=0
│  GRUB_TIMEOUT=5
│ -GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
│ +GRUB_DISTRIBUTOR=`( . /etc/os-release && echo ${NAME} )`
│  GRUB_CMDLINE_LINUX_DEFAULT="consoleblank=0 systemd.show_status=true elevator=noop console=tty1 console=ttyS0"
│  GRUB_CMDLINE_LINUX=""
│
│ +# If your computer has multiple operating systems installed, then you
│ +# probably want to run os-prober. However, if your computer is a host
│ +# for guest OSes installed via LVM or raw disk devices, running
│ +# os-prober can cause damage to those guest OSes as it mounts
│ +# filesystems to look for things.
│ +#GRUB_DISABLE_OS_PROBER=false
│ +
│  # Uncomment to enable BadRAM filtering, modify to suit your needs
│  # This works with Linux (no patch required) and with any kernel that obtains
│  # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
│  #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
│
│ -# Uncomment to disable graphical terminal (grub-pc only)
│ -GRUB_TERMINAL=console
│ +# Uncomment to disable graphical terminal
│ +#GRUB_TERMINAL=console
│
│  # The resolution used on graphical terminal
│ -# note that you can use only modes which your graphic card supports via VBE
│ -# you can see them in real GRUB with the command `vbeinfo'
│ +# note that you can use only modes which your graphic card supports via VBE/GOP/UGA
│ +# you can see them in real GRUB with the command `videoinfo'
│  #GRUB_GFXMODE=640x480

Some issues afterwards:

postfix[18307]: postfix/postlog: warning: /var/spool/postfix/etc/ssl/certs/GlobalSign_ECC_Root_CA_-_R4.pem and /etc/ssl/certs/GlobalSign_ECC_Root_CA_-_R4.pem differ

I said “postconf compatibility_level=3.6”:

# postconf compatibility_level=3.6
# postfix reload
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtp_use_tls" will be removed; instead, specify "smtp_tls_security_level"
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtpd_use_tls" will be removed; instead, specify "smtpd_tls_security_level"
postfix/postfix-script: refreshing the Postfix mail system

And I then did as the warnings told me:

# smtp_use_tls=yes
# smtpd_use_tls=yes

smtp_tls_security_level = may
smtpd_tls_security_level = may

Thunderbird said “An error occurred while sending mail: Outgoing server (SMTP) error. The server responded: TLS not available due to local problem.” – this was because i had a typo in the smtpd_tls_cert_file filename. A “systemctl status postfix” told me this in a funny way:

apr   11 10:21:20 laudate postfix/submission/smtpd[19025]: warning: cannot get RSA certificate from file "/etc/letsencrypt/live/laudate.ee/fulchain.pem": disabling TLS support
apr   11 10:21:20 laudate postfix/submission/smtpd[19025]: warning: TLS library problem: error:80000002:system library::No such file or directory:../crypto/bio/bss_file.c:288:calling fopen(/etc/letsencrypt/live/laudate.ee/fulchai>
apr   11 10:21:20 laudate postfix/submission/smtpd[19025]: warning: TLS library problem: error:10080002:BIO routines::system lib:../crypto/bio/bss_file.c:291:
apr   11 10:21:20 laudate postfix/submission/smtpd[19025]: warning: TLS library problem: error:0A080002:SSL routines::system lib:../ssl/ssl_rsa.c:503:

Thunderbird said “Unable to communicate securely with peer: requested domain name does not match the server’s certificate. The configuration related to mail.laudate.ee must be corrected.”

Saturday, March 21, 2026

Sat, 21 Mar 2026 00:00:00 +0000

Saturday, March 21, 2026¶

I’m still trying to understand why emails sent with farm mailto fail to arrive to my inbox when sent from nora but not when sent from LF. And OTOH emails sent from jane (on nora) do arrive to my inbox.

Mail-related settings on Jane are:

ADMINS: [['Luc Saffre', 'luc@saffre-rumma.net']]
DEFAULT_FROM_EMAIL = noreply@mylino.net
EMAIL_BACKEND = django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST = localhost
EMAIL_HOST_PASSWORD =
EMAIL_HOST_USER =
EMAIL_PORT = 25
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = None
EMAIL_SUBJECT_PREFIX = [jane]
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = False
SERVER_EMAIL = noreply@mylino.net

Here is the output of journalctl:

$ journalctl -f -n 40 … märts 21 09:15:06 nora jane[2713141]: Send email ‘[jane] Status report’ from noreply@mylino.net to [‘luc@saffre-rumma.net’] märts 21 09:15:06 nora postfix/smtpd[2713200]: connect from localhost[127.0.0.1] märts 21 09:15:06 nora postfix/smtpd[2713200]: ACFF0AA03F: client=localhost[127.0.0.1] märts 21 09:15:06 nora postfix/cleanup[2713203]: ACFF0AA03F: message-id=<177408450669.2713141.1866157075527590826@nora> märts 21 09:15:06 nora postfix/smtpd[2713200]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 märts 21 09:15:06 nora postfix/qmgr[2673489]: ACFF0AA03F: from=<noreply@mylino.net>, size=5291, nrcpt=1 (queue active) märts 21 09:15:06 nora jane[2713141]: Done manage.py status -s luc@saffre-rumma.net (PID 2713141) märts 21 09:15:08 nora postfix/smtp[2713204]: ACFF0AA03F: to=<luc@saffre-rumma.net>, relay=smtp.zone.eu[85.234.244.110]:587, delay=1.5, delays=0.01/0.04/0.68/0.8, dsn=2.0.0, status=sent (250 Message queued as 19d0fad47370002b2c) märts 21 09:15:08 nora postfix/qmgr[2673489]: ACFF0AA03F: removed

The From header of an email received from Jane says:

From: luc@saffre-rumma.net

Monday, March 9, 2026

Mon, 09 Mar 2026 00:00:00 +0000

Monday, March 9, 2026¶

I’m working on #6654 (Language switching and page_name for songbook).

The only problem is that the EN page with page_name “laulud” cannot make a link to the corresponding page in ET. I want the text of the English page to say “This content is available only in Estonian”, followed by a [page 20 Click here] or [ref page:20 Click here].

lino/modlib/bootstrap5/config/bootstrap5/base.html calls ar.get_request_url() for each language except the current one. ar.get_request_url() then calls Renderer.obj2url()
The ref memo command calls MemoReferrable.memo2html(), which calls Model.as_summary_item(), which calls ar.obj2html(), which calls Renderer.obj2url().
Renderer.obj2url() calls Model.get_publisher_pk()

Friday, February 20, 2026

Fri, 20 Feb 2026 00:00:00 +0000

Friday, February 20, 2026¶

Configure postfix for sending email via an external SMTP relay server

https://dohost.us/index.php/2025/07/27/configuring-postfix-to-send-outgoing-mail-relay-hosts-and-smart-hosts/

apt-get install libsasl2-modules

is needed for the authentication

smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd smtp_sasl_security_options = noanonymous

Sunday, January 18, 2026

Sun, 18 Jan 2026 00:00:00 +0000

Sunday, January 18, 2026¶

After upgrading SR from bookworm to trixie we have the issue that emails don’t work. SR accepts them but then they vanish, and the mail.log says:

2026-01-18T13:48:22.282711+02:00 saffre-rumma postfix/postdrop[5083]: \
  warning: mail_queue_enter: create file maildrop/281740.5083: Read-only file system

The authentication seems to work, Thunderbird reports no error, but journalctl -t dovecot reports that my Thunderbird logged in and disconnected:

jaan  18 19:25:28 saffre-rumma.net dovecot[712]: pop3-login: Logged in: user=<luc>, method=PLAIN, rip=90.191.158.109, lip=51.68.71.43, mpid=1476, TLS, sessi>
jaan  18 19:25:28 saffre-rumma.net dovecot[712]: pop3(luc)<1476><EOZH3KxIPJJav55t>: Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0

I ran postfix check and it reported some issues:

# postfix check
postfix: Postfix is using backwards-compatible default settings
postfix: See https://www.postfix.org/COMPATIBILITY_README.html for details
postfix: To disable backwards compatibility use "postconf compatibility_level=3.6" and "postfix reload"
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtpd_use_tls" will be removed; instead, specify "smtpd_tls_security_level"
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtpd_use_tls" will be removed; instead, specify "smtpd_tls_security_level"
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtpd_use_tls" will be removed; instead, specify "smtpd_tls_security_level"
/usr/sbin/postconf: warning: /etc/postfix/main.cf: support for parameter "smtpd_use_tls" will be removed; instead, specify "smtpd_tls_security_level"
postfix/postfix-script: warning: group or other writable: /etc/postfix/./redirect.regexp
postfix/postfix-script: warning: group or other writable: /etc/postfix/./access
postfix/postfix-script: warning: group or other writable: /etc/postfix/./address.txt
postfix/postfix-script: warning: group or other writable: /etc/postfix/./redirect.pcre

In my /etc/postfix/main.cf I replaced smtpd_use_tls=yes by smtpd_tls_security_level = may.
I said chmod g-w /etc/postfix/./redirect.regexp etc for the four mentioned files.

Now postfix check no longer warns.

In journalctl -t postfix I saw the following series of warnings:

jaan  18 16:41:49 saffre-rumma.net postfix[11008]: postfix/postlog: warning: not owned by root: /var/spool/postfix/.
...

And yes, most of the files below /var/spool/postfix have postfix as owner.

# ll /var/spool/postfix/ total 2828 drwx—— 2 postfix postfix 1232896 18. jaan 18:07 active drwx—— 2 postfix postfix 4096 6. jaan 16:04 bounce drwx—— 2 postfix postfix 4096 15. märts 2021 corrupt drwx—— 18 postfix postfix 4096 29. dets 2021 defer drwx—— 18 postfix postfix 4096 29. dets 2021 deferred drwxr-xr-x 2 root root 4096 18. jaan 15:43 dev drwxr-xr-x 2 postfix postfix 4096 18. jaan 12:53 etc drwx—— 2 postfix postfix 4096 15. märts 2021 flush drwx—— 2 postfix postfix 4096 15. märts 2021 hold drwx—— 2 postfix postfix 1581056 18. jaan 18:07 incoming drwxr-xr-x 2 postfix postfix 4096 18. jaan 12:53 lib drwx-wx–T 2 postfix postdrop 4096 18. jaan 18:07 maildrop drwxrwxr-x 2 postfix postfix 4096 18. jaan 14:05 opendkim drwxr-xr-x 2 postfix postfix 4096 1. jaan 2022 photos drwxr-xr-x 2 postfix postfix 4096 21. apr 2022 pid drwx—— 2 postfix postfix 4096 18. jaan 16:41 private drwx–s— 2 postfix postfix 4096 18. jaan 16:41 public -rw-r–r– 1 postfix postfix 0 21. märts 2025 restart drwx—— 2 postfix postfix 4096 15. märts 2021 saved drwx—— 2 postfix postfix 4096 18. jaan 16:09 trace drwxr-xr-x 3 postfix postfix 4096 15. märts 2021 usr

Should I change their owner to root? I changed it only for maildrop:

# chown root /var/spool/postfix/maildrop

No, this caused a fatal error:

postsuper: fatal: scan_dir_push: open directory maildrop: Permission denied

I removed the sticky bit from this directory as well:

# chmod -t /var/spool/postfix/maildrop

No, that didn’t fix my issue. I reverted the change.

I changed also to root the owner of the other directories mentioned by postfix check.

Now postfix check gives a new warning:

postsuper: warning: bogus file name: maildrop/NS_CANARY.8142

Okay:

mv /var/spool/postfix/maildrop/NS_CANARY.8142 /tmp/

Now postfix gives no more warnings:

# postfix check
postfix: Postfix is using backwards-compatible default settings
postfix: See https://www.postfix.org/COMPATIBILITY_README.html for details
postfix: To disable backwards compatibility use "postconf compatibility_level=3.6" and "postfix reload"

To disable backwards compatibility, I also said:

# postconf compatibility_level=3.6

Now, “postfix check” gives no output at all. Nice, but my problem isn’t fixed. Seems that these warnings weren’t the cause of my problem.

I don’t believe that it is caused by a read-only filesystem. I guess that the warning “Read-only file system” actually means “permission denied”. But let’s verify:

# fsck -n -f
fsck from util-linux 2.41
e2fsck 1.47.2 (1-Jan-2025)
Warning!  /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 64 extent tree (at level 1) could be shorter.  Optimize? no
Deleted inode 131393 has zero dtime.  Fix? no
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (5344474, counted=5795181).
Fix? no
Inode bitmap differences:  -131393
Fix? no
Free inodes count wrong (2322711, counted=2322773).
Fix? no
/dev/sda1: ********** WARNING: Filesystem still has errors **********
/dev/sda1: 298729/2621440 files (0.3% non-contiguous), 5141025/10485499 blocks

So there are indeed some errors in the file system. But (1) “fsck -n -f” reports similar errors also on my notebook and (2) it is difficult to run fsck on a busy partition:

# fsck /dev/sda
fsck from util-linux 2.41
e2fsck 1.47.2 (1-Jan-2025)
/dev/sda is in use.
e2fsck: Cannot continue, aborting.

# umount /dev/sda1
umount: /: target is busy.

I tried restarting the server in rescue mode, but there fsck fails with “Bad magic number in super-block”:

[RESCUE] root@vps-331a9923:~ $ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  2.9G  0 disk
`-sda1   8:1    0  2.9G  0 part /
sdb      8:16   0   40G  0 disk
`-sdb1   8:17   0   40G  0 part

[RESCUE] root@vps-331a9923:~ $ fsck /dev/sdb
fsck from util-linux 2.38.1
e2fsck 1.47.0 (5-Feb-2023)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sdb
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>
Found a dos partition table in /dev/sdb

At this point I gave up for today and restored once more the backup from January 16 9:19 UTC. So the server runs again on Debian 12, and I don’t know when I will a next attempt to upgrade to 13. I have been warned:

In order to avoid potentially extended downtime, you are strongly encouraged to port your configuration in a staging environment before beginning the upgrade of a production mail system.

—https://www.debian.org/releases/trixie/release-notes/issues.en.html

I still don’t believe that our problem is caused by a read-only filesystem, I still believe that the warning “Read-only file system” is not an original message from the operating system but actually means “permission denied”.

Wednesday, January 14, 2026

Wed, 14 Jan 2026 00:00:00 +0000

Wednesday, January 14, 2026¶

The `request_handler()` method¶

The request_handler() method of an action request builds and returns the Javascript code that would execute this request on the client.

It obviously depends on the front-end renderer. The publisher renderer returns None.

It is needed by WrappedAction, ShowSlaveTable, RequestFieldElement, Renderer.href_to_request(). It is also needed by ShowSuggestions or ShowEntriesByDay, which are similar to ShowSlaveTable

It is needed by WrappedAction to build the URI for an action at runtime:

class WrappedAction(Action):
  ...
  def run_from_ui(self, ar, **kw):
      sar = self.bound_action.request_from(ar)
      js = ar.renderer.request_handler(sar)
      ar.set_response(eval_js=js)

WrappedAction is a kind of dynamic URL, it does nothing but return a Javascript snippet to be executed on the client.

The WrappedAction class is used:

by QuickLinksList (the class of the object sitting on settings.SITE.quicklinks)
by custom implementations of collect_extra_actions(), e.g. the “Pretty view” action on a blog entry, the “Import from VCard” action on a partner, or the actions used by the calendar view to switch between daily, weekly and monthly view.

Sunday, January 4, 2026

Sun, 04 Jan 2026 00:00:00 +0000

Sunday, January 4, 2026¶

The site_config property is now defined on the new SiteConfigCache class, which is inherited by Site as before, and also by User and AnonymousUser.

The config_id attribute is renamed to tenant_id. This corresponds to the value of the SiteConfig.id. The default tenant_id is 1.

On a multi-tenant site, User.tenant_id will become a database field.

The today(), atoday() and now() functions are also on the new SiteConfigCache class.

But we keep a limited version of today(), that does not consider SiteConfig, because we use it e.g. in lino.modlib.system.mixins:

dd.update_field(RecurrenceSet, "start_date", default=dd.today)

A callable field default value must be serializable, so it cannot depend on the tenant.

We no longer install a signal handler that calls Site.clear_site_config() when setting_changed, testcase_setup, connection_created or post_migrate are emitted.

BuildMethods.get_system_default() no longer looks up SiteConfig, but the new SiteConfigCache.get_printing_build_method does this.

Waiting for a miracle

Sun, 14 Dec 2025 00:00:00 +0000

Waiting for a miracle¶

Sunday, December 14, 2025

Last month brought at least four sad pieces of news for Lino:

Noémy decided, after a promising first meeting, to not start using Lino. The obvious reason is the bus factor: their Lino site would have more than 200 users, and they can’t put such a responsibility on the shoulders of a almost-one-man project.

Sharif wants to immigrate to Europe, together with his wife. Which means that he will be lost for Lino because I cannot afford a European salary for him. Of course he will find a job, sooner or later, because he is good. He’s both intelligent and able to work in a team. I endorse him. I recommended him to Thorgate, who were reluctant at first because an immigrant means administrative effort for them, but upon my word they reconsidered the case and invited him to a job interview. Sharif sincerely plans to continue working for Lino at the weekends, but I know how it will go: after 40 hours staring at a screen he will prefer doing things with his wife and their children. At least I hope that he will. I would worry if he wouldn’t. Today I rose priority for ticket #5239 (Document & test the React front end) on his to-do list. This ticket becomes urgent because his first job interview will be next Friday, and things could get out of hand quickly if Thorgate decide to engage him.

Astrid had been planning to to work a few hours per week for Lino, besides her full-time job as a mother of three children. But two days ago she heard that she will need to find a “real” job in January 2026 because the Estonian government “optimized” the rules about paying health insurance for parents and no longer considers her a full-time mother.

Andres introduced me to a potential new Lino user, and during the meeting it turned out that publishing the source code is not an option for them. They were unable to understand why proprietary software is not an option for me. I spent multiple hours with Andres to explain him the Synodalsoft vision and why I’m doing Lino. But people don’t believe that software should be free.

As a result, I currently feel a bit resigned: I failed to bring Synodalsoft to the next level. The Synodalsoft vision was too big for a single human life. And I was not enough of a manager for such a big vision. It is not enough to publish all your software under the GNU Affero General Public License (AGPL) (as I wrote in The Synodalsoft project); if you want your intellectual work to be “sustainably free, usable for everybody and forever”, you also need to do the administrative stuff of institutionalizing yourself. Software needs more than a human, it also needs legal person to care for it. When I will be gone, Lino will be just be an orphaned software project, and there is nothing I can do about it, except keep watching out for a miracle, which includes doing my best for providing perfect service to my customers.

Tuesday, November 25, 2025

Tue, 25 Nov 2025 00:00:00 +0000

Tuesday, November 25, 2025¶

fields needs params, actions, actors, vfields, fakefield, basefields
elems needs layouts
utils needs atable
atable needs actors
frames needs actors
actors needs params, layouts, fields
actions needs layouts, choosers, params
layouts needs vfields, fakefield, store
store needs fakefield, vfields, storefields, atomizer
atomizer needs basefields, utils, storefields
basefields needs fakefield, vfields
storefields needs fakefield, vfields, actions
choosers needs vfields, dbutils
vfields needs fakefield, basefields
basefields needs
fakefield needs dbutils
dbutils needs nothing
params needs nothing

Insert a local table of contents into a HTML page

Fri, 31 Oct 2025 00:00:00 +0000

Insert a local table of contents into a HTML page¶

Friday, October 31, 2025.

How to insert a local table of contents into a HTML page? Except for some basic examples I couldn’t find something that actually inserts the TOC in a marked place, complete with backlinks, and later updates that TOC.

from difflib import Differ
from lino.utils.soup import sanitized_soup

TOC_MARKER = "=TOC_HERE="
html = """\
<html><body><p>This page is an example.</p>
<p>=TOC_HERE=</p>
<h1>First approach</h1>
<p>The first approach has 2 steps</p>
<h2>First</h2>
<p>In a <b>first</b> step ...</p>
<h2>Second</h2>
<p>In a second step ...</p>
<h1>Second approach</h1>
<h2>Third</h2>
<p>In a third step ...</p>
<h2>Fourth</h2>
<p>In a fourth step ...</p></body></html>"""


def add_toc(html):
    toc_tag = None
    headers = []
    soup = sanitized_soup(html)
    for tag in soup.children:
        if tag.name is None:
            # print(f"Oops: {repr(tag)}")
            continue
        tag_name = tag.name.lower()
        if tag_name == "div" and tag.get('id', None) == "contents":
            tag.clear()
            toc_tag = tag
            continue
        elif tag.string and tag.string.strip() == TOC_MARKER:
            tag.name = 'div'
            tag.clear()
            toc_tag = tag
            continue
        elif tag_name == "h1":
            headers.append(tag.string)
            # todo add an anchor and a backlink
            # if tag.contents[-1].name == "a"
            # <a class="headerlink" href="#kuhu-minna" title="Link to this heading">¶</a>

    if toc_tag is not None:
        toc_tag['id'] = "contents"
        ul = soup.new_tag('ul')
        for h in headers:
            ul.append(soup.new_tag('li', string=h))
            toc_tag.append(ul)

    return str(soup)


html2 = add_toc(html)
print(html2)
html3 = add_toc(html2)
if html2 != html3:
    print("Oops there are differences:")
    d = Differ()
    for ln in d.compare(html2.splitlines(), html3.splitlines()):
        print(ln)

That was fun, but it’s rather useless, because there are JS scripts for doing this automatically and better.

Wonderful world

Wed, 29 Oct 2025 00:00:00 +0000

Wonderful world¶

Wednesday, October 29, 2025

I found the translators package and thought by myself: what a wonderful world.

>>> title = "Wonderful world"
>>> body = """I hear babies cry, I watch them grow, they'll learn much more than I'll ever know, and I think to myself: What a wonderful world."""
>>> html = f'''
... <!DOCTYPE html><html><head><title>{title}</title></head>
... <body>
... <p></p>{body}</body></html>
... '''

>>> import translators as ts

Optionally caching sessions in advance can help improve access speed:

>>> # ts.preaccelerate_and_speedtest()

>>> # help(ts.translate_text)

>>> print(ts.translators_pool)
['alibaba', 'apertium', 'argos', 'baidu', 'bing', 'caiyun', 'cloudTranslation',
'deepl', 'elia', 'google', 'hujiang', 'iciba', 'iflytek', 'iflyrec',
'itranslate', 'judic', 'languageWire', 'lingvanex', 'niutrans', 'mglip',
'mirai', 'modernMt', 'myMemory', 'papago', 'qqFanyi', 'qqTranSmart', 'reverso',
'sogou', 'sysTran', 'tilde', 'translateCom', 'translateMe', 'utibet',
'volcEngine', 'yandex', 'yeekit', 'youdao']

>>> kwargs = dict(translator='alibaba', to_language="et")
>>> print(ts.translate_text(body, **kwargs))
Ma kuulen beebid nutmas, vaatan neid kasvamas, nad õpivad palju rohkem, kui ma
kunagi tean. Ja ma mõtlen, milline imeline maailm.

>>> kwargs = dict(translator='google', to_language="et")
>>> print(ts.translate_text(body, **kwargs))
Ma kuulen imikute nuttu, vaatan, kuidas nad kasvavad, nad õpivad palju rohkem,
kui mina kunagi teada saan, ja ma mõtlen endamisi: milline imeline maailm.

>>> kwargs = dict(translator='deepl', to_language="et")
>>> print(ts.translate_text(body, **kwargs))
Ma kuulen, kuidas lapsed nutavad, ma vaatan, kuidas nad kasvavad, nad õpivad
palju rohkem, kui mina kunagi teada saan, ja ma mõtlen endamisi: Milline imeline
maailm.

>>> kwargs = dict(translator='bing', to_language="et")
>>> print(ts.translate_text(body, **kwargs))
Ma kuulen beebide nutmist, ma vaatan, kuidas nad kasvavad, nad õpivad palju
rohkem, kui mina kunagi tean, ja ma mõtlen endamisi: Kui imeline maailm.

>>> kwargs = dict(translator='google', to_language="bn")
>>> print(ts.translate_text(body, **kwargs))
আমি বাচ্চাদের কান্না শুনতে পাই, আমি তাদের বড় হতে দেখি, তারা আমি যা জানতাম তার
চেয়েঅনেক বেশি শিখবে, এবং আমি মনে মনে ভাবি: কী চমৎকার পৃথিবী।

Possible exceptions:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://www2.deepl.com/jsonrpc?method=LMT_split_text
translators.server.TranslatorError: Unsupported to_language[et]

Sunday, September 14, 2025

Sun, 14 Sep 2025 00:00:00 +0000

Sunday, September 14, 2025¶

User cannot set predicate or smiley directly from the as_paragraph view of ElementResponsesBySection. in the detail of an SectionResponse.

rating_buttons

Both methods call Ratable.get_rating_buttons on an ElementResponse to generate this HTML.

set_score_action_directly is available on 4 models: ChallengeRating FinalRating ElementResponse SectionResponse

The URL in the href of the link (generated by row_action_button) is almost the same in both cases, only difference is the “onMain” keyword, which is false in the first case and true in the other case.

(a) javascript:window.App.runAction({ “action_full_name”: “ratings.ChallengeRatings.set_score_action_directly”, “actorId”: “cert.ElementResponsesBySection”, “onMain”: true, “rp”: “weak-key-0”, “sr”: 1453, “status”: { “fv”: [ null, “2” ] } })

(b) javascript:window.App.runAction({ “action_full_name”: “ratings.ChallengeRatings.set_score_action_directly”, “actorId”: “cert.ElementResponsesBySection”, “onMain”: false, “rp”: “weak-key-0”, “sr”: 1453, “status”: { “fv”: [ null, “2” ] } })

But the Ajax call that gets sent by App.runAction then differs, more precisely the values mk=462 and mt=29 are missing (and that’s probably our problem):

(a) GET /api/cert/ElementResponsesBySection/1453?an=set_score_action_directly &fmt=json&fv&fv=2&sr=1453

(b) GET /api/cert/ElementResponsesBySection/1453?an=set_score_action_directly &dm=detail&fmt=json&fv&fv=2&mk=462&mt=29&sr=1453&wt=d

Monday, August 18, 2025

Mon, 18 Aug 2025 00:00:00 +0000

Monday, August 18, 2025¶

To be meditated: What should be the default value for the group field of a PrivacyRelevant? When an end-user writes a comment, and the group is not obvious from the context, Lino can set the user’s current_group. But if that happens to be empty as well? Okay technically the group field may be empty, but right now a comment without a group wouldn’t be visible even to its author. I could change that rule and say that comments without a group are visible to everybody, but that sounds dangerous on sites with very confidential data. Okay we have the plugin setting lino.modlib.users.private_default.

Answer: a comment without a group should be visible (1) only to its author (and to SiteAdmin) if it’s marked private, and (2) to everybody if it’s not marked private.

Friday, August 1, 2025

Fri, 01 Aug 2025 00:00:00 +0000

Friday, August 1, 2025¶

I’m trying to understand why Ctrl+Shift+U doesn’t work in LibreOffice on my computer. Somebody suggested to say:

$ flatpak install flathub org.libreoffice.LibreOffice
Looking for matches…

org.libreoffice.LibreOffice permissions:
    ipc      network      fallback-x11      pulseaudio      wayland      x11      dri      file access [1]      dbus access [2]     bus ownership [3]

    [1] host, xdg-config/fontconfig:ro, xdg-config/gtk-3.0, xdg-run/gvfsd
    [2] com.canonical.AppMenu.Registrar, org.gtk.vfs.*
    [3] org.libreoffice.LibreOfficeIpc0

        ID                                            Branch           Op           Remote            Download
 1.     org.gtk.Gtk3theme.Breeze                      3.22             u            flathub           < 192,6 kB
 2.     org.libreoffice.LibreOffice.Locale            stable           i            flathub            < 89,4 MB (partial)
 3.     org.libreoffice.LibreOffice                   stable           i            flathub           < 321,1 MB

Proceed with these changes to the system installation? [Y/n]:

But what does that mean?

Saturday, July 5, 2025

Sat, 05 Jul 2025 00:00:00 +0000

Saturday, July 5, 2025¶

My first communication with a matrix server (after reading the specs):

import json
import requests
from pprint import pprint
server_url = "https://matrix.org"
url = server_url + "/.well-known/matrix/client"
session = requests.Session()
response = session.get(url)
assert response.status_code in {200, 201, 202, 204, 400}
pprint(json.loads(response.text))

The output is:

{'m.homeserver': {'base_url': 'https://matrix-client.matrix.org'},
 'm.identity_server': {'base_url': 'https://vector.im'},
 'org.matrix.msc2965.authentication': {'account': 'https://account.matrix.org/account/',
                                       'issuer': 'https://account.matrix.org/'},
 'org.matrix.msc4143.rtc_foci': [{'livekit_service_url': 'https://livekit-jwt.call.matrix.org',
                                  'type': 'livekit'}]}

BeautifulSoup and lxml

Fri, 04 Jul 2025 00:00:00 +0000

BeautifulSoup and lxml¶

Friday, July 4, 2025

When you build a BeautifulSoup from a plain text, that text was getting wrapped into a  tag when the installed lxml version is earlier than 6, but isn’t any more with the new version of lxml.

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup("Foo", "lxml")

The following snippet passes with lxml before 6 …

>>> print(soup)  # lxml<6
<html><body><p>Foo</p></body></html>

… but after lxml 6 the output changes:

>>> print(soup)  # lxml>=6
<html><body>Foo</body></html>

This is surprising because the behaviour of lxml itself hasn’t changed. The following snippet passes with both versions of lxml:

>>> from lxml import etree
>>> etree.fromstring("Foo")
Traceback (most recent call last):
...
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

How can the behaviour of BeautifulSoup change with the lxml version when the behaviour of lxml itself does not change? The answer seems to be hidden somewhere in the deep waters of the BeautifulSoup source code. I didn’t understand it fully. But here are some more observations (they all pass with both lxml versions):

>>> etree.fromstring("")
Traceback (most recent call last):
...
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1

>>> from pprint import pprint
>>> from bs4 import builder_registry
>>> pprint(builder_registry.builders_for_feature)
defaultdict(<class 'list'>,
            {'fast': [<class 'bs4.builder._lxml.LXMLTreeBuilder'>,
                      <class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>],
             'html': [<class 'bs4.builder._lxml.LXMLTreeBuilder'>,
                      <class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>],
             'html.parser': [<class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>],
             'lxml': [<class 'bs4.builder._lxml.LXMLTreeBuilder'>,
                      <class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>],
             'lxml-html': [<class 'bs4.builder._lxml.LXMLTreeBuilder'>],
             'lxml-xml': [<class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>],
             'permissive': [<class 'bs4.builder._lxml.LXMLTreeBuilder'>,
                            <class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>],
             'strict': [<class 'bs4.builder._htmlparser.HTMLParserTreeBuilder'>],
             'xml': [<class 'bs4.builder._lxml.LXMLTreeBuilderForXML'>]})

>>> soup.builder
<bs4.builder._lxml.LXMLTreeBuilder object at ...>

>>> print(soup.builder.test_fragment_to_document("Foo"))
<html><body>Foo</body></html>

Thursday, June 26, 2025

Thu, 26 Jun 2025 00:00:00 +0000

Thursday, June 26, 2025¶

I was wondering… yes there is a pitfall when using Jinja. When your class defines a property, and when, while computing this property, it raises an AttributeError, then Jinja catches this exception and reports that your object doesn’t have the requested property instead of showing the full traceback.

from jinja2 import Environment, DictLoader

A = False
B = True


class Foo:

    _bar = None

    @property
    def bar(self):
        if self._bar is None:
            self._bar = Answer(self)
        return self._bar


class Answer:

    def __init__(self, foo):
        if A:
            raise Exception("Oops")
        elif B:
            raise AttributeError("Oops")
        self.answer = 42


TEMPLATE = "The answer is {{foo.bar.answer}}."


env = Environment(
    loader=DictLoader({'1': TEMPLATE}),
)
tpl = env.get_template('1')
print(tpl.render(foo=Foo()))

Output when there’s no bug:

The answer is 42.

Output with exception of type A:

Traceback (most recent call last):
  File "/home/luc/work/blog/docs/blog/2025/0626.py", line 35, in <module>
    print(tpl.render(foo=Foo()))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 490, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/luc/work/blog/docs/blog/2025/0626.py", line 14, in bar
    self._bar = Answer(self)
                ^^^^^^^^^^^^
  File "/home/luc/work/blog/docs/blog/2025/0626.py", line 22, in __init__
    raise Exception("Oops")
Exception: Oops

Output with exception of type B:

Traceback (most recent call last):
  File "/home/luc/work/blog/docs/blog/2025/0626.py", line 35, in <module>
    print(tpl.render(foo=Foo()))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/jinja2/environment.py", line 490, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: '__main__.Foo object' has no attribute 'bar'

Monday, June 23, 2025

Mon, 23 Jun 2025 00:00:00 +0000

Monday, June 23, 2025¶

Rename FinalExam to FinalExamTemplate

New model FinalExam(template, group)

Friday, June 20, 2025

Fri, 20 Jun 2025 00:00:00 +0000

Friday, June 20, 2025¶

About the doctest failures in specs/calview.rst and specs/avanti/db.rst (avanti1):

The number of cal.Event changed from 278 to 533 (+155). The new events are suggested meetings with clients according to their event_policy. Explanation: the generate_calendar_entries procedure was not being run by pm prep, now it is being run. Good.

About the doctest failures in projects/voga2.rst and plugins/checkdata.rst (voga2):

(1) plugins/checkdata.rst modified the database but didn’t run check_virgin(), causing nasty surprises

(2) The checkdata procedure (which finds conflicting events) was being run before generate_calendar_entries (which generates them):

$ pm show linod.Procedures
=========================== =========================== =========================== ================== ===========================
 value                       name                        text                        Task class         Suggested recurrency
--------------------------- --------------------------- --------------------------- ------------------ ---------------------------
 checkdata                   checkdata                   checkdata                   linod.SystemTask   every=1, every_unit=daily
 generate_calendar_entries   generate_calendar_entries   generate_calendar_entries   linod.SystemTask   every=1, every_unit=daily
 checksummaries              checksummaries              checksummaries              linod.SystemTask   every=1, every_unit=daily
 run_invoicing_tasks         run_invoicing_tasks         run_invoicing_tasks         invoicing.Task     every=1, every_unit=daily
 delete_older_changes        delete_older_changes        delete_older_changes        linod.SystemTask   every=1, every_unit=daily
 update_publisher_pages      update_publisher_pages      update_publisher_pages      linod.SystemTask   every=1, every_unit=daily
=========================== =========================== =========================== ================== ===========================

Which means that checkdata messages appeared only during a second loop of the task runner.

Both checkdata and generate_calendar_entries are daily background tasks.

Tuesday, June 3, 2025

Tue, 03 Jun 2025 00:00:00 +0000

Tuesday, June 3, 2025¶

I’m working on miscellaneous optimizations for tim2lino and Peppol.

What if a customer becomes send_peppol after the onboarding date, and had already invoices sent before the onboarding date? For example I activate my customer as a supplier, they start sending invoices to some of their customers. One month later, two customers of my customer onboard and ask to receive e-invoices. My customer also has sales invoices to these two customers during the first month, these invoices were sent outside of peppol via email. Right now Lino would send the invoices of the first month to these two customers.

This is why we now have PeppolJournal.last_sending.

Q: What happens when a supplier has peppolReceiver set to false and nevertheless: asks for inbound documents? Will Ibanity simply return an empty list? Or will it return an error?

A: It returns an empty list

I asked two more questions on the Ibanity forum:

Is is allowed that a supplier sends their invoices with several days of delay? One of my customer wants this as a feature. They would generate a sales invoice, send it as a pro-forma invoice to their customer via email and give them two days for verifying the content. The real invoice comes only after this verification. But they want the official invoice to be dated on the same date as the pro-forma invoice. They know that they cannot send any other sales invoices while waiting for feedback from the pro-forma invoices. Is this technically possible? Is it legally allowed?
And another question. Another customer has multiple journals for issuing sales invoices, used by different salesmen. One salesman goes online every day and sends their invoices via Peppol. The second salesman goes online only once a week. Is the second salesman allowed to send an invoice dated three days ago when the first salesman has sent an invoice yesterday?

Both got the same answer: Out of scope of Peppol/API. Peppol doesn’t care when/what you send. So, it’s technically allowed. Regulatory, you’ll need to find some info per region. My gut feeling is that it’s not a problem.

TODO:

Lino does not yet delete a supplier on Ibanity when I remove them from our list of suppliers.

Monday, May 19, 2025

Mon, 19 May 2025 00:00:00 +0000

Monday, May 19, 2025¶

Puppeteer¶

I tried to run the puppeteer test suite in react:

(dev) luc@frodo:~/work/react$ npx puppeteer browsers install chrome
Downloading chrome 131.0.6778.204 - 165.9 MB [====================] 100% 0.0s
chrome@131.0.6778.204 /home/luc/.cache/puppeteer/chrome/linux-131.0.6778.204/chrome-linux64/chrome

(dev) luc@frodo:~/work/react$ npm test

> lino_react@25.4.2 test
> jest

Error: Jest: Got error running globalSetup - /home/luc/work/react/lino_react/react/testSetup/setupJEST.js, reason: Failed to launch the browser process!

[1185962:1185962:0519/073804.739181:FATAL:zygote_host_impl_linux.cc(128)] No
usable sandbox! If you are running on Ubuntu 23.10+ or another Linux distro
that has disabled unprivileged user namespaces with AppArmor, see
https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md.
Otherwise see
https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md
for more information on developing with the (older) SUID sandbox. If you want
to live dangerously and need an immediate workaround, you can try using
--no-sandbox.

TROUBLESHOOTING: https://pptr.dev/troubleshooting

    at ChildProcess.onClose (/home/luc/work/react/node_modules/@puppeteer/browsers/src/launch.ts:486:11)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess.emit (node:domain:488:12)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)

I read the mentioned Chromium Docs and did as instructed there:

echo 0 | sudo tee /proc/sys/kernel/apparmor_restrict_unprivileged_userns

Now the test suite works:

(dev) luc@frodo:~/work/react$ npm test

> lino_react@25.4.2 test
> jest

 PASS  lino_react/react/components/__tests__/integrity.ts (14.057 s)
  integrity.ts
    ✓ load landing page (1599 ms)
    ✓ sign in ok (5476 ms)

 PASS  lino_react/react/components/__tests__/cloneUI.ts (17.069 s)
  cloneUI.ts
    ✓ clone tickets.AllTickets (10185 ms)

 FAIL  lino_react/react/components/__tests__/test_URLContext.ts (39.691 s)
  test_URLContext.ts
    ✓ trading.InvoicesByJournal workflow button (11581 ms)
    ✕ test grid_put (21015 ms)

  ● test_URLContext.ts › test grid_put

    Could not satisfy condition

      18 |     }
      19 |
    > 20 |     let err = Error("Could not satisfy condition");
         |               ^
      21 |     throw err;
      22 | }
      23 |

      at Object.<anonymous>.global.waitToMeet (lino_react/react/testSetup/setupTests.ts:20:15)
      at Object.dataContextReady (lino_react/react/testSetup/setupTests.ts:38:9)
      at Object.<anonymous>.global.signIn (lino_react/react/testSetup/setupTests.ts:78:5)
      at Object.<anonymous> (lino_react/react/components/__tests__/test_URLContext.ts:48:9)

 FAIL  lino_react/react/components/__tests__/testParamChanges.ts (63.668 s)
  testParamChanges.ts
    ✕ test #5792 (56206 ms)

  ● testParamChanges.ts › test #5792

    TypeError: Cannot read properties of null (reading 'getProperty')

      45 |
      46 |         header = await page.$("div.l-detail-header>span");
    > 47 |         headerText = await (await header.getProperty("textContent")).jsonValue();
         |                                          ^
      48 |
      49 |         expect(headerText).toBe("Tickets to work (Assigned to Luc) » #101 (Foo never bars)");
      50 |         header.dispose();

      at Object.<anonymous> (lino_react/react/components/__tests__/testParamChanges.ts:47:42)

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.
Test Suites: 2 failed, 2 passed, 4 total
Tests:       2 failed, 4 passed, 6 total
Snapshots:   0 total
Time:        65.011 s
Ran all test suites.

This looks well! I won’t dive further now, I’m glad to have Sharif for the React front end.

SalesRule becomes TradeRule¶

invoicing.SalesRule becomes trading.TradeRule

I tried lilyponddist

Thu, 01 May 2025 00:00:00 +0000

I tried lilyponddist¶

Thursday, May 1, 2025

Today I discovered the lilyponddist package and tried it. It works:

>>> import requests
>>> import lilyponddist
>>> import subprocess

>>> url = "https://lilypond.org/ly-examples/bach-bwv610.ly"
>>> src = "/tmp/tmp.ly"
>>> out = "/tmp/tmp"
>>> r = requests.get(url)
>>> open(src, "wt").write(r.text)
4940
>>> subprocess.call([lilyponddist.lilypondbin(), '--pdf', '-o', out, src])
0
>>> subprocess.call(["xdg-open", out + '.pdf'])
0

The console output is something like this:

Processing `tmp.ly'
Parsing...
Interpreting music...[8]
Preprocessing graphical objects...
Finding the ideal number of pages...
Fitting music on 1 or 2 pages...
Drawing systems...
Converting to `tmp.pdf'...
Success: compilation successfully completed

I’m positively impressed by the ease of installation and execution speed. It also works with svg or png instead of pdf. Cool.

We could this in Lino. Not yet sure how exactly. Wikimedia has Extension:Score but the Lino way would probably be to have a music plugin with a music.Score model, or maybe even a music.Song model with individual text fields like soprano, alto, tenor, bass, lyrics, etc, as I had in songbook.

Sources:

Tuesday, April 22, 2025

Tue, 22 Apr 2025 00:00:00 +0000

Tuesday, April 22, 2025¶

Today I had the rare occasion of working on a Windows machine. I tried a few things for How to import TIM data to Lino. Here are some screenshots.

Microsoft Windows [version 10.0.26100.3775]
(c) Microsoft Corporation. Tous droits réservés.

C:\Windows\System32>wsl --install
Téléchargement en cours : Sous-système Windows pour Linux 2.4.13
Installation en cours : Sous-système Windows pour Linux 2.4.13
Sous-système Windows pour Linux 2.4.13 a été installé.
Installation du composant facultatif Windows : VirtualMachinePlatform

Outil Gestion et maintenance des images de déploiement
Version : 10.0.26100.1150

Version de l’image : 10.0.26100.3775

Activation de la ou des fonctionnalités
[==========================100.0%==========================]
L’opération a réussi.
L’opération demandée est réussie. Les modifications ne seront pas effectives avant que le système ne soit réamorcé.
L’opération demandée est réussie. Les modifications ne seront pas effectives avant que le système ne soit réamorcé.

This wasn’t needed:

C:\Windows\System32>wsl --install Debian
Téléchargement : Debian GNU/Linux
Installation : Debian GNU/Linux
La distribution a été installée. Il peut être lancé via 'wsl.exe -d Debian'

This one didn’t help:

C:\Windows\System32>ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\utilisateur/.ssh/id_rsa):
Created directory 'C:\\Users\\utilisateur/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\utilisateur/.ssh/id_rsa
Your public key has been saved in C:\Users\utilisateur/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:LdA41... utilisateur@DESKTOP-K1MP4G1
The key's randomart image is:
+---[RSA 3072]----+
|     .  .. +.o.o*|
|      o+  = + +.=|
|      +*.. = o +o|
|      +o*.o +   o|
|     . =S..+ . . |
|    . + =.+   +  |
|     . E o = o o |
|      .   + . . .|
|                 |
+----[SHA256]-----+

Friday, April 18, 2025

Fri, 18 Apr 2025 00:00:00 +0000

Friday, April 18, 2025¶

I worked on #5754 (How to install esteid digidoc on Tuxedo OS) and #6007 (eidreader test suite fails with libbeidpkcs11.so.0: No such file or directory). Seems that both issues are related. Difficult to get help because nobody feels responsible. After finding and installing the pyasice package <https://pypi.org/project/pyasice/#quickstart> I wrote a script asice.py, which is supposed to print the names of the people who signed an ASICE file:

import click
from pyasice import Container

@click.command()
@click.argument("filenames", nargs=-1)
def main(filenames: tuple[str, ...]):
    """Show the signers of the specified .asice file."""
    for fn in filenames:
        try:
            cont = Container.open(fn)
            text = "Signatures: "
            for n, sig in enumerate(cont.iter_signatures()):
                text += f"\n{n+1}) {sig.get_certificate().x509}"
        except Exception as e:
            text = str(e)
        print(fn, ":", text)

if __name__ == '__main__':
    main()

I had the following traceback:

Traceback (most recent call last):
  File "/home/luc/bin/asice.py", line 6, in <module>
    from pyasice import Container, XmlSignature
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/pyasice/__init__.py", line 11, in <module>
    from .container import Container
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/pyasice/container.py", line 8, in <module>
    from oscrypto.asymmetric import Certificate, load_certificate
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/asymmetric.py", line 19, in <module>
    from ._asymmetric import _unwrap_private_key_info
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/_asymmetric.py", line 27, in <module>
    from .kdf import pbkdf1, pbkdf2, pkcs12_kdf
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/kdf.py", line 9, in <module>
    from .util import rand_bytes
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/util.py", line 14, in <module>
    from ._openssl.util import rand_bytes
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/_openssl/util.py", line 6, in <module>
    from ._libcrypto import libcrypto, libcrypto_version_info, handle_openssl_error
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/_openssl/_libcrypto.py", line 9, in <module>
    from ._libcrypto_cffi import (
  File "/home/luc/venvs/dev/lib/python3.12/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 44, in <module>
    raise LibraryNotFoundError('Error detecting the version of libcrypto')
oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto

After reading this thread on github, worked around this for my personal use by saying:

$ pip uninstall oscrypto
$ pip install git+https://github.com/wbond/oscrypto.git@d5f3437

Interesting to note that the author of oscrypto, Will Bond, seems a bit special. Smells like frustration or burnout. In August 2023 somebody asked “I saw you already fixed that in d5f3437, but as this is not released to pypi our Tool does not fully work. Would it be possible to release a bug-fix version to pypi?”, a request that was then backed by half a dozen of other people who had the same problem. And in October he finally found some time to write an answer: “No, this is a free-time project for me. The commit is there, so it is certainly possible to use most Python packaging tools to grab the sha of the commit.”

Thursday, April 17, 2025

Thu, 17 Apr 2025 00:00:00 +0000

Thursday, April 17, 2025¶

Today I continued testing Ibanity synchronization using real data in integration mode.

My first error message was:

No transmissionId in {
'data': {
  'id': 'd772574a-bddb-4a7b-9b60-4921736daaca',
  'type': 'peppolInvoice',
  'attributes': {
    'createdAt': '2025-04-17T06:43:37.760Z',
    'errors': [{
        'code': 'invalid-schematron',
        'detail': 'Belgian VAT number MUST bestated in the correct format.
        (/Invoice/cac:AccountingSupplierParty/cac:Party/cbc:EndpointID)\nBelgian VAT
        number MUST be stated in the correct format.
        (/Invoice/cac:AccountingCustomerParty/cac:Party/cbc:EndpointID)'}],
    'status': 'invalid',
    'technicalStatus': 'invalid'},
  'relationships': {
    'supplier': {'data': {
      'id': '249e6232-5d7d-4654-8547-c8b252309698',
      'type': 'supplier'}}}}}

Okay, this is probably yet another trace of having validated against the wrong rule set. Seems that in Belgium they require the two first letters of the VAT number to be specified in <cbc:EndpointID>`. Why not.

Next error message was:

“Supplier data in document can’t be matched with data of supplier.
Value found in //cac:AccountingSupplierParty/cac:Party/cac:PartyName/cbc:Name can’t be matched with internal supplier data: FIRMA SOUNDSO AG”

Indeed, I had registered them using the name “Soundso A.G.”, not “FIRMA SOUNDSO AG”. After changing the site owner’s name to “SOUNDSO A.G.” it worked.

Here is the next error message:

No transmissionId in {'data': {'id': '94aa14c3-a5c6-49a3-9ec6-a13d66822087',
'type': 'peppolInvoice', 'attributes': {'createdAt':
'2025-04-17T07:09:44.016Z', 'errors': [{'code':
'error-customer-not-registered', 'detail': "Identifier
'iso6523-actorid-upis::9925:be 1234567890' is not registered in SML."}],
'status': 'send-error', 'technicalStatus': 'send-error'}, 'relationships':
{'supplier': {'data': {'id': '249e6232-5d7d-4654-8547-c8b252309698', 'type':
'supplier'}}}}}

This makes sense because the recipient of that invoice obviously didn’t yet register to the Peppol network. The SML means Service Metadata Locator and because we said SchemeID “9925” (which means “Belgian VAT number”), they probably queried the Belgian VAT office to ask whether the organization with VAT id 1234567890 has already onboarded to the Peppol network.

TODO:

The end user should not need to read the log messages of the SyncPeppol action. Such error messages should be stored in the OutboundDocument instance.
When the VAT Regime is subject or cocontractor, then the end user doesn’t need to check PeppolPartner.send_peppol because that’s mandatory.

Monday, April 14, 2025

Mon, 14 Apr 2025 00:00:00 +0000

Monday, April 14, 2025¶

For one of my demo invoices the UBL validator says: [BR-Z-01]-An Invoice that contains an Invoice line (BG-25), a Document level allowance (BG-20) or a Document level charge (BG-21) where the VAT category code (BT-151, BT-95 or BT-102) is “Zero rated” shall contain in the VAT breakdown (BG-23) exactly one VAT category code (BT-118) equal with “Zero rated”.

And yes this invoice sells a bottle of gas with normal VAT rate, and the bottle itself without VAT (because the bottle is actually not sold, the customer gets this money back when he returns the empty bottle).

Does this mean that it’s not allowed to issue invoices with a mixture of normal-vat and zero-vat?

To be exact, my example is even a bit more complex, it contains three different VAT categories “normal”, “reduced” and “exempt”.

>>> from lino import startup
>>> startup('lino_book.projects.cosi1.settings')
>>> from lino.api.doctest import *

>>>

>>> obj = trading.VatProductInvoice.objects.get(pk=111)
>>> print(obj)
SLS 6/2014

>>> rt.show(trading.ItemsByInvoice, obj, language="en")
==================== ================ ========== ======== ======== ============== ========
 Product              Heading          Discount   Pr       Qty      TotExcl        InvObj
-------------------- ---------------- ---------- -------- -------- -------------- --------
 Book                 Book                        29,90    10       299,00
 Stamp                Stamp                       1,40     8        11,20
 Wooden table         Tisch aus Holz              199,99   4        799,96
 **Total (3 rows)**                                        **22**   **1 110,16**
==================== ================ ========== ======== ======== ============== ========

>>> rt.show(vat.MovementsByVoucher, obj, language="en")
================== ==================== ============== ============== ================================= ================ =========
 Account            Partner              Debit          Credit         VAT class                         Match            Cleared
------------------ -------------------- -------------- -------------- --------------------------------- ---------------- ---------
 (7000) Sales                                           299,00         Goods at reduced VAT rate (12%)                    Yes
 (4510) VAT due                                         35,88          Goods at reduced VAT rate (12%)                    Yes
 (7000) Sales                                           11,20          Goods exempt from VAT                              Yes
 (7000) Sales                                           799,96         Services                                           Yes
 (4510) VAT due                                         167,99         Services                                           Yes
 (4000) Customers   Garage Mergelsberg   1 314,03                                                        **SLS 6/2014**   Yes
                                         **1 314,03**   **1 314,03**
================== ==================== ============== ============== ================================= ================ =========

Later it told me “Specification identifier MUST have the value ‘urn:cen.eu:en16931:2017#conformant#urn:UBL.BE:1.0.0.20180214’.” Aha. But what is a Specification identifier? https://docs.peppol.eu/edelivery/specs/reporting/tsr/trn-transaction_statistics/semantic-model/trbt-001/

Error message: [BR-CL-17]-Invoice tax categories MUST be coded using UNCL5305 code list

Friday, April 11, 2025

Fri, 11 Apr 2025 00:00:00 +0000

Friday, April 11, 2025¶

I made an XML file named “VKR-2025-33.xml”, then POSTed it to https://api.ibanity.com/einvoicing/peppol/suppliers/249e6232-5d7d-4654-8547-c8b252309698/invoices?

And the response was {‘id’: ‘7031b2c6-acfa-450b-84ab-4ac557fd6f69’, ‘type’: ‘peppolInvoice’, ‘attributes’: {‘createdAt’: ‘2025-04-11T13:34:20.548Z’, ‘status’: ‘created’, ‘technicalStatus’: ‘created’}, ‘relationships’: {‘supplier’: {‘data’: {‘id’: ‘249e6232-5d7d-4654-8547-c8b252309698’, ‘type’: ‘supplier’}}}}

And then I request GET https://api.ibanity.com/einvoicing/peppol/suppliers/249e6232-5d7d-4654-8547-c8b252309698/invoices/e62dd6dc-b094-49b9-af02-8fd630171dab to see the status of the document.

And the response is {‘data’: {‘id’: ‘e62dd6dc-b094-49b9-af02-8fd630171dab’, ‘type’: ‘peppolInvoice’, ‘attributes’: {‘createdAt’: ‘2025-04-11T12:53:53.702Z’, ‘errors’: [{‘code’: ‘invalid-format’, ‘detail’: ‘Document format is not supported’}], ‘status’: ‘invalid’, ‘technicalStatus’: ‘invalid’}, ‘relationships’: {‘supplier’: {‘data’: {‘id’: ‘249e6232-5d7d-4654-8547-c8b252309698’, ‘type’: ‘supplier’}}}}}

But AFAICS my XML file is valid according to https://ecosio.com/en/peppol-and-xml-document-validator/

What am I missing?

Python 3.13 breaks sending of emails

Wed, 02 Apr 2025 00:00:00 +0000

Python 3.13 breaks sending of emails¶

Wednesday, April 2, 2025

TIM has a single field PAR->email with the customer’s email addresses. And TIM users know that they can put multiple email addresses into this field if they separate them by a semicolon. A semicolon?! According to RFC 5322 the correct delimiter for recipients in the To: header is the comma, not a semicolon. Yes, but a semicolon works as well. Or at least it used to work as well. But some time ago (I guess in version 3.2) the Python email module has been modernized and no longer accepts strange values in the To: header.

TIM uses the sendmail.py script to actually send the emails.

>>> import email
>>> from email.utils import getaddresses

Here is an example of an email generated by TIM (the sendmail.py script reads this from a file and cares about the special cp850 encoding, but we can simplify this here):

>>> TEMPLATE = """From: foo@example.be
... To: {recipient}
... Subject: Rechnung
...
... Sehr geehrter Kunde,
... anbei unsere Rechnung mit Gesamtbetrag 407,90
... """

And here is a function to test how sendmail.py parses the recipients from the To: header:

>>> def test(to):
...     msg = email.message_from_string(TEMPLATE.format(recipient=to))
...     return getaddresses(msg.get_all('to'))

And yes, semicolons aren’t okay while commas are okay:

>>> test("bar@example.be;baz@example.be")
[('', '')]

>>> test("bar@example.be,baz@example.be")
[('', 'bar@example.be'), ('', 'baz@example.be')]

Until some time ago, the first example would also have returned a list of parser addresses. And now poof, more than a hundred customers of my customer didn’t get their invoice. Because two weeks ago my customer had inadvertently deleted the Python from their PC, and I had reinstalled it. The new version is 3.13 and I think that the old one was 3.10 but I’m not sure.

And I didn’t find a way to restore the old behaviour. The compat32 policy should do it (according to the docs) but doesn’t:

>>> from email.policy import compat32
>>> s = TEMPLATE.format(recipient="bar@example.be;baz@example.be")
>>> msg = email.message_from_string(s, policy=compat32)
>>> getaddresses(msg.get_all('to', []))
[('', '')]

I could customize the header_factory, but I’m afraid that this might change in the future.

So I’ll rather ask the TIM user to replace their semicolons by commas.

More precisely I wrote a TIM script for them:

// activate from command line using "tim /exec:actexec('parscan')"
DlgDefine("b",{||(PAR->email:=strtran(PAR->email,";",","))!=NIL})
Confirm(MsgSure())
DbfScan({oPar()},1,NIL,NIL,NIL,{||TryRecLock(DlgValue('b'))})
SetMsg(MsgDone())

More tests:

>>> getaddresses(["foo@example.com; bar@example.org"])
[('', '')]

>>> getaddresses(["foo@example.com; bar@example.org"], strict=False)
[('', 'foo@example.com'), ('', ''), ('', 'bar@example.org')]

>>> import sys
>>> sys.version
'3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]'

Monday, March 31, 2025

Mon, 31 Mar 2025 00:00:00 +0000

Monday, March 31, 2025¶

Some days ago I upgraded the software for using Estonian ID cards and now eidreader is broken, inv test says:

File "docs/usage.rst", line 191, in usage.rst
Failed example:
    shell("eidreader")  #doctest: +NORMALIZE_WHITESPACE
Expected:
    {"eidreader_version": "1.0.8", "success": false, "message": "Could not find any reader with a card inserted"}
Got:
    {"eidreader_version": "1.0.8", "success": false, "message": "Middleware not properly installed"}
    src/dyn_unix.c:34:SYS_dyn_LoadLibrary() libbeidpkcs11.so.0: cannot open shared object file: No such file or directory

The following approach did not work:

$ sudo apt remove libbeidpkcs11-bin
...
The following packages will be REMOVED
  libbeidpkcs11-bin
0 to upgrade, 0 to newly install, 1 to remove and 51 not to upgrade.
...
Removing libbeidpkcs11-bin (5.1.18v5.1.18-0u2204-1) ...

$ sudo apt install libbeidpkcs11-bin
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package libbeidpkcs11-bin is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

I tried to reinstall Belgian eid software according to Installation. But they don’t work on my computer, which I bought from a German company, Tuxedo, who use install own variant of Debian called “Tuxedo OS”. I asked from Tuxedo for help, but they answered promptly that “Unfortunately we are unable to offer support for the individual setup, configuration or use of software.”

Thursday, March 27, 2025

Thu, 27 Mar 2025 00:00:00 +0000

Thursday, March 27, 2025¶

The Ibanity staff activated our application, now I will configure Jane to use it.

Creating a new application client certificate is more complicated for “live” applications than for “sandbox” applications. The developer portal asks me

to generate an RSA key pair:

openssl genrsa -aes256 -out private_key.pem 2048

and then to generate a Certificate Signing Request:

openssl req -new -sha256 -key private_key.pem -out ibanity.csr \
-subj "/C=BE/O=Isabel Group/OU=Isabel Group Certificate Authority/CN=Lino application account (abc123)/serialNumber=123456"

This creates a file ibanity.csr

and then to upload the ibanity.csr file to their server.

In response, I receive a zip file with two files ca_chain.pem and certificate.pem.

I assume that I should run these commands in the project directory of my Lino site.

Thursday, March 13, 2025

Thu, 13 Mar 2025 00:00:00 +0000

Thursday, March 13, 2025¶

We had another occurrence of #5965 (SyntaxError: unterminated string literal) in the daily cron job that runs pull_demos.sh on lino-framework.org.

The reason was that autopep8 had transformed

if response.status_code not in {200, 201, 202}:
    raise Exception(f"{meth_name} {url} returned unexpected status code {response.status_code}")

into

if response.status_code not in {200, 201, 202}:
    raise Exception(f"{meth_name} {url} returned unexpected status code {
        response.status_code}")

This is indeed valid Python code under Python 3.12, but on a machine with Python 3.9 it caused a syntax error. The new syntax was introduced in 3.12.

As a result, our public demo sites were down for several hours.

Must I run all tests on both Python versions before pushing? The whole Lino test suite (pp inv prep test clean -b bd) lasts almost an hour on my computer.

I might have changed the line into:

if response.status_code not in {200, 201, 202}:
    raise Exception(f"{meth_name} {url} returned unexpected status code " \
        f"{response.status_code}")

But I finally refactored it yet a bit more:

if response.status_code not in {200, 201, 202}:
    msg = f"{meth_name} {url} returned unexpected status code " \
        f"{response.status_code}"
    raise Exception(msg)

This “wastes” a bit of processor time for creating a variable name, but I prefer this construct because it is more readable, because it’s easier to extend the message, for example like this:

msg += "and the content is {response.content}"

or to insert a temporary debugging message:

print(msg)

The underlying reason for these issues is that I recently started to use automated PEP8 code formatting. My Pulsar now has pulsar-ide-python installed.

Deep in my heart I fully agree with E501 (line too long). It’s not because of my eyes. Okay they are getting older and I need larger fonts than young people, but I stopped using two-column code editing already quite some time ago, so I could work on lines of 100 chars without problem. The reason is rather this:

“when people let themselves write long lines, they stop caring about proper abstraction in their code. Everything turns into uber-nested unreadable soup. If a line is too long, it should probably get refactored.” — 1668553684 on reddit.com

Monday, March 10, 2025

Mon, 10 Mar 2025 00:00:00 +0000

Monday, March 10, 2025¶

Collect outbound invoices into outbox Send outbound documents Check outbound documents Check our inbox No inbound documents to download.

Collect outbound documents into outbox Send outbound documents Check status of sent documents Check for new inbound documents Download inbound documents.

No inbound documents to download.

Thursday, March 6, 2025

Thu, 06 Mar 2025 00:00:00 +0000

Thursday, March 6, 2025¶

I’m working on #5964 (pm prep in avanti1 says ‘str’ object has no attribute ‘get_layout_handle’)

In lino_xl/lib/courses/models.py we had:

@dd.requestfield(_("Enrolments"))
def enrolments(self, ar):
    return self.get_enrolments(start_date=dd.today())

def get_enrolments(self, **pv):
    return rt.models.courses.EnrolmentsByCourse.create_request(self,
                                                        param_values=pv)

The issue went away when I added “if ar is None: return” in the enrolments virtual field before calling get_enrolments().

But why? And why did it start only now?

And the same error message happens when I say doctest docs/specs/avanti/courses.rst, this time it is when courses/Course/presence_sheet.weasy.html says:

{% for enrolment in obj.get_enrolments(
   start_date=ar.action_param_values.start_date,
   end_date=ar.action_param_values.end_date) %}

Note that get_enrolments() accesses rt.models.courses.EnrolmentsByCourse. This is an actor, more precisely a table. And avanti redefines this table.

Lino is trying to setup the actor handle for the EnrolmentsByCourse from lino_xl.lib.courses instead of lino_avanti.lib.courses.

Aha! Explanation: It’s an aftermath of #5956 (python-tools no longer works). One of those automatic formatters that came with pulsar-ide-python, which I installed to replace the orphaned python-tools, insists on moving all global imports to the top of the file. Including these ones:

from .ui import *

But these imports must come at the end of the models.py file, at least for lino_avanti.lib.courses.

No. They don’t need to be at the end of models.py file, it’s enough if they are after the:

from lino_xl.lib.courses.models import *

Should we stop using `from .ui import *` ?¶

Most people say that from x import * is bad code style. But we use from .ui import * as a feature in a well-defined context in order to follow another code style principle which is “don’t repeat yourself”. We deliberately merge database models and actors into a same pot (the plugin’s namespace) because it makes documentation more concise. For application developers it makes no sense to differentiate between models, tables, choicelists because these are implementation details. But the more I think about this I realize that it’s a controversial topic, and for a developer who is used to relying on their IDE it can be a serious obstacle for diving into Lino.

I started to move all from .ui import * lines from the bottom of the models.py file to its top. But actually I have no clear plan yet. Even if we would name them explicitly, pyflakes would still complain because we import them and then don’t use them.

There is another topic, related to this: we actually have two namespaces per plugin: dd.plugins.foo and rt.models.foo. Their difference is that dd.plugins is available as soon as the plugins have been installed while rt.models is available only when Django has populated its database models. The items of lino.api.dd.plugins are instances of lino.core.plugins.Plugin while the items of lino.api.rt.models are the models.py module objects.

Once upon a time we had the actors collected into a namespace rt.actors, separately from rt.models. This used a automagic behaviour that we copied from Django: Django, when starting up, loops over INSTALLED_APPS and tries to import a module appname + ".models".

I had a look at django.apps.registry and I am tempted to try to review the Lino startup process because Django has evolved during the last 10 years. Let lino.core.plugin.Plugin inherit from AppConfig and use apps.get_app_config() instead of dd.plugins. Override AppConfig.import_models() so that it also tries to import a “ui” module, similar to what it does with a “models” module.

That would be fun! I’m really tempted. But there are more urgent things to do right now.

Thursday, February 27, 2025

Thu, 27 Feb 2025 00:00:00 +0000

Thursday, February 27, 2025¶

The Ecosio validator says “[BR-CL-17]-Invoice tax categories MUST be coded using UNCL5305 code list”, but there are two such code lists: A and B. One of them has a code “AA” for reduced VAT and the other one hasn’t. And my invoice contains indeed a line with reduced VAT.

Explanation: Haha! I was asking a rule set for Australia and New-Zealand. With the “UBL Invoice 2.4” rule set, the invoices generated by Lino now pass.

Wednesday, February 26, 2025

Wed, 26 Feb 2025 00:00:00 +0000

Wednesday, February 26, 2025¶

Element/context:	Error message:
cac:InvoiceLine[1]/cbc:InvoicedQuantity[1]	[BR-CL-23]-Unit code MUST be coded according to the UN/ECE Recommendation 20 with Rec 21 extension
cac:TaxTotal[1]/cac:TaxSubtotal[1]/cac:TaxCategory[1]/cbc:ID[1]	[BR-CL-17]-Invoice tax categories MUST be coded using UNCL5305 code list
cac:AccountingCustomerParty[1]/cac:Party[1]/cac:PartyLegalEntity[1]/cbc:CompanyID[1]	[BR-CL-11]-Any registration identifier identification scheme identifier MUST be coded using one of the ISO 6523 ICD list.

Saturday, February 22, 2025

Sat, 22 Feb 2025 00:00:00 +0000

Saturday, February 22, 2025¶

I am meditating on how to integrate into Lino the Participant Identifier Schemes published as part of the OpenPeppol eDEC Code Lists.

For example the <cbc:EndpointID> for my employer Rumma & Ko OÜ is:

<cbc:EndpointID schemeID="9931">100588749</cbc:EndpointID>

The 9931 is the ICD value of item 68 of the list of Participant Identifier Schemes, which has “Estonia VAT number” as Scheme Name. This item also has a Scheme ID of EE:VAT.

Yes, our VAT number is 100588749.

But we also have a company number, which is different from the VAT number: 10501849. And there is another item for Estonia in that list, ICD value 0191, named “Company code”. So here is another valid <cbc:EndpointID> to point to my employer:

<cbc:EndpointID schemeID="0191">10501849</cbc:EndpointID>

So the lino_xl.lib.vat.VatVoucher.fill_from_xml() method will potentially get two different <cbc:EndpointID> for a same partner, either “9931:100588749” or “0191:10501849”.

>>> url = "https://docs.peppol.eu/edelivery/codelists/v9.0/Peppol%20Code%20Lists%20-%20Participant%20identifier%20schemes%20v9.0.json"
>>> import requests
>>> res = requests.get(url).json()
>>> for scheme in sorted(res['values'], key=lambda x: x['country']):
...        print("{country} {iso6523} {schemeid} {scheme-name}".format(**scheme))
AD 9922 AD:VAT Andorra VAT number
AL 9923 AL:VAT Albania VAT number
AT 9914 AT:VAT Österreichische Umsatzsteuer-Identifikationsnummer
AT 9915 AT:GOV Österreichisches Verwaltungs bzw. Organisationskennzeichen
AT 9916 AT:CID Firmenidentifikationsnummer der Statistik Austria
AT 9919 AT:KUR Kennziffer des Unternehmensregisters
AU 0151 AU:ABN Australian Business Number (ABN) Scheme
BA 9924 BA:VAT Bosnia and Herzegovina VAT number
BE 0193 UBLBE UBL.BE Party Identifier
BE 0208 BE:EN Numero d'entreprise / ondernemingsnummer / Unternehmensnummer
BE 9925 BE:VAT Belgium VAT number
BE 9956 BE:CBE Belgian Crossroad Bank of Enterprise number
BG 9926 BG:VAT Bulgaria VAT number
CH 0183 CH:UIDB Swiss Unique Business Identification Number (UIDB)
CH 9927 CH:VAT Switzerland VAT number
CY 9928 CY:VAT Cyprus VAT number
CZ 9929 CZ:VAT Czech Republic VAT number
DE 0204 DE:LWID Peppol-Leitweg-ID
DE 9930 DE:VAT Germany VAT number
DE 9958 DE:LID Peppol-Leitweg-ID
DK 0096 DK:P The Danish Business Authority - P-number (DK:P)
DK 0184 DK:DIGST The Danish Business Authority - CVR-number (DK:CVR)
DK 0198 DK:ERST The Danish Business Authority - SE-number (DK:SE)
DK 9901 DK:CPR Danish Ministry of the Interior and Health
DK 9902 DK:CVR The Danish Commerce and Companies Agency
DK 9904 DK:SE Danish Ministry of Taxation, Central Customs and Tax Administration
DK 9905 DK:VANS Danish VANS providers
EE 0191 EE:CC Company code
EE 9931 EE:VAT Estonia VAT number
ES 9920 ES:VAT Agencia Española de Administración Tributaria
FI 0037 FI:OVT LY-tunnus
FI 0212 FI:ORG Finnish Organization Identifier
FI 0213 FI:VAT Finnish Organization Value Add Tax Identifier
FI 0215 FI:NSI Net service ID
FI 0216 FI:OVT2 OVTcode
FR 0002 FR:SIRENE System Information et Repertoire des Entreprise et des Etablissements: SIRENE
FR 0009 FR:SIRET SIRET-CODE
FR 9957 FR:VAT French VAT number
GB 9932 GB:VAT United Kingdom VAT number
GR 9933 GR:VAT Greece VAT number
HR 9934 HR:VAT Croatia VAT number
HU 9910 HU:VAT Hungary VAT number
IE 9935 IE:VAT Ireland VAT number
IS 0196 IS:KTNR Icelandic identifier
IS 9917 IS:KT Icelandic National Registry
IT 0097 IT:FTI FTI - Ediforum Italia
IT 0135 IT:SIA SIA Object Identifiers
IT 0142 IT:SECETI SECETI Object Identifiers
IT 0201 IT:CUUO Codice Univoco Unità Organizzativa iPA
IT 0205 IT:COD CODDEST
IT 0210 IT:CFI CODICE FISCALE
IT 0211 IT:IVA PARTITA IVA
IT 9906 IT:VAT Ufficio responsabile gestione partite IVA
IT 9907 IT:CF TAX Authority
IT 9921 IT:IPA Indice delle Pubbliche Amministrazioni
JP 0188 JP:SST Corporate Number of The Social Security and Tax Number System
JP 0221 JP:IIN The registered number of the qualified invoice issuer
LI 9936 LI:VAT Liechtenstein VAT number
LT 0200 LT:LEC Legal entity code
LT 9937 LT:VAT Lithuania VAT number
LU 9938 LU:VAT Luxemburg VAT number
LV 0218 LV:URN Unified registration number
LV 9939 LV:VAT Latvia VAT number
MC 9940 MC:VAT Monaco VAT number
ME 9941 ME:VAT Montenegro VAT number
MK 9942 MK:VAT Macedonia, the former Yugoslav Republic of VAT number
MT 9943 MT:VAT Malta VAT number
MY 0230 MY:EIF National e-Invoicing Framework
NL 0106 NL:KVK Vereniging van Kamers van Koophandel en Fabrieken in Nederland (Association of
Chambers of Commerce and Industry in the Netherlands), Scheme
NL 0190 NL:OINO Organisatie-identificatienummer (OIN)
NL 9944 NL:VAT Netherlands VAT number
NL 9954 NL:OIN Dutch Originator's Identification Number
NO 0192 NO:ORG Organisasjonsnummer
NO 9908 NO:ORGNR Enhetsregisteret ved Bronnoysundregisterne
NO 9909 NO:VAT Norwegian VAT number
PL 9945 PL:VAT Poland VAT number
PT 9946 PT:VAT Portugal VAT number
RO 9947 RO:VAT Romania VAT number
RS 9948 RS:VAT Serbia VAT number
SE 0007 SE:ORGNR Organisationsnummer
SE 9955 SE:VAT Swedish VAT number
SG 0195 SG:UEN Singapore Nationwide E-Invoice Framework
SI 9949 SI:VAT Slovenia VAT number
SK 9950 SK:VAT Slovakia VAT number
SM 9951 SM:VAT San Marino VAT number
TR 9952 TR:VAT Turkey VAT number
US 9959 US:EIN US Employer ID Number
VA 9953 VA:VAT Holy See (Vatican City State) VAT number
international 0060 DUNS Data Universal Numbering System (D-U-N-S Number)
international 0088 GLN Global Location Number
international 0130 EU:NAL Directorates of the European Commission
international 0199 LEI Legal Entity Identifier (LEI)
international 0209 GS1 GS1 identification keys
international 9912 EU:VAT National ministries of Economy
international 9913 EU:REID Business Registers Network
international 9918 IBAN SOCIETY FOR WORLDWIDE INTERBANK FINANCIAL, TELECOMMUNICATION S.W.I.F.T

Countries like Andorra or Albania are simple and clear, the have only one identifier scheme. Italians are the most complex administration with not less than 10 schemes.

For outbound documents things are simple: we can decide which EAS to use, and we opt for using the VAT number. I used the code snippets of this page to write the commondata.peppolcodes module.

But what about inbound documents? When processing incoming documents we don’t want to do per document yet another Ibanity request in order to find the VAT id of the partner. Do we need another database model “Customer”?

Answer: No, because the name of the supplier is given in the invoice. As an end user, for your business partners who send sou invoices you should fill the peppol_id with the one they (your partner) use when sending invoices. Otherwise Lino will create a partner for you based on the name. If you then discover that you have a duplicate, you can merge them manually.

Wednesday, February 19, 2025

Wed, 19 Feb 2025 00:00:00 +0000

Wednesday, February 19, 2025¶

How to reinstall my Lino developer environment after dist-upgrade¶

I did “pip freeze > 20250218.txt” before the dist-upgrade, but then didn’t use it because I preferred to say:

go getlino
pip install -e .
getlino configure

I had a series of SyntaxWarning “invalid escape sequence” in a few files because my Python is now Python 12 (it was 3.10 before). There is a beautiful discussion on python.org about why this is good.

getlino should maybe call pip install -e .[test]

Tuesday, February 18, 2025

Tue, 18 Feb 2025 00:00:00 +0000

Tuesday, February 18, 2025¶

How to validate a Peppol XML file¶

crux (“Command-line Refuter of Unshapely XML”) is a command-line tool “for XML schema and Schematron validation for any platform with a Java runtime”.

$ sudo apt install default-jre
$ java -v
openjdk version "11.0.26" 2025-01-21
OpenJDK Runtime Environment (build 11.0.26+4-post-Ubuntu-1ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.26+4-post-Ubuntu-1ubuntu122.04, mixed mode, sharing)

I downloaded SaxonHE12-5J.zip to ~/Downloads/saxon from https://github.com/Saxonica/Saxon-HE/tree/main/12/Java

$ sudo apt install maven

https://www.saxonica.com/html/documentation10/using-xsl/commandline/index.html

$ pytest /home/luc/work/book/docs/apps/cosi/ibanity.rst

Friday, February 14, 2025

Fri, 14 Feb 2025 00:00:00 +0000

Friday, February 14, 2025¶

I continued to explore the Ibanity API. Thanks to Geoffrey who answered my questions on their support platform.

Q: The Ibanity API documentation has three very similar sections about outbound documents: Invoice, Credit Note and Document. Everything seems to be the same for these three sections, except for the value of type. Why is this differentiation?

A: The outbound-document resource exists only to describe the response of list-peppol-outbound-documents, which returns a list of all outbound documents, both invoices and credit notes mixed up. Note that you cannot POST via this resource; when posting a document you must specify either Invoice or CreditNote.

Q: What would happen if I would POST a credit note via the create-peppol-outbound-invoice call?

A: It would return “invalid format” because Ibanity looks at the root element of the UBL file, which is either <Invoice> or <CreditNote>, and it will not match that of the POST. In the sandbox you cannot see this because the sandbox doesn’t do such verification.

Thursday, February 13, 2025

Thu, 13 Feb 2025 00:00:00 +0000

Thursday, February 13, 2025¶

About CLI libraries¶

I read about Typer, which is an extension of Click, which we use at different places in Lino, most notably getlino. In atelier and picsel we still use argh, which is older than click.

GUI libraries for console terminals¶

I continue to dream about writing a command-line frontend for Lino. Right now it remains just a dream because it would be very experimental and nobody would ever pay money for it…

… but still it would be a great project for a volunteer contributor. Here are some libraries that we would consider to base our work upon.

Computing with durations¶

>>> from lino.utils.quantities import Duration
>>> def func(values):
...     values = values.split()
...     print(sum([Duration(x) for x in values]))

>>> func("-1:37 10:00 -6:31")
1:52

>>> func("-4:42 10:00 -0:11")
5:07

Tuesday, February 11, 2025

Tue, 11 Feb 2025 00:00:00 +0000

Tuesday, February 11, 2025¶

A user story about writing comments¶

A user writes a comment about a minor issue under a sticky ticket because he doesn’t estimate necessary to create a whole ticket for such a little issue. But the issue gets a series of replies and other comments. It turns out that it wasn’t such a minor thing. At this moment we can decide to create an independent ticket for it and to move the series of comments from the sticky ticket to the new one.

This story just happened on Jane where I created #5921 (GL reports “Failed to sign in as robin” for noi1r and tera1) in order to regroup a series of three comments that until then had been under #881 (Review test suites after a series of changes). How to move a comment from one ticket to another: by simply modifying the owner_key field. I did this for comments 15564, 15566 and 15567. Yes, it is possible to “change the topic” within a thread of comments. I think that this feature makes sense.

(“Owner” and “Topic” are synonym in the case of comments: we do dd.update_field(Comment, "owner", verbose_name=_("Topic")).)

Note that comment 15800 still has 881 as owner because it is just a reply to one of the tickets that were moved. I could have moved it as well, but I didn’t care because the default CommentsByRFC table shows only top-level comments.

It would be easy to extend comments.CommentChecker so that it would correct this by setting the owner of a comment to the owner of its parent. But do we really want this? At least in the current situation it doesn’t disturb, and it leaves a “memory” of the fact that this discussion had originally started on #881, 2 weeks before #5921 was created.

Code review for #5910¶

I tried to review Sharif’s commit 211cd2c09, which fixes #5910 (Detail link shows the wrong ticket in Jane). To review a change means to say something useful about it. Here is what I’ve been thinking. Though I’m not sure whether it is useful…

Here is the code after the change:

interceptBrowserBF(event) {
    const [pathname, search] = document.URL.split('#')[1].split('?'),
        params = this.ex.queryString.parse(search),
        { URLContext } = this;
    if (pathname === URLContext.value.path) return;
    if (URLContext.filled(params.rs)) {
        if (URLContext.history.has(params.rs))
            URLContext.history.load({rs: params.rs, lazy: true})
        else URLContext.history.pushPath({
            pathname: pathname, params: params, lazy: true});
    } else {
        params.rs = URLContext.newSlug();
        this.navigate(URLContext.makePath({path: pathname, ...params}),
            {replace: true})
        URLContext.history.pushPath({
            pathname: pathname, params: params, lazy: true});
    }
}

Here is what I learned from it:

The App.interceptBrowserBF() method is our handler of the window.popstate.
The this.ex.queryString refers to the query-string module. The this.ex tells me that it has been loaded dynamically using the ImportPoolRegistry. As expected it parses the URL string into a JS object.
I wonder whether there is a difference between saying const { URLContext } = this; and const URLContext = this.URLContext;. It’s one of these funny Javascript magics…
Anyway, what is URLContext? It is used so often in the Lino React frontend but still a quite unknown beast to me. It has been initialized during App.prepare() using the following expression:
```
this.URLContext = new this.ex.nc.Context
```
Where nc refers to the NavigationControl module.

My summary so far: our interceptBrowserBF() method does some magic to avoid loading a new page when it’s not necessary. It searches the browser history for the current “random string” (the string stored in the rs URL parameter) and jumps back when possible. And the bug was that this mechanism didn’t test for the case of an empty rs. I have the feeling that this mechanism is quite magical and fragile. But my feeling isn’t very founded.

Monday, February 10, 2025

Mon, 10 Feb 2025 00:00:00 +0000

Monday, February 10, 2025¶

More bugs in LeafComponentInput.focus()¶

I stumbled into #5919 (Cannot change the ticket number of a session in WorkedHours). This is obviously yet another side effect of #5893 (The grid cell editor should start by selecting all its content).

We can see it on Jane. Click on the Ticket cell of some existing session. More precisely it starts editing, correctly selects the whole content of the field, but then it never lets me enter more than one character because after every character it selects the whole thing again. This is the same problem as I saw for password fields.

I had a look at the two changes.

202505202 : Fixed #5893 (The grid cell editor should start by selecting all its content) with commit 3f6d577d (“optimize for: #5893 (The grid cell editor should start by selecting all its content)”)
20250207 : Fixed #5915 (Cannot manually enter a password) with commit 4c7a1024 (“allow input.select only in grid cell(s)”)

Here is the LeafComponentInput.focus() method before and after these changes.

Before:

focus() {
  let ref = this.inputEl;
  if (ref.focus) ref.focus()
  else if (ref.focusInput) ref.focusInput.focus()
  else if (ref.inputRef) ref.inputRef.current.focus()
}

First change (Fixed #5893 (The grid cell editor should start by selecting all its content)):

focus() {
  let ref = this.inputEl, input;
  if (ref.focusInput) ref = ref.focusInput
  else if (ref.inputRef) ref = ref.inputRef.current;
  if (ref.focus) {
      ref.focus()
      if (ref.hasOwnProperty('select')) {ref.select(); return}
      if (this.container) input = this.container.getElementsByTagName('input')[0];
      if (input) input.select();
}

Second change (Fixed #5915 (Cannot manually enter a password)):

focus() {
  let ref = this.inputEl, input;
  if (ref.focusInput) ref = ref.focusInput
  else if (ref.inputRef) ref = ref.inputRef.current;
  if (ref.focus) {
      ref.focus()
      if (this.props[constants.URL_PARAM_WINDOW_TYPE] === constants.WINDOW_TYPE_TABLE) {
          if (ref.hasOwnProperty('select')) {ref.select(); return}
          if (this.container) input = this.container.getElementsByTagName('input')[0];
          if (input) input.select();
      }
  }
}

LeafComponentInput is the base class for all input fields. It is defined in LinoComponentUtils.jsx where it inherits from LeafComponentBase and has itself a subclass LeafComponentInputChoices. It is subclassed by all input fields, including AutoComplete, DTFieldElement, TextFieldElement, ChoiceListFieldElement, ChoicesFieldElement, etc.

OMG! What are the meanings of the attributes inputEl, focusInput and inputRef? I started to read How to set focus on an input field after rendering in React. Trying to understand these things would require me several man days! I am glad to have Sharif!

But if you ask me, the second change, making it depend on the window type, is bullshit. I think you can safely undo this change. It just disables the new behaviour when we aren’t in a grid. This condition is unrelated. We want Lino to select the content of a field, including password fields, each time the field gets focus, also in an action dialog, also in a detail form. One exception: we do not want this in a TextField field.

The problem after the first change was that it selects the whole thing again after each keyboard stroke. It seems that the focus event is called again when selection changes.

But here I stop. Sharif, I hope that my thoughts helped you to continue.

P.S. : Sharifs answer is here: Commit f979de77.

About plugin dependency¶

For #5916 I had to (wanted to) put a text “Screenshot : [file 12]” into the description of some tickets.

These demo tickets get generated by the demo fixture of the noi plugin.

Adding references to screenshots in the description of tickets requires these tickets to exist already.

The demo screenshots get created in the demo fixtures of uploads.

So I said that the noi demo should load after the uploads demo.

For this I changed the lino_noi.lib.tickets and lino_noi.lib.cal plugins to no longer declare lino_noi.lib.noi as needed.

I commented out the needs_plugins = [‘lino_noi.lib.noi’] in lino_noi.lib.cal:

from lino_xl.lib.cal import Plugin
class Pugin(Plugin)
    # needs_plugins = ['lino_noi.lib.noi']
    ...

This had the unwanted side effect of reactivating the needs_plugins of lino_xl.lib.cal (but I understood this only later).

I think that my fundamental mistake was to increase the complexity of the noi demo fixture.

So I now load the screenshots in a new demo2 fixture of noi.

En passant I changed the xl cal plugin to no longer need checkdata and linod.

Saturday, February 8, 2025

Sat, 08 Feb 2025 00:00:00 +0000

Saturday, February 8, 2025¶

>>> from bs4 import BeautifulSoup
>>> from pprint import pprint
>>> html = '''<p>The <a href="foo"><b>quick</b></a>, brown fox jumps over a lazy dog.</p>'''
>>> # soup = BeautifulSoup(html, "html.parser")
>>> soup = BeautifulSoup(html, features="lxml")
>>> p = soup.find('p')
>>> print(p.string)
None
>>> list(p.children)
['The ', <a href="foo"><b>quick</b></a>, ', brown fox jumps over a lazy dog.']
>>> pprint([c.__class__ for c in p.children])
[<class 'bs4.element.NavigableString'>,
 <class 'bs4.element.Tag'>,
 <class 'bs4.element.NavigableString'>]
>>> a = p.find('a')
>>> b = a.find('b')
>>> assert b == p.find('b')

>>> list(a.children)
[<b>quick</b>]
>>> list(b.children)
['quick']

>>> a.string.__class__
<class 'bs4.element.NavigableString'>
>>> b.string.__class__
<class 'bs4.element.NavigableString'>

>>> a.string
'quick'
>>> b.string
'quick'

>>> a.string.string
'quick'
>>> a.string.string.string.string
'quick'

>>> a
<a href="foo"><b>quick</b></a>
>>> b
<b>quick</b>

>>> old = a.string.replace_with("foo")
>>> a
<a href="foo"><b>foo</b></a>

>>> old = a.replace_with("bar")
>>> a
<a href="foo"><b>foo</b></a>

>>> b.string.string.replace_with("bar")
'foo'
>>> a
<a href="foo"><b>bar</b></a>
>>> b
<b>bar</b>

>>> html = '<p>Before <a href="foo"><b>bar</b></a> and after.</p>'
>>> soup = BeautifulSoup(html, features="lxml")
>>> p = soup.find('p')
>>> list(p.descendants)
['Before ', <a href="foo"><b>bar</b></a>, <b>bar</b>, 'bar', ' and after.']
>>> print(p.string)
None
>>> print(list(p))
['Before ', <a href="foo"><b>bar</b></a>, ' and after.']
>>> a = soup.find('a')
>>> print(a.string)
bar
>>> list(a.children)
[<b>bar</b>]

>>> html = '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>'
>>> soup = BeautifulSoup(html, features="lxml")
>>> p = soup.find('p')
>>> s = p.string
>>> print(s)
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
>>> unused = p.string.replace_with("foo")
>>> print(p)
<p>foo</p>

>>> html = "<p>A <b>bold</b></p><p>\n\n and <i>italic</i> thing.</p>"
>>> soup = BeautifulSoup(html, features="lxml")
>>> p = soup.find('p')
>>> list(p.children)
['A ', <b>bold</b>]

>>> html = """<p>foo <html><head><base href="bar" target="_blank"></head><body></p><p>baz</p>"""
>>> soup = BeautifulSoup(html, features="lxml")
>>> for c in soup.children:
...     print(c)
<html><body><p>foo </p><base href="bar" target="_blank"/><p>baz</p></body></html>

Lino sites with multiple “systems”

Sun, 02 Feb 2025 00:00:00 +0000

Lino sites with multiple “systems”¶

Sunday, February 2, 2025.

Here is a set of unfinished ideas about #5904 (Lino sites with multiple “systems”), which is the result of my medittions on #5901 and #5902. This blog entry is meant to be integrated into the Developer Guide when the time has come. There is no need to hurry.

The lino_xl.lib.ledgers plugin expresses something important, but the name “ledgers” is misleading. It’s about more than accounting. And it is even more than a plugin, it is a new core feature, causing changes in many places of the Lino core. This is not a job for a simple plugin. The lino_xl.lib.ledgers plugin will be replaced by a new site attribute. The Lino world will get a new fundamental concept: we knew already the difference between a Lino server and a Lino site, but now we will also talk about Lino systems.

Lino system¶: A subset of a Lino site that is accessible only to end users who are part of this system. Each system can be subject to an individual hosting agreement with a customer.

With this new feature, a hosting provider can run a single Lino site and share it among multiple customers. Which is potentially much more lucrative.

Until now all Lino sites are single-system. They have one and only one instance of the lino.modlib.system.SiteConfig model. We will rename this model to system.System and introduce a new site attribute multisystem.

multisystem¶

Whether this Lino site hosts multiple Lino systems. You activate it in your settings.py file:

class Site(Site):
    multisystem = True

Or should it rather be plugin setting? In this case the settings.py would look like this:

def get_plugin_configs(self):
    yield super().get_plugin_configs()
    yield ('system', 'multisystem', True)

Whether site attribute or plugin setting, the idea is that on a multisystem site there can me more than one system.System instance.

On a multisystem site, most database models get an additional field named system, which is a non-nullable foreign key to system.System. So every database row belongs to a given system. And this field is invisible to normal users. It is automatically set when the user creates a new database row, and Lino automatically guarantees to never show to a normal user any database rows belonging to an alien system.

Normal users are not aware that a Lino site hosts multiple systems.

We will have a new user role SystemAware. A system-aware user can see and edit this special field system. A multisystem Lino site should define a dedicated user type who does nothing but managing systems. The system manager of a Lino site is responsible for

creating new systems and initializing them with demo data or default data from fixtures or from from templates
creating at least one user for every system
deactivating or deleting systems that are no longer used

The lino.modlib.users.Users table will get the following required roles:

required_roles = {(SiteAdmin, SystemAware)}

A tuple in required_roles means that one OR the other role is required.

The system.Systems table will differ depending on whether multisite is True or False. This might get a bit magical.

I said that most database models get an additional field named system because at least contenttypes.ContentType and sessions.Session do not need this field.

We will probably say known_values.update(system=self.user.system) in Request.__init__() or Request.setup().

We will have to fight quite some dragons.

Once more: there is no need to hurry. Even the term “system” might change. Try for example to re-read this post and replace “system” by “community”.

Playing with Ibanity (continued)

Thu, 23 Jan 2025 00:00:00 +0000

Playing with Ibanity (continued)¶

Thursday, January 23, 2025

Geoffrey helped me to find the mistakes in my yesterday script: content type should be application/vnd.api+json, there was a mistake in my URL, and the payload should get specified using the json keyword argument instead of data (I didn’t know this argument, so I looked up the API docs).

Here, finally, is the working script:

# See https://luc.lino-framework.org/blog/2025/0123.html
from pathlib import Path
import requests
import base64
import sys
import json
import uuid
from pprint import pprint

# Define client ID and secret
client_id = "69d85961-7d68-474d-9ac2-426fdc71bab8"
client_secret = "valid_client_secret"

# Define certificate and key file paths
cert_dir = Path(f"~/Documents/ibanity/application_{client_id}/").expanduser()
cert_file = cert_dir / "certificate.pem"
key_file = cert_dir / "decrypted_private_key.pem"

# Validate if the key and certificate files exist
if not cert_file.exists():
    raise Exception(f"Error: Certificate file not found at {cert_file}")

if not key_file.exists():
    raise Exception(f"Error: Private key file not found at {key_file}")

# Base64 encode client_id and client_secret for Basic Auth
client_credentials = f"{client_id}:{client_secret}"
encoded_credentials = base64.b64encode(client_credentials.encode()).decode()

# Create an HTTPS session
s = requests.Session()

# Attach client certificate and key to the session
s.cert = (cert_file, key_file)

# Add the Authorization header
headers = {
    "Authorization": f"Basic {encoded_credentials}",
    "Content-Type": "application/x-www-form-urlencoded",  # Required for OAuth2 requests
}

url = "https://api.ibanity.com/einvoicing/oauth2/token"
data = {"grant_type": "client_credentials"}
response = s.post(url, data=data, headers=headers)
if response.status_code != 200:
    raise Exception(f"Unexpected response status code: {response.status_code}\n{response.text}")
rv = json.loads(response.text)
access_token = rv['access_token']

# Customer search. Check whether my customer exists.
# Belgian participants are registered with the Belgian company number, for which
# identifier 0208 can be used. Optionally, the customer can be registered with
# their VAT number, for which identifier 9925 can be used.

print("Customer search")
url = "https://api.ibanity.com/einvoicing/peppol/customer-searches"
eas = "9925"
vat_id = "BE0010012671"
# The Flowin sandbox contains hard-coded fake data.  Using another reference as
# customerReference will in result a 404

headers = {
    "Accept": "application/vnd.api+json",
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/vnd.api+json"
}
# pprint(headers)
data = {
    "type": "peppolCustomerSearch",
    # "id": str(uuid.uuid4()),
    "attributes": {
        "customerReference": f"{eas}:{vat_id}",
        # "supportedDocumentFormats": doc_formats
    }
}
data = {"data": data}
# pprint(data)
response = s.post(url, headers=headers, json=data)
if response.status_code != 200:
    msg = f"POST {url} returned {response.status_code}\n{response.text}\n"
    raise Exception(msg)
rv = json.loads(response.text)
pprint(rv)

And here is the output:

$ python docs/blog/2025/0123.py
Customer search
{'data': {'attributes': {'customerReference': '9925:BE0010012671',
                         'supportedDocumentFormats': [{'customizationId': 'urn:cen.eu:en16931:2017#compliant#urn:fdc:peppol.eu:2017:poacc:billing:3.0',
                                                       'localName': 'Invoice',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:Invoice-2',
                                                       'ublVersionId': '2.1'},
                                                      {'customizationId': 'urn:cen.eu:en16931:2017#conformant#urn:UBL.BE:1.0.0.20180214',
                                                       'localName': 'Invoice',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:Invoice-2',
                                                       'ublVersionId': '2.1'},
                                                      {'customizationId': 'urn:cen.eu:en16931:2017#conformant#urn:UBL.BE:1.0.0.20180214',
                                                       'localName': 'CreditNote',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2',
                                                       'ublVersionId': '2.1'},
                                                      {'customizationId': 'urn:cen.eu:en16931:2017#compliant#urn:fdc:peppol.eu:2017:poacc:billing:3.0',
                                                       'localName': 'CreditNote',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2',
                                                       'ublVersionId': '2.1'},
                                                      {'customizationId': 'urn:cen.eu:en16931:2017#compliant#urn:fdc:nen.nl:nlcius:v1.0',
                                                       'localName': 'Invoice',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:Invoice-2',
                                                       'ublVersionId': '2.1'},
                                                      {'customizationId': 'urn:cen.eu:en16931:2017#compliant#urn:fdc:nen.nl:nlcius:v1.0',
                                                       'localName': 'CreditNote',
                                                       'profileId': 'urn:fdc:peppol.eu:2017:poacc:billing:01:1.0',
                                                       'rootNamespace': 'urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2',
                                                       'ublVersionId': '2.1'}]},
          'id': '99988e77-cc4a-4583-bd4a-288095c5566e',
          'type': 'peppolCustomerSearch'}}

Okay so far. Now let’s go on.

Playing with Ibanity (continued)

Wed, 22 Jan 2025 00:00:00 +0000

Playing with Ibanity (continued)¶

Wednesday, January 22, 2025

Yesterday Geoffrey helped me to get a bit more of information:

{"errors":[{"code":"invalidPayload","detail":"The submitted payload could not be parsed.",
"meta":{"requestId":"00015ABF903CA9960A01063301BB678FE5E7E08D82DF"}}]}

His explanation to this error message is “Well, that we couldn’t parse the payload. an issue with the syntax. Sadly we can’t see what, because we couldn’t parse it.”

But what can be the problem? A blatant syntax problem is unlikely since I specify the payload as a dictionary and the Python requests module converts this to a stream. I have only the Ibanity API documentation, and I cannot see anything that helps me.

I reduced my script to focus on my problem:

# See https://luc.lino-framework.org/blog/2025/0122.html
from pathlib import Path
import requests
import base64
import sys
import json
import uuid
from pprint import pprint

# Define client ID and secret
client_id = "69d85961-7d68-474d-9ac2-426fdc71bab8"
client_secret = "valid_client_secret"

# Define certificate and key file paths
cert_dir = Path(f"~/Documents/ibanity/application_{client_id}/").expanduser()
cert_file = cert_dir / "certificate.pem"
key_file = cert_dir / "decrypted_private_key.pem"

# Validate if the key and certificate files exist
if not cert_file.exists():
    raise Exception(f"Error: Certificate file not found at {cert_file}")

if not key_file.exists():
    raise Exception(f"Error: Private key file not found at {key_file}")

# Base64 encode client_id and client_secret for Basic Auth
client_credentials = f"{client_id}:{client_secret}"
encoded_credentials = base64.b64encode(client_credentials.encode()).decode()

# Create an HTTPS session
s = requests.Session()

# Attach client certificate and key to the session
s.cert = (cert_file, key_file)

# Add the Authorization header
headers = {
    "Authorization": f"Basic {encoded_credentials}",
    "Content-Type": "application/x-www-form-urlencoded",  # Required for OAuth2 requests
}

url = "https://api.ibanity.com/einvoicing/oauth2/token"
data = {"grant_type": "client_credentials"}
response = s.post(url, data=data, headers=headers)
if response.status_code != 200:
    raise Exception(f"Unexpected response status code: {response.status_code}\n{response.text}")
rv = json.loads(response.text)
access_token = rv['access_token']

# Customer search. Check whether my customer exists.
# Belgian participants are registered with the Belgian company number, for which
# identifier 0208 can be used. Optionally, the customer can be registered with
# their VAT number, for which identifier 9925 can be used.

print("Customer search")
url = "https://api.ibanity.com/einvoicing/customer-searches"
eas = "9925"
vat_id = "BE0010012671"
# vat_id = "BE0840559537"
headers = {
    "Accept": "application/vnd.api+json",
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/x-www-form-urlencoded",  # Required for OAuth2 requests
    # "Content-Type": "application/vnd.api+json"
}
# pprint(headers)

doc_formats = [
{
    "customizationId": "urn:cen.eu:en16931:2017#compliant#urn:fdc:peppol.eu:2017:poacc:billing:3.0",
    "localName": "Invoice",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:Invoice-2",
    "ublVersionId": "2.1"
    },
    {
    "customizationId": "urn:cen.eu:en16931:2017#conformant#urn:UBL.BE:1.0.0.20180214",
    "localName": "Invoice",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:Invoice-2",
    "ublVersionId": "2.1"
    },
    {
    "customizationId": "urn:cen.eu:en16931:2017#conformant#urn:UBL.BE:1.0.0.20180214",
    "localName": "CreditNote",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2",
    "ublVersionId": "2.1"
    },
    {
    "customizationId": "urn:cen.eu:en16931:2017#compliant#urn:fdc:peppol.eu:2017:poacc:billing:3.0",
    "localName": "CreditNote",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2",
    "ublVersionId": "2.1"
    },
    {
    "customizationId": "urn:cen.eu:en16931:2017#compliant#urn:fdc:nen.nl:nlcius:v1.0",
    "localName": "Invoice",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:Invoice-2",
    "ublVersionId": "2.1"
    },
    {
    "customizationId": "urn:cen.eu:en16931:2017#compliant#urn:fdc:nen.nl:nlcius:v1.0",
    "localName": "CreditNote",
    "profileId": "urn:fdc:peppol.eu:2017:poacc:billing:01:1.0",
    "rootNamespace": "urn:oasis:names:specification:ubl:schema:xsd:CreditNote-2",
    "ublVersionId": "2.1"
    }
]

data = {
    "type": "peppolCustomerSearch",
    # "id": str(uuid.uuid4()),
    "attributes": {
        "customerReference": f"{eas}:{vat_id}",
        # "supportedDocumentFormats": doc_formats
    }
}
# data = {"data": data}
pprint(data)
response = s.post(url, headers=headers, data=data)
if response.status_code != 200:
    msg = f"POST {url} returned {response.status_code}\n{response.text}\n"
    raise Exception(msg)
rv = json.loads(response.text)
pprint(rv)

Out of intuition I tried to change the “Content-Type” header from “application/vnd.api+json” to “application/x-www-form-urlencoded”.

And with the new Content-Type I get a new result:

Exception: POST https://api.ibanity.com/einvoicing/customer-searches returned 404
{"errors":[{"code":"resourceNotFound","detail":"The route you are trying to reach was not found."}]}

I tried all kind of variants of structuring the payload, and I added more information copied from the API docs example (supportedDocumentFormats and id), but the response is always the same.

I don’t even know whether this result is better than yesterday or not :-/

The story goes on here: Playing with Ibanity (continued).

Playing with Ibanity (continued)

Sat, 18 Jan 2025 00:00:00 +0000

Playing with Ibanity (continued)¶

Saturday, January 18, 2025

I continued to play with Ibanity, based on what I learned some days ago. I successfully requested an access token, but how can I use this to send an outbound invoice? I do have an XML file generated by TIM or Lino as described in Lino and eInvoicing (PEPPOL).

Here is the script I used today:

from pathlib import Path
import requests
import base64
import sys
import json
from pprint import pprint

# Set EXAMPLE to 1 or 2, see https://luc.lino-framework.org/blog/2025/0118.html
EXAMPLE = 2

# Define client ID and secret
client_id = "69d85961-7d68-474d-9ac2-426fdc71bab8"
client_secret = "valid_client_secret"

# Define certificate and key file paths
cert_dir = Path(f"~/Documents/ibanity/application_{client_id}/").expanduser()
cert_file = cert_dir / "certificate.pem"
key_file = cert_dir / "decrypted_private_key.pem"

# Validate if the key and certificate files exist
if not cert_file.exists():
    raise Exception(f"Error: Certificate file not found at {cert_file}")

if not key_file.exists():
    raise Exception(f"Error: Private key file not found at {key_file}")

# Base64 encode client_id and client_secret for Basic Auth
client_credentials = f"{client_id}:{client_secret}"
encoded_credentials = base64.b64encode(client_credentials.encode()).decode()

# Create an HTTPS session
s = requests.Session()

# Attach client certificate and key to the session
s.cert = (cert_file, key_file)

# Add the Authorization header
headers = {
    "Authorization": f"Basic {encoded_credentials}",
    "Content-Type": "application/x-www-form-urlencoded",  # Required for OAuth2 requests
}

url = "https://api.ibanity.com/einvoicing/oauth2/token"
data = {"grant_type": "client_credentials"}
try:
    response = s.post(url, data=data, headers=headers)
except requests.exceptions.SSLError as e:
    raise Exception(f"SSL Error: {e}. Please ensure your certificate and private key are correct and properly configured.")
except requests.exceptions.RequestException as e:
    raise Exception(f"Request Error: {e}")
if response.status_code != 200:
    raise Exception(f"Unexpected response status code: {response.status_code}")
rv = json.loads(response.text)
access_token = rv['access_token']

# Same headers dict used in all examples
headers = {
    "Accept": "application/vnd.api+json",
    "Authorization": f"Bearer {access_token}"
}

# I don't know whether the access token allows multiple requests, so I skip
# this one for example 2.
if EXAMPLE == 1:
    print("Get a list of suppliers")
    url = "https://api.ibanity.com/einvoicing/suppliers"
    response = s.get(url, headers=headers)
    if response.status_code != 200:
        raise Exception(f"Unexpected status code {response.status_code} for GET {url}")
    rv = json.loads(response.text)
    pprint(rv)
    sys.exit()

# Customer search. Check whether my customer exists.
# Belgian participants are registered with the Belgian company number, for which
# identifier 0208 can be used. Optionally, the customer can be registered with
# their VAT number, for which identifier 9925 can be used.

if EXAMPLE == 2:
    print("Customer search")
    url = "https://api.ibanity.com/einvoicing/customer-searches"
    eas = "9925"
    vat_id = "BE0010012671"
    # vat_id = "BE0840559537"
    # headers are the same as before, plus a new header "Content-Type"
    headers["Content-Type"] = "application/vnd.api+json"
    # pprint(headers)
    data = {
        "type": "peppolCustomerSearch",
        "attributes": {
          "customerReference": f"{eas}:{vat_id}"
        }
    }
    # data = {"data": data}
    pprint(data)
    response = s.post(url, headers=headers, data=data)
    if response.status_code != 200:
        print(response.text) # line added 20250121
        raise Exception(f"Unexpected status code {response.status_code} for POST {url}")
    rv = json.loads(response.text)
    pprint(rv)

When EXAMPLE is 1 in this script, it asks for a list of suppliers. And this works. Here is the output

{'data': [{'attributes': {'city': 'Leuven',
                          'companyNumber': '1234567890',
                          'contactEmail': 'contact@example.be',
                          'country': 'Belgium',
                          'createdAt': '2025-01-18T15:10:56.719813Z',
                          'email': 'someone@example.be',
                          'enterpriseIdentification': {'enterpriseNumber': '1234567890',
                                                       'vatNumber': 'BE1234567890'},
                          'homepage': 'www.home.com',
                          'ibans': [{'id': 'bdfa52c6-2b50-4690-8b8d-24541a92c578',
                                     'value': 'BE68539007547034'},
                                    {'id': 'dcb9f7c2-be2c-4b52-8d77-3ed2bc05c5f8',
                                     'value': 'BE68539007547034'}],
                          'names': [{'id': '3dffba33-97af-4477-9fab-2d9d2dc31cee',
                                     'value': 'Company'},
                                    {'id': '99e410cc-d6f0-4f36-8096-949741ea8ec3',
                                     'value': 'Company S.A.'}],
                          'onboardingStatus': 'CREATED',
                          'peppolReceiver': False,
                          'phoneNumber': '+3254321121',
                          'street': 'Street',
                          'streetNumber': '2',
                          'supportEmail': 'support@example.be',
                          'supportPhone': '+3212345121',
                          'supportUrl': 'www.support.com',
                          'zip': '3000'},
           'id': '273c1bdf-6258-4484-b6fb-74363721d51f',
           'type': 'supplier'}],
 'meta': {'paging': {'number': 0, 'size': 2000, 'total': 1}}}

But until further notice I hope that TIM doesn’t need to care about suppliers because our users will use their online banking application to see their purchase invoices, and they will simply enter them manually into TIM as they are used to do right now.

What TIM users will need to do is a check whether their customer exists. That’s documented in Peppol Customer Search.

Today’s script does exactly this when EXAMPLE is 2. But until now I always get 400 (Bad Request):

Customer search
{'attributes': {'customerReference': '9925:BE0010012671'},
 'type': 'peppolCustomerSearch'}
Traceback (most recent call last):
  File ".../0118.py", line 93, in <module>
    raise Exception(f"Unexpected status code {response.status_code} for POST {url}")
Exception: Unexpected status code 400 for POST https://api.ibanity.com/einvoicing/customer-searches

What am I missing?

20250121: Geoffrey helped me to get a bit more of information. I forgot to look at response.text when response.status_code isn’t 200. Here is the missing bit:

{"errors":[{"code":"invalidPayload","detail":"The submitted payload could not be parsed.",
"meta":{"requestId":"00015ABF903CA9960A01063301BB678FE5E7E08D82DF"}}]}

Lino Prima next sprint

Wed, 15 Jan 2025 00:00:00 +0000

Lino Prima next sprint¶

Wednesday, January 15, 2025

In lino_prima.projects.prima1 demo project. Signed in as madeleine.carbonez.

TALK:

Perioden werden als Nummer angezeigt: das kam wahrscheinlich, weil site data nicht aktuell war. Mit Shift+Ctrl+F5 geht es weg.
Grüne Bausteintitel unlesbar: scheint browserabhängig zu sein, bei mir ist er deutlich lesbar weiß auf grün. Gemeinsam: Andere Farben ausprobieren. Hat Edge eine Inspect-Funktion bzw. Entwickler-Modus (Shift+Ctrl+I im Firefox), damit ich sehen kann was da falsch läuft.
Wo ist das Zeugnis? Gemeinsam testen.
Piktogramme pro Fach.

DONE:

Unsinniger Text “Bausteine als Bausteine” in Schule ‣ Meine Lehrerrollen
Übersichtstabelle Bausteine pro Klasse: keine Fortschrittskästchen und keine Prozentzahl anzeigen, lediglich ein Symbol “⚒” und orange Farbe, um anzuzeigen, dass der Baistein gestartet ist.
Lino wusste bisher nicht, dass es in jedem Schuljahr einen Kurs “Religion” geben muss. Für Religion in der 6B wurden in der App noch keine Tests erstellt. Der neue GroupChecker prüft das jetzt nach und erstellt fehlende Kurse. Für jedes Fach, das “Erweitert” angekreuzt hat und für das eine Sektion in der Zeugnisvorlag existiert, muss ein Kurs existieren.
Lino zeigte unsinnigerweise alle drei Möglichkeiten Smiley/Prädikat/Bewertung an.

TODO:

MyGroups occurs twice in dashboard

Playing with Ibanity (continued)

Tue, 14 Jan 2025 00:00:00 +0000

Playing with Ibanity (continued)¶

Tuesday, January 14, 2025

I asked in the Ibanity support forum what’s wrong with my yesterday’s script and Pedro gave me a more promising version:

from pathlib import Path
import requests
import base64
import sys
import json
from pprint import pprint

# Define client ID and secret
client_id = "69d85961-7d68-474d-9ac2-426fdc71bab8"
client_secret = "valid_client_secret"

# Define certificate and key file paths
cert_dir = Path(f"~/Documents/ibanity/application_{client_id}/").expanduser()
cert_file = cert_dir / "certificate.pem"
key_file = cert_dir / "decrypted_private_key.pem"

# Validate if the key and certificate files exist
if not cert_file.exists():
    print(f"Error: Certificate file not found at {cert_file}")
    sys.exit(1)

if not key_file.exists():
    print(f"Error: Private key file not found at {key_file}")
    sys.exit(1)

# Base64 encode client_id and client_secret for Basic Auth
client_credentials = f"{client_id}:{client_secret}"
encoded_credentials = base64.b64encode(client_credentials.encode()).decode()

# Create an HTTPS session
s = requests.Session()

try:
    # Attach client certificate and key to the session
    s.cert = (cert_file, key_file)

    # Add the Authorization header
    headers = {
        "Authorization": f"Basic {encoded_credentials}",
        "Content-Type": "application/x-www-form-urlencoded",  # Required for OAuth2 requests
    }

    # Make the POST request
    response = s.post(
        "https://api.ibanity.com/einvoicing/oauth2/token",
        data={"grant_type": "client_credentials"},
        headers=headers
    )

    # Handle response
    print(f"Response status: {response.status_code}")
    # print(f"Response body: {response.text}")
    rv = json.loads(response.text)
    print("Response text:")
    pprint(rv)

except requests.exceptions.SSLError as ssl_error:
    print(f"SSL Error: {ssl_error}")
    print("Please ensure your certificate and private key are correct and properly configured.")

except requests.exceptions.RequestException as req_error:
    print(f"Request Error: {req_error}")

Great! This version works:

Response status: 200
Response text:
{'access_token': 'access_token_1737210675',
 'expires_in': 300,
 'not-before-policy': 0,
 'refresh_expires_in': 300,
 'refresh_token': 'refresh_token_1737210675',
 'scope': 'email profile',
 'session_state': '94bdf4ce-0dea-4ab7-abda-a53c7ba4bc87',
 'token_type': 'bearer'}

TIL that providing my credentials is more complex than just giving a certificate and a private key, it also requires Basic access authentication.

I continue to read Flowin e-invoicing Services.

Some sentences in this document make me believe that maybe our customers can access the Peppol network via their online banking application: “All the customer has to do is register to Zoomit via his preferred online banking application.” So there is some hope that our customers can simply instruct their bank to accept incoming purchase invoices, they would then continue to enter them manually into TIM as they are used to do. Making this step automatically would involve considerable changes in configuration, we don’t want this right now.

TIM can already generate an invoice or credit note into Peppol XML format. The next steps TIM should learn to do:

Customer Search – check if a customer is registered on the Peppol network (It seems that the sender is responsible for verifying this, which makes sense)
Send document - sending the invoice or credit note in XML format

The story continues here: Playing with Ibanity (continued).

Playing with Ibanity

Mon, 13 Jan 2025 00:00:00 +0000

Playing with Ibanity¶

Monday, January 13, 2025

We are not going to become a PEPPOL Access Point, but we can become partner with Ibanity. We pay them a yearly fee of 2400€ and for this we can give access to up to 200 customers. It’s not cheap but it’s a reasonable price.

Getting started¶

I opened an account on their developer portal. This is where we will create credentials for our customers.
They do not provide a web interface like Hermes where our customers would themselves manage their invoices. We must implement this interface in TIM. The TIM of my customers will connect to Ibanity using the credentials they get from us, and TIM must handle all events of the PEPPOL network: receiving invoices, sending invoices, verifying whether a business partner exists, …
They provide a sandbox application, which I can use to write and test how our software will access their API.
API Reference
They have a Python Library, which “provides wrappers around the Ibanity API.”

I must now extend our accounting software TIM so that it uses the Ibanity API to access the Peppol network. I will create credentials for my customers in the Ibanity web interface. The TIM of my customers will connect to Ibanity using the credentials they get from me, and TIM must handle the events of the PEPPOL network: receiving invoices, sending invoices, verifying whether a business partner exists, …

I see that Ibanity had a Python library for accessing the API. But I also see that the last commit in this library is 7 years old, and this library is not published on PyPI, and there is no Python-specific documentation. I also know that I can make raw API calls from within Python. So here is my first question: Is this Python library actually being used by somebody? Is it a recommended way to use the Ibanity API?

My first script¶

Here is the first script I used for playing with Ibanity:

from pathlib import Path
import requests

client_id = "69d85961-7d68-474d-9ac2-426fdc71bab8"
pth = Path(f"~/.ssh/ibanity/application_{client_id}/")
pth = pth.expanduser()

cert_file = pth / "certificate.pem"
key_file = pth / "decrypted.private_key.pem"

s = requests.Session()
# s.cert = (key_file, cert_file)
s.cert = (cert_file, key_file)

r = s.get("https://api.ibanity.com/einvoicing/suppliers?")
print(r)

I stored the client cert files in a directory below my ~/.ssh.

I read https://requests.readthedocs.io/en/latest/user/advanced/#client-side-certificates

I had to decrypt the private key because Python’s requests module doesn’t support encrypted keys:

$ cd ~/.ssh/ibanity
$ openssl rsa -in private_key.pem -out decrypted_private_key.pem

Now I am getting:

ssl.SSLError: [SSL] PEM lib (_ssl.c:3900)

Maybe I specified key_file and cert_file in the wrong order. After inverting the order I get at least a response:

<Response [401]>

And 401 means “wrong credentials”. So what is wrong with my credentials?

I discovered the place in the python-ibanity library that does the connection: https://github.com/ibanity/ibanity-python/blob/master/ibanity/Ibanity.py

I see that the python-ibanity package doesn’t use requests. Why? See Playing with Ibanity (continued).

Sunday, January 12, 2025

Sun, 12 Jan 2025 00:00:00 +0000

Sunday, January 12, 2025¶

Thoughts about detail links¶

Imagine you are building the dashboard for user X in a Lino Noi. You render the table of recent comments. At some moment you are there, with an instance of comments.Comment and want to render a link to the comments.Comment.owner of this comment. You call ar.obj2html(), which calls Model.as_str() to build the displayed text and obj2url() to get the URL. Let’s talk about that URL. It uses the “default table”. Model.get_default_table()

A different thing is when a row disappears from the list because of the end user’s change. For example, you are in the list of “Active tickets”, showing the detail view of one active ticket, and then you click on “Closed”. Lino currently gives you a “bad request” warning in this case,, which is wrong. The correct behaviour here should be to skip to the next row, maybe with a warning “Ticket X is no longer in Active tickets after your change” (although there are situations where even this warning should get suppressed). If Lino cannot skip to the next row (because you were on the last one), it should skip back to the previous row.

Sharif is back

Fri, 03 Jan 2025 00:00:00 +0000

Sharif is back¶

Friday, January 3, 2025

Today is a historic day for Lino: Sharif and I had our first meeting after half a year (see Sharif, where are you?). We took a screenshot to celebrate this moment:

I am profoundly glad that he is healthy and back. The Synodalsoft team is more like a family than a company because we are such a small team with such a big and vague vision.

Another thing that makes us different from normal software development companies is that we do not have any clear plan. We have only 20 customers, and we manage quite easily to satisfy their needs because Lino focuses on maintainability and stability. So most of our work is research and development. We advance step by step following our inner voice. We have a lot of work and we need new team members.

Sunday, December 22, 2024

Sun, 22 Dec 2024 00:00:00 +0000

Sunday, December 22, 2024¶

I found an USB stick in my desktop drawer. When I insert it to my notebook, nothing happens. In a terminal I can see it:

$ sudo fdisk -l
...
Disk /dev/sda: 14,46 GiB, 15525216256 bytes, 30322688 sectors
Disk model: USB DISK 2.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8399da61

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        8064 30322687 30314624 14,5G  c W95 FAT32 (LBA)

Saturday, December 21, 2024

Sat, 21 Dec 2024 00:00:00 +0000

Saturday, December 21, 2024¶

I stumbled into what looks like a regression on the cosi1e online demo. I sign in as robin and then click on the “749 Sales invoices (SLS)” in the dashboard, … but nothing happens. Or more precisely the JS console says:

Uncaught (in promise) TypeError: n is undefined
  computeDefaults NavigationControl.js:289
  _ NavigationControl.js:572
  pushPath NavigationControl.js:731
  runAction ActionHandler.tsx:739
  checkAndRunAction ActionHandler.tsx:688
  runAction App.jsx:559
  <anonymous> :1
NavigationControl.js:289:47
  pushPath NavigationControl.js:731
  AsyncFunctionThrow self-hosted:804
  (Async: async)
  runAction ActionHandler.tsx:739
  InterpretGeneratorResume self-hosted:1413
  AsyncFunctionNext self-hosted:800
  (Async: async)
  checkAndRunAction ActionHandler.tsx:688
  runAction App.jsx:559
  <anonymous> :1

The URL under the “749 Sales invoices (SLS)” text in the dashboard is:

javascript:window.App.runAction({ "actorId": "trading.InvoicesByJournal",
"an": "grid", "rp": null, "status": { "base_params": { "mk": 1, "mt": 23 },
"param_values": { "cleared": null, "clearedHidden": null, "end_period": null,
"end_periodHidden": null, "partner": null, "partnerHidden": null,
"start_period": null, "start_periodHidden": null, "state": null,
"stateHidden": null } } })

I had a look at the mentioned source code App.jsx and ActionHandler.tsx (both are in react/lino_react/react/components).

Then I discovered that the problem doesn’t occur on my machine. NB: of course I had hit Shift+Ctrl+R before starting to investigate. So I cleared the cookies and site data for the online demo site, signed in again, hit Shift+Ctrl+R once more, and the issue was gone.

So I just stumbled into a situation where Lino does not automatically detect that the site data needs to get cleared.

Friday, December 20, 2024

Fri, 20 Dec 2024 00:00:00 +0000

Friday, December 20, 2024¶

Today I continued to work on Ticket #5670 (Support e-invoices using PEPPOL).

I decided to play with the source code of the peppol-py package. Thanks for publishing your work, Anders Rune Jensen!

Is it possible that my customers can (and must) become an “access point” of the Peppol network? So that I just need to install them something like this script? And when they register a sales invoice in TIM, TIM would call this script, which would connect to the Peppol network and say “Hello, here I have a new invoice”.

Before I actually started playing, I worked on the topic page Peppol support (e-Invoicing) in Lino, trying to get a more or less satisfying understanding of PEPPOL into my little brain. And I got quickly lost.

For example the first sentence of the README.md file says that it is a “python implementation for sending peppol eDelivery AS4 documents”. So I wanted to know what this AS4 actually means. On docs.peppol.eu I found found the following definition:

In short, AS4 is used in the Peppol network for transmission of asynchronous messages between corner 2 and corner 3 in a Four Corner Topology using the Peppol PKI for signature and encryption on AS4 message level and SMP/SML for dynamic discovery.

Did you understand anything? I didn’t. This is a good example of why I need my own definitions… Surf, surf, okay, everybody except me seems to know what this “AS4” stands for, but let’s not get stuck any longer because of this. Let’s have a look at this source code instead.

The main script sender.py wants at least two arguments: a --filename (I guess that this should be the XML file of the invoice) and a --receiver (which is then referred to as their_id). And what does it do with this receiver?

I copied sml.py and smp.py into a file 1220.py (see below) in order to play with it.

The README gives an example “9922:NGTBCNTRLP1001” for the --receiver argument, and I guess that 9922 is an EAS code (meaning “Andorra VAT number”). For my tests I’d rather use 9931 (Estonia VAT number) and our own VAT number, which is EE100588749.

Here is my playground script 1220.py:

import hashlib
from lxml import etree
import urllib.request
import urllib.parse

# copied from sml.py:
# SML: receiver -> domain (DNS)

def get_domain_using_http(receiver, test):
    smp_id = 'B-' + hashlib.md5((receiver.lower()).encode("utf-8")).hexdigest()
    return f'{smp_id}.iso6523-actorid-upis.{get_server(test)}'

def get_server(test):
    if test:
        return 'acc.edelivery.tech.ec.europa.eu'
    else:
        return 'edelivery.tech.ec.europa.eu'

# copied from smp.py:
# SMP: domain + path -> xml with service descriptions

def get_smp_info(domain, receiver):
    # all the available interfaces (invoice, credit note etc.)
    url = 'http://' + domain + "/iso6523-actorid-upis::" + receiver
    print(f"request from url {url}")
    contents = urllib.request.urlopen(url).read()
    print(contents)
    return contents

invoice_end = urllib.parse.quote("billing:3.0::2.1")

def find_invoice_smp_document(smp_contents):
    root = etree.fromstring(smp_contents)
    for child in root:
        for el in child:
            if el.get('href').endswith(invoice_end):
                return el.get('href')

def extract_as4_information(smp_contents):
    invoice_url = find_invoice_smp_document(smp_contents)
    invoice_smp = urllib.request.urlopen(invoice_url).read()
    root = etree.fromstring(invoice_smp)
    #id = root.findall(".//{*}ParticipantIdentifier")[0].text
    as4_endpoint = root.findall(".//{*}EndpointReference")[0][0].text
    certificate = root.findall(".//{*}Certificate")[0].text
    return [as4_endpoint, certificate]


their_id = "9931:EE100588749"  # Rumma & Ko OÜ
test = True

smp_domain = get_domain_using_http(their_id, test)
print(f"smp_domain is {smp_domain}")
smp_contents = get_smp_info(smp_domain, their_id)
url, their_cert = extract_as4_information(smp_contents)
print(url, their_cert)

#their_certfile = '/tmp/their-cert.pem'
#with open(their_certfile, 'w') as f:
#    f.write('-----BEGIN CERTIFICATE-----\n' + their_cert + '\n-----END CERTIFICATE-----')

Here is the output of my script:

smp_domain is B-e0aa140770a8690c4a3512bc5cdfaeb9.iso6523-actorid-upis.acc.edelivery.tech.ec.europa.eu
request from url http://B-e0aa140770a8690c4a3512bc5cdfaeb9.iso6523-actorid-upis.acc.edelivery.tech.ec.europa.eu/iso6523-actorid-upis::9931:EE100588749
Traceback (most recent call last):
...
socket.gaierror: [Errno -2] Name or service not known

So basically it builds a URL that requests from acc.edelivery.tech.ec.europa.eu but gets an error “Name or service not known” as response.

That server (acc.edelivery.tech.ec.europa.eu) seems to exist (at least it has an IP address) but doesn’t answer to ping:

$ ping acc.edelivery.tech.ec.europa.eu
PING acc.edelivery.tech.ec.europa.eu (147.67.35.45) 56(84) bytes of data.
^C
--- acc.edelivery.tech.ec.europa.eu ping statistics ---
109 packets transmitted, 0 received, 100% packet loss, time 110629ms

I must be missing something.

I guess that the --receiver is not my customer (the receiver of the invoice) but a certified “Service Metadata Publisher” (SMP). To become an SMP they need to join OpenPeppol AISBL. That’s definitively not something my customers will do.

So these SMPs or APs are certified by Beppol authorities. Every country has a PEPPOL authority. For Belgium it is BOSA: https://bosa.belgium.be

And now I see that Hermes is back! I thought that it had vanished!

https://bosa.belgium.be/fr/applications/hermes

And my customers in Belgium can probably sign in to Hermes:

https://hermes-belgium.be/hermeslogin?lang=fr

Next step then is to explore Hermes with my customers and find out how Lino can upload their XML invoices to it.

Thursday, December 19, 2024

Thu, 19 Dec 2024 00:00:00 +0000

Thursday, December 19, 2024¶

Lino can generate a PEPPOL-compliant invoice, i.e. an XML file that follows the EN 16931-1 standard (Peppol support (e-Invoicing) in Lino). But how is the site user supposed to do with this file? As far as I understood the system, they must “upload this file into the PEPPOL network”.

Compliance with the European standard on eInvoicing. Explaining the standard is a good introduction to PEPPOL. Interesting, but doesn’t answer my question.

EESPA, the European Association of E-Invoicing Service Providers, is an international not-for-profit association (AISBL/IVZW) organised under Belgian law. And they offer a partner programme. But haha, this is obviously not for Lino: the one-time joining fee is €1000, plus a yearly membership fee which was €2750 in for 2023.

A company in Denmark, named Bizbrains A/S, offer a partner programme that seems to be what I need. On their website they write:

Is your company an ERP house, an e-commerce provider or an EDI service provider (VAN)? Or do you sell products where you need to be able to send and/or receive PEPPOL invoices? If so, you have much to gain by partnering up with Peppol.com and establishing a close integration with our PEPPOL services – via our API, for example. As a Peppol.com partner, you can provide your users with the opportunity to send and receive e-invoices. We can also handle transformations between the PEPPOL format and the format you use in-house.

I filled their contact form, we’ll see where this leads.

The peppol-py package might be interesting. It is a “python implementation for sending peppol eDelivery AS4 documents”.

Sunday, November 24, 2024

Sun, 24 Nov 2024 00:00:00 +0000

Sunday, November 24, 2024¶

When I run inv bd in prima, I get this:

There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.10/pkgutil.py", line 417, in get_importer
    importer = sys.path_importer_cache[path_item]
KeyError: PosixPath('/home/luc/work/prima/docs')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/sphinx/config.py", line 529, in eval_config_file
    exec(code, namespace)  # NoQA: S102
  File "/home/luc/work/prima/docs/conf.py", line 8, in <module>
    configure(globals(), 'lino_prima.projects.prima1.settings')
  File "/home/luc/work/lino/lino/sphinxcontrib/__init__.py", line 112, in configure
    startup(django_settings_module)
  File "/home/luc/work/lino/lino/__init__.py", line 158, in startup
    django.setup()
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/django/conf/__init__.py", line 81, in __getattr__
    self._setup(name)
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/django/conf/__init__.py", line 68, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/django/conf/__init__.py", line 166, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/luc/work/prima/lino_prima/projects/prima1/settings.py", line 23, in <module>
    SITE = Site(globals())
  File "/home/luc/work/lino/lino/core/site.py", line 472, in __init__
    self.load_plugins()
  File "/home/luc/work/lino/lino/core/site.py", line 888, in load_plugins
    install_plugin(app_name)
  File "/home/luc/work/lino/lino/core/site.py", line 837, in install_plugin
    app_mod = import_module(app_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/bulma/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution, DistributionNotFound
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3668, in <module>
    def _initialize_master_working_set() -> None:
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3651, in _call_aside
    f(*args, **kwargs)
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3680, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 647, in _build_master
    ws = cls()
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 640, in __init__
    self.add_entry(entry)
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 696, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/luc/venvs/dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2276, in find_distributions
    importer = get_importer(path_item)
  File "/usr/lib/python3.10/pkgutil.py", line 421, in get_importer
    importer = path_hook(path_item)
  File "<frozen importlib._bootstrap_external>", line 1632, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 1504, in __init__
  File "<frozen importlib._bootstrap_external>", line 182, in _path_isabs
AttributeError: 'PosixPath' object has no attribute 'startswith'

It happens when the conf.py says:

from lino.sphinxcontrib import configure
configure(globals(), 'lino_prima.projects.prima1.settings')

Or more precisely when lino.sphinxcontrib.configure() calls lino.startup(). But the error does not occur when I do this in an interactive normal Python session:

>>> from lino import startup
>>> startup("lino_prima.projects.prima1.settings")

>>> from importlib import import_module
>>> import_module("bulma")
<module 'bulma' from '/home/luc/venvs/dev/lib/python3.10/site-packages/bulma/__init__.py'>

It might have to do with the fact that Bulma uses pkg_resources as an API, which is deprecated:

from pkg_resources import get_distribution, DistributionNotFound
<stdin>:1: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

I worked around the problem by modifying my copy of the bulma/__init__.py file as follows.

Before:

from pkg_resources import get_distribution, DistributionNotFound
try:
    __version__ = get_distribution('django-bulma').version
except DistributionNotFound:
    # package is not installed
    pass

After:

__version__  = '0.9.0'
# from pkg_resources import get_distribution, DistributionNotFound
# try:
#     __version__ = get_distribution('django-bulma').version
# except DistributionNotFound:
#     # package is not installed
#     pass

I considered already contacting Tim Kamanin, the maintainer of django-bulma and maybe even help him to move away from the deprecated usage of pkg_resources.

But I can only ask him to change something if I can tell him how to reproduce my problem. So I created a virgin Sphinx project in a temporary directory and added the triggering code to the conf.py file:

from lino.sphinxcontrib import configure
configure(globals(), 'lino_prima.projects.prima1.settings')
from lino import startup
startup("lino_prima.projects.prima1.settings")

And this lead to undisclose the real culprit: it’s my own code! No need to bother Tim. My rstgen.sphinxconf.configure() function did this:

docs_root = Path(...)
sys.path.append(docs_root)

And the (deprecated) pkg_resources modules expects sys.path to be a list of strings.

So the solution was to change above line in rstgen.sphinxconf.configure() into:

sys.path.append(str(docs_root))

Thursday, November 14, 2024

Thu, 14 Nov 2024 00:00:00 +0000

Thursday, November 14, 2024¶

I stumbled into a surprisingly easy solution for #4851 (Bleach is no longer being maintained): I wrote a new function lino.utils.soup.sanitize_html(), which replaces bleach.clean(). Here are some first test cases I used for the new function. (But these are now merged into Bleaching).

>>> from lino.utils.soup import sanitize_html

>>> content = """
... No tag at beginning of text.
... bla bLTaQSTyI80t2t8l
... foo bar.
... And here is some <b>bold</b> text.
...
... """
>>> print(sanitize_html(content))
No tag at beginning of text.
bla bLTaQSTyI80t2t8l
foo bar.
And here is some <b>bold</b> text.

>>> content = """
... <p align="right">First paragraph</p>
... <p onclick="kill()">Second paragraph</p>
... """
>>> print(sanitize_html(content))
<p align="right">First paragraph</p>
<p>Second paragraph</p>

>>> content = """
... <!DOCTYPE html>
... <html>
...   <head>
...     <meta http-equiv="content-type" content="text/html; charset=UTF-8">
...     <title>Baby</title>
...   </head>
...   <body>
...     This is a descriptive text with <b>some</b> formatting.<br>
...     <br>
...     Here is a second paragraph.<br>
...     <br>
...   </body>
... </html>
... """

>>> print(sanitize_html(content))
This is a descriptive text with <b>some</b> formatting.<br/>
<br/>
    Here is a second paragraph.<br/>
<br/>

A series of optimizations while working on Lino Prima

Thu, 31 Oct 2024 00:00:00 +0000

A series of optimizations while working on Lino Prima¶

Thursday, October 31, 2024

During the last week I did the following changes, which I plan to push soon.

We have a new attribute lino.core.actors.Actor.row_template, which is a kind of a context-sensitive template for __str__() .

An avalanche of changes started because a piece of code in lino_react.react.views.ApiList.get() said:

ar = action_request(app_label, actor, request, request.GET, True)
ar.renderer = settings.SITE.kernel.default_renderer

Some of the results:

The default value for lino.core.requests.BaseRequest.renderer is now determined a bit differently. Traditionally it is either default_renderer (when the action request was created from an incoming HTTP request) or text_renderer (e.g. during a doctest).
The rt.login function is no longer implemented as a method of lino.core.site.Site but as a stand-alone function in lino.core.utils.
The rt.show function has a wrapper rt.shows, which returns the output as a string instead of printing it to stdout. In unit tests we had code like this:
```
s = cal.GuestsByEvent.create_request(evt).to_rst()
```
This has been replaced by:
```
s = rt.shows(cal.GuestsByEvent, evt)
```

The signature of lino.core.actors.Actor.row_as_summary() is a bit more explicit. Before:

def row_as_summary(cls, ar, obj, **kwargs):

After:

def row_as_summary(cls, ar, obj, text=None, **kwargs):

The lino.core.actors.Actor.group_by is back, but in a new form. If specified, it must be a callable, and it currently only for the display mode “list” and only in the React front end. It uses a utility class lino.utils.html.Grouper. First usage example is in Lino Prima:

class ChallengeRatingsByProject(ChallengeRatings):
    ...
    group_by = [lambda obj: obj.challenge.project_section]

There are also a few subtle optimizations in React front end for Lino.

Friday, October 25, 2024

Fri, 25 Oct 2024 00:00:00 +0000

Friday, October 25, 2024¶

>>> url = "beid://https//xxxxxx.xxxx/receivedata.php?id=123456&date=2024-10-24"
>>> from urllib.parse import quote, unquote
>>> url
'beid://https//xxxxxx.xxxx/receivedata.php?id=123456&date=2024-10-24'
>>> quote(url)
'beid%3A//https//xxxxxx.xxxx/receivedata.php%3Fid%3D123456%26date%3D2024-10-24'
>>> unquote(quote(url))
'beid://https//xxxxxx.xxxx/receivedata.php?id=123456&date=2024-10-24'
>>> unquote(quote(url)) == url
True
>>> unquote(url) == url
True

Should we use urlparse rather than doing our own parsing? Like this:

>>> from urllib.parse import urlparse
>>> urlparse(url)
ParseResult(scheme='beid', netloc='https', path='//xxxxxx.xxxx/receivedata.php', params='', query='id=123456&date=2024-10-24', fragment='')

But this doesn’t relieve us from having to unquote the URL:

>>> urlparse(quote(url))
ParseResult(scheme='', netloc='', path='beid%3A//https//xxxxxx.xxxx/receivedata.php%3Fid%3D123456%26date%3D2024-10-24', params='', query='', fragment='')

So I prefer staying with our simple split() because using urlparse() might introduce new unknown issues,

Wednesday, October 23, 2024

Wed, 23 Oct 2024 00:00:00 +0000

Wednesday, October 23, 2024¶

Thursday, October 10, 2024

Thu, 10 Oct 2024 00:00:00 +0000

Thursday, October 10, 2024¶

I am working on #5777 (ParameterStore (…) expects a list of 3 values but got 5), which requires me to dive into the NavigationControl.js and Base.ts files. This page contains my reading notes.

The localStorage read-only property of window allows you to access a Storage object for the Document‘s origin; the stored data is saved across browser sessions. localStorage is similar to sessionStorage, except that sessionStorage gets cleared when the page session ends — that is, when the page is closed. (Source: mozilla.org) that

Base.ts¶

The Base.ts file:

defines the DynDep class, which adds dynamic dependency management to a class. The constructor() loads the modules specified by requiredModules and then calls prepare(), sets ready to true and finally calls onReady() (which subclasses can override).
it also defines a Component class, which extends the bare React.Component to use DynDep.

The file also creates two contexts URLContextType and DataContextType, using React.createContext(), and their default value is an (empty) object. A context lets a parent component provide data to the entire tree below it. (Source: react.dev)

They are assigned to the contextType of our components. More precisely:

URLContextType is used by: GridFilter, LinoBbar, LinoBody, LinoLayout, LinoDialog, LinoPaginator, LinoParamsPanel, …
DataContextType is used by: GridElement, LinoComponents.UnknownElement

IOW, in a GridElement instance, e.g. in the GridElement.arrowSelect() method, this.context points to the DataContextType and we can say this.context.rows to access the array of rows. Two questions:

Do all GridElement instances share the same context? I guess that React actually makes a copy of contextType for each new instance.
Where do we define that a DataContextType has an attribute rows?

I see that React “recommend defining components as functions instead of classes.” So some day we will probably want to migrate. But that won’t be as easy as they describe it because we use an enhanced Component base class.

NavigationControl.js¶

The NavigationControl.js file:

defines the Context class, which extends DynDep. A Context is a controller for all navigation features. It as an attribute contextType, which contains one of the following values (defined in constants.js):
```
CONTEXT_TYPE_SINGLE_ROW = "SingleRow";
CONTEXT_TYPE_MULTI_ROW = "MultiRow";
CONTEXT_TYPE_ACTION = "Action";
```
defines the History class, with attributes like state (contains the “router state”, an instance of State), and context (the context of its owner).

History.pushPath()
defines Delegate
defines State, which represents a router state

The `ActionHandler`¶

class ActionHandler()¶

ActionHandler.singleRow(event, pk, where, status)¶: Open a detail window on the given primary key pk in the current table.

The ActionHandler.singleRow() method is called by:

GridElement.showDetail() (which is called when you click on the first cell of a row)
when you click on the title of a LinoCards item

class GridElement()¶

defines

DynDep.reflect()

RegisterImportPool,

A new Lino: Lino Prima

Mon, 30 Sep 2024 00:00:00 +0000

A new Lino: Lino Prima¶

Monday, September 30, 2024

Note: This page is obsolete and unmaintained. You probably prefer to read the up-to-date article in Lino Prima User’s Guide.

Lino Prima is for storing evaluation results and printing certificates in a primary school.

Manage evaluation results and certificates in a primary school

Existing plugins: users

Database models in the prima plugin:

Teacher and Pupil are just types of users
Year: A period of time, usually lasting from September to June Examples: 23-24, 24-25
Periods: P1 and P2
Group (Klasse): A group of ↗pupils who work together for a given ↗year, following a given ↗course. Examples: 1A, 1B, 2A, 2B, 2C, …
Enrolment (Einschreibung): when a given ↗pupil is member of a given ↗group. Fields: ↗pupil, ↗group.
Role (Lehrerrolle): a specific role to be assumed by a teacher in a group. Examples: Klassenleiter, Sportlehrer,…
Cast (Rollenbelegung) : when a given ↗teacher assumes a given ↗role in a given ↗group. Fields: ↗teacher, ↗group, ↗role
Subject (Unterrichtsfach). Fields: name, ☑complex Examples: Wissenschaften, Kunst, Musik, Sport, Schriftbild, Heimatkunde/Sachkunde, Französisch, Mathematik, Religion, Deutsch

Every subject has a set of ↗skills. “complex” means a complex schema for evaluations using ↗skills, ↗blocks and ↗tests
Vintage (Jahrgang): 1, 2, … 6
Skill (Hauptkompetenz): a subdivision of a ↗subject. Fields: name, ↗subject Examples: Deutsch:Schreiben, Deutsch:Lesen, Mathematik:Arithmetik
Block (Baustein): a subdivision of a ↗skill. Fields: name, ref, ↗subject, ↗skill, ↗color, ↗vintage Examples: Erlebnisse spannend erzählen (Erle), Rund um Tiere (Tier), Berichten (Ber), Geschichten aus dem Leben erzählen (Leb)
Course (Kursus): a set of ↗subjects and ↗skills to be taught to a number of ↗pupils. Fields : name, ↗vintage. Examples: 1. Schuljahr, 2. Schuljahr, …
Probe: A test that has been made in a given group of pupils Fields: name, ↗group, ↗period, ↗subject

Each probe has a list of “skill probes”
SkillProbe (Leistung): when a given ↗skill is being measured during a given ↗probe

Fields: ↗probe, ↗skill, max_rating (Höchstpunktzahl)
Evaluation (Bewertung)
Certificate (Zeugnis)

Wednesday, September 25, 2024

Wed, 25 Sep 2024 00:00:00 +0000

Wednesday, September 25, 2024¶

Before:

display_mode = ((None, constants.DISPLAY_MODE_SUMMARY),)

after:

default_display_modes = { None: constants.DISPLAY_MODE_SUMMARY }

Tables to test manually:

checkdata.MessagesByOwner
dupable.SimilarObjects
comments.ReactionsByComment
notify.MyMessages
published.TranslationsByPage
uploads.AreaUploads
debts.PrintEntriesByBudget
integ.UsersWithClients
albums.FilesByAlbum
humanlinks.LinksByHuman

Tuesday, September 24, 2024

Tue, 24 Sep 2024 00:00:00 +0000

Tuesday, September 24, 2024¶

Today I fixed the warning A props object containing a “key” prop is being spread into JSX and I wrote a section “How to prepare lino_react for commit” in Build the static files.

A props object containing a “key” prop is being spread into JSX¶

Full warning:

Warning: A props object containing a "key" prop is being spread into JSX:
  let props = {key: someKey, window_layout: ..., wt: ..., editing_mode: ..., tabIndex: ..., elem: ..., parent: ..., urlParams: ..., children: ...};
  <ChoicesFieldElement {...props} />
  React keys must be passed directly to JSX without using spread:
  let props = {window_layout: ..., wt: ..., editing_mode: ..., tabIndex: ..., elem: ..., parent: ..., urlParams: ..., children: ...};
  <ChoicesFieldElement key={someKey} {...props} />

The cause was actually quite easy, but it took me a while to find it:

let pss = {style: style, key: i, className: this.ex.classNames("l-component")}
if (resizable_panel) return <this.ex.prSplitter.SplitterPanel {...pss}>
    {child}
</this.ex.prSplitter.SplitterPanel>
return <div {...pss}>

Here is the correct code:

let pss = {style: style, className: this.ex.classNames("l-component")}
if (resizable_panel) return <this.ex.prSplitter.SplitterPanel {...pss} key={i}>
    {child}
</this.ex.prSplitter.SplitterPanel>
return <div {...pss} key={i}>

I fixed the editor bug September 2024!

Mon, 23 Sep 2024 00:00:00 +0000

I fixed the editor bug September 2024!¶

Monday, September 23, 2024

We had a bug in the text field editor: when you start editing a text field that has already some content, the field gets cleared before you receive the focus. I don’t remember when I saw it for the first time. It’s there already several weeks or maybe months.

Yesterday I changed the quill version in package.json from “^1.3.7” to “^2.0.2”, but I’d actually want to try upgrading all dependencies to their latest version. How to do that?

$ npm outdated
Package                                     Current   Wanted   Latest  Location                                                 Depended by
@babel/cli                                   7.24.8   7.25.6   7.25.6  node_modules/@babel/cli                                  react
@babel/core                                  7.24.9   7.25.2   7.25.2  node_modules/@babel/core                                 react
@babel/plugin-transform-runtime              7.14.3   7.25.4   7.25.4  node_modules/@babel/plugin-transform-runtime             react
@babel/preset-env                            7.14.4   7.25.4   7.25.4  node_modules/@babel/preset-env                           react
@babel/preset-react                         7.13.13   7.24.7   7.24.7  node_modules/@babel/preset-react                         react
@electron-forge/cli                           7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/cli                         react
@electron-forge/maker-deb                     7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/maker-deb                   react
@electron-forge/maker-rpm                     7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/maker-rpm                   react
@electron-forge/maker-squirrel                7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/maker-squirrel              react
@electron-forge/maker-zip                     7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/maker-zip                   react
@electron-forge/plugin-auto-unpack-natives    7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/plugin-auto-unpack-natives  react
@electron-forge/plugin-fuses                  7.4.0    7.5.0    7.5.0  node_modules/@electron-forge/plugin-fuses                react
@testing-library/jest-dom                     6.4.6    6.5.0    6.5.0  node_modules/@testing-library/jest-dom                   react
@testing-library/react                       14.3.1   14.3.1   16.0.1  node_modules/@testing-library/react                      react
@types/jest                                 29.5.12  29.5.13  29.5.13  node_modules/@types/jest                                 react
@types/react                                 18.3.3   18.3.8   18.3.8  node_modules/@types/react                                react
babel-loader                                  9.1.3    9.2.1    9.2.1  node_modules/babel-loader                                react
css-loader                                   6.11.0   6.11.0    7.1.2  node_modules/css-loader                                  react
css-minimizer-webpack-plugin                  5.0.1    5.0.1    7.0.0  node_modules/css-minimizer-webpack-plugin                react
electron                                     29.4.5   29.4.6   32.1.2  node_modules/electron                                    react
electron-log                                  5.1.7    5.2.0    5.2.0  node_modules/electron-log                                react
esbuild                                     0.19.12  0.19.12   0.24.0  node_modules/esbuild                                     react
i18next                                     23.13.0  23.15.1  23.15.1  node_modules/i18next                                     react
i18next-browser-languagedetector              7.2.1    7.2.1    8.0.0  node_modules/i18next-browser-languagedetector            react
i18next-http-backend                          2.5.2    2.6.1    2.6.1  node_modules/i18next-http-backend                        react
jest-puppeteer                                9.0.2    9.0.2   10.1.1  node_modules/jest-puppeteer                              react
primeicons                                    6.0.1    6.0.1    7.0.0  node_modules/primeicons                                  react
puppeteer                                   21.11.0  21.11.0   23.4.0  node_modules/puppeteer                                   react
query-string                                  8.2.0    8.2.0    9.1.0  node_modules/query-string                                react
quill-mention                                 4.1.0    4.1.0    6.0.1  node_modules/quill-mention                               react
rc-collapse                                   3.7.3    3.8.0    3.8.0  node_modules/rc-collapse                                 react
react-i18next                                14.1.3   14.1.3   15.0.2  node_modules/react-i18next                               react
react-router-dom                             6.26.1   6.26.2   6.26.2  node_modules/react-router-dom                            react
style-loader                                  3.3.4    3.3.4    4.0.0  node_modules/style-loader                                react
ts-jest                                      29.2.4   29.2.5   29.2.5  node_modules/ts-jest                                     react
typescript                                    5.5.4    5.6.2    5.6.2  node_modules/typescript                                  react
uglify-js                                    3.19.2   3.19.3   3.19.3  node_modules/uglify-js                                   react

::

$ npx npm-check-updates -u Need to install the following packages: npm-check-updates@17.1.3 Ok to proceed? (y)

Upgrading /home/luc/work/react/package.json [====================] 76/76 100%

@babel/cli ^7.24.1 → ^7.25.6 @babel/core ^7.13.1 → ^7.25.2 @babel/plugin-transform-runtime ^7.13.10 → ^7.25.4 @babel/preset-env ^7.13.5 → ^7.25.4 @babel/preset-react ^7.12.13 → ^7.24.7 @babel/preset-typescript ^7.23.3 → ^7.24.7 @electron-forge/cli ^7.3.0 → ^7.5.0 @electron-forge/maker-deb ^7.3.0 → ^7.5.0 @electron-forge/maker-rpm ^7.3.0 → ^7.5.0 @electron-forge/maker-squirrel ^7.3.0 → ^7.5.0 @electron-forge/maker-zip ^7.3.0 → ^7.5.0 @electron-forge/plugin-auto-unpack-natives ^7.3.0 → ^7.5.0 @electron-forge/plugin-fuses ^7.3.0 → ^7.5.0 @electron/fuses ^1.7.0 → ^1.8.0 @testing-library/jest-dom ^6.1.5 → ^6.5.0 @testing-library/react ^14.1.2 → ^16.0.1 @types/jest ^29.5.11 → ^29.5.13 @types/lodash ^4.14.202 → ^4.17.7 @types/react ^18.2.28 → ^18.3.8 @types/react-dom ^18.2.13 → ^18.3.0 autoprefixer ^10.2.4 → ^10.4.20 babel-loader ^9.1.3 → ^9.2.1 builder-util-runtime ^9.2.4 → ^9.2.5 classnames ^2.2.6 → ^2.5.1 css-loader ^6.8.1 → ^7.1.2 css-minimizer-webpack-plugin ^5.0.1 → ^7.0.0 electron ^29.1.0 → ^32.1.2 electron-log ^5.1.1 → ^5.2.0 electron-squirrel-startup ^1.0.0 → ^1.0.1 electron-updater ^6.1.8 → ^6.3.4 esbuild ^0.19.8 → ^0.24.0 html-webpack-plugin ^5.5.3 → ^5.6.0 i18next ^23.10.1 → ^23.15.1 i18next-browser-languagedetector ^7.2.1 → ^8.0.0 i18next-http-backend ^2.5.0 → ^2.6.1 i18next-scanner ^4.4.0 → ^4.5.0 jest-puppeteer ^9.0.2 → ^10.1.1 primeicons ^6.0.1 → ^7.0.0 prop-types ^15.7.2 → ^15.8.1 puppeteer ^21.6.1 → ^23.4.0 query-string ^8.1.0 → ^9.1.0 quill-mention ^4.1.0 → ^6.0.1 rc-collapse ^3.1.0 → ^3.8.0 react ^18.2.0 → ^18.3.1 react-i18next ^14.1.0 → ^15.0.2 react-router-dom ^6.21.1 → ^6.26.2 react-test-renderer ^18.2.0 → ^18.3.1 react-transition-group ^4.4.1 → ^4.4.5 style-loader ^3.3.3 → ^4.0.0 ts-jest ^29.1.1 → ^29.2.5 typescript ^5.3.3 → ^5.6.2 uglify-js ^3.17.4 → ^3.19.3 weak-key ^1.0.2 → ^1.0.3 webpack ^5.53.0 → ^5.94.0 whatwg-fetch ^3.6.1 → ^3.6.20 workbox-precaching ^7.0.0 → ^7.1.0 workbox-webpack-plugin ^7.0.0 → ^7.1.0

Run npm install to install new versions. $

$ npm install

added 32 packages, removed 43 packages, changed 169 packages, and audited 1687 packages in 1m

201 packages are looking for funding
  run `npm fund` for details

7 vulnerabilities (1 high, 6 critical)

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
$

$ npm audit fix

up to date, audited 1687 packages in 3s

201 packages are looking for funding
  run `npm fund` for details

# npm audit report

babel-traverse  *
Severity: critical
Babel vulnerable to arbitrary code execution when compiling specifically crafted malicious code - https://github.com/advisories/GHSA-67hx-6x53-jw92
fix available via `npm audit fix --force`
Will install babel-plugin-transform-class-properties@6.10.2, which is a breaking change
node_modules/babel-traverse
  babel-helper-function-name  *
  Depends on vulnerable versions of babel-template
  Depends on vulnerable versions of babel-traverse
  node_modules/babel-helper-function-name
    babel-plugin-transform-class-properties  >=6.11.5
    Depends on vulnerable versions of babel-helper-function-name
    Depends on vulnerable versions of babel-template
    node_modules/babel-plugin-transform-class-properties
  babel-template  *
  Depends on vulnerable versions of babel-traverse
  node_modules/babel-template

json5  <1.0.2
Severity: high
Prototype Pollution in JSON5 via Parse Method - https://github.com/advisories/GHSA-9c47-m6qq-7p4h
fix available via `npm audit fix --force`
Will install webpack-typescript@0.5.5, which is a breaking change
node_modules/webpack-typescript/node_modules/json5
  loader-utils  <=1.4.0
  Depends on vulnerable versions of json5
  node_modules/webpack-typescript/node_modules/loader-utils
    webpack-typescript  >=0.5.6
    Depends on vulnerable versions of loader-utils
    node_modules/webpack-typescript


7 vulnerabilities (1 high, 6 critical)

To address all issues (including breaking changes), run:
  npm audit fix --force
$

$ npm audit fix --force
npm warn using --force Recommended protections disabled.
npm warn audit Updating webpack-typescript to 0.5.5, which is a SemVer major change.
npm warn audit Updating babel-plugin-transform-class-properties to 6.10.2, which is a SemVer major change.

removed 25 packages, changed 2 packages, and audited 1662 packages in 2s

201 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$

After this upgrade, I had 2 build errors in cloneUI.ts and integrity.ts in lino_react/react/components/__tests__, the same ones reported already by Joy when she had upgraded her dependencies. So it was easy to fix them.

Now the front-end builds, and works at first glance… and surprise: the bug is gone!

But there are at least two new problems:

quill Cannot import modules/mention. Are you sure it was registered?¶

When a textfield editor starts, the JS console says “quill Cannot import modules/mention. Are you sure it was registered?”

I saw that quill-mention has evolved from version 4 to 6 and that the first thing they instruct in their README is to import ‘quill-mention/autoregister’;, and that Lino’s quillmodules.jsx file doesn’t do this. So I added that line and the warning went away. No need to dive deeper right now.

Uncaught (in promise) TypeError: n is null¶

The New Client quick link is broken. When I click it, the JS console says:

Uncaught (in promise) TypeError: n is null,
  onReady SiteContext.jsx:148
  setReady Base.ts:181
  componentDidMount Base.ts:153

This issue is actually old, and it goes away when you reload the dashboard. It comes only when you had previously opened another table and then returned to the main page. I think it is a bug in our logic for retrieving the parent’s context. I won’t dive into this now because no customer has ever complained about it.

It seems you are still using the old wait option¶

Full warning in the JS console:

react-i18next:: It seems you are still using the old wait option, you may migrate to the new useSuspense behaviour.

I had seen this warning before, and since it is still there after the upgrade, I decided to investigate a bit.

As suggested e.g. here, I replaced 'wait' by 'useSuspense' in components/i18n.js and the warning was gone. And I didn’t dive deeper.

Another issue related to i18n is that the client requests GET http://127.0.0.1:8000/static/react/locales/en/translation.json and receives “404 Not found”. I didn’t yet dive into this.

Support for defaultProps will be removed from function components in a future major release¶

We have indeed a Labeled component, defined in LinoComponentUtils.jsx, and not as a class but as a function.

Thanks to Matt Pocock, who explains how to handle this issue in a short video Stop using defaultProps (React 18.3 preview) After watching this video I was able to convert our code.

How to install esteid software on Tuxedo OS

Fri, 20 Sep 2024 00:00:00 +0000

How to install esteid software on Tuxedo OS¶

Friday, September 20, 2024

I want to install the Estonian eid software (“DigiDoc client”) on my notebook, but the install script says:

$ sh install-open-eid.sh
tuxedo is not supported

All other Linux software installed nicely on my Tuxedo notebook from .deb packages because Tuxedo OS is actually just an optimized Ubuntu.

More exactly it’s an Ubuntu 22.04, codename “jammy”:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Tuxedo
Description:    TUXEDO OS 3
Release:        22.04
Codename:       jammy

Here is what I did.

Modify the install script:

$ nano install-open-eid.sh

Find the place where it says:

case $distro in
   debian)
      ...
   ubuntu|neon|zorin)
      ...
   linuxmint)

Change the case “ubuntu|neon|zorin)” into:

ubuntu|neon|zorin|tuxedo)

For normal people this would probably have worked. But I am a Belgian, and this caused a little problem. It seems that the Belgian eid software conflicts with the Estonian one:

$ sh install-open-eid.sh
Adding RIA repository to APT sources list (/etc/apt/sources.list.d/ria-repository.list)
[sudo] password for luc:
deb [signed-by=/usr/share/keyrings/ria-repository.gpg] https://installer.id.ee/media/ubuntu/ jammy main
Adding key to trusted key set
0xC6C83D68 'RIA Software Signing Key <signing@ria.ee>'
Installing software (apt update && apt install open-eid)
Hit:1 https://linux.teamviewer.com/deb stable InRelease
Get:2 https://installer.id.ee/media/ubuntu jammy InRelease
Get:3 https://installer.id.ee/media/ubuntu jammy/main amd64 Packages [6 094 B]
Hit:4 http://txos-extra.tuxedocomputers.com/ubuntu jammy InRelease
Hit:5 https://txos.tuxedocomputers.com/ubuntu jammy InRelease
Hit:6 https://deb.tuxedocomputers.com/ubuntu jammy InRelease
Get:7 https://mirrors.tuxedocomputers.com/ubuntu/mirror/archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Hit:8 https://plasma6.tuxedocomputers.com jammy InRelease
Hit:9 https://txos.tuxedocomputers.com/ubuntu jammy-plasma InRelease
Hit:10 https://files.eid.belgium.be/debian jammy InRelease
Hit:11 https://mirrors.tuxedocomputers.com/ubuntu/mirror/security.ubuntu.com/ubuntu jammy-security InRelease
Hit:12 https://mirrors.tuxedocomputers.com/ubuntu/mirror/archive.ubuntu.com/ubuntu jammy-updates InRelease
Fetched 279 kB in 1s (200 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up-to-date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies.
 libbeidpkcs11-0 : Conflicts: opensc-pkcs11 but 0.22.0-1ubuntu2 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

So I uninstalled the Belgian eid software:

sudo apt remove eid-viewer eid-mw libbeidpkcs11-0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libbeidpkcs11-bin libnss3-tools libphonon4qt5-4 libqt5keychain1 libqt5quickwidgets5 libqt5webengine-data libqt5webenginecore5 libqt5webenginewidgets5 libqt5websockets5 libre2-9
  phonon-backend-gstreamer-common phonon4qt5
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED
  beid-mozilla-extension beid-mozilla-webext libbeidpkcs11-0 libeidviewer0
0 to upgrade, 0 to newly install, 4 to remove and 0 not to upgrade.
After this operation, 621 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 350378 files and directories currently installed.)
Removing beid-mozilla-extension (5.1.18v5.1.18-0u2204-1) ...
Removing beid-mozilla-webext (5.1.18v5.1.18-0u2204-1) ...
Removing libeidviewer0:amd64 (5.1.18v5.1.18-0u2204-1) ...
Removing libbeidpkcs11-0:amd64 (5.1.18v5.1.18-0u2204-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...

And then tried again:

$ sudo apt install open-eid

Now it worked. And when Estonian software was installed first, the Belgian software installs as well:

$ sudo apt install eid-viewer eid-mw

Voilà.

Afterwards I noticed that maybe both (Estonian) DigiDoc and (Belgian) eid viewer had been known to Tuxedo OS before and all this wasn’t necessary. But I’m not sure.

sudo apt remove eid-viewer eid-mw libbeidpkcs11-0

Wednesday, September 11, 2024

Wed, 11 Sep 2024 00:00:00 +0000

Wednesday, September 11, 2024¶

I have a copy of my data, made with rsync and using the -a option:

--archive, -a            archive mode is -rlptgoD (no -A,-X,-U,-N,-H)

How restore my data do my new notebook computer? I’d like to keep all permissions. But of course the new computer has different values for UID and GID. I would have to do some mapping.

Welcome at Synodalsoft, Joy and Abdullah!

Sat, 31 Aug 2024 00:00:00 +0000

Welcome at Synodalsoft, Joy and Abdullah!¶

Saturday, August 31, 2024

The Synodalsoft team welcomes two new contributors, Joy and Abdullah. I updated our list of individual contributors.

They come from very different places: Joy lives in Montevideo, 6 hours after Tallinn time zone, and Abdullah lives in Melbourne, 7 hours before Tallinn. Lino is extending to the ends of the world ;-)

They joined Synodalsoft at an interesting moment of our history.

When Joy contacted me, on March 4, I was actually in hospital, recovering from an accident where I broke my hip. This accident forced me to step back from my social volunteer engagements for the next months. As a religious person I constantly talk with God (some call it “praying”), so I asked him “What does that mean? What are you telling me through this?” and he answered in an unusually clear way: “Luc, it’s more than 20 years that you are trying to explain to church people in Estonia how they should do their work. They don’t want to hear you. Leave them in peace. Your vocation is Lino. Stop wasting your time with useless things.”

Note that “Lino” and “Synodalsoft” are synonym here. Synodalsoft is the team, Lino is our flagship product. I registered the synodalsoft.net domain name because the idea behind Lino is bigger than a single software product. Synodalsoft is about how software should ideally be developed, it is the name I suggest for the foundation to create if the project becomes bigger than Rumma & Ko can handle.

I had not yet fully recovered from my hip surgery when another frightening thing happened: Sharif, our second core developer, disappeared. He left his wife to visit a friend, and hasn’t answered his phone since then. This was more than three months ago and we still have no news about him. (Details in my blog.)

During the first weeks I was under shock and considered abandoning Lino if Sharif doesn’t come back. He had become like a son to me after 3 years of working together. But despite the catastrophic human aspect of this story, I started to realize that Lino continues to live, and that I will continue to develop it. Lino has always been developing slowly. Slow development is actually one of its strengths. For the first time in my life I seriously went down into the React front end. This part of Lino is quite different from the rest of Lino, most of it is written in Javascript. It had been written by Tonis and Hamza, and I never had to get my hands dirty with it. Because Sharif took over their job and was better than me at it.

Joy and Abdullah will remain volunteer contributors because I don’t plan to build up a third team of paid workers. I am now 56 years old and if you want Synodalsoft to continue when I will be gone, you can count on my help, but don’t ask me to be the motor.

Thursday, August 29, 2024

Thu, 29 Aug 2024 00:00:00 +0000

Thursday, August 29, 2024¶

Continued from Wednesday, August 28, 2024. Working on #4381 (Uploading photos or other media files via email (postfix)).

Hey, all my yesterday’s plans are probably bullshit! It’s much more straightforward to let Postfix deliver all emails to photos locally, as if photos was a normal user. And all analyzing is done by Lino in the background task. What I don’t know yet is how to remove mails from a mailbox.

https://docs.python.org/3/library/mailbox.html

Yes, it seems like the mailbox module is what we need. And I copy their warning:

Be very cautious when modifying mailboxes that might be simultaneously changed by some other process. The safest mailbox format to use for such tasks is Maildir; try to avoid using single-file formats such as mbox for concurrent writing. If you’re modifying a mailbox, you must lock it by calling the lock() and unlock() methods before reading any messages in the file or making any changes by adding or deleting a message. Failing to lock the mailbox runs the risk of losing messages or corrupting the entire mailbox.

And now I remember that we have the lino_xl.lib.inbox plugin, which also accesses mailboxes but doesn’t write to them. It just looks for incoming emails that are a reply to an existing comments.Comment.

I had a look at the django-mailbox plugin. This plugin (they call it a “Django application”) consumes messages from POP3, IMAP, Office365 API or local mailboxes into a Django database. It defines three database models Mailbox``(name, uri, from_email, active, last_polling), ``Message``(mailbox, subject, message_id, in_reply_to. from_header, to_header, encoded, processed, read, raw_message_content) and ``MessageAttachment``(message, headers, document) as well as an django-admin command ``getmail and a Django signal message_received.

The value of django-mailbox is that it provides a few additional transport classes compared to the standard mailbox package (see source code).

But for #4381 we don’t need django-mailbox because we don’t want to see the messages themselves. We actually want to avoid moving the messages too much from one place to another.

Here is a copy of the last version of the /home/photos/bin/photos.py script on SR, which I won’t maintain any more:

import logging; logger = logging.getLogger("photos")
import sys
from pathlib import Path
import email
import mimetypes
from email.policy import default
import datetime
import getpass

# store_dir = Path("/var/mail/photos")
# store_dir = Path("/home/photos")
store_dir = Path.home()

allowed_chars = "_-+"
def saniiyze(filename):
    filename = filename.replace(" ", "_")
    filename = "".join(c for c in filename if c.isalpha() or c.isdigit() or c in allowed_chars).strip()
    return filename


def main():

    log_file = store_dir / "photos.log"
    print("I am", getpass.getuser(), "writing to", str(log_file))  # "I am photos writing to /home/photos"

    log_file.write_text("hello")

    input_str = "".join(sys.stdin)

    msg = email.message_from_string(input_str, policy=default)
    log_file.write_text("{}\n\n{}\n\n".format(sys.argv, msg))
    now = datetime.datetime.now()
    log_file.write_text("{} Write message from {} to {}.\n".format(now, msg['From'], store_dir))
    # log_file.write_text("Write {} to {}.\n".format(msg, store_dir))
    log_file.write_text("my username: {}".format(getpass.getuser()))

    counter = 1
    for part in msg.walk():
        # multipart/* are just containers
        if part.get_content_maintype() == 'multipart':
            continue
        # Applications should really sanitize the given filename so that an
        # email message can't be used to overwrite important files
        filename = sanitize(part.get_filename())
        if not filename:
            ext = mimetypes.guess_extension(part.get_content_type())
            if ext:
                filename = f'part-{counter:03d}{ext}'
        counter += 1
        with open(store_dir / filename, 'wb') as fp:
            fp.write(part.get_payload(decode=True))

        #print("Wrote {} parts to {}".format(counter, filename))
    print("Wrote {} parts to {}.".format(counter, store_dir))


if __name__ == '__main__':
    main()

Wednesday, August 28, 2024

Wed, 28 Aug 2024 00:00:00 +0000

Wednesday, August 28, 2024¶

I reactivated ticket #4381 (Uploading photos or other media files via email (postfix)).

I am plying with the /home/photos/bin/photos.py script on SR. It is called because the /etc/aliases defines an alias:

photos:             "|/home/photos/bin/photos.py"

That script is increasing in complexity and I should definitively publish it somewhere in a git repository rather than editing it with nano in a terminal.

Should I publish it as a django-admin command? No, because we don’t want to start the whole Django machine for every incoming email. Let it be an independent little command-line tool. But how to name it? Maybe mail2mime?

While meditating on this I stumbled into a question on SO postfix - access to incoming mail info inside of python script where the author uses a file .procmailrc and a tool ripmime. Am I reinventing the wheel?

Seems that ripmime is not enough for our use case. We want a few features that are quite specific to Lino:

The tool should read a list of allowed recipients from a plain text file and refuse any incoming mail from other recipients.
The plain text file would be generated by the Lino site and give the Lino user name for every allowed recipient email address.
The tool should read the subject and the body of the email, and pass them to an application-specific Lino command.

A new mail2json script would write a json file containing subject, body, sender, timestamp, and all attachments to a directory. And the Lino site runs a background task that watches this directory for new json files. And each new json file will create an uploads.Upload row for each attachment, and one comments.Comment for each json file.

The story continues here: Thursday, August 29, 2024.

Saturday, August 24, 2024

Sat, 24 Aug 2024 00:00:00 +0000

Saturday, August 24, 2024¶

Today I pushed my first changeset in lino-react. I also released this to PyPI.

My first changeset in lino-react

Detailed list of changes:

Updated the docs so that they build again with inv bd. Some time ago we tried to use sphinxjs, but that turned out to not work as easily as expected, and now I boldly decided to try documenting in prosa style.
Removed the “Pinned URL(s)” button in the toolbar of the main page.
Changed the “!” into a “.” in a translatable text message (mostly to test i18n).
In <GridElement>, the logic for finding the next cell to edit when a cell editor has finished, is no longer run by the onKeyDown event of the wrapper div but by the onCellEditComplete hook of the <DataTable>.
Bump primereact version from 10.6 to 10.8.2. (Add editMode attribute to <DataTable>)

Wednesday, August 21, 2024

Wed, 21 Aug 2024 00:00:00 +0000

Wednesday, August 21, 2024¶

Today I wanted to see a balance from my accounting system, which has been running stable as a rock for the last 20 years.

I use TIM for my accounting. TIM has one disadvantage: it runs only under Windows. That’s why I still have a copy of the Windows XP from my last Windows computer, which I run in a VirtualBox VM without Internet connection.

But that box didn’t open today. It said:

On https://forums.virtualbox.org/viewtopic.php?t=111917 I found the probable explanation: I am still using VB version 6.1, which came to its end of life in December 2023. “If you’ve applied a recent Linux kernel upgrade then roll back to the previous working kernel. VirtualBox 6.1.50 is out of date and no longer supported; so running newer Linux kernels is likely to result in problems if VirtualBox is not upgraded.”

Thanks to concise upgrade instructions on kifarunix.com it took me only half an hour to get it running again. The remaining part of this post are just my console sessions.

I don’t have the VirtualBox Extension Pack:

$ sudo vboxmanage list extpacks
Extension Packs: 1
Pack no. 0:   VNC
Version:      6.1.50
Revision:     161033
Edition:
Description:  VNC plugin module
VRDE Module:  VBoxVNC
Usable:       true
Why unusable:

Uninstall the old version:

$ sudo apt remove --purge --auto-remove virtualbox virtualbox-6.1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package 'virtualbox-6.1' is not installed, so not removed
The following packages will be REMOVED:
  dctrl-tools* dkms* libcairo-5c0* libgsoap-2.8.117* linux-headers-6.2.0-39-generic* linux-hwe-6.2-headers-6.2.0-39* python-tinycss2-common* python3-cairocffi*
  python3-cairosvg* python3-cffi* python3-cssselect2* python3-defusedxml* python3-ply* python3-pycparser* python3-tinycss2* python3-xcffib* virtualbox*
  virtualbox-dkms* virtualbox-qt*
0 upgraded, 0 newly installed, 19 to remove and 6 not upgraded.
After this operation, 293 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 312295 files and directories currently installed.)
Removing virtualbox-qt (6.1.50-dfsg-1~ubuntu1.22.04.3) ...
Removing virtualbox (6.1.50-dfsg-1~ubuntu1.22.04.3) ...
Removing virtualbox-dkms (6.1.50-dfsg-1~ubuntu1.22.04.3) ...
Module virtualbox-6.1.50 for kernel 6.5.0-45-generic (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

vboxdrv.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.5.0-45-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxnetadp.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.5.0-45-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxnetflt.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.5.0-45-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...
Module virtualbox-6.1.50 for kernel 6.8.0-40-generic (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

vboxdrv.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.8.0-40-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxnetadp.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.8.0-40-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxnetflt.ko:
 - Uninstallation
   - Deleting from: /lib/modules/6.8.0-40-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...
Deleting module virtualbox-6.1.50 completely from the DKMS tree.
Removing dkms (2.8.7-2ubuntu2.2) ...
Removing dctrl-tools (2.24-3build2) ...
Removing libcairo-5c0:amd64 (1.21) ...
Removing libgsoap-2.8.117:amd64 (2.8.117-2build1) ...
Removing linux-headers-6.2.0-39-generic (6.2.0-39.40~22.04.1) ...
dpkg: warning: while removing linux-headers-6.2.0-39-generic, directory '/lib/modules/6.2.0-39-generic' not empty so not removed
Removing linux-hwe-6.2-headers-6.2.0-39 (6.2.0-39.40~22.04.1) ...
Removing python3-cairosvg (2.5.2-1) ...
Removing python3-cssselect2 (0.4.1-2) ...
Removing python3-tinycss2 (1.1.1-2) ...
Removing python-tinycss2-common (1.1.1-2) ...
Removing python3-cairocffi (1.2.0-2) ...
Removing python3-xcffib (0.11.1-2) ...
Removing python3-cffi (1.15.0-1build2) ...
Removing python3-defusedxml (0.7.1-1) ...
Removing python3-pycparser (2.21-1) ...
Removing python3-ply (3.11-5) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for shared-mime-info (2.1-2) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
(Reading database ... 281825 files and directories currently installed.)
Purging configuration files for dkms (2.8.7-2ubuntu2.2) ...
Purging configuration files for virtualbox (6.1.50-dfsg-1~ubuntu1.22.04.3) ...
Purging configuration files for dctrl-tools (2.24-3build2) ...

Create a file /etc/apt/sources.list.d/virtualbox.list with a single line of text:

deb [arch=amd64] http://download.virtualbox.org/virtualbox/debian jammy contrib

Install gnupg2 (I didn’t ask why):

$ sudo apt install gnupg2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  gnupg2
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 5 548 B of archives.
After this operation, 52,2 kB of additional disk space will be used.
Get:1 http://ee.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 gnupg2 all 2.2.27-3ubuntu2.1 [5 548 B]
Fetched 5 548 B in 0s (123 kB/s)
Selecting previously unselected package gnupg2.
(Reading database ... 281792 files and directories currently installed.)
Preparing to unpack .../gnupg2_2.2.27-3ubuntu2.1_all.deb ...
Unpacking gnupg2 (2.2.27-3ubuntu2.1) ...
Setting up gnupg2 (2.2.27-3ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...

Add the key to a file /etc/apt/trusted.gpg.d/virtualbox.gpg:

$ wget -qO- https://www.virtualbox.org/download/oracle_vbox_2016.asc | \
sudo gpg --dearmor --yes -o /etc/apt/trusted.gpg.d/virtualbox.gpg

Update my apt cache:

$ sudo apt update
Hit:1 http://ee.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://ee.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://ee.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 https://updates.signal.org/desktop/apt xenial InRelease
Hit:5 https://installer.id.ee/media/ubuntu jammy InRelease
Hit:6 http://deb.anydesk.com all InRelease
Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease
Get:8 http://download.virtualbox.org/virtualbox/debian jammy InRelease [4 428 B]
Hit:9 https://files.eid.belgium.be/debian jammy InRelease
Get:10 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 Packages [1 495 B]
Hit:11 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu jammy InRelease
Hit:12 https://linux.teamviewer.com/deb stable InRelease
Fetched 5 923 B in 3s (2 103 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.

Install the new version:

$ sudo apt install virtualbox-7.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libsdl-ttf2.0-0
The following packages will be REMOVED:
  virtualbox-guest-additions-iso
The following NEW packages will be installed:
  libsdl-ttf2.0-0 virtualbox-7.0
0 upgraded, 2 newly installed, 1 to remove and 6 not upgraded.
Need to get 92,9 MB of archives.
After this operation, 157 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ee.archive.ubuntu.com/ubuntu jammy/universe amd64 libsdl-ttf2.0-0 amd64 2.0.11-6 [15,1 kB]
Get:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy [92,9 MB]
Ign:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy
Get:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy [92,9 MB]
Ign:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy
Get:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy [92,9 MB]
Ign:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy
Get:2 http://download.virtualbox.org/virtualbox/debian jammy/contrib amd64 virtualbox-7.0 amd64 7.0.20-163906~Ubuntu~jammy [92,9 MB]
Fetched 18,4 MB in 3min 33s (86,5 kB/s)
Preconfiguring packages ...
(Reading database ... 281798 files and directories currently installed.)
Removing virtualbox-guest-additions-iso (6.1.50-1~ubuntu1.22.04.1) ...
Selecting previously unselected package virtualbox-7.0.
(Reading database ... 281793 files and directories currently installed.)
Preparing to unpack .../virtualbox-7.0_7.0.20-163906~Ubuntu~jammy_amd64.deb ...
Unpacking virtualbox-7.0 (7.0.20-163906~Ubuntu~jammy) ...
Selecting previously unselected package libsdl-ttf2.0-0:amd64.
Preparing to unpack .../libsdl-ttf2.0-0_2.0.11-6_amd64.deb ...
Unpacking libsdl-ttf2.0-0:amd64 (2.0.11-6) ...
Setting up virtualbox-7.0 (7.0.20-163906~Ubuntu~jammy) ...
Adding group `vboxusers' (GID 136) ...
Done.
Setting up libsdl-ttf2.0-0:amd64 (2.0.11-6) ...
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for shared-mime-info (2.1-2) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...

I also had to do:

$ sudo adduser luc vboxusers

Explanation to my riddle

Tue, 20 Aug 2024 00:00:00 +0000

Explanation to my riddle¶

Tuesday, August 20, 2024

I found the explanation to my riddle (Grid cells don’t start editing when I click on them).

When Sharif built the front end in June, he had of course an older version of some dependency. The package-lock.json file is not included in the git repo (because otherwisde we would also have to include the node_modules), but I hope that webpack somehow writes the list of included dependencies and their versions somewhere in the distributed output.

No, it seems that there is no such file. I might write it myself by extending our npm run build command to say something like “npm ls > … versions.txt” when making a build.

So I have to guess. And since the Lino grid view is made using PrimeReact datatable, let’s start with downgrading PrimeReact.

Indeed, according to the PrimeReact changelog, 10.6.6 was the latest PR version on June 17 when Sharif built the React front end that is currently visible in the repository.

Using npm ls I can see which version is being used on my machine:

(dev) luc@yoga:~/work/react$ npm ls
lino_react@24.5.0 /home/luc/work/react
├── @babel/cli@7.24.8
...
├── primereact@10.8.2
...

The difference comes because in the package.json we have:

"primereact": "latest",

I changed that line into:

"primereact": "^10.6.6",

Afterwards I also had to repeat myself, saying:

npm install primereact@10.6.6

And when I now build the front end, the cell editor is back! So the explanation is that meanwhile PrimeReact has evolved.

Of course this explanation is not yet a solution for our problem. We are not going to freeze Lino onto version 10.6.6 of PR, are we? We must rather adapt Lino to work with the latest PR version.

So the next step is to carefully read the PrimeReact release notes and find out which change caused things to brake. After reading the release notes quickly, I realize that this won’t be trivial. I need to dive into how we use the PR library.

Grid cells don’t start editing when I click on them

Sat, 17 Aug 2024 00:00:00 +0000

Grid cells don’t start editing when I click on them¶

Saturday, August 17, 2024

The npm run build command now works as expected, I made some easy changes and can see the resulting behaviour in the version built on my machine. Here are my first changes:

In components/ActionHandler.tsx I changed the message “No modified data detected!”, replacing the exclamation mark (“!”) by a period (“.”). I also ran npm run mm, which caused the files below lino_react/translations/extracts/i18n to get updated.
In components/AppInlineProfile.jsx and components/constants.js I removed the “Pinned URL(s)” button and related code. Sharif had added this feature recently, but I don’t like it. The basic idea is great, but I believe that we should implement it on the server side and store the pinned urls in the database.

But I am having yet another riddle before I can say that I understood how to build the React front end. One detail is broken: the grid cell editor doesn’t start when I click on a cell. I even undid my easy changes and returned back to exactly the source code as in the gitlab repo, but the grid cell remains unimpressed by my mouse clicks. No message in the JS console.

The main page

Thu, 15 Aug 2024 00:00:00 +0000

The main page¶

Thursday, August 15, 2024

The main page of every Lino application is returned by the lino_react.views.Index view (source), which just renders the react/main.html template using Jitsi. This rendering happens at runtime for each incoming HTTP request.

The react/main.html template is itself rendered from another template file lino_react/react/components/index.html (source). But this rendering is done only once, by webpack, during npm build. The two only statements replaced by webpack are <%= webpack_comment %> and <%= webpack_injects %>.

The <%= webpack_comment %> statement expands into a comment reminding us to not modify this file directly, the <%= webpack_injects %> statement expands into something like this:

<script defer src="{{site.build_static_url('react/main.runtime.ddd08114b7e8cedd5863.js')}}"></script>
<script defer src="{{site.build_static_url('react/main.2454e643ddf1a330e98b.js')}}"></script>

The defer attribute means that the script should be downloaded in parallel to parsing the page, and executed after the page has finished parsing.

More precisely, above rendering is done by the HtmlWebpackPlugin plugin, as specified in the webpack.config.js. We use this plugin mostly because our webpack bundle includes a hash in the filename, which changes every compilation.

Now that we have explored the “administrative” aspects, let’s have a look at the the actually “relevant” parts. The body of the main page is very short:

<body>
  <noscript>You need to enable JavaScript to run this app.</noscript>
  <div id="root">
    <div class="splash-screen">
      <div class="splash-container">
        <div class="splash-double-bounce1"></div>
        <div class="splash-double-bounce2"></div>
      </div>
    </div>
  </div>
</body>

There is also the following little chunk:

<script>
    window.Lino = Object.assign(window.Lino || {}, {
        lv: {{ lino_version }},
        site_name: "{{ site_name }}",
    });
</script>

(To be continued)

Wednesday, August 7, 2024

Wed, 07 Aug 2024 00:00:00 +0000

Wednesday, August 7, 2024¶

I read npm vs npx — What’s the Difference?, here are my reading notes.

npx cowsay Hello works as described in the tutorial. It says:

 _______
< Hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

And npx lino-react, as expected, doesn’t work, it says “npm error could not determine executable to run”, which is normal because the React front end is not an executable command-line tool, it is a Python package to be used as the lino.core.site.Site.default_ui.

The next step in above tutorial is npx create-react-app:

$ cd ~/work
$ npx create-react-app

Creating a new React app in /home/luc/work/my-app.
Installing packages. This might take a couple of minutes.
Installing react, react-dom, and react-scripts with cra-template...
added 1483 packages in 35s
261 packages are looking for funding
  run `npm fund` for details
Installing template dependencies using npm...
added 54 packages in 3s
261 packages are looking for funding
  run `npm fund` for details
Removing template package using npm...
removed 1 package, and audited 1537 packages in 2s
261 packages are looking for funding
  run `npm fund` for details
8 vulnerabilities (2 moderate, 6 high)
To address all issues (including breaking changes), run:
  npm audit fix --force
Run `npm audit` for details.
Success! Created my-app at /home/luc/work/my-app
Inside that directory, you can run several commands:

  npm start
    Starts the development server.

  npm run build
    Bundles the app into static files for production.

  npm test
    Starts the test runner.

  npm run eject
    Removes this tool and copies build dependencies, configuration files
    and scripts into the app directory. If you do this, you can’t go back!

We suggest that you begin by typing:

  cd my-app
  npm start

Happy hacking!

But when I do as instructed, I get an error message:

$ cd my-app
$ npm start
> my-app@0.1.0 start
> react-scripts start

Cannot find module 'ajv/dist/compile/codegen'
Require stack:
- /home/luc/work/my-app/node_modules/ajv-keywords/dist/definitions/typeof.js
- /home/luc/work/my-app/node_modules/ajv-keywords/dist/keywords/typeof.js
- /home/luc/work/my-app/node_modules/ajv-keywords/dist/keywords/index.js
- /home/luc/work/my-app/node_modules/ajv-keywords/dist/index.js
- /home/luc/work/my-app/node_modules/schema-utils/dist/validate.js
- /home/luc/work/my-app/node_modules/schema-utils/dist/index.js
- /home/luc/work/my-app/node_modules/webpack-dev-server/lib/Server.js
- /home/luc/work/my-app/node_modules/react-scripts/scripts/start.js

I had to manually say npm install --save-dev ajv to get it to work. Now it works:

Compiled successfully!

You can now view my-app in the browser.

  Local:            http://localhost:3000
  On Your Network:  http://192.168.1.115:3000

Note that the development build is not optimized.
To create a production build, use npm run build.

webpack compiled successfully

And the browser launches and shows the screen described in the tutorial.

But on the console I see the following warnings:

One of your dependencies, babel-preset-react-app, is importing the
"@babel/plugin-proposal-private-property-in-object" package without
declaring it in its dependencies. This is currently working because
"@babel/plugin-proposal-private-property-in-object" is already in your
node_modules folder for unrelated reasons, but it may break at any time.

babel-preset-react-app is part of the create-react-app project, which
is not maintianed anymore. It is thus unlikely that this bug will
ever be fixed. Add "@babel/plugin-proposal-private-property-in-object" to
your devDependencies to work around this error. This will make this message
go away.

The create-react-app project is not maintained any more?! That sounds strange! Maybe the freecodecamp.org tutorial is obsolete? But also the “official” Getting Started page recommends the same steps.

The create-react-app project is hosted on GitHub, so I started to file an issue. It asks me what type of issue: a bug report.

The issue template recommends to do:

$ sudo npm install -g npm@latest

Maybe this is the source of all my problems! I ran it, and it said:

changed 13 packages in 3s
22 packages are looking for funding
run `npm fund` for details

But the issue remained the same.

So today I submitted my first issue to the facebook group on GitHub:

https://github.com/facebook/create-react-app/issues/13643

I ran npm uninstall -g html-webpack-plugin n react-dom typedoc to remove everything except corepack and npm from my global environment:

$ npm ls -g
/usr/local/lib
├── corepack@0.28.1
└── npm@10.8.2

TODO: The second warning above (“babel-preset-react-app is part of the create-react-app project, which is not maintianed anymore….”) seems to be a false alert. That might become my second issue. Furthermore it contains a type “maintianed” instead of “maintained”.

Tuesday, August 6, 2024

Tue, 06 Aug 2024 00:00:00 +0000

Tuesday, August 6, 2024¶

Continued from Monday, August 5, 2024.

I started to read introductions to Webpack: Getting Started and Core Concepts.

I also continued to update Introduction to npm, npx, NodeJS and React.

Ha! I’m unlocked after two weeks of feeling stuck! Tonis donated half an hour of his spare time and had a look with me at my problem. Not even a video meeting, just a text chat. And he found the missing piece!

The missing piece was this: We can assume that these error messages are actually not relevant, at least not immediately, they don’t prevent the build from happening, they are experimental and Sharif probably also had them. I believed that they prevented the build from happening because npm run build didn’t create any static files. But that is normal because there is no actual change in the source code. After adding a “foo bar” string somewhere in the toolbar, npm run build did create a whole series of files in the lino_react/react/static directory.

Okay I’m unlocked, but I still have quite a few things to learn before I’ll get productive :-)

Q: When Sharif did his last change on June 16, there were 6 static files changed. Why do I now have, in my lino_react/react/static, 41 untracked files named main.*.js plus a .js.map for each of them?

A: That’s because I ran “npm audit fix –force”, to “address all issues (including breaking changes)”, which updated a series of packages.

Q: I tried to verify this answer by going back to the current GitLab version, adding again that “foo bar” string somewhere in the toolbar, run npm run build. I should see a few files changes. But nothing changes. Why?

A: Maybe because the audit report has critical issues, it says “12 vulnerabilities (2 moderate, 4 high, 6 critical)”.

So I ran npm audit fix --force again:

$ npm audit fix --force
npm warn using --force Recommended protections disabled.
npm warn audit Updating quill to 2.0.2, which is a SemVer major change.
npm warn audit Updating webpack-typescript to 0.5.5, which is a SemVer major change.
npm warn audit Updating quill-mention to 6.0.1, which is a SemVer major change.
npm warn audit Updating babel-plugin-transform-class-properties to 6.10.2, which is a SemVer major change.
npm warn audit Updating puppeteer to 22.15.0, which is a SemVer major change.

added 8 packages, removed 31 packages, changed 15 packages, and audited 1684 packages in 4m

208 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

And now npm run build should build again those 41 static .js files.

But it doesn’t! WTF!? I restored the same situation as two hours ago! Why did it stop working?

Monday, August 5, 2024

Mon, 05 Aug 2024 00:00:00 +0000

Monday, August 5, 2024¶

Having no news from Sharif (see Sharif, where are you?) is a good reason for me to seriously dive into React, npm, webpack, jest and all these tools.

I still didn’t get npm run build to execute (see Monday, July 29, 2024), so I tried to grasp the problem from the roots: I read a series of basic introductions to React and JSX:

What I don’t understand: How does a browser “know” whether a script is written in JSX and not in plain JavaScript?

The story continues on Tuesday, August 6, 2024.

Sharif, where are you?

Fri, 02 Aug 2024 00:00:00 +0000

Sharif, where are you?¶

Friday, August 2, 2024

The Lino community is currently worried about Sharif, who has been our second core developer since February 2021, for more than 3.5 years.

On Friday 2024-06-28 he left his home by motorcycle to visit a friend, and since then his cell phone is switched off. His family filed a missing person report at the police.

We hope that he had just a psychological crisis and decided to escape from everyday life. He is indeed an introvert person with tendencies to melancholy. It seems that people have seen him in Dhaka (the capital, quite far away from his home) and since then his family are angry with him because disappearing without explanations is obviously not a nice behaviour.

I have no contact with him. I know about all this only from chats with his wife, his sister and some friends. The last times he registered work sessions in Jane was 2024-06-22 (0:30 hours), 2024-06-21 (0:30 h) and 2024-06-16 (3:30 h).

I am pretty sure that Sharif’s disappearing has nothing to do with the political tensions in Bangladesh (see the news, e.g. Bangladesh bans Jamaat-e-Islami party following violent protests that left more than 200 dead).

Q: Are you having any problem without Sharif? Who does his work now?

A: Oh yes I am having problems without him! He was the main responsible for developing the Lino React front end, which is written in JavaScript, which is quite different from Python. I was glad to delegate this task to him. And now I need to dive into things like Node.js, npm or webpack. Fortunately there is no critical issue to fix, so I can take it calmly. My biggest problem is psychological: I am still under shock, mentally paralyzed. This story shakes my visions about remote asynchronous teamwork. There are moments where I am tempted to give up. But I won’t do that because Lino is my life’s work.

Q: Will he be able to work in your company again if he comes back after a long time?

A: Depends on how long. Right now it would still be possible. He owes us more than 5 weeks of work. So if he reappears, he will have to work some time before we pay him new money! I don’t plan to engage anybody else. But of course I don’t know the future.

If he comes back, we will learn from this and find rules to ensure a minimum of discipline. It is important that paid workers get enough pressure to work the agreed number of hours per week.

Let’s hope that he will soon reappear and give explanations, that his wife can forgive him and that this whole story turns out as an unpleasant adventure with a good end, Inshallah.

P.S.: See Sharif is back

Monday, July 29, 2024

Mon, 29 Jul 2024 00:00:00 +0000

Monday, July 29, 2024¶

Continued from Saturday, July 20, 2024.

Here is my current output of npm run build:

$ npm run build

> lino_react@24.5.0 build
> NODE_OPTIONS='--max-old-space-size=8192' webpack --mode production

assets by status 3.81 MiB [cached] 51 assets
Entrypoint main = main.runtime.c483990e94f9f59f6c76.js main.2454e643ddf1a330e98b.js 2 auxiliary assets
orphan modules 1.24 MiB [orphan] 715 modules
runtime modules 12.2 KiB 14 modules
cacheable modules 5.02 MiB (javascript) 241 KiB (asset)
  modules by path ./node_modules/ 4.53 MiB (javascript) 241 KiB (asset)
    javascript modules 4.53 MiB 221 modules
    asset modules 210 bytes (javascript) 241 KiB (asset)
      ./node_modules/primeicons/fonts/primeicons.eot 42 bytes (javascript) 70.7 KiB (asset) [built] [code generated]
      ./node_modules/primeicons/fonts/primeicons.woff2 42 bytes (javascript) 29.5 KiB (asset) [built] [code generated]
      + 3 modules
  modules by path ./lino_react/react/ 494 KiB 38 modules
  modules by mime type image/png 357 bytes
    data:image/png;base64,iVBORw0KGgoAAAAN.. 179 bytes [built] [code generated]
    data:image/png;base64,iVBORw0KGgoAAAAN.. 178 bytes [built] [code generated]

LOG from InjectManifest
<i> The service worker at ../../config/react/service-worker.js will precache
<i>         0 URLs, totaling 0 B.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
7:7-17
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(7,8)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
8:7-19
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(8,8)
      TS2339: Property 'WAIT_TIMEOUT' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
10:7-17
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(10,8)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
12:44-56
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(12,45)
      TS2339: Property 'WAIT_TIMEOUT' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
21:7-11
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(21,8)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
23:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(23,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
29:21-25
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(29,22)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
30:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(30,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
35:7-13
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(35,8)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
36:17-21
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(36,18)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
38:17-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(38,18)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
54:17-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(54,18)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
61:17-21
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(61,18)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
9:42-50
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(9,43)
      TS2322: Type '"new"' is not assignable to type 'boolean | "shell"'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
14:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(14,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
15:21-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(15,22)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
19:21-25
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(19,22)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
20:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(20,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
35:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(35,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
9:42-50
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(9,43)
      TS2322: Type '"new"' is not assignable to type 'boolean | "shell"'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
14:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(14,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
18:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(18,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
19:21-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(19,22)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

23 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

webpack 5.93.0 compiled with 23 errors in 15121 ms

It seems that I need to learn more about webpack and jest.

The story continues on Monday, August 5, 2024.

Tuesday, July 23, 2024

Tue, 23 Jul 2024 00:00:00 +0000

Tuesday, July 23, 2024¶

Jason Cook made yet another pull request for sphinxfeed, which I accepted. Jason has obviously more experience with pytest than me.

When I was exploring the sphinx.testing.utils.SphinxTestApp, I stumbled into the following comment:

# see https://github.com/sphinx-doc/sphinx/pull/12089 for the
# discussion on how the signature of this class should be used

I have a problem with this because it because it causes a part of the Sphinx documentation to be copyrighted by GitHub. It triggers my Daniel syndrome.

Saturday, July 20, 2024

Sat, 20 Jul 2024 00:00:00 +0000

Saturday, July 20, 2024¶

Continued from Wednesday, July 17, 2024.

Here is my current problem when I try to run npm run build:

(dev) luc@yoga:~/work/react$ npm run build

> lino_react@24.5.0 build
> NODE_OPTIONS='--max-old-space-size=8192' webpack --mode production

assets by status 3.82 MiB [cached] 50 assets
Entrypoint main = main.runtime.430b48360e2c8f3437a2.js main.30cfd4b61f5cb674be23.js 2 auxiliary assets
orphan modules 438 KiB [orphan] 58 modules
runtime modules 12.1 KiB 14 modules
modules by path ./node_modules/ 4.48 MiB (javascript) 241 KiB (asset)
  javascript modules 4.48 MiB 165 modules
  asset modules 210 bytes (javascript) 241 KiB (asset)
    ./node_modules/primeicons/fonts/primeicons.eot 42 bytes (javascript) 70.7 KiB (asset) [built] [code generated]
    ./node_modules/primeicons/fonts/primeicons.woff2 42 bytes (javascript) 29.5 KiB (asset) [built] [code generated]
    + 3 modules
modules by path ./lino_react/react/ 494 KiB 38 modules
modules by mime type image/png 357 bytes
  data:image/png;base64,iVBORw0KGgoAAAAN.. 179 bytes [built] [code generated]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 178 bytes [built] [code generated]

LOG from InjectManifest
<i> The service worker at ../../config/react/service-worker.js will precache
<i>         0 URLs, totaling 0 B.

ERROR in ./lino_react/react/components/App.jsx 11:7-72
Module not found: Error: Can't resolve 'react-dom/client' in '/home/luc/work/react/lino_react/react/components'
Did you miss the leading dot in 'resolve.extensions'? Did you mean '[".tsx",".js",".json",".html",".ts",".jsx",".css",".mjs",".bmp",".gif",".jpg",".jpeg",".png",".woff",".woff2",".eot",".ttf",".otf"]' instead of '[".tsx",".js",".json",".html",".ts",".jsx",".css",".mjs",".bmp",".gif",".jpg",".jpeg",".png","woff","woff2","eot","ttf","otf"]'?
 @ ./lino_react/react/index.js 1:0-42 2:0-6

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
7:7-17
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(7,8)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
8:7-19
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(8,8)
      TS2339: Property 'WAIT_TIMEOUT' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
10:7-17
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(10,8)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
12:44-56
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(12,45)
      TS2339: Property 'WAIT_TIMEOUT' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
21:7-11
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(21,8)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
23:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(23,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
29:21-25
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(29,22)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
30:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(30,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
35:7-13
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(35,8)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
36:17-21
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(36,18)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
38:17-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(38,18)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
54:17-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(54,18)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts
61:17-21
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/setupTests.ts(61,18)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
14:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(14,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
15:21-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(15,22)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
19:21-25
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(19,22)
      TS2339: Property 'wait' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
20:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(20,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts
35:21-31
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/cloneUI.ts(35,22)
      TS2339: Property 'waitToMeet' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
14:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(14,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
18:31-41
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(18,32)
      TS2339: Property 'SERVER_URL' does not exist on type 'Global & typeof globalThis'.

ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts
19:21-27
[tsl] ERROR in /home/luc/work/react/lino_react/react/components/__tests__/integrity.ts(19,22)
      TS2339: Property 'signIn' does not exist on type 'Global & typeof globalThis'.

22 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

webpack 5.93.0 compiled with 22 errors in 15425 ms

The message “Did you mean ‘[“.tsx”,”.js”,”.json”,”.html”,”.ts”,”.jsx”,”.css”,”.mjs”,”.bmp”,”.gif”,”.jpg”,”.jpeg”,”.png”,”.woff”,”.woff2”,”.eot”,”.ttf”,”.otf”]’ instead of ‘[“.tsx”,”.js”,”.json”,”.html”,”.ts”,”.jsx”,”.css”,”.mjs”,”.bmp”,”.gif”,”.jpg”,”.jpeg”,”.png”,”woff”,”woff2”,”eot”,”ttf”,”otf”]’?” made me change the webpack.config.js file where indeed the last five extensions didn’t start with a dot. Seems that this was just an irrelevant warning.

The packages.json talks much about “electron”. What’s that? Google gave me https://www.electron.build/ and I tried to install it:

yarn add electron-builder --dev

Lots of output, here are some chunks:

! Corepack is about to download https://registry.yarnpkg.com/yarn/-/yarn-1.22.22.tgz
? Do you want to continue? [Y/n] Y

! The local project doesn't define a 'packageManager' field. Corepack will now add one referencing yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e.
! For more details about this field, consult the documentation at https://nodejs.org/api/packages.html#packagemanager

yarn add v1.22.22
info No lockfile found.
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
...

[3/4] Linking dependencies...
warning " > autoprefixer@10.4.19" has unmet peer dependency "postcss@^8.1.0".
warning " > ts-node@10.9.2" has unmet peer dependency "@types/node@*".
warning "electron-builder > app-builder-lib@24.13.3" has unmet peer dependency "electron-builder-squirrel-windows@24.13.3".
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 885 new dependencies.
info Direct dependencies
├─ @babel/cli@7.24.8
├─ @babel/core@7.24.9
├─ @babel/plugin-transform-runtime@7.24.7
...

├─ yauzl@2.10.0
├─ yn@3.1.1
└─ yocto-queue@0.1.0
Done in 152.53s.

The warning “The local project doesn’t define a ‘packageManager’ field. Corepack will now add one…” makes me think that this is probably not the right way. If Sharif had been using electron builder, the packageManager field would already exist. So I removed that field again.

I tried “npm audit fix”:

$ npm audit fix

up to date, audited 1831 packages in 2s

215 packages are looking for funding
  run `npm fund` for details

# npm audit report

babel-traverse  *
Severity: critical
Babel vulnerable to arbitrary code execution when compiling specifically crafted malicious code - https://github.com/advisories/GHSA-67hx-6x53-jw92
fix available via `npm audit fix --force`
Will install babel-plugin-transform-class-properties@6.10.2, which is a breaking change
node_modules/babel-traverse
  babel-helper-function-name  *
  Depends on vulnerable versions of babel-template
  Depends on vulnerable versions of babel-traverse
  node_modules/babel-helper-function-name
    babel-plugin-transform-class-properties  >=6.11.5
    Depends on vulnerable versions of babel-helper-function-name
    Depends on vulnerable versions of babel-template
    node_modules/babel-plugin-transform-class-properties
  babel-template  *
  Depends on vulnerable versions of babel-traverse
  node_modules/babel-template

json5  <1.0.2
Severity: high
Prototype Pollution in JSON5 via Parse Method - https://github.com/advisories/GHSA-9c47-m6qq-7p4h
fix available via `npm audit fix --force`
Will install webpack-typescript@0.5.5, which is a breaking change
node_modules/loader-utils/node_modules/json5
  loader-utils  <=1.4.0
  Depends on vulnerable versions of json5
  node_modules/loader-utils
    webpack-typescript  >=0.5.6
    Depends on vulnerable versions of loader-utils
    node_modules/webpack-typescript


quill  <=1.3.7
Severity: moderate
Cross-site Scripting in quill - https://github.com/advisories/GHSA-4943-9vgg-gr5r
fix available via `npm audit fix --force`
Will install quill@2.0.2, which is a breaking change
node_modules/quill
  quill-mention  <=4.1.0
  Depends on vulnerable versions of quill
  node_modules/quill-mention

ws  8.0.0 - 8.17.0
Severity: high
ws affected by a DoS when handling a request with many HTTP headers - https://github.com/advisories/GHSA-3h5v-q93c-6h6q
fix available via `npm audit fix --force`
Will install puppeteer@22.13.1, which is a breaking change
node_modules/puppeteer-core/node_modules/ws
  puppeteer-core  11.0.0 - 22.11.1
  Depends on vulnerable versions of ws
  node_modules/puppeteer-core
    puppeteer  18.2.0 - 22.11.1
    Depends on vulnerable versions of puppeteer-core
    node_modules/puppeteer

12 vulnerabilities (2 moderate, 4 high, 6 critical)

To address all issues (including breaking changes), run:
  npm audit fix --force

I ran “npm audit fix –force” several times, the first time it gave a lot of output similar to above, but after two or three runs it got kind of satisfied:

$ npm audit fix --force
npm warn using --force Recommended protections disabled.
npm warn audit Updating babel-plugin-transform-class-properties to 6.10.2, which is a SemVer major change.

removed 20 packages, changed 2 packages, and audited 1796 packages in 3s

215 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

$ npm audit fix --force
npm warn using --force Recommended protections disabled.

up to date, audited 1796 packages in 2s

215 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

I wondered whether it is maybe actually working. The only errors are in lino_react/react/components/setupTests.ts, it’s possible that that they aren’t important. Sharif started to experiment with testing frameworks only recently.

But no. Because the files under :file: lino_react/react/static haven’t changed. I also made a minor change to one source file and (LinoToolbar.jsx), ran npm run build again: still no change in the static directory. So it’s sure that I need to also get rid of the remaining error messages.

The story continues on Monday, July 29, 2024.

Friday, July 19, 2024

Fri, 19 Jul 2024 00:00:00 +0000

Friday, July 19, 2024¶

Today I ran inv mm in lino, obviously for the first time since 20240521, because the Spanish .po file was broken, it contained hundreds of Git conflict markers. I guess that something went wrong with Joy’s first commit.

I wrote a little Python script to remove the second part of every chunk:

import sys
from pathlib import Path

mark_start = "<<<<<<< HEAD\n"
mark_middle = "=======\n"
mark_end = ">>>>>>> upstream/master\n"

def remove_upstream(s):
    chunks1 = s.split(mark_start, 1)
    if len(chunks1) != 2:
        return s
    s1 = chunks1[0]
    chunks2 = chunks1[1].split(mark_middle, 1)
    if len(chunks2) != 2:
        return s
    chunks3 = chunks2[1].split(mark_end, 1)
    if len(chunks3) != 2:
        return s
    return chunks1[0] + chunks3[0] + chunks3[1]

def remove_all(s):
    while True:
        s2 = remove_upstream(s)
        if s2 == s:
            return s
        s = s2

def main():
    if len(sys.argv) < 2:
        print("You must specify at least one file to repair")
        sys.exit(-1)
    for fn in sys.argv[1:]:
        pth = Path(fn)
        s = pth.read_text()
        s = remove_all(s)
        pth.write_text(s)

if __name__ == "__main__":
    main()

Wednesday, July 17, 2024

Wed, 17 Jul 2024 00:00:00 +0000

Wednesday, July 17, 2024¶

We are worried about Sharif. He left his home on June 28 to visit a friend, but since then he has vanished, his mobile phone is not reachable. His sister and his wife filed a missing person report at the police. I still hope that he had just a psycholigical crisis or problems with his wife (they married only 1 year ago) and doesn’t dare to speak about it. That would be a strange behaviour, but I’d prefer such an explanation over hearing that they found his motorcycle and his body somewhere in the jungle.

We are in the third week without Sharif, I start to imagine the worst case: that I must learn how to build the React front end. Until now I had successfully avoided the dirty details of Node, npm, React and webpack because Sharif was better than me with this part of Lino development.

Even when we assume that Sharif will soon come back, it’s a good thing that I am forced to do his job for some time. I started to read and review Introduction to npm, npx, NodeJS and React.

Of course my npm environment needs a upgrade:

$ npm -v
8.5.1
$ node -v
v12.22.9

$ sudo npm install -g n
[sudo] password for luc:

added 1 package, and audited 2 packages in 560ms

found 0 vulnerabilities

$ sudo n stable
  installing : node-v20.15.1
       mkdir : /usr/local/n/versions/node/20.15.1
       fetch : https://nodejs.org/dist/v20.15.1/node-v20.15.1-linux-x64.tar.xz
     copying : node/20.15.1
   installed : v20.15.1 (with npm 10.7.0)

Note: the node command changed location and the old location may be remembered in your current shell.
         old : /usr/bin/node
         new : /usr/local/bin/node
If "node --version" shows the old version then start a new shell, or reset the location hash with:
hash -r  (for bash, zsh, ash, dash, and ksh)
rehash   (for csh and tcsh)

$ npm -v
10.7.0
$ node -v
v20.15.1

I also did the following (don’t ask me why):

$ sudo npm upgrade -g react

Let’s try:

$ npm run build

> lino_react@24.5.0 build
> NODE_OPTIONS='--max-old-space-size=8192' webpack --mode production

[webpack-cli] Failed to load '/home/luc/work/react/webpack.config.js' config
[webpack-cli] Error: Cannot find module 'html-webpack-plugin'
Require stack:
- /home/luc/work/react/webpack.config.js
- /home/luc/work/react/node_modules/webpack-cli/lib/webpack-cli.js
- /home/luc/work/react/node_modules/webpack-cli/lib/bootstrap.js
- /home/luc/work/react/node_modules/webpack-cli/bin/cli.js
- /home/luc/work/react/node_modules/webpack/bin/webpack.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Module.require (node:internal/modules/cjs/loader:1233:19)
    at require (/home/luc/work/react/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at Object.<anonymous> (/home/luc/work/react/webpack.config.js:2:27)
    at Module._compile (/home/luc/work/react/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Module.require (node:internal/modules/cjs/loader:1233:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/luc/work/react/webpack.config.js',
    '/home/luc/work/react/node_modules/webpack-cli/lib/webpack-cli.js',
    '/home/luc/work/react/node_modules/webpack-cli/lib/bootstrap.js',
    '/home/luc/work/react/node_modules/webpack-cli/bin/cli.js',
    '/home/luc/work/react/node_modules/webpack/bin/webpack.js'
  ]
}

The message “Cannot find module ‘html-webpack-plugin’” makes me try the following:

$ sudo npm install -g html-webpack-plugin

added 44 packages in 2s

10 packages are looking for funding
  run `npm fund` for details

I am not sure about whether the difference between “sudo npm install -g” and “npm install” is important.

This story continues on Saturday, July 20, 2024.

Review the API for get_table_summary()

Sat, 13 Jul 2024 00:00:00 +0000

Review the API for get_table_summary()¶

I did #5685 (Review the API for get_table_summary(). The following tables had a customized Actor.get_table_summary() method, which I converted to Actor.table_as_summary():

lino_avanti.lib.courses.PresencesByEnrolment
lino.mixins.dupable.SimilarObjects
lino.modlib.dupable.SimilarObjects
lino.modlib.notify.MyMessages
lino_xl.lib.accounting.JournalsOverview
lino_xl.lib.accounting.VouchersByPartnerBase
lino_xl.lib.accounting.MovementsByPartner
lino_xl.lib.blogs.EntriesByController (currently not used anywhere)
lino_xl.lib.cal.Guests and lino_xl.lib.cal.Events use the same code.
lino_xl.lib.cv.HistoryByPerson
lino_xl.lib.cv.LanguageKnowledgesByPerson
lino_xl.lib.households.SiblingsByPerson
lino_xl.lib.households.MembersByPerson
lino_xl.lib.humanlinks.SiblingsByPerson (including an optimization of the presentation)
lino_xl.lib.phones.ContactDetailsByPartner
lino_xl.lib.tickets.TicketsByEndUser (which had a bug that was not covered; I fixed the bug and added coverage)
lino_xl.lib.working.SessionsByTicket

The following table summaries: also have a customized Actor.get_table_summary() method, but I dit not yet convert it to an Actor.table_as_summary():

lino.modlib.uploads.AreaUploads (I prefer to leave this for later)
lino_xl.lib.deploy.DeploymentsByTicket (not used anywhere)
lino_xl.lib.excerpts.ExcerptsByOwner is not used anywhere directly, but inherited by lino_xl.lib.excerpts.ExcerptsByProject. I wonder whether the current implementation is useful in real life.
lino_xl.lib.polls.AnswersByResponseEditor, lino_xl.lib.polls.AnswersByResponsePrint, lino_xl.lib.polls.ResponsesByPartner, lino_xl.lib.skills.DemandsByDemander, lino_xl.lib.skills.OffersByDemander and a few more.

Side effects:

lino.utils.AttrDict now supports setting an attribute.
The Model.overview field didn’t yet support getting a safe html string from Model.get_overview_elems().
New plugin setting printing.print_demo_objects and model attribute lino.modlib.printing.Printable.print_demo_objects.
I started to move away from using the ar.html_text() method. Under ExtJS this method wraps the given html fragment into a <div class="htmlText">, which specifies that this fragment contains simple html text inside an ExtJS component. ExtJS does a lot of CSS magic to neutralize the “usual” effects of most html tags. This htmlText class is required to restore a half-way normal look. Under any other front end the ar.html_text() method does nothing. My plan is to add the class="htmlText" unconditionally, also under React.

I reviewed Customize the summary view.

Blog entry started on Friday, July 12, 2024. Changes are in lino, xl, noi, cosi, care, avanti, tera, book, welfare and amici.

Thursday, July 4, 2024

Thu, 04 Jul 2024 00:00:00 +0000

Thursday, July 4, 2024¶

I did some more tests for #5670 (Electronic sales invoices) and started a topics page about Peppol support (e-Invoicing) in Lino.

The following code snippet shows that pyvat knows Belgian VAT numbers but not Belgian company numbers. I use a selection of companies found on companyweb.be.

>>> import pyvat
>>> from stdnum.be import vat
>>> examples = [
... ["Abakus It-Solutions Belgium BV", '0443493896', 'BE'],
... ["BSDG V.o.G.", '0441635753', 'BE'],
... ["Assurances DEKEMPENER NV", '0437272535', 'BE']]

The first company is subject to VAT, the other two aren’t.

>>> for name, vat_number, country_code in examples:
...     res = pyvat.check_vat_number(vat_number, country_code)
...     print(f"{country_code} {vat_number} {res.is_valid} {res.business_name}")
BE 0443493896 True SRL ABAKUS IT-SOLUTIONS Belgium
BE 0441635753 False None
BE 0437272535 False None

The python-stdnum module validates also the company numbers (but it doesn’t do any online check).

>>> for name, vat_number, country_code in examples:
...     print(vat.validate(vat_number))
0443493896
0441635753
0437272535

I currently assume that the schemeID

Saturday, June 29, 2024

Sat, 29 Jun 2024 00:00:00 +0000

Saturday, June 29, 2024¶

Yesterday I sent a next newsletter to my friends. I manage the recipient list in Thunderbird and set it as the bcc field (because we don’t want everybody to see the other recipients).

My list holds 375 addresses, 43 of which are at skynet.be. And Skynet (which is now a part of Proximus) accepts a maximum number recipients per connection. After the 5th recipient it says “Too many recipients”. But the Postfix on our mail server doesn’t give up. It tries again after 5 minutes, then after ten minutes, after 20 minutes… Here is how that looks in the mail.log file:

$ sudo grep joe.doe /var/log/mail.log
Jun 28 21:57:35 saffre-rumma postfix/smtp[2252573]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.22.25]:25, delay=22, delays=20/0.97/0.14/0.24, dsn=4.0.0, status=deferred (host mx.glb.proximus.be[195.238.22.25] said: 459 Too many recipients (RL) (in reply to RCPT TO command))
Jun 28 22:03:21 saffre-rumma postfix/smtp[2252792]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.20.25]:25, delay=367, delays=367/0.08/0.24/0.19, dsn=4.0.0, status=deferred (host mx.glb.proximus.be[195.238.20.25] said: 459 Too many recipients (RL) (in reply to RCPT TO command))
Jun 28 22:13:21 saffre-rumma postfix/smtp[2253003]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.20.25]:25, delay=967, delays=966/0.04/0.82/0.16, dsn=4.0.0, status=deferred (host mx.glb.proximus.be[195.238.20.25] said: 459 Too many recipients (RL) (in reply to RCPT TO command))
Jun 28 22:33:20 saffre-rumma postfix/smtp[2253231]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.20.25]:25, delay=2166, delays=2166/0.03/0.11/0.13, dsn=4.0.0, status=deferred (host mx.glb.proximus.be[195.238.20.25] said: 459 Too many recipients (RL) (in reply to RCPT TO command))
Jun 28 23:13:21 saffre-rumma postfix/smtp[2253484]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.22.25]:25, delay=4568, delays=4567/0.04/0.44/0.66, dsn=4.0.0, status=deferred (host mx.glb.proximus.be[195.238.22.25] said: 452 #4.5.3 Too many recipients. (in reply to RCPT TO command))
Jun 29 00:23:22 saffre-rumma postfix/smtp[2254227]: 3B984211FB: to=<joe.doe@skynet.be>, relay=mx.glb.proximus.be[195.238.20.25]:25, delay=8768, delays=8767/0.03/0.19/0.47, dsn=2.0.0, status=sent (250 ok:  Message 3727612 accepted)

Thursday, June 27, 2024

Thu, 27 Jun 2024 00:00:00 +0000

Thursday, June 27, 2024¶

Content moved to Peppol support (e-Invoicing) in Lino.

Monday, June 24, 2024

Mon, 24 Jun 2024 00:00:00 +0000

Monday, June 24, 2024¶

I would like to validate the e-invoices generated by my Python program. The Peppol BIS Billing 3.0 - November 2023 Release page provides two Schematron files for download. And LXML knows how to validate an XML file using Schematron files. So theoretically everything is clear…

… but both Schematron files from docs.peppol.eu fail to load into LXML. Here is what I tried.

First I wrote a utility function that downloads the file and then instantiates an LXML schematron schema from it.

>>> import requests
>>> from lxml import isoschematron
>>> def load(url, target):
...     r = requests.get(url + target, stream=True)
...     with open(target, "wb") as f:
...         for chunk in r.iter_content(chunk_size=16 * 1024):
...             f.write(chunk)
...     schematron = isoschematron.Schematron(file=target)

The first schematron file, “Schematron for PEPPOL rules (UBL)” is diagnosed by LXML as being invalid:

>>> load("https://docs.peppol.eu/poacc/billing/3.0/files/", "PEPPOL-EN16931-UBL.sch")
Traceback (most recent call last):
...
lxml.etree.SchematronParseError: invalid schematron schema: <string>:498:0:ERROR:RELAXNGV:RELAXNG_ERR_ELEMNAME: Expecting element pattern, got let
<string>:0:0:ERROR:RELAXNGV:RELAXNG_ERR_INTEREXTRA: Extra element let in interleave
<string>:8:0:ERROR:RELAXNGV:RELAXNG_ERR_ELEMNAME: Expecting element ns, got title
<string>:7:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element schema failed to validate content

The second schematron file, “Schematron for TC434 rules (UBL)” seems to be using the “xslt2” query language:

>>> load("https://docs.peppol.eu/poacc/billing/3.0/files/", "CEN-EN16931-UBL.sch")
Traceback (most recent call last):
...
lxml.etree.XSLTApplyError: Fail: This implementation of ISO Schematron does not work with
    schemas using the "xslt2" query language.

Friday, June 21, 2024

Fri, 21 Jun 2024 00:00:00 +0000

Friday, June 21, 2024¶

Today I fixed about 6 bugs and documentation issues. None of them were revolutionary.

getlino failed when no `www-data` group exists ¶

Joy reported that getlino : the Lino installer breaks in a developer environment when no user group www-data exists. So I made it more resistant: getlino.utils.Installer.check_permissions() no longer tries to change the group owner of the file to www-data when no such group exists. More precisely www-data is the group name specified in usergroup. The user group is needed only on a production site with a web-server.

>>> from getlino.utils import grpexists
>>> help(grpexists)
Help on function grpexists in module getlino.utils:

grpexists(name)
    Check whether the user group with the given ``name`` exists.

>>> grpexists("foo")
False
>>> grpexists("luc")
True
>>> grpexists("www-data")
True

Cannot edit “today” date of a user plan ¶

The today field of a UserPlan is no longer read-only (editable=False) because Lino no longer automatically sets it each time you open your invoicing plan. When you start an invoicing plan today and finish it tomorrow, you might not want Lino to update that field, e.g. because you want all generated documents on the same date. But if you started a service report three weeks ago, got interrupted and now continue, you want to have the real date in this field.

#5671 (bad argument type: proxy([some heading of a dashboard item]))¶

I have seen this error before. Here is how to get this error message:

>>> from lino import startup
>>> startup("lino_book.projects.min2.settings")
>>> from lino.api import _
>>> from lino.utils.html import E
>>> E.h2(_("Foo"))
Traceback (most recent call last):
  File "/usr/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest 0621.rst[9]>", line 1, in <module>
    E.h2(_("Foo"))
  File "src/lxml/builder.py", line 219, in lxml.builder.ElementMaker.__call__
TypeError: bad argument type: __proxy__('Foo')

I am quite sure it is caused by my recent changes with safetext and elementtree. But I don’t see how to reproduce it.

And I guess that it is caused by lino.core.elems.DisplayElement.value2html(), which is indeed a strange piece of code.

I cleaned up the code and changed the call to logger.error() into logger.exception()

#5672 (A sales invoice must DEBIT the account but CREDIT the storage)¶

The reason for this ticket was actually just a configuration problem in Jane.

The title of this ticket is NOT a problem. Whether the storage (more exactly the provision) gets debited or credited depends on the lino_xl.lib.storage.TransferRule.

In lino_xl.lib.accounting DcAmountField now uses DC.normalized_amount() rather than the hard-coded meaning of debit and credit.

“A sales invoice debits the customer’s account” (as explained in the end-user docs), but the Primary booking direction field of a Journal Journal.dc field says “Credit” for sales invoices. I tried whether I can optimize this.

#5673 (Internal Server Error: /api/invoicing/MyPlan/row)¶

invoicing.MyPlan.get_row_by_pk() does not check for permissions because that’s done later during the request, but accesses the database using current user as argument, which fails when the authentication info has been cleared for some reason, causing current user to be an AnonymousUser instance. Django then says Field 'id' expected a number but got <SimpleLazyObject: <lino.core.auth.utils.AnonymousUser object at 0x7f9e52076d90>>.

Multiple invoicing tasks and help_text ¶

When a site has multiple invoicing tasks, Lino adds one “Start invoicing” action for each of them to the partner model and to the invoicing order model. These buttons have the same icon (a shopping basket), but now they have at least different help texts.

Merge obj2href() and as_summary_row() to as_summary_item()

Wed, 12 Jun 2024 00:00:00 +0000

Merge obj2href() and as_summary_row() to as_summary_item()¶

Written between 2024-06-12 and 2024-06-14.

I merged the obj2href() and as_summary_row() methods of Model into a single method named as_summary_item.

They did almost the same, both returned a link (an <a href="...">...</a> fragment) that opens the detail window of this object. One difference between them was that obj2href() returned an ElementTree element while as_summary_row() returned a safe string. The as_summary_item() method returns an ElementTree element. ElementTree is better than safe HTML for specifying small HTML chunks because it is more flexible and more fool-proof (and thus actually safer). And at this level we won’t use any templating nor store the result in a database, so there is no advantage of using safe html. So I converted the as_summary_row() methods from generating safe html to using ElementTree.

The as_summary_item() method expects the text of the link as an optional positional argument after the ar.

Note: We should also rename the obj2html() method. But this is another topic because application developers aren’t supposed to override it. obj2html() just returns a link to

Documentation about this is still poor. I started to move docstrings from lino.core.requests to lino.api.core.Request, but then stopped because this is yet another huge set of changes. I prefer to do this another time.

Usage examples:

lino_xl.lib.cal.Event customizes the title of the link
Polymorphic.obj2href() potentially loops over navigatable MTI models (_mtinav_models)
Voucher.obj2href() finds the MTI leaf (the implementing document type) before linking to it.
Note.obj2href() customizes the default text to use.
Ticket.obj2href() customizes both title and default text.
ProjectRelated.as_summary_row() adds, between parentheses, a clickable link to the related project.
coachings.Coaching.as_summary_row() adds, between parentheses, a clickable link to the related client.
comments.Reaction.as_summary_row()
publisher.Page.as_summary_row()
cal.Component.as_summary_row() adds a bit more information (the start_time, state and summary).
notes.Note.as_summary_row() adds a bit more information (the subject).

Calls:

as_summary_row() is called by

Actor.row_as_summary()
Request.row_as_summary()

obj2href() is called by

TableRow.as_summary_row and get_overview_elems() and as_paragraph()

Changes¶

converted every as_summary_row() to use ElementTree instead of safe html.
Renamed all as_summary_row() and obj2href() to as_summary_item().
as_summary_item() supports a second optional positional argument text
As a consequence, row_as_summary() (on both Actor and Request) now returns an ET element instead of safe html.

TODO: Rename obj2href() to as_summary_item() or something similar, and row_as_summary() to row_as_summary_item(). I leave these renames to a later commit to keep the changes as transparent as possible.

Saturday, June 8, 2024

Sat, 08 Jun 2024 00:00:00 +0000

Saturday, June 8, 2024¶

Today I had a video meeting with Hamza who gave me quick intro to the PoS module of Odoo. Oh yes, there is so much work to do in Lino! Most people might loose courage! But the answer is clear: Odoo is no free software, so it makes no sense to compare. It doesn’t even qualify as a candidate.

It’s sad to think that Odoo was free in its beginnings. Yes, as long as we don’t make sure that our software is forever free, sooner or later it will get jeopardized into a proprietary product because greedy giants have more power than those who care for everybody.

There is Tryton. I had another quick look at it. It seems to do well and they obviously have a better sales team than Lino. But my statement on this is the same as six years ago: We do Lino because we don’t believe that Tryton can replace Lino. Tryton is just a big application with plugins while Lino is a framework. Lino has some unique features and ways of doing things, which differ from Tryton, and I guess (but didn’t verify) that I wouldn’t be satisfied with other approaches. We know that we don’t know it, we just believe it. It would take months of research work to prove this. In case you don’t believe me, try to configure a Tryton site in a way that it has a chance to replace a Lino Presto or any other production Lino application of your choice. If you succeed, you might get me and the whole Lino team join the Tryton project (excerpts from 2018-12-11 and book/docs/tickets/51.rst)

After these lamentations I had another look at our public demo page and the documentation about Noi and Così, and I updated the following pages:

No, it’s not finished (it will never be finished ;-) but it got again a bit better.

A Lino for Alvatal

Sun, 02 Jun 2024 00:00:00 +0000

A Lino for Alvatal¶

June 2–3, 2024

I am working on #5636 (A Lino for Alvatal). After a first brainstorming I assume they need a Lino Noi, but with content management (publisher, blogs albums) and without accounting (invoicing, service reports).

So I created #5637 (New options with_accounting and with_cms for Noi) and added two new plugin settings noi.with_accounting and noi.with_cms. When one of these options is False, then a series of plugins is set to “hidden”:

noi.with_accounting : accounting, vat, trading, …
noi.with_cms : sources, blogs, albums, …

I created a new demo project noi2 : Noi with publisher and without accounting.

Because the concept of hiding plugins is relatively new, this revealed a few places in the core where Lino did not yet manage well with hidden plugins. When a plugin is hidden, then it remains installed (especially for the database schema) but is not visible to the end users.

For example the lino_xl.lib.products.Product.vat_class field is hidden in noi2 because the vat plugin is hidden. Hidden means that it is there in the database, but not visible to end users. products.Products.column_names has a field vat_class:

>>> products.Products.column_names
'id name category vat_class *'
>>> rt.show(products.Products)
==== ============= ================== ==========
 ID   Designation   Designation (et)   Category
---- ------------- ------------------ ----------
 2    Extra         Extra
 3    Free          Free
 1    Regular       Regular
==== ============= ================== ==========

After playing with this for some time (and fixing most but not all issues), I decided or realized that it’s better to implement these two options by not installing the related plugins rather than hiding them. Which also meant to implement the two new settings as site settings, not as plugin settings. An option that decides which plugins are to get installed cannot be a plugin setting because it must be known when the Site instantiates.

So Lino Noi has now two site settings (or “application options”) with_accounting and with_cms.

I am not sure whether it would have been possible to implement it by just hiding the relevant plugins. I didn’t dig further into that direction because I can’t imagine any advantage in being able to change these options without also having to migrate the database.

Some days ago I merged the pages plugin into the publisher plugin as a side effect of #5633 (Special pages). Now I discovered a side effect of this side effect: the publisher plugin now influences the application menu, it causes a “Publisher” menu to appear, and in noi2 it appeared even before the “Contacts” menu. That was disturbing. I fixed this by calling the super() method in Noi’s get_installed_plugins at the end, not at the beginning. I reviewed the docs about How Lino builds the INSTALLED_APPS setting.

A side effect of #5637 (New options with_accounting and with_cms for Noi) is that Noi no longer has the publisher plugin installed when with_cms is False. Which makes sense, the publisher plugin in the “classical” Noi was rather accidentally there.

Alvatal moved from GitHub to GitLab

Thu, 30 May 2024 00:00:00 +0000

Alvatal moved from GitHub to GitLab¶

Thursday, May 30, 2024

The alvatal.ee website is now hosted on GitLab.

Alvatal, short for “Estonian Free and Open-Source Software Association” is a non-profit umbrella organization for Estonian open-source and free software initiatives.

Alvatal is led by Edmund Laugasson who dedicated many years of his life as full-time activist helping deciders to opt for FOSS and companies to offer professional services.

The website is made using Jekyll. Lauri Võsandi created it in December 2014 and maintained it until September 2016 (initial commit).

Website: https://www.alvatal.ee
Project page on GitLab: https://gitlab.com/alvatal/www

At least one problem is still open: the present blog post does not appear in the news section even though I added my blog to the planet_en.yml file and GitLab reports “Parsing https://luc.lino-framework.org/rss.xml”. But planet then ignores all blog posts, even this one, which starts with “Alvatal” (the keyword defined in whitelisted_tags).

I converted my Sphinx newsfeed generator (sphinxfeed) to generate ATOM instead of RSS (a task for which I had to read the docs and the code of feedgen), but that too didn’t fix the issue.

Right now i do not know any further…

2024-06-01¶

I guess that RSS or ATOM has some concept of tags, and sphinxfeed doesn’t create such tags. I now checked the xml files of the two blog entries that get selected, and indeed the atom.xml file of one of the selected blog posts has a following <category> element:

<feed xmlns='http://www.w3.org/2005/Atom' ...>
  ...
      <entry>
  <category scheme="http://www.blogger.com/atom/ns#" term="alvatal"/>
              <title type='text'>alvatal.ee lehel hakkas tööle HTTPS-protokoll</title>

So I extended sphinxfeed to look for two new fields category and tags in the metadata of the page. If it does, sphinxfeed now uses the feedgen.FeedEntry.category() method to add <category> elements to the feed item.

The difference between category and tags is that the category of a blog post may contain spaces while the tags metadata field is itself a space-separated list of tags, so each tag must be a single word. Both the category and each tag will become a <category> element in the feed.

One feed entry can have multiple <category> elements (according to the ATOM specification), but Sphinx metadata is designed to have a field at most once per page.

The rss.xml file of my blog now declares three “categories” for this entry:

<entry>
<id>https://luc.lino-framework.org/blog/2024/0530.html</id>
<title>Alvatal moved from GitHub to GitLab</title>
<updated>2024-06-01T07:05:14.936148+00:00</updated>
<content>...</content>
<link href="https://luc.lino-framework.org/blog/2024/0530.html"/>
<category term="alvatal"/>
<category term="estonia"/>
<category term="foss"/>
<published>2024-05-30T23:59:00+00:00</published>
</entry>

And yes! After restarting the pipeline on GitLab, here finally Alvatal also lists my blog entry: https://alvatal.ee/blogs_en/2024/05/30/alvatal-moved-from-github-to-gitlab/

What’s the difference between a “category” and a “tag”? WordPress as a leading free blogging software can be considered an authority for answering this question:

“WordPress lets you put a post into multiple categories. (…) It’s possible that having your post in multiple categories could cause some SEO issues due to duplicate content. If you do use multiple categories, then try to avoid putting one post into two or more main (parent) categories. Each post should fit within one main category.”
“WordPress itself doesn’t have any limit on the number of tags you can have on each post. ()…) We suggest that you normally stick to a maximum of 10 tags per post.”

(Excerpt from Categories vs Tags – SEO Best Practices for Sorting Your Content.

A miracle in cron?

Thu, 16 May 2024 00:00:00 +0000

A miracle in cron?¶

Thursday, May 16, 2024

I am working on #5588.

On a Lino server with two sites a and b, we have two scripts make_snapshot_a.sh and make_snapshot_b.sh in /etc/cron.daily/, both generated by getlino startsite as follows:

#!/bin/sh
# generated by getlino
sudo service supervisor stop
/usr/local/lino/lino_local/a/make_snapshot.sh > /dev/null
sudo service supervisor start

But the daily snapshots were not being made. And no error message. We moved the two scripts to /usr/local/bin/ and created a new script /etc/cron.daily/linobackups with the following content:

#!/bin/sh
/usr/local/bin/make_snapshot_a.sh > /dev/null
sleep 5
/usr/local/bin/make_snapshot_b.sh > /dev/null

And after this change the daily snapshots started to work again! A miracle!?

That’s the kind of situations where Hannes stops worrying and where I start worrying. Because this would mean that getlino startsite does something wrong. And because I don’t believe in miracles. I am pretty sure that the additional linobackups script was not the explanation.

So I undid this move:

$ mkdir ~/20240516
$ cd ~/20240516
$ sudo mv /etc/cron.daily/linobackups .
$ sudo mv /usr/local/bin/make_snapshot_a.sh /etc/cron.daily/
$ sudo mv /usr/local/bin/make_snapshot_b.sh /etc/cron.daily/

P.S. 2024-05-31:

And yes, after my change the daily snapshots stopped again to work. And I explained to Hannes that I don’t want a solution, I want an explanation. And after two weeks of research, Hannes found this explanation (in a discussion on SO): run-parts (which is used by cron) ignores files with a period in the name.

So yes, getlino startsite did something wrong. I made the change, it was a single line of code.

Yes, the stupidest errors are the hardest to find!

Wednesday, May 8, 2024

Wed, 08 May 2024 00:00:00 +0000

Wednesday, May 8, 2024¶

I am working on #5590 (Render href in tested docs more consistently).

Until now the ;meth:rt.show function in a doctest rendered clickable cell content sometimes using *emphasis*, sometimes using **boldface** and sometimes as `reStruturedText links <some_url>`__. Now it uses consistently the latter.

Here is an example of what this means to many tested documents:

-=============================== =================== ===========
- When                            Calendar entry      State
-------------------------------- ------------------- -----------
- Fri 06/11/2015-Sun 08/11/2015   *Activity #26  5*   Suggested
- Fri 02/10/2015-Sun 04/10/2015   *Activity #26  4*   Suggested
- Fri 28/08/2015-Sun 30/08/2015   *Activity #26  3*   Suggested
- Fri 24/07/2015-Sun 26/07/2015   *Activity #26  2*   Suggested
- Fri 19/06/2015-Sun 21/06/2015   *Activity #26  1*   Suggested
-=============================== =================== ===========
+=============================== ======================== ===========
+ When                            Calendar entry           State
+------------------------------- ------------------------ -----------
+ Fri 06/11/2015-Sun 08/11/2015   `Activity #26  5 <…>`__   Suggested
+ Fri 02/10/2015-Sun 04/10/2015   `Activity #26  4 <…>`__   Suggested
+ Fri 28/08/2015-Sun 30/08/2015   `Activity #26  3 <…>`__   Suggested
+ Fri 24/07/2015-Sun 26/07/2015   `Activity #26  2 <…>`__   Suggested
+ Fri 19/06/2015-Sun 21/06/2015   `Activity #26  1 <…>`__   Suggested
+=============================== ======================== ===========
 <BLANKLINE>

Monday, May 6, 2024

Mon, 06 May 2024 00:00:00 +0000

Monday, May 6, 2024¶

New or updated rules.

Lino no longer tests for if s.startswith("<") for deciding whether a string s contains safe html. If you don’t want it to get escaped, you must mark it as safe using mark_safe().

TODO: Rename Model.as_summary_row to Model.as_summary_item

Model.as_paragraph() should include a link to the detail. Lino no longer adds such a link automatically in certain places. (add_detail_link() has been removed).

Wednesday, April 24, 2024

Wed, 24 Apr 2024 00:00:00 +0000

Wednesday, April 24, 2024¶

#5570 (Jane doesn’t send email notifications) is not trivial. It was (of course) caused by my changes in the background task runner. I am really still too naive regarding asynchronous programming!

Should we add test coverage for asynchronous operations. A unit test case in noi1r that creates a log directory (in order to activate the socket logger), launches a pm linod process and some django-admin command and then checks the content of the lino.log file…

Here is an excerpt from lino_xl.lib.invoicing.models:

class Task(Runnable, UserAuthored):

    target_journal = dd.ForeignKey(
        'ledger.Journal',
        verbose_name=_("Target journal"),
        related_name="invoicing_task_targets")

    def __str__(self):
        return _("Make {}").format(self.target_journal)

class Runnable(Sequenced, RecurrenceSet):

    async def start_task(self, ar):
        ...

Arguments: (Unprintable Task(pk=1,error=SynchronousOnlyOperation(‘You cannot call this from an async context - use a thread or sync_to_async.’),)

Do we really need the asynchronous versions of ar.debug, ar.info and ar.warning?

Edit 20240426: Yes we do. Because these methods potentially lead to I/O operations. When calling them from an async context, code execution would potentially continue before they have done their work. I tried to reproduce the problem but without success. Maybe the problem exists only with Django?

"""
Trying to show why the logging module needs an async interface.
"""

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

import asyncio
from time import sleep

from asgiref.sync import sync_to_async


info = logger.info
ainfo = sync_to_async(logger.info)

async def task1():
    info("- Run task1")
    sleep(1)

async def task2(name):
    info("- Run %s", name)
    sleep(1)


async def main():
    # await ainfo("Start task runner ")
    info("Start task runner ")
    count = 0
    while count < 2:
        count += 1
        # await ainfo("Start loop %s.", count)
        info("Start loop %s.", count)
        await task1()
        await task2("foo")
        await task2("bar")

    # await ainfo("Done task runner ")
    info("Done task runner ")

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

“If you want to call a part of Django that is still synchronous, you will need to wrap it in a sync_to_async() call. If you accidentally try to call a part of Django that is synchronous-only from an async view, you will trigger Django’s asynchronous safety protection to protect your data from corruption.”

“Transactions do not yet work in async mode. If you have a piece of code that needs transactions behavior, we recommend you write that piece as a single synchronous function and call it using sync_to_async().”

Fictive VAT numbers are different on GitLab

Tue, 16 Apr 2024 00:00:00 +0000

Fictive VAT numbers are different on GitLab¶

April 16–17, 2024

I’m investigating for #5542 (Two VAT doctests fail because generated VAT numbers differ).

The demo fixture of the vat plugin assigns a fictive and randomly generated (but syntactically valid) VAT number to each business partner. For some reason, the generated VAT numbers differ between my computer and GitLab CI, causing two doctests (docs/plugins/eevat.rst and docs/plugins/bevats.rst) to fail.

What causes this difference? That’s the question of #5542!

The VAT numbers are assigned by VatNumberManager.generate_vid. I added a diagnostic log message in this method:

dd.logger.info("20240416 generated VAT id {} for {}".format(self.obj.vat_id, self.obj))

Then I run inv prep with the new --verbose option. The first demo database to get prepared is cosi2. The resulting messages are always the same:

20240416 generated VAT id BE 7088.996.857 for Bäckerei Ausdemwald
20240416 generated VAT id BE 4685.739.309 for Bäckerei Mießen
20240416 generated VAT id BE 4181.505.692 for Bäckerei Schmitz
20240416 generated VAT id BE 9045.438.159 for Garage Mergelsberg
20240416 generated VAT id NL 220.876.686B01 for Donderweer BV
20240416 generated VAT id NL 451.948.587B01 for Van Achter NV
20240416 generated VAT id DE 143.956.862 for Hans Flott & Co
20240416 generated VAT id DE 135.079.295 for Bernd Brechts Bücherladen
20240416 generated VAT id DE 138.433.397 for Reinhards Baumschule
20240416 generated VAT id FR 86.915.334.564 for Moulin Rouge
20240416 generated VAT id FR 66.435.589.280 for Auto École Verte
20240416 generated VAT id EE 848.217.541 for Maksu- ja Tolliamet
20240416 generated VAT id BE 4018.258.949 for Electrabel Customer Solutions

The same messages also come on GitLab, but with other VAT numbers:

20240416 generated VAT id BE 2914.517.428 for Bäckerei Ausdemwald
20240416 generated VAT id BE 8750.836.192 for Bäckerei Mießen
20240416 generated VAT id BE 1958.116.531 for Bäckerei Schmitz
20240416 generated VAT id BE 4534.589.652 for Garage Mergelsberg
20240416 generated VAT id NL 237.725.353B01 for Donderweer BV
20240416 generated VAT id NL 643.080.485B01 for Van Achter NV
20240416 generated VAT id DE 928.188.312 for Hans Flott & Co
20240416 generated VAT id DE 593.748.463 for Bernd Brechts Bücherladen
20240416 generated VAT id DE 618.180.575 for Reinhards Baumschule
20240416 generated VAT id FR 65.449.289.186 for Moulin Rouge
20240416 generated VAT id FR 40.268.455.901 for Auto École Verte
20240416 generated VAT id EE 211.892.074 for Maksu- ja Tolliamet
20240416 generated VAT id BE 7659.012.310 for Electrabel Customer Solutions

The seed() method initializes the random number generator, and if you use the same seed value twice you will get the same random number twice. I verify this in Generating fictive VAT numbers (which passes both on my machine and on GitLab).

The generate_vid is the only place in Lino where Python’s random module is used (there is another import in the users plugin to generate the verification code, but this is never called during inv prep).

Calling seed() without argument would take the system time (and therefore yield different random numbers). But we deliberately call random.seed(1) at the global context of lino_xl.lib.vat.choicelists, i.e. when that module is imported. I added another log message at that place:

20240417 random.seed(1)

NB: Until now the import and the random.seed(1) call had been conditional (only when dd.is_installed("vat")), I removed this condition because it’s not needed and because it adds complexity. But that didn’t fix our problem.

One theoretic possibility was that for some reason the sorting order of the business partners might differ when they get their VAT id. By comparing the output between my machine and GitLab we can now exclude this possibility (IOW we are advancing ;-)

I also had a closer look at the code in lino_xl.lib.vat.choicelists and noticed this:

for cc, length in {
        "HR": 11,
        "DK": 8,
        "EE": 9,
        "FI": 8,
        "FR": 11,
        "DE": 9,
        "EL": 9,
        "HU": 8,
        "IT": 11,
        "LV": 11,
        "LT": 12,
        "LU": 8,
}.items():
    vat_origins.add_item(cc, VatOrigin(cc, length))

Which means that VatOrigin objects are instantiated in an order that can vary. I don’t say that this is the culprit, but it is suspicious… I made a series of bold simplifications to the code. Result: none.

Explanation¶

I found the explanation on April 20.

While showing the problem to Sharif I had the idea that I could “patch” the seed() method of the random generator so that it logs every time when it is called.

I added the following to lino/__init__.py:

import random
import inspect
def seed(self, a=None, version=2):
    stk = "\n".join(["{}:{}".format(s.filename, s.lineno) for s in inspect.stack()[1:3]])
    logger.info("20240420 random.seed(%s, %s) is called from %s", a, version, stk)
    self.original_seed(a=a, version=version)
random.Random.original_seed = random._inst.seed
random.Random.seed = seed
random.seed = random._inst.seed

And now:

(dev) luc@yoga:~/work/book/lino_book/projects/cosi2$ pm prep
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/home/luc/virtualenvs/dev/lib/python3.10/site-packages/sympy/core/random.py:29
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/home/luc/virtualenvs/dev/lib/python3.10/site-packages/sympy/core/symbol.py:419
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/home/luc/virtualenvs/dev/lib/python3.10/site-packages/sympy/ntheory/ecm.py:7
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/home/luc/virtualenvs/dev/lib/python3.10/site-packages/sympy/ntheory/qs.py:8
20240420 random.seed(1, 2) is called from /home/luc/work/xl/lino_xl/lib/vat/choicelists.py:27
<frozen importlib._bootstrap>:241
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/usr/lib/python3.10/tempfile.py:285
20240420 random.seed(None, 2) is called from /usr/lib/python3.10/random.py:125
/usr/lib/python3.10/tempfile.py:285
We are going to flush your database (/home/luc/work/book/lino_book/projects/cosi2/settings/default.db)
AND REMOVE ALL FILES BELOW /home/luc/work/book/lino_book/projects/cosi2/settings/media.
Are you sure (y/n) ? [Y,n]?

This made me realize that after calling random.seed() from lino_xl.lib.vat.choicelists, it gets called two more times from tempfile. Ha! No need to dig more! We must simply use our own random generator in lino_xl.lib.vat.choicelists!

Helo command rejected: need fully-qualified hostname

Tue, 09 Apr 2024 00:00:00 +0000

Helo command rejected: need fully-qualified hostname¶

Tuesday, April 9, 2024

Uff! #5535 (How to send emails from a developer environment) is finally done! Today I learned a lot, but the only visible result is this blog post and two paragraphs in The email settings of a Lino site.

For my tests, I wrote a script quick_mail_test.py in the noi1r demo project. My email settings are of course not published, I store them in my lino_local.settings module.

Problem: When I try to send emails from my developer environment, our mail server refuses them saying “Helo command rejected: need fully-qualified hostname”.

That’s because in our /etc/postfix/main.cf we have:

smtpd_helo_restrictions = reject_invalid_helo_hostname,
        reject_non_fqdn_helo_hostname,
        reject_unknown_helo_hostname

I got my mail to pass by commenting out the smtpd_helo_restrictions in the /etc/postfix/main.cf file. But that’s not a solution. These restrictions are probably a good choice, and they don’t cause Thunderbird to fail. So the question is: what does Thunderbird do differently from Django when sending emails?

I increased the verbosity of smtpd (in /etc/postfix/master.cf) and then observed our /var/log/mail.log. But that didn’t help. Postfix doesn’t seem to consider the HELO command something worth to log. Here are some excerpts.

Submit an email from my Thunderbird:

25:12 xyz postfix/submission/smtpd[944041]: connect from 211-169-190-90.dyn.estpak.ee[90.190.169.211]
25:13 xyz postfix/submission/smtpd[944041]: Anonymous TLS connection established from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
  TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
25:13 xyz postfix/submission/smtpd[944041]: F22F2209BB: client=211-169-190-90.dyn.estpak.ee[90.190.169.211], sasl_method=PLAIN, sasl_username=luc
25:14 xyz postfix/cleanup[944055]: F22F2209BB: message-id=<b52dde5e-165a-4b73-89aa-bb1bde876303@saffre-rumma.net>
25:14 xyz opendkim[573]: F22F2209BB: DKIM-Signature field added (s=mail, d=saffre-rumma.net)
25:14 xyz postfix/qmgr[943857]: F22F2209BB: from=<luc@saffre-rumma.net>, size=1308, nrcpt=1 (queue active)
25:14 xyz postfix/smtp[944057]: F22F2209BB: to=<luc.saffre@gmx.net>, relay=mx01.emig.gmx.net[212.227.17.5]:25, delay=1, delays=0.38/0.02/0.28/0.35, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=1MzjvV-1sgRrf2LZJ-0152Y8)
25:14 xyz postfix/qmgr[943857]: F22F2209BB: removed
25:19 xyz postfix/submission/smtpd[944041]: disconnect from 211-169-190-90.dyn.estpak.ee[90.190.169.211] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=8

TB asking for new mail:

13:25:46 xyz dovecot: pop3-login: Login: user=<luc>, method=PLAIN, rip=90.190.169.211, lip=51.68.71.43, mpid=944130, TLS, session=<NOJbWKcV3tRavqnT>
13:25:47 xyz dovecot: pop3(luc)<944130><NOJbWKcV3tRavqnT>: Disconnected: Logged out top=0/0, retr=1/12373, del=1/1, size=12355

Submit via quick_mail_test.py with EMAIL_USERNAME=”luc”:

26:02 xyz postfix/submission/smtpd[944041]: connect from 211-169-190-90.dyn.estpak.ee[90.190.169.211]
26:02 xyz postfix/submission/smtpd[944041]: Anonymous TLS connection established from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
  TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
26:03 xyz postfix/submission/smtpd[944041]: NOQUEUE: reject: RCPT from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
5.5.2 <yoga>: Helo command rejected: need fully-qualified hostname; from=<root@localhost> to=<luc.saffre@gmx.net> proto=ESMTP helo=<yoga>
26:03 xyz postfix/submission/smtpd[944041]: disconnect from 211-169-190-90.dyn.estpak.ee[90.190.169.211] ehlo=2 starttls=1 mail=1 rcpt=0/1 rset=1 quit=1 commands=6/7

Submit via quick_mail_test.py with EMAIL_USERNAME=””:

29:47 xyz postfix/submission/smtpd[944183]: connect from 211-169-190-90.dyn.estpak.ee[90.190.169.211]
29:49 xyz postfix/submission/smtpd[944183]: Anonymous TLS connection established from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
  TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
29:49 xyz postfix/submission/smtpd[944183]: NOQUEUE: reject: RCPT from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
5.5.2 <yoga>: Helo command rejected: need fully-qualified hostname; from=<root@localhost> to=<luc.saffre@gmx.net> proto=ESMTP helo=<yoga>
29:49 xyz postfix/submission/smtpd[944183]: disconnect from 211-169-190-90.dyn.estpak.ee[90.190.169.211] ehlo=2 starttls=1 mail=1 rcpt=0/1 rset=1 quit=1 commands=6/7
29:57 xyz postfix/smtps/smtpd[943972]: warning: unknown[194.169.175.10]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
30:03 xyz postfix/smtps/smtpd[943972]: lost connection after AUTH from unknown[194.169.175.10]
30:03 xyz postfix/smtps/smtpd[943972]: disconnect from unknown[194.169.175.10] ehlo=1 auth=0/1 rset=1 commands=2/3

Submit via quick_mail_test.py with EMAIL_USERNAME=””:

02:48 xyz postfix/submission/smtpd[945692]: connect from 211-169-190-90.dyn.estpak.ee[90.190.169.211]
02:48 xyz postfix/submission/smtpd[945692]: Anonymous TLS connection established from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
  TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
02:48 xyz postfix/submission/smtpd[945692]: NOQUEUE: reject: RCPT from 211-169-190-90.dyn.estpak.ee[90.190.169.211]:
5.5.2 <yoga>: Helo command rejected: need fully-qualified hostname; from=<noreply@saffre-rumma.net> to=<luc.saffre@gmx.net> proto=ESMTP helo=<yoga>
02:48 xyz postfix/submission/smtpd[945692]: disconnect from 211-169-190-90.dyn.estpak.ee[90.190.169.211] ehlo=2 starttls=1 mail=1 rcpt=0/1 rset=1 quit=1 commands=6/7

But then I had the idea to increase Thunderbird’s logging verbosity. Here is how to do that:

Open Preferences ‣ Settings
Search for “Config editor” and change mailnews.smtp.loglevel from “Warn” to “All”
Hit Ctrl+Shift+J to open the Error Console

And there is the answer to my question: Thunderbird sends the following FQDN in the EHLO request:

mailnews.smtp: C: EHLO [192.168.1.115]

See also the two paragraphs in The email settings of a Lino site.

Thanks to¶

Saturday, April 6, 2024

Sat, 06 Apr 2024 00:00:00 +0000

Saturday, April 6, 2024¶

I made some test to check whether redirection to stderr and/or stdout work as expected. It does so, perfectly.

$ pip install -U django
Requirement already satisfied: django in .../lib/python3.10/site-packages (5.0.4)
Requirement already satisfied: asgiref<4,>=3.7.0 in .../lib/python3.10/site-packages (from django) (3.7.2)
Requirement already satisfied: sqlparse>=0.3.1 in .../lib/python3.10/site-packages (from django) (0.4.2)
Requirement already satisfied: typing-extensions>=4 in .../lib/python3.10/site-packages (from asgiref<4,>=3.7.0->django) (4.8.0)
$ pip install -U django > tmp.txt
$ cat tmp.txt
Requirement already satisfied: django in .../lib/python3.10/site-packages (5.0.4)
Requirement already satisfied: asgiref<4,>=3.7.0 in .../lib/python3.10/site-packages (from django) (3.7.2)
Requirement already satisfied: sqlparse>=0.3.1 in .../lib/python3.10/site-packages (from django) (0.4.2)
Requirement already satisfied: typing-extensions>=4 in .../lib/python3.10/site-packages (from asgiref<4,>=3.7.0->django) (4.8.0)

$ pip install -U djangofoo
ERROR: Could not find a version that satisfies the requirement djangofoo (from versions: none)
ERROR: No matching distribution found for djangofoo
$ pip install -U djangofoo > tmp.txt
ERROR: Could not find a version that satisfies the requirement djangofoo (from versions: none)
ERROR: No matching distribution found for djangofoo
$ cat tmp.txt
$

When I deliberately introduce a mistake into settings.py so that pm prep fails, then the traceback is sent to stderr (not to stdout):

$ go chatter
$ pm prep --noinput > tmp.txt
$ nano settings.py # (modify settings.py)
$ pm prep --noinput > tmp.txt
Traceback (most recent call last):
  File ".../projects/chatter/manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File ".../lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File ".../lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File ".../lib/python3.10/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File ".../lib/python3.10/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
  File ".../lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File ".../projects/chatter/settings/demo.py", line 5, in <module>
    class Site(Site):
  File ".../projects/chatter/settings/demo.py", line 7, in Site
    the_demo_date = i2d(201410231)
  File ".../rstgen/rstgen/utils.py", line 84, in i2d
    raise Exception("Invalid date specification {0}.".format(i))
Exception: Invalid date specification 201410231.
$ cat tmp.txt
$

Thursday, February 8, 2024

Thu, 08 Feb 2024 00:00:00 +0000

Thursday, February 8, 2024¶

It seems that we want a new database model for contracts in Lino Welfare. This new contract model is called “Activation SINE”. Question for a brainstorming: which database fields should this new model have?

>>> from lino import startup
>>> startup('lino_welfare.projects.mathieu.settings.demo')
>>> from lino.api.doctest import *

>>> def fmt(f):
...   if hasattr(f, 'verbose_name'):
...     return "{} ({}) {}".format(f.name, f.verbose_name, f.__class__.__name__)
...   return "{} ({})".format(f.name, f)

>>> base_fields = set([f.name for f in isip.ContractBase._meta.get_fields()])
>>> from lino.core.fields import VirtualField

>>> def show(f):
...     if f.name in base_fields: return False
...     if isinstance(f, VirtualField): return False
...     return True

Classes de contrats ¶

Here is a list of the contracts classes we currently differentiate in Lino

>>> langs = ["en", "fr"]
>>> rows = []
>>> for m in rt.models_by_base(isip.ContractBase):
...     cells = [dd.full_model_name(m)]
...     for lng in langs:
...       with translation.override(lng):
...         cells.append(str(m._meta.verbose_name))
...     rows.append(cells)
>>> print(rstgen.table(["model"]+langs, rows))
==================== ======================== =========================
 model                en                       fr
-------------------- ------------------------ -------------------------
 art60.Contract       Art60§7 job supplyment   Mise à l'emploi art60§7
 art61.Contract       Art61 job supplyment     Mise à l'emploi art.61
 immersion.Contract   Immersion training       Stage d'immersion
 isip.Contract        ISIP                     PIIS
==================== ======================== =========================

List of fields that are common to all contracts:

>>> for f in isip.ContractBase._meta.get_fields():
...     print(fmt(f))
signer1 (Secrétaire) ForeignKey
signer2 (Président) ForeignKey
user (Titulaire (SI)) ForeignKey
printed_by (Imprimé) ForeignKey
client (client) ForeignKey
language (Langue) LanguageField
applies_from (Début de contrat) DateField
applies_until (Fin prévue) DateField
date_decided (Décidé le) DateField
date_issued (Date de signature) DateField
user_asd (Titulaire (SSG)) ForeignKey
exam_policy (exam policy) ForeignKey
ending (ending) ForeignKey
date_ended (Date de fin) DateField

Remarks:

art60.Contract is the only model with multiple conventions per contract.

Mise à l’emploi art60§7 ¶

>>> print(art60.Contract._meta.verbose_name)
Mise à l'emploi art60§7
>>> for f in art60.Contract._meta.get_fields():
...   if show(f):
...     print(fmt(f))
convention (<ManyToOneRel: art60.convention>)
id (ID) BigAutoField
duration (Jours ONSS) IntegerField
remark (Remarque) TextField
type (Type) ForeignKey

>>> for f in art60.Convention._meta.get_fields():
...   if not isinstance(f, VirtualField):
...     print(fmt(f))
id (ID) BigAutoField
contact_person (Représenté par) ForeignKey
contact_role (En tant que) ForeignKey
reference_person (persone de référence) CharField
responsibilities (responsabilités) RichTextField
company (Service utilisateur) ForeignKey
contract (Mise à l'emploi art60§7) ForeignKey
job (Poste de travail) ForeignKey
start_date (Date début) DateField
regime (Régime de travail) ForeignKey
schedule (Horaire) ForeignKey
monthly_refund (Rétrocession mensuelle) PriceField
hourly_rate (coût horaire) PriceField
refund_rate (tarif de remboursement) CharField

Mise à l’emploi art.61 ¶

>>> print(art61.Contract._meta.verbose_name)
Mise à l'emploi art.61
>>> for f in art61.Contract._meta.get_fields():
...   if show(f):
...     print(fmt(f))
id (ID) BigAutoField
contact_person (Représenté par) ForeignKey
contact_role (En tant que) ForeignKey
sector (Secteur d'activité) ForeignKey
function (Fonction professionnelle) ForeignKey
reference_person (persone de référence) CharField
responsibilities (responsabilités) RichTextField
duration (Jours ONSS) IntegerField
remark (Remarque) TextField
company (Employeur) ForeignKey
type (Type) ForeignKey
job_title (Fonction professionnelle) CharField
status (Statut) ForeignKey
cv_duration (Type de contrat) ForeignKey
regime (Régime de travail) ForeignKey
sub_10_amount (Aide à l'emploi) PriceField
sub_10_start (Depuis) DateField
sub_10_end (Jusqu'à) DateField
sub_20_amount (Tutorat) PriceField
sub_20_start (Depuis) DateField
sub_20_end (Jusqu'à) DateField
sub_30_amount (Région Wallonne) PriceField
sub_30_start (Depuis) DateField
sub_30_end (Jusqu'à) DateField

PIIS ¶

>>> print(isip.Contract._meta.verbose_name)
PIIS
>>> for f in isip.Contract._meta.get_fields():
...   if show(f):
...     print(fmt(f))
contractpartner (<ManyToOneRel: isip.contractpartner>)
id (ID) BigAutoField
type (Type de contrat) ForeignKey
study_type (Type d'éducation) ForeignKey
stages (stages) RichTextField
goals (buts) RichTextField
duties_asd (Obligations du service social) RichTextField
duties_dsbe (Obligations du service d'insertion) RichTextField
duties_pcsw (Obligations du CPAS) RichTextField
duties_person (Obligations du bénéficiaire) RichTextField
user_dsbe (Titulaire (SI)) ForeignKey

Stage d’immersion ¶

>>> print(immersion.Contract._meta.verbose_name)
Stage d'immersion
>>> for f in immersion.Contract._meta.get_fields():
...   if show(f):
...     print(fmt(f))
id (ID) BigAutoField
contact_person (Représenté par) ForeignKey
contact_role (En tant que) ForeignKey
sector (Secteur d'activité) ForeignKey
function (Fonction professionnelle) ForeignKey
company (Organisation) ForeignKey
type (Type de stage d'immersion) ForeignKey
goal (Objectif) ForeignKey
reference_person (persone de référence) CharField
responsibilities (responsabilités) RichTextField
remark (Remarque) TextField

Différence entre “Service utilisateur” et “Employeur”¶

Les postes de travail ne devraient-ils pas être offerts par les employeurs plutot que par les services utilisateurs?

jobs.Job.provider is currently a pointer to JobProvider but shouldn’t it point to Employer instead?

>>> print(jobs.Job._meta.verbose_name)
Poste de travail

>>> print(jobs.JobProvider._meta.verbose_name)
Service utilisateur

>>> print(jobs.Employer._meta.verbose_name)
Employeur

Tuesday, February 6, 2024

Tue, 06 Feb 2024 00:00:00 +0000

Tuesday, February 6, 2024¶

How we configured the server for zeugnisse

sudo hostnamectl set-hostname pdg2
Create users lsaffre and hliivat.
Add “umask 002” in /etc/bash.bashrc
sudo apt install git python3-venv python3-pip libffi-dev
cd /usr/local
sudo mkdir synodal
cd synodal
sudo chown www-data .
sudo chgrp www-data .
sudo chmod g+ws .
git clone https://gitlab.com/synodalsoft/zeugnisse.git .
nano bash_aliases #
nano ~/.bashrc # and add a line “. /usr/local/synodal/bash_aliases”

Ctrl-D and come back

go # cd /usr/local/synodal
python3 -m venv env
a
pip install -r requirements.txt

It was interesting to read this but don’t follow it blindly. We install uwsgi using the Debian package, not using the Python package.

sudo apt install nginx uwsgi uwsgi-plugin-python3 postgresql

Hannes defined a domain name zeugnisse.mylino.net pointing to pdg2.

Create config files for nginx and uwsgi:

sudo nano /etc/nginx/sites-available/zeugnisse.conf

sudo nano /etc/uwsgi/apps-available/zeugnisse.ini

Enable them:

sudo nano ln -s /etc/nginx/sites-available/zeugnisse.conf /etc/nginx/sites-enabled

sudo nano ln -s /etc/uwsgi/apps-available/zeugnisse.ini /etc/uwsgi/apps-enabled

How to copy a file from

$ scp pdg:/home/django/django-seite/django_seite/secret/* pdg2:/usr/local/synodal/django_seite/secret/

sudo -u postgres psql -c “CREATE DATABASE django;”

On old server as user django:

cd
pg_dump django > 20240210.psql

On my machine:

scp pdg:/home/django/20240210.psql pdg2:/usr/local/synodal/

On new server:

sudo -u postgres psql -c ‘CREATE USER “django”;’
sudo -u postgres psql -c ‘GRANT ALL PRIVILEGES ON DATABASE django TO “django”;’
sudo -u postgres psql django < 20240210.psql
sudo mkdir /var/log/zeugnisse
sudo chown www-data:www-data /var/log/zeugnisse
sudo chmod g+ws /var/log/zeugnisse
nano django_seite/settings/settings_prod.py

‘filename’: ‘/var/log/zeugnisse/django.log’,
apt install libpango-1.0-0 libpangoft2-1.0-0
apt install python3-certbot-nginx
sudo certbot –nginx -d zeugnisse.mylino.net

Git:

nano ~/.gitconfig
cd /usr/local/synodal
nano .gitignore
git rm -r static

Tuesday, January 16, 2024

Tue, 16 Jan 2024 00:00:00 +0000

Tuesday, January 16, 2024¶

The following output (issued by lino.utils.diag.Anayzer.show_db_overview()) helped me to understand the MRO problem with ledger.Voucher and ledger.RegistrableVoucher:

class VatProductInvoice(TradingVoucher, Matching, InvoicingTargetVoucher, StorageTransferer):

Model:
      lino_xl.lib.trading.models.VatProductInvoice
Duplicate base:
      LedgerRegistrable
Inheritance trees:
      LedgerRegistrable.StorageTransferer.VatProductInvoice
      LedgerRegistrable.VatDocument.VatVoucher.TradingVoucher.VatProductInvoice
----------------------------------------
Duplicate base:
      Voucher
Inheritance trees:
      Voucher.LedgerRegistrable.StorageTransferer.VatProductInvoice
      Voucher.LedgerRegistrable.VatDocument.VatVoucher.TradingVoucher.VatProductInvoice
----------------------------------------
Duplicate base:
      UserAuthored
Inheritance trees:
      UserAuthored.Voucher.LedgerRegistrable.StorageTransferer.VatProductInvoice
      UserAuthored.Voucher.LedgerRegistrable.VatDocument.VatVoucher.TradingVoucher.VatProductInvoice

Please reinstall virtualbox-dkms package

Thu, 11 Jan 2024 00:00:00 +0000

Please reinstall virtualbox-dkms package¶

Thursday, January 11, 2024

I use virtualbox to run a venerable legacy accounting software under Windows XP from within my Ubuntu system. Works like a charm. But after the latest system upgrade (or at least I cannot see any other reason) it was broken. When I try to launch the VM, it says:

Kernel driver not installed (rc=-1908) The VirtualBox Linux kernel driver is either not loaded or not set up correctly. Please reinstall virtualbox-dkms package and load the kernel module by executing ‘modprobe vboxdrv’ as root.

If your system has EFI Secure Boot enabled you may also need to sign the kernel modules (vboxdrv, vboxnetflt, vboxnetadp, vboxpci) before you can load them. Please see your Linux system’s documentation for more information.

where: suplibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) - The support driver is not installed. On linux, open returned ENOENT.

And later:

Failed to open a session for the virtual machine Windows XP. The virtual machine ‘Windows XP’ has terminated unexpectedly during startup with exit code 1 (0x1).

Result Code: NS_ERROR_FAILURE (0x80004005)

Component: MachineWrap

Interface: IMachine {85632c68-b5bb-4316-a900-5eb28d3413df}

I have had this message years ago (on another computer) and remember that a “reinstall virtualbox-dkms” helped. But this time it didn’t. I tried to remove and reinstall the virtualbox packages without success.

Here are some things I tried.

$ sudo apt remove virtualbox-guest-additions-iso virtualbox-guest-utils virtualbox-qt virtualbox-source
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  kbuild module-assistant
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  virtualbox-guest-additions-iso virtualbox-guest-utils virtualbox-qt virtualbox-source
0 upgraded, 0 newly installed, 4 to remove and 3 not upgraded.
1 not fully installed or removed.
After this operation, 131 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 317868 files and directories currently installed.)
Removing virtualbox-guest-additions-iso (6.1.38-1~ubuntu1.22.04.1) ...
Removing virtualbox-guest-utils (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Removing virtualbox-qt (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
dpkg: virtualbox-source: dependency problems, but removing anyway as you requested:
 virtualbox depends on virtualbox-dkms (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-source (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-modules; however:
  Package virtualbox-dkms is not configured yet.
  Package virtualbox-source is to be removed.
  Package virtualbox-modules is not installed.
  Package virtualbox-dkms which provides virtualbox-modules is not configured yet.

Removing virtualbox-source (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Setting up virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Removing old virtualbox-6.1.38 DKMS files...
Deleting module virtualbox-6.1.38 completely from the DKMS tree.
Loading new virtualbox-6.1.38 DKMS files...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/virtualbox-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.5.0-14-generic (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.38/build/make.log for more information.
dpkg: error processing package virtualbox-dkms (--configure):
 installed virtualbox-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for shared-mime-info (2.1-2) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Errors were encountered while processing:
 virtualbox-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

$ sudo cat /var/lib/dkms/virtualbox/6.1.38/build/make.log
DKMS make.log for virtualbox-6.1.38 for kernel 6.5.0-14-generic (x86_64)
N 11 jaan  2024 07:49:57 EET
make: Entering directory '/usr/src/linux-headers-6.5.0-14-generic'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  You are using:           gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/linux/SUPDrv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrvGip.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrvSem.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrvTracer.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPLibAll.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/alloc-r0drv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/initterm-r0drv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/memobj-r0drv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/mpnotification-r0drv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/powernotification-r0drv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/assert-r0drv-linux.o
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrvTracer.o: warning: objtool: SUPR0TracerFireProbe+0x7: indirect jump found in RETPOLINE build
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/SUPDrvTracer.o: warning: objtool: supdrvTracerProbeFireStub+0x0: 'naked' return found in RETHUNK build
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/alloc-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/initterm-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memuserkernel-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/mp-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/mpnotification-r0drv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/process-r0drv-linux.o
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjNativeLockUser’:
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1228:18: error: too many arguments to function ‘get_user_pages’
 1228 |             rc = get_user_pages(R3Ptr,                  /* Where from. */
      |                  ^~~~~~~~~~~~~~
In file included from /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/the-linux-kernel.h:102,
                 from /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:31:
./include/linux/mm.h:2430:6: note: declared here
 2430 | long get_user_pages(unsigned long start, unsigned long nr_pages,
      |      ^~~~~~~~~~~~~~
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1261:33: error: passing argument 6 of ‘get_user_pages_remote’ from incompatible pointer type [-Werror=incompatible-pointer-types]
 1261 |                                 papVMAs                 /* vmas */
      |                                 ^~~~~~~
      |                                 |
      |                                 struct vm_area_struct **
./include/linux/mm.h:2400:33: note: expected ‘int *’ but argument is of type ‘struct vm_area_struct **’
 2400 |                            int *locked);
      |                            ~~~~~^~~~~~
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1245:18: error: too many arguments to function ‘get_user_pages_remote’
 1245 |             rc = get_user_pages_remote(
      |                  ^~~~~~~~~~~~~~~~~~~~~
./include/linux/mm.h:2397:6: note: declared here
 2397 | long get_user_pages_remote(struct mm_struct *mm,
      |      ^~~~~~~~~~~~~~~~~~~~~
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1304:39: error: assignment of read-only member ‘vm_flags’
 1304 |                 papVMAs[rc]->vm_flags |= VM_DONTCOPY | VM_LOCKED;
      |                                       ^~
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjNativeMapUser’:
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1774:35: error: assignment of read-only member ‘vm_flags’
 1774 |                     vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
      |                                   ^~
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o] Error 1
make[3]: *** Waiting for unfinished jobs....
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: VBoxHost_RTR0MemKernelCopyTo+0x13: redundant CLD
/var/lib/dkms/virtualbox/6.1.38/build/vboxdrv/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: VBoxHost_RTR0MemKernelCopyFrom+0x13: redundant CLD
make[2]: *** [scripts/Makefile.build:488: /var/lib/dkms/virtualbox/6.1.38/build/vboxdrv] Error 2
make[1]: *** [/usr/src/linux-headers-6.5.0-14-generic/Makefile:2037: /var/lib/dkms/virtualbox/6.1.38/build] Error 2
make: *** [Makefile:234: __sub-make] Error 2
make: Leaving directory '/usr/src/linux-headers-6.5.0-14-generic'

$ sudo apt install virtualbox
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
virtualbox is already the newest version (6.1.38-dfsg-3~ubuntu1.22.04.1).
The following packages were automatically installed and are no longer required:
  kbuild module-assistant
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Removing old virtualbox-6.1.38 DKMS files...
Deleting module virtualbox-6.1.38 completely from the DKMS tree.
Loading new virtualbox-6.1.38 DKMS files...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/virtualbox-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.5.0-14-generic (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.38/build/make.log for more information.
dpkg: error processing package virtualbox-dkms (--configure):
 installed virtualbox-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
 virtualbox-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

::: $ ls -l /var/crash/virtualbox-dkms.0.crash -rw-r–r– 1 root whoopsie 6048 jaan 11 07:29 /var/crash/virtualbox-dkms.0.crash

I was able to remove all virtualbox packages:

$ sudo apt remove virtualbox-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  dctrl-tools dkms linux-headers-5.15.0-60 linux-headers-5.15.0-60-generic linux-headers-5.19.0-50-generic linux-hwe-5.19-headers-5.19.0-50
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  virtualbox-source
The following packages will be REMOVED:
  virtualbox-dkms
The following NEW packages will be installed:
  virtualbox-source
0 upgraded, 1 newly installed, 1 to remove and 3 not upgraded.
1 not fully installed or removed.
Need to get 836 kB of archives.
After this operation, 4 789 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://ee.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 virtualbox-source amd64 6.1.38-dfsg-3~ubuntu1.22.04.1 [836 kB]
Fetched 836 kB in 0s (3 211 kB/s)
Selecting previously unselected package virtualbox-source.
(Reading database ... 317674 files and directories currently installed.)
Preparing to unpack .../virtualbox-source_6.1.38-dfsg-3~ubuntu1.22.04.1_amd64.deb ...
Unpacking virtualbox-source (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
dpkg: virtualbox-dkms: dependency problems, but removing anyway as you requested:
 virtualbox depends on virtualbox-dkms (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-source (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-modules; however:
  Package virtualbox-dkms is to be removed.
  Package virtualbox-source is not configured yet.
  Package virtualbox-modules is not installed.
  Package virtualbox-dkms which provides virtualbox-modules is to be removed.
 virtualbox depends on virtualbox-dkms (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-source (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-modules; however:
  Package virtualbox-dkms is to be removed.
  Package virtualbox-source is not configured yet.
  Package virtualbox-modules is not installed.
  Package virtualbox-dkms which provides virtualbox-modules is to be removed.
(Reading database ... 317679 files and directories currently installed.)
Removing virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Deleting module virtualbox-6.1.38 completely from the DKMS tree.
Setting up virtualbox-source (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
(dev) luc@yoga:~$ sudo apt remove virtualbox
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  dctrl-tools dkms kbuild libgsoap-2.8.117 linux-headers-5.15.0-60 linux-headers-5.15.0-60-generic linux-headers-5.19.0-50-generic linux-hwe-5.19-headers-5.19.0-50
  module-assistant virtualbox-source
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  virtualbox
0 upgraded, 0 newly installed, 1 to remove and 3 not upgraded.
After this operation, 112 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 317402 files and directories currently installed.)
Removing virtualbox (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Processing triggers for man-db (2.10.2-1) ...

$ sudo apt remove virtualbox-source
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  dctrl-tools dkms kbuild libgsoap-2.8.117 linux-headers-5.15.0-60 linux-headers-5.15.0-60-generic linux-headers-5.19.0-50-generic linux-hwe-5.19-headers-5.19.0-50
  module-assistant
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  virtualbox-source
0 upgraded, 0 newly installed, 1 to remove and 3 not upgraded.
After this operation, 936 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 317207 files and directories currently installed.)
Removing virtualbox-source (6.1.38-dfsg-3~ubuntu1.22.04.1) ...

But now I can’t install it any more:

$ sudo apt install virtualbox
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  kbuild module-assistant
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  virtualbox-dkms virtualbox-qt
Suggested packages:
  vde2 virtualbox-guest-additions-iso
The following NEW packages will be installed:
  virtualbox virtualbox-dkms virtualbox-qt
0 upgraded, 3 newly installed, 0 to remove and 3 not upgraded.
Need to get 22,1 MB/46,1 MB of archives.
After this operation, 178 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ee.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 virtualbox-qt amd64 6.1.38-dfsg-3~ubuntu1.22.04.1 [22,1 MB]
Fetched 22,1 MB in 4s (5 703 kB/s)
Selecting previously unselected package virtualbox-dkms.
(Reading database ... 317202 files and directories currently installed.)
Preparing to unpack .../virtualbox-dkms_6.1.38-dfsg-3~ubuntu1.22.04.1_amd64.deb ...
Unpacking virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Selecting previously unselected package virtualbox.
Preparing to unpack .../virtualbox_6.1.38-dfsg-3~ubuntu1.22.04.1_amd64.deb ...
Unpacking virtualbox (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Selecting previously unselected package virtualbox-qt.
Preparing to unpack .../virtualbox-qt_6.1.38-dfsg-3~ubuntu1.22.04.1_amd64.deb ...
Unpacking virtualbox-qt (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Setting up virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Loading new virtualbox-6.1.38 DKMS files...
Building for 6.5.0-14-generic
Building initial module for 6.5.0-14-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/virtualbox-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.5.0-14-generic (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.38/build/make.log for more information.
dpkg: error processing package virtualbox-dkms (--configure):
 installed virtualbox-dkms package post-installation script subprocess returned error exit status 10
dpkg: dependency problems prevent configuration of virtualbox:
 virtualbox depends on virtualbox-dkms (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-source (>= 6.1.38-dfsg-3~ubuntu1.22.04.1) | virtualbox-modules; however:
  Package virtualbox-dkms is not configured yet.
  Package virtualbox-source is not installed.
  Package virtualbox-modules is not installed.
  Package virtualbox-dkms which provides virtualbox-modules is not configured yet.

dpkg: error processing package virtualbox (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because the error message indicates its a followup error from a previous failure.
dpkg: dependency problems prevent configuration of virtualbox-qt:
 virtualbox-qt depends on virtualbox (= 6.1.38-dfsg-3~ubuntu1.22.04.1); however:
  Package virtualbox is not configured yet.

dpkg: error processing package virtualbox-qt (--configure):
 dependency problems - leaving unconfigured
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for shared-mime-info (2.1-2) ...
Errors were encountered while processing:
 virtualbox-dkms
 virtualbox
 virtualbox-qt
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried some cleanup actions, with success:

$ sudo apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  dctrl-tools dkms kbuild libgsoap-2.8.117 linux-headers-5.15.0-60 linux-headers-5.15.0-60-generic linux-headers-5.19.0-50-generic linux-hwe-5.19-headers-5.19.0-50
  module-assistant virtualbox-dkms
0 upgraded, 0 newly installed, 10 to remove and 3 not upgraded.
1 not fully installed or removed.
After this operation, 222 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 317479 files and directories currently installed.)
Removing virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) ...
Deleting module virtualbox-6.1.38 completely from the DKMS tree.
Removing dkms (2.8.7-2ubuntu2.2) ...
Removing dctrl-tools (2.24-3build2) ...
Removing kbuild (1:0.1.9998svn3564+dfsg-1) ...
Removing libgsoap-2.8.117:amd64 (2.8.117-2build1) ...
Removing linux-headers-5.15.0-60-generic (5.15.0-60.66) ...
dpkg: warning: while removing linux-headers-5.15.0-60-generic, directory '/lib/modules/5.15.0-60-generic' not empty so not removed
Removing linux-headers-5.15.0-60 (5.15.0-60.66) ...
Removing linux-headers-5.19.0-50-generic (5.19.0-50.50) ...
dpkg: warning: while removing linux-headers-5.19.0-50-generic, directory '/lib/modules/5.19.0-50-generic' not empty so not removed
Removing linux-hwe-5.19-headers-5.19.0-50 (5.19.0-50.50) ...
Removing module-assistant (0.11.10) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...

::: $ sudo dpkg –purge virtualbox virtualbox-qt virtualbox-dkms (Reading database … 260368 files and directories currently installed.) Removing virtualbox-qt (6.1.38-dfsg-3~ubuntu1.22.04.1) … Removing virtualbox (6.1.38-dfsg-3~ubuntu1.22.04.1) … Purging configuration files for virtualbox (6.1.38-dfsg-3~ubuntu1.22.04.1) … Removing virtualbox-dkms (6.1.38-dfsg-3~ubuntu1.22.04.1) … Deleting module virtualbox-6.1.38 completely from the DKMS tree. Processing triggers for man-db (2.10.2-1) … Processing triggers for shared-mime-info (2.1-2) … Processing triggers for hicolor-icon-theme (0.17-2) … Processing triggers for mailcap (3.70+nmu1ubuntu1) … Processing triggers for gnome-menus (3.36.0-1ubuntu3) … Processing triggers for desktop-file-utils (0.26-1ubuntu3) …

System diagnostics:

$ dpkg -l | grep -i virtualb
iU  virtualbox                                    6.1.38-dfsg-3~ubuntu1.22.04.1           amd64        x86 virtualization solution - base binaries
iF  virtualbox-dkms                               6.1.38-dfsg-3~ubuntu1.22.04.1           amd64        x86 virtualization solution - kernel module sources for dkms
rc  virtualbox-guest-utils                        6.1.38-dfsg-3~ubuntu1.22.04.1           amd64        x86 virtualization solution - non-X11 guest utilities
iU  virtualbox-qt                                 6.1.38-dfsg-3~ubuntu1.22.04.1           amd64        x86 virtualization solution - Qt based user interface

$ uname -a
Linux yoga 6.5.0-14-generic #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -crid
Distributor ID:       Ubuntu
Description:  Ubuntu 22.04.3 LTS
Release:      22.04
Codename:     jammy

$ sudo dmesg | grep -i -e virtualb -e vbox
(no output)

$ find /lib/modules -iname vboxdrv.ko -ls
(no output)

Thursday, January 4, 2024

Thu, 04 Jan 2024 00:00:00 +0000

Thursday, January 4, 2024¶

A subscription is a time-based or periodic invoice generator.

A storage filler is not periodic. It generates “invoices” (or similar vouchers) when a given storage has reached a minimum quantity.

Tuesday, January 2, 2024

Tue, 02 Jan 2024 00:00:00 +0000

Tuesday, January 2, 2024¶

Here is my first Django session on the Zeugnisse Server:

In [1]: from zeugnisse.models import BausteinArbeitBewertung
In [2]: qs = BausteinArbeitBewertung.objects.filter(schüler=361, baustein=27)
In [16]: [b.arbeit for b in qs]
Out[16]:
[<BausteinArbeit: Arbeitsverhalten>,
 <BausteinArbeit: Sauberkeit>,
 <BausteinArbeit: Korrektur>,
 <BausteinArbeit: Zeitmanagement>,
 <BausteinArbeit: Arbeitsverhalten>]

In [17]: [b.arbeit.id for b in qs]
Out[17]: [4, 1, 2, 3, 4]

In [20]: qs = BausteinArbeitBewertung.objects.filter(schüler=361, baustein=27, arbeit=4)

In [21]: qs.count()
Out[21]: 2

In [22]: [b.punkte for b in qs]
Out[22]: [Decimal('2.0'), Decimal('2.0')]

In [23]: [b.id for b in qs]
Out[23]: [319012, 319022]

In [23]: [b.id for b in qs]
Out[23]: [319012, 319022]

In [24]: b = BausteinArbeitBewertung.objects.get(id=319022)

In [25]: b.punkte
Out[25]: Decimal('2.0')

In [26]: b.arbeit
Out[26]: <BausteinArbeit: Arbeitsverhalten>

In [27]: b.schüler
Out[27]: <Schüler: Musterman Max>

In [28]: b.delete()
Out[28]: (2, {'zeugnisse.BausteinArbeitBewertung': 1, 'zeugnisse.Bewertung': 1})

Optimizing RolesByCompany caused an avalanche

Wed, 27 Dec 2023 00:00:00 +0000

Optimizing RolesByCompany caused an avalanche¶

December 27-29, 2023

I did some fundamental cleanup in the following methods. Initially it was just for #5346 (Optimize RolesByCompany), which was trivial: there was no insert layout and therefore it was difficult to add a new contact person for an organization.

But working on this caused an avalanche of #5353 (Use mark_safe instead of ElementTree).

ElementTree versus mark_safe()¶

The fundamental new thing is that we started to use mark_safe() function. When a virtual field or any function returns some content, then the caller needs to know whether that content is (1) plain text or (2) HTML-formatted text. During the first years of Lino, I discovered ElementTree and fell in love. And I still say that it’s a great tool for generating HTML. And I based most function calls on the convention that when it’s a str then it is plain text (and thus needs to get escaped when rendered as HTML), and when a function wants to return formatted text, then it must return an ElementTree.

But this convention is suboptimal when we return content of a rich text field, it that is already a chunk of HTML, verified and bleached and validated. In that case a method that is expected to return ElementTree currently has to parse the stored HTML using lxml.html.fromstring() in order to get an ET from it. Which is a bit stupid because that ET is used for nothing else but to generate HTML.

I had been feeling this issue for quite some time already, but now I finally created a separate ticket for it: #5353 (Use mark_safe instead of ElementTree).

qs2summary() no longer returns an etree but an html string. As a result, Actor.summary_sep no longer needs to be a callable, we can remove the “from lino.core.utils import comma”.
The default implementation of Model.as_story_item() returned as_summary_row() but now returns as_paragraph()
summary_row() replaced by as_summary_row()
row_as_summary()
ar.obj2htmls(self)
Model.as_search_item()

About mark_safe()¶

The Django docs about mark_safe() say that a “safe” string becomes unsafe as soon as it is modified. I wondered what happens if you just concatenate two safe strings. Answer the result is still safe.

>>> from django.utils.html import mark_safe
>>> s1 = mark_safe("<b>Foo</b>")
>>> s2 = mark_safe("<b>Bar</b>")
>>> s3 = mark_safe("<em>{} and {}</em>")

>>> type(s1+s2)
<class 'django.utils.safestring.SafeString'>

But already the following requires another call to mark_safe():

>>> s4 = s3.format(s1, s2)
>>> type(s4)
<class 'str'>

Note that the word “safe” here has nothing to do with actual security. A SafeString can contain dangerous HTML.

Tuesday, December 26, 2023

Tue, 26 Dec 2023 00:00:00 +0000

Tuesday, December 26, 2023¶

I am trying to understand #5278 (Jane doesn’t send mails (because linod is stuck). Yesterday and today I used “clinical software development”: I replaced a call to sync_to_async() by database_sync_to_async().

class Procedure(dd.Choice):
    ...
    async def run(self, ar):
        if True:
          return await sync_to_async(self.func)(ar)
        else:
          return await database_sync_to_async(self.func)(ar)

When calling sync_to_async(), pm prep in the noi1e demo project works:

Loading data from /home/luc/work/lino/lino/modlib/help/fixtures/demo2.py
Fixture lino.dpy_tmp_3876721588043865475 has no attribute 'objects'
Loading data from /home/luc/work/noi/lino_noi/lib/users/fixtures/demo2.py
Loading data from /home/luc/work/xl/lino_xl/lib/excerpts/fixtures/demo2.py
appy.pod render /home/luc/work/xl/lino_xl/lib/contacts/config/contacts/Person/TermsConditions.odt -> /home/luc/work/book/lino_book/projects/noi1e/settings/media/cache/appypdf/excerpts.Excerpt-1.pdf
weasy2pdf render ['subscriptions/Subscription/default.weasy.html', 'excerpts/default.weasy.html'] -> /home/luc/work/book/lino_book/projects/noi1e/settings/media/cache/weasy2pdf/subscriptions.Subscription-1.pdf ('en', {})
weasy2pdf render ['contacts/Partner/payment_reminder.weasy.html', 'excerpts/payment_reminder.weasy.html'] -> /home/luc/work/book/lino_book/projects/noi1e/settings/media/cache/weasy2pdf/excerpts.Excerpt-3.pdf ('en', {})
Loading data from /home/luc/work/lino/lino/modlib/comments/fixtures/demo2.py
Loading data from /home/luc/work/xl/lino_xl/lib/lists/fixtures/demo2.py
Loading data from /home/luc/work/lino/lino/modlib/notify/fixtures/demo2.py
Loading data from /home/luc/work/xl/lino_xl/lib/pages/fixtures/demo2.py
Update published pages...
72 pages have been updated.
Loading data from /home/luc/work/xl/lino_xl/lib/addresses/fixtures/demo2.py
Loading data from /home/luc/work/xl/lino_xl/lib/phones/fixtures/demo2.py
Loading data from /home/luc/work/xl/lino_xl/lib/ledger/fixtures/demo_bookings.py
Loading data from /home/luc/work/xl/lino_xl/lib/invoicing/fixtures/demo_bookings.py
Loading data from /home/luc/work/lino/lino/modlib/summaries/fixtures/checksummaries.py

But when I replace the True by False, it fails:

Loading data from /home/luc/work/xl/lino_xl/lib/pages/fixtures/demo2.py
Update published pages...
Traceback (most recent call last):
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 294, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 190, in create_cursor
    return self.connection.cursor(factory=SQLiteCursorWrapper)
sqlite3.ProgrammingError: Cannot operate on a closed database.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/luc/work/book/lino_book/projects/noi1e/manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/luc/work/lino/lino/management/commands/prep.py", line 55, in handle
    call_command('initdb', *args, **kwargs)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/luc/work/lino/lino/management/commands/initdb.py", line 271, in handle
    call_command('loaddata', *fixtures, **options)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata
    self.load_label(fixture_label)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label
    for obj in objects:
  File "/home/luc/work/lino/lino/utils/dpy.py", line 400, in deserialize
    for o in self.deserialize_module(module, **options):
  File "/home/luc/work/lino/lino/utils/dpy.py", line 413, in deserialize_module
    for obj in objects():
  File "/home/luc/work/xl/lino_xl/lib/pages/fixtures/demo2.py", line 10, in objects
    async_to_sync(Procedures.update_publisher_pages.run)(ar)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/asgiref/sync.py", line 277, in __call__
    return call_result.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/asgiref/sync.py", line 353, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/luc/work/lino/lino/modlib/linod/choicelists.py", line 28, in run
    return await database_sync_to_async(self.func)(ar)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
    ret: _R = await loop.run_in_executor(
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/channels/db.py", line 13, in thread_handler
    return super().thread_handler(loop, *args, **kwargs)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
    return func(*args, **kwargs)
  File "/home/luc/work/xl/lino_xl/lib/pages/models.py", line 344, in update_publisher_pages
    for obj in Page.get_publisher_pages():
  File "/home/luc/work/xl/lino_xl/lib/pages/models.py", line 266, in get_publisher_pages
    for root in cls.objects.filter(parent__isnull=True):
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/models/query.py", line 1926, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor = self.connection.cursor()
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 316, in cursor
    return self._cursor()
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 293, in _cursor
    with self.wrap_database_errors:
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/base/base.py", line 294, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 190, in create_cursor
    return self.connection.cursor(factory=SQLiteCursorWrapper)
django.db.utils.ProgrammingError: Problem installing fixture '/home/luc/work/xl/lino_xl/lib/pages/fixtures/demo2.py': Cannot operate on a closed database.

Saturday, December 16, 2023

Sat, 16 Dec 2023 00:00:00 +0000

Saturday, December 16, 2023¶

Strange. On Thursday morning around 8:00 I changed the level of the “Let background tasks sleep for 8.694031 seconds” logger message in BackgroundTask.start_task_runner() from “debug” to “info” so that Jane since then faithfully wrote this message every 10 seconds to the lino.log file. These messages appear until Thu at 20:07:09. And then they stop suddenly without any visible reason. This is also the moment when Jane stopped to send email notifications.

202312-14 09:35:31 INFO [lino 125638 140314165638912] : run_action luc tickets.Tickets.submit_detail None [Ticket #5278 ("#5278 (Jane doesn't send mails)")]
202312-14 10:11:56 INFO [lino 136977 139740051445568] : Started manage.py linod --force (using lino_local.jane.settings) --> PID 136977
202312-14 10:12:00 INFO [django.channels.worker 136977 139739903883008] : Running worker for channels ['linod_jane']
202312-14 10:12:00 INFO [lino 136977 139739903883008] : Start the log server...
202312-14 10:12:00 INFO [lino 136977 139739903883008] : Start the background tasks runner using <Logger lino (INFO)>...
202312-14 10:12:00 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.984451 seconds.
202312-14 10:12:03 INFO [lino 136972 140550985377600] : Started /usr/local/lino/lino_local/jane/env/bin/daphne -u /run/daphne/jane_daphne1.sock --fd 0 --access-log - --proxy-headers asgi:application (using lino_local.jane.settings) --> PID 136972
202312-14 10:12:03 INFO [lino 136971 140023566563136] : Started /usr/local/lino/lino_local/jane/env/bin/daphne -u /run/daphne/jane_daphne0.sock --fd 0 --access-log - --proxy-headers asgi:application (using lino_local.jane.settings) --> PID 136971
202312-14 10:12:10 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.99448 seconds.
202312-14 10:12:20 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996263 seconds.
202312-14 10:12:30 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995986 seconds.
202312-14 10:12:40 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.994652 seconds.
...
202312-14 10:16:03 INFO [lino 136971 140023299036928] : run_action luc tickets.Tickets.submit_detail None [Ticket #5278 ("#5278 (Jane doesn't send mails)")]
202312-14 10:16:11 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995164 seconds.
...
202312-14 10:20:22 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996292 seconds.
202312-14 10:20:28 INFO [lino 136972 140550819161856] : run_action luc tickets.Tickets.end_session None [Ticket #5278 ("#5278 (Jane doesn't send mails)")]
202312-14 10:20:32 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996359 seconds.
...
202312-14 19:53:10 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995296 seconds.
202312-14 19:53:20 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995327 seconds.
202312-14 19:53:30 INFO [lino 136977 139739903883008] : Let background tasks sleep for 4.892348 seconds.
202312-14 19:58:08 INFO [lino~ 136977 139739903883008] : Start Background task #8 generate_calendar_entries...
202312-14 19:58:08 INFO [lino~ 136977 139739903883008] : Start Background task #9 update_publisher_pages...
202312-14 19:58:08 INFO [lino~ 136977 139739873892096] : Update published pages...
202312-14 19:58:08 INFO [lino~ 136977 139739873892096] : 0 pages have been updated.
202312-14 19:58:08 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.563372 seconds.
202312-14 19:58:17 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996238 seconds.
202312-14 19:58:27 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996709 seconds.
...
202312-14 20:00:48 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995795 seconds.
202312-14 20:00:58 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996443 seconds.
202312-14 20:01:08 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996272 seconds.
202312-14 20:01:16 INFO [lino 136971 140023273858816] : run_action luc working.Sessions.end_session None [Session #26243 ('14.12.2023 16:50')]
202312-14 20:01:18 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996541 seconds.
202312-14 20:01:28 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.995331 seconds.
...
202312-14 20:06:49 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996506 seconds.
202312-14 20:06:59 INFO [lino 136977 139739903883008] : Let background tasks sleep for 9.996184 seconds.
202312-14 20:07:09 INFO [lino 136977 139739903883008] : Let background tasks sleep for 8.694031 seconds.
202312-14 20:23:10 WARNING [django.request 136971 140023307429632] : Not Found: /.well-known/traffic-advice
202312-14 21:29:11 INFO [lino 136972 140550714943232] : run_action luc working.MySessionsByDay.grid_put <lino_xl.lib.working.ui.Day object at 0x7fd4826495e0> [Session #26243 ('14.12.2023 16:50')]
202312-14 21:29:24 INFO [lino 136971 140022899271424] : run_action luc working.MySessionsByDay.grid_put <lino_xl.lib.working.ui.Day object at 0x7f59bc1520a0> [Session #26242 ('14.12.2023 10:20')]
202312-14 21:40:16 INFO [lino 136972 140550706550528] : run_action luc working.MySessionsByDay.grid_put <lino_xl.lib.working.ui.Day object at 0x7fd481615e50> [Session #26245 ('14.12.2023 21:40')]
202312-14 22:07:37 INFO [lino 136971 140023273858816] : run_action hannes comments.CommentsByRFC.submit_insert Ticket #5311 ('#5311 (Install a Zulip instance 2023)') []
202312-14 22:07:44 INFO [lino 136972 140550671939328] : run_action hannes working.Sessions.end_session None [Session #26244 ('14.12.2023 19:03')]
202312-14 22:20:43 INFO [lino 136972 140550706550528] : run_action luc working.Sessions.end_session None [Session #26245 ('14.12.2023 21:30')]
202312-14 22:20:58 INFO [lino 136972 140550706550528] : run_action luc tickets.AllTickets.submit_insert None []
202312-14 22:21:13 INFO [lino 136972 140550706550528] : run_action luc tickets.AllTickets.submit_detail None [Ticket #5319 ("#5319 (AttributeError: 'PosixPath' object has no attribute 'startswith')")]
202312-14 22:21:23 INFO [lino 136972 140550706550528] : run_action luc tickets.AllTickets.end_session None [Ticket #5319 ("#5319 (AttributeError: 'PosixPath' object has no attribute 'startswith')")]
202312-14 22:23:33 INFO [lino 136971 140023273858816] : run_action luc tickets.AllTickets.submit_insert None []
202312-14 22:24:01 INFO [lino 136972 140550706550528] : run_action luc tickets.AllTickets.submit_detail None [Ticket #5320 ("#5320 (unable to access 'https://gitlab.com/lino-framework/book.git/')")]
202312-14 22:24:11 INFO [lino 136971 140023273858816] : run_action luc tickets.AllTickets.end_session None [Ticket #5320 ("#5320 (unable to access 'https://gitlab.com/lino-framework/book.git/')")]
202312-15 00:31:15 WARNING [django.request 136972 140550819161856] : Not Found: /.well-known/traffic-advice

Friday, December 8, 2023

Fri, 08 Dec 2023 00:00:00 +0000

Friday, December 8, 2023¶

Yesterday I started a new Lino Amici production site for the “Association of Estonian Taizé Friends”, a small group of volunteers. My idea is to give them a collaboration and file sharing platform similar to Slack or Microsoft Teams. I am the only one to believe in this idea. I wouldn’t even say that I “believe” in it, I’d rather say I “feel a call to try” it.

Why not Mattermost? Because Mattermost is not really free any more. Okay, but why not Zulip? Led by Tim Abbott, Zulip seems very similar to Lino. His wife Alya Abbott is quite good with promoting the idea (read e.g. When “free forever” means “free for the next 4 months”). And Zulip has more than 1000 code contributors, their Contributing guide is obviously better than ours.

In other words, reasonably spoken, I should recommend Zulip to my Estonian Taizé friends rather than burdening them with a Lino.

My problem is that reality is not always reasonable.

SynchronousOnlyOperation

Sat, 02 Dec 2023 00:00:00 +0000

SynchronousOnlyOperation¶

Saturday, December 2, 2023

Hannes upgraded Jane yesterday, Jane still doesn’t send mails (#5278) The status of send_pending_emails_often is still Not started (because I had clicked on “Run now”) for this tasks. The other tasks are “Scheduled to run at 2023-12-02T03:31:14 (3 hours ago)” or “Running since 2023-10-21T19:53:32 (1 month, 1 week ago)”.

Hmm… did I forget to set lino.modlib.linod.use_channels to True? No.

Tilt! Here is probably the explanation:

$ sudo cat /var/log/supervisor/jane_linod-stdout---supervisor-kis1sukm.log
Started manage.py linod --force (using lino_local.jane.settings) --> PID 218446
Running worker for channels ['linod_jane']
Start the log server...
Start the background tasks runner using <Logger lino (INFO)>...
Task exception was never retrieved
future: <Task finished name='Task-32' coro=<LinodConsumer._run_background_tasks() done, defined at /usr/local/lino
/lino_local/jane/env/lib/python3.9/site-packages/lino/modlib/linod/consumers.py:88> exception=SynchronousOnlyOpera
tion('You cannot call this from an async context - use a thread or sync_to_async.')>
Traceback (most recent call last):
  File ".../python3.9/site-packages/lino/modlib/linod/consumers.py", line 96, in _run_background_tasks
    next_dt = await BackgroundTask.run_them_all(ar)
  File ".../python3.9/site-packages/lino/modlib/linod/models.py", line 91, in run_them_all
    self.full_clean()
  File ".../python3.9/site-packages/lino/mixins/sequenced.py", line 286, in full_clean
    super(Sequenced, self).full_clean(*args, **kw)
  File ".../python3.9/site-packages/lino/modlib/system/mixins.py", line 260, in full_clean
    super().full_clean(*args, **kw)
  File ".../python3.9/site-packages/django/db/models/base.py", line 1465, in full_clean
    self.validate_unique(exclude=exclude)
  File ".../python3.9/site-packages/django/db/models/base.py", line 1207, in validate_unique
    errors = self._perform_unique_checks(unique_checks)
  File ".../python3.9/site-packages/django/db/models/base.py", line 1317, in _perform_unique_checks
    if qs.exists():
  File ".../python3.9/site-packages/django/db/models/query.py", line 1226, in exists
    return self.query.has_results(using=self.db)
  File ".../python3.9/site-packages/django/db/models/sql/query.py", line 592, in has_results
    return compiler.has_results()
  File ".../python3.9/site-packages/django/db/models/sql/compiler.py", line 1366, in has_results
    return bool(self.execute_sql(SINGLE))
  File ".../python3.9/site-packages/django/db/models/sql/compiler.py", line 1396, in execute_sql
    cursor = self.connection.cursor()
  File ".../python3.9/site-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.
Started /usr/local/lino/lino_local/jane/env/bin/daphne -u /run/daphne/jane_daphne1.sock --fd 0 --access-log - --proxy-headers asgi:application (using lino_local.jane.settings) --> PID 218443
Started /usr/local/lino/lino_local/jane/env/bin/daphne -u /run/daphne/jane_daphne0.sock --fd 0 --access-log - --proxy-headers asgi:application (using lino_local.jane.settings) --> PID 218442

I upgraded Django (from 4.1.7 to 4.2.7)

Blog entries should be just another type of node

Sun, 05 Nov 2023 00:00:00 +0000

Blog entries should be just another type of node¶

Sunday, November 5, 2023

Here is my summary after having played a full week with #5193 (Blog entries should be just another type of node).

Visible changes in publisher: Hierarchical pages now have previous/next links. Blog entries now have the same layout as hierarchical pages.

After my playing I think that we actually do not want previous/next links on blog entries. In the admin view yes, there we want to navigate also in display mode “detail”, but not on a publisher link. Such links are meant to be shared in social media where people get quickly distracted by every useless detail.

The lino.modlib.publisher plugin is now used as menu group for the three other CMS-specific plugins (blogs, pages and albums).

The new setting lino.modlib.publisher.locations replaces the PublisherViews choicelist.

A bug caused the “Start now” button on a background task to not work. We can now run these tasks manually.

TODO: Think about #5204 (About Site.user_linod, checkdata, checksummaries & Co)

I reviewed the __init__() method of lino.modlib.linod.Procedure. Now it takes just the func object. That’s much easier to remember for application developers. This optimization revealed an issue in lino.core.choicelists: the deconstruct() method of a ChoiceListField must remove the choices keyword because the list of choices of a ChoiceListField does not influence whether we need a migration or not. As a side effect, Choice no longer needs to be @deconstructible. (See Friday, November 3, 2023 for details)

Removed duplicate definition of requests.BaseRequest.obj2url().

lino.modlib.users.User is now lino.modlib.publisher.Publishable

lino_xl/lib/pages/fixtures/demo2.py shows how to run a background task from a script.

I changed Plugin._active to Plugin.hidden and renamed Plugin.deactivate() to Plugin.hide(). You can now hide a plugin in the get_plugin_configs. I removed the obsolete Plugin.disables_plugins attribute.

The lino_xl.lib.topics plugin is now more easy to integrate because the InterestsByController slave panel becomes invisible when the plugin gets hidden.

The add_interest action is no longer needed since InterestsByController has an insert button in the slave summary.

The files utils.py and views.py in lino_xl.lib.pages were no longer used. I removed them.

When the plugin specified as the menu group of another plugin has itself a menu_group, then get_menu_group() now returns the menu_group of the menu group.

Friday, November 3, 2023

Fri, 03 Nov 2023 00:00:00 +0000

Friday, November 3, 2023¶

I had a not-so-trivial failure:

(dev) luc@yoga:~/work/book$ dt docs/specs/migrate.rst
**********************************************************************
File "docs/specs/migrate.rst", line 87, in migrate.rst
Failed example:
    shell("python manage.py prep --noinput")
    #doctest: +ELLIPSIS +REPORT_UDIFF +NORMALIZE_WHITESPACE
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,256 @@
    -`initdb std minimal_ledger demo demo2 demo_bookings checksummaries checkdata` started on database .../default.db.
    -...
    -Installed 4030 object(s) from 31 fixture(s)
    +`initdb std minimal_ledger demo demo2 demo_bookings checksummaries checkdata` started on database /home/luc/work/book/lino_book/projects/migs/default.db.
    +Migrations for 'calview':
    +  migrations/calview/0001_initial.py
    +    - Create model DailyPlannerRow
    +Migrations for 'countries':
    +  migrations/countries/0001_initial.py
    +    - Create model Country
    +    - Create model Place
    +Migrations for 'linod':
    +  migrations/linod/0001_initial.py
    +    - Create model BackgroundTask
    +Migrations for 'pages':
    +  migrations/pages/0001_initial.py
    +    - Create model Node
    +    - Create model Page
    +Migrations for 'userstats':
    +  migrations/userstats/0001_initial.py
    +    - Create model UserStat
    +Migrations for 'cal':
    +  migrations/cal/0001_initial.py
    +    - Create model Calendar
    +    - Create model EntryRepeater
    +    - Create model Event
    +    - Create model EventPolicy
    +    - Create model EventType
    +    - Create model Guest
    +    - Create model GuestRole
    +    - Create model RecurrentEvent
    +    - Create model RemoteCalendar
    +    - Create model Room
    +    - Create model Subscription
    +    - Create model Task
    +  migrations/cal/0002_initial.py
    +    - Add field user to task
    +    - Add field calendar to subscription
    +    - Add field user to subscription
    +    - Add field calendar to room
    +    - Add field company to room
    +    - Add field contact_person to room
    +    - Add field contact_role to room
    +    - Add field event_type to recurrentevent
    +    - Add field user to recurrentevent
    +    - Add field event to guest
    +    - Add field partner to guest
    +    - Add field role to guest
    +    - Add field event_type to eventpolicy
    +    - Add field assigned_to to event
    +    - Add field company to event
    +    - Add field contact_person to event
    +    - Add field contact_role to event
    +    - Add field event_type to event
    +    - Add field owner_type to event
    +    - Add field previous_page to event
    +    - Add field room to event
    +    - Add field user to event
    +    - Add field cal_entry to entryrepeater
    +    - Alter unique_together for guest (1 constraint(s))
    +Migrations for 'changes':
    +  migrations/changes/0001_initial.py
    +    - Create model Change
    +  migrations/changes/0002_initial.py
    +    - Add field user to change
    +Migrations for 'checkdata':
    +  migrations/checkdata/0001_initial.py
    +    - Create model Message
    +  migrations/checkdata/0002_initial.py
    +    - Add field user to message
    +Migrations for 'comments':
    +  migrations/comments/0001_initial.py
    +    - Create model Comment
    +    - Create model CommentType
    +    - Create model Reaction
    +  migrations/comments/0002_initial.py
    +    - Add field user to reaction
    +    - Add field comment_type to comment
    +    - Add field owner_type to comment
    +    - Add field reply_to to comment
    +    - Add field user to comment
    +Migrations for 'contacts':
    +  migrations/contacts/0001_initial.py
    +    - Create model CompanyType
    +    - Create model Partner
    +    - Create model RoleType
    +    - Create model Company
    +    - Create model Person
    +    - Create model Role
    +  migrations/contacts/0002_initial.py
    +    - Add field payment_term to partner
    +    - Add field purchase_account to partner
    +    - Add field region to partner
    +    - Add field company to role
    +    - Add field person to role
    +    - Add field type to company
    +Migrations for 'lists':
    +  migrations/lists/0001_initial.py
    +    - Create model List
    +    - Create model ListType
    +    - Create model Member
    +    - Add field list_type to list
    +Migrations for 'system':
    +  migrations/system/0001_initial.py
    +    - Create model SiteConfig
    +Migrations for 'users':
    +  migrations/users/0001_initial.py
    +    - Create model User
    +    - Create model Authority
    +Migrations for 'dashboard':
    +  migrations/dashboard/0001_initial.py
    +    - Create model Widget
    +Migrations for 'groups':
    +  migrations/groups/0001_initial.py
    +    - Create model Group
    +    - Create model Membership
    +Migrations for 'tinymce':
    +  migrations/tinymce/0001_initial.py
    +    - Create model TextFieldTemplate
    +Migrations for 'excerpts':
    +  migrations/excerpts/0001_initial.py
    +    - Create model ExcerptType
    +    - Create model Excerpt
    +Migrations for 'invoicing':
    +  migrations/invoicing/0001_initial.py
    +    - Create model FollowUpRule
    +    - Create model Item
    +    - Create model Plan
    +    - Create model SalesRule
    +    - Create model Tariff
    +  migrations/invoicing/0002_initial.py
    +    - Add field product to tariff
    +    - Add field invoice_recipient to salesrule
    +  migrations/invoicing/0003_initial.py
    +    - Add field paper_type to salesrule
    +    - Add field order to plan
    +    - Add field partner to plan
    +    - Add field user to plan
    +    - Add field generator_type to item
    +    - Add field invoice to item
    +    - Add field partner to item
    +    - Add field plan to item
    +    - Add field source_journal to followuprule
    +Migrations for 'ledger':
    +  migrations/ledger/0001_initial.py
    +    - Create model Account
    +    - Create model AccountingPeriod
    +    - Create model FiscalYear
    +    - Create model Journal
    +    - Create model LedgerInfo
    +    - Create model PaymentTerm
    +    - Create model Voucher
    +    - Create model Movement
    +    - Create model MatchRule
    +  migrations/ledger/0002_initial.py
    +    - Add field uploads_volume to journal
    +    - Add field year to accountingperiod
    +    - Alter unique_together for matchrule (1 constraint(s))
    +Migrations for 'vat':
    +  migrations/vat/0001_initial.py
    +    - Create model VatAccountInvoice
    +    - Create model InvoiceItem
    +Migrations for 'memo':
    +  migrations/memo/0001_initial.py
    +    - Create model Mention
    +Migrations for 'notify':
    +  migrations/notify/0001_initial.py
    +    - Create model Message
    +Migrations for 'products':
    +  migrations/products/0001_initial.py
    +    - Create model Category
    +    - Create model Product
    +    - Create model PriceRule
    +  migrations/products/0002_initial.py
    +    - Add field selector to pricerule
    +    - Add field parent to category
    +Migrations for 'sales':
    +  migrations/trading/0001_initial.py
    +    - Create model PaperType
    +    - Create model VatProductInvoice
    +    - Create model InvoiceItem
    +Migrations for 'subscriptions':
    +  migrations/subscriptions/0001_initial.py
    +    - Create model Subscription
    +    - Create model SubscriptionPeriod
    +    - Create model SubscriptionItem
    +Migrations for 'tickets':
    +  migrations/tickets/0001_initial.py
    +    - Create model Site
    +    - Create model TicketType
    +    - Create model Ticket
    +    - Create model CheckListItem
    +Migrations for 'working':
    +  migrations/working/0001_initial.py
    +    - Create model SessionType
    +    - Create model UserSummary
    +    - Create model SiteSummary
    +    - Create model Session
    +    - Create model ServiceReport
    +Migrations for 'storage':
    +  migrations/storage/0001_initial.py
    +    - Create model TransferRule
    +    - Create model Provision
    +    - Create model Movement
    +    - Create model Filler
    +    - Create model DeliveryNote
    +    - Create model DeliveryItem
    +    - Create model Component
    +Migrations for 'uploads':
    +  migrations/uploads/0001_initial.py
    +    - Create model UploadType
    +    - Create model Volume
    +    - Create model Upload
    +Traceback (most recent call last):
    +  File "/home/luc/work/book/lino_book/projects/migs/manage.py", line 9, in <module>
    +    execute_from_command_line(sys.argv)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    +    utility.execute()
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    +    self.fetch_command(subcommand).run_from_argv(self.argv)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    +    self.execute(*args, **cmd_options)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    +    output = self.handle(*args, **options)
    +  File "/home/luc/work/lino/lino/management/commands/prep.py", line 43, in handle
    +    super().handle(**options)
    +  File "/home/luc/work/lino/lino/management/commands/initdb.py", line 261, in handle
    +    call_command('migrate', '--run-syncdb', **options)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/__init__.py", line 194, in call_command
    +    return command.execute(*args, **defaults)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    +    output = self.handle(*args, **options)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/base.py", line 106, in wrapper
    +    res = handle_func(*args, **kwargs)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    +    executor = MigrationExecutor(connection, self.migration_progress_callback)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
    +    self.loader = MigrationLoader(self.connection)
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
    +    self.build_graph()
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/migrations/loader.py", line 229, in build_graph
    +    self.load_disk()
    +  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/django/db/migrations/loader.py", line 120, in load_disk
    +    migration_module = import_module(migration_path)
    +  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    +    return _bootstrap._gcd_import(name[level:], package, level)
    +  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    +  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    +  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    +  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    +  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    +  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    +  File "/home/luc/work/book/lino_book/projects/migs/migrations/linod/0001_initial.py", line 17, in <module>
    +    class Migration(migrations.Migration):
    +  File "/home/luc/work/book/lino_book/projects/migs/migrations/linod/0001_initial.py", line 195, in Migration
    +    lino.modlib.linod.choicelists.Procedure(
    +  File "/home/luc/work/lino/lino/modlib/linod/choicelists.py", line 20, in __init__
    +    name = func.__name__
    +AttributeError: 'NoneType' object has no attribute '__name__'. Did you mean: '__ne__'?

This error was because the migration file linod/0001_initial.py had something like this:

(
    "procedure",
    lino.core.choicelists.ChoiceListField(
        choicelist=lino.modlib.linod.choicelists.Procedures,
        choices=[
            (
                lino.modlib.linod.choicelists.Procedure(
                    lino_xl.lib.cal.models.event_notification_scheduler,
                    every=300,
                    every_unit="secondly",
                ),
                "event_notification_scheduler",
            ),
            ...
        ],
        help_text="Pointer to an instance of Procedure.",
        max_length=100,
        unique=True,
        verbose_name="Background procedure",
    ),
),

I fixed it by reviewing the deconstruct() method of the lino.core.choicelists.ChoiceListField class. It now removes the ‘choices’ key from kwargs:

  def deconstruct(self):
      name, path, args, kwargs = super().deconstruct()
      kwargs['choicelist'] = self.choicelist
+     kwargs.pop('choices', None)
      return name, path, args, kwargs

Because the available choices are not relevant for the database structure, and Django might have problems for representing them in a migration.

This didn’t disturb until now, but I changed the signature of Procedure.__init__(): it now takes just the function object. And function objects are not always serializable.

Wednesday, November 1, 2023

Wed, 01 Nov 2023 00:00:00 +0000

Wednesday, November 1, 2023¶

When a site has its calendar events, or products or users…, published as PublisherViews, then we probably do not want calendar entries or users to become an instance of pages.Node (with fields like title and body, with potential child nodes, etc). But since they (potentially) generate a page on their own, we want them to have previous/next links. I am not so sure any more whether these previous/next links should integrate them into a “one and only sequence of pages”. We can say that a Lino site with publisher has one pagination suite per PublisherView.

Saturday, October 28, 2023

Sat, 28 Oct 2023 00:00:00 +0000

Saturday, October 28, 2023¶

I am meditating and juggling with the database structure of pages and blogs.

The big difference with established systems is that in Lino we don’t provide a WYSIWYG editor. Every section or subsection of a page is a node in the Lino database. Lino CMS is not like SimpleCMS but rather like CMSimple. Why is this so cool? See this screencast by Black Scorpion.

I am meditating about questions like: Is there one big pagination sequence, which includes special pages like automatically generated monthly overviews? Can blog entries have multiple pages? Are there pages where we want the children to sorted alphabetically by name? Should we rename the current concept of ContentFillers to NodeGenerator?

- Node(Polymorphic, Hierarchical, Sequenced, Previewable, Commentable):
  - title
  - body
  - child_node_depth
  - node_type
  - filler

- Publishable:
  - publishing_state
  - language

- pages.Page(Node, Publishable):
  - ref (or name)
  - translated_from

- blogs.Entry(Node, Publishable):
  - pub_date
  - pub_time
  - author

Some fundamental changes in background jobs and logging

Sun, 15 Oct 2023 00:00:00 +0000

Some fundamental changes in background jobs and logging¶

Sunday, October 15, 2023

I pushed yet another series of changes in lino.modlib.linod and lino.core.logging: 2023-10-15 16:10. They are a follow-up of my changes from 4 days ago: 2023-10-11 09:05

I have been working more than a week on this. It all started with a customer who observed that a background job apparently was not running. Ticket #5172 (lino_runworker is running but doesn’t work).

We no longer have a history of the individual background job runs. If I want that, I can consult the lino.log file. What we want is a detailed log of the last run. And we want a way to manually set a higher logging level for an individual job.

A side effect is the log server. I partly rewrote how we had been doing things for #4595 (Multiple Lino processes logging to single file). And this time it is also about #5179 (Write to lino.log even when log server is not running). The decision whether a Lino process writes directly to the lino.log or logs via a socket server does not depend on whether we are in an async environment. The processes now simply check whether the socket file exists. If it exists, they connect to the log server, otherwise they write directly to the file. On a production site the log server is supposed to run as a consumer in linod. The linod process is responsible for starting the log server. All other processes become clients of that server when the socket file exists and otherwise write to the file themselves.

Tuesday, August 22, 2023

Tue, 22 Aug 2023 00:00:00 +0000

Tuesday, August 22, 2023¶

This page contains code snippets (lines starting with >>>), which are being tested during our development workflow. The following snippet initializes the demo project used throughout this page.

>>> import datetime
>>> from lino_book.projects.noi1r.startup import *
>>> from lino.core.diff import ChangeWatcher
>>> from etgen.html import tostring
>>> from lino.modlib.notify.mixins import ChangeNotifier

>>> obj = tickets.Ticket.objects.first()
>>> cw = None
>>> cw = ChangeWatcher(obj)
>>> ar = rt.login("robin", renderer=settings.SITE.kernel.default_renderer, permalink_uris=True)
>>> obj.summary = "foo"

Just to verify that our change watcher works:

>>> cw.is_dirty()
True
>>> tostring(cw.get_updates_html())
'<li><b>Summary</b> : Föö fails to bar when baz --&gt; foo</li>'

The reason for #5038 (Links in email notifications are escaped) on 2023-08-22 was that the <a href ...> in the text returned by get_change_body() was getting escaped. This is fixed now:

>>> obj.get_change_body(ar, cw)
'<div><p>Robin Rood modified <a href="/#/api/tickets/Tickets/1">#1 (foo)</a>:</p><ul><li><b>Summary</b> : Föö fails to bar when baz --&gt; foo</li></ul></div>'

That’s fun! Let’s repeat this for other objects in order to increase test coverage!

>>> pprint(rt.models_by_base(ChangeNotifier))
[<class 'lino_noi.lib.cal.models.Event'>,
 <class 'lino_xl.lib.cal.models.Task'>,
 <class 'lino.modlib.comments.models.Comment'>,
 <class 'lino_noi.lib.groups.models.Group'>,
 <class 'lino_noi.lib.tickets.models.Site'>,
 <class 'lino_noi.lib.tickets.models.Ticket'>]

>>> def test(obj, **kwargs):
...     cw = ChangeWatcher(obj)
...     print(obj.get_change_body(ar, None))
...     for k, v in kwargs.items():
...         setattr(obj, k, v)
...     print(obj.get_change_body(ar, cw))

>>> test(tickets.Site.objects.first(), name="Foo")
<div><p>Robin Rood created <a href="/#/api/tickets/Sites/1">pypi</a></p>.</div>
<div><p>Robin Rood modified <a href="/#/api/tickets/Sites/1">pypi</a>:</p><ul><li><b>Designation</b> : pypi --&gt; Foo</li></ul></div>

>>> test(cal.Event.objects.first(), start_time=datetime.time(10,5))
<div><p>Robin Rood created <a href="/#/api/cal/Events/1">New Year's Day (01.01.2013)</a></p>.</div>
<div><p>Robin Rood modified <a href="/#/api/cal/Events/1">New Year's Day (01.01.2013 10:05)</a>:</p><ul><li><b>Start time</b> : None --&gt; 10:05:00</li></ul></div>

>>> test(comments.Comment.objects.first(), body="Foo")
Robin Rood commented on <a href="/#/api/groups/Groups/1">Developers</a>:<br>Styled comment <span>pasted from word!</span>
Robin Rood modified comment on <a href="/#/api/groups/Groups/1">Developers</a>:<br>Foo

Friday, August 4, 2023

Fri, 04 Aug 2023 00:00:00 +0000

Friday, August 4, 2023¶

Most of the Belgian IBAN numbers in lino_xl.lib.sepa.fixtures.sample_ibans were actually invalid. Their IBAN check digit was correct, but the the programmers at www.mobilefish.com did not know that Belgian bank account numbers have their own check digit. I wrote the following doctest snippet to show only IBANs that are really valid. Statistically only one of 97 is valid. And indeed, I had to get 1000 fictive IBANS from mobilefish in order to harvest 10 good ones.

Edit 2023-08-06 Seems that Montenegro, Norway and Spain are like Belgium: they have check digits in national bank account numbers. And django-localflavor now also knows about them. So I need to update our sample_ibans.py file even mode.

>>> from lino_book.projects.voga2.startup import *
>>> from django.core.exceptions import ValidationError

>>> partner = rt.models.contacts.Partner.objects.first()
>>> Account = rt.models.sepa.Account
>>> ibans = """
... BE28724246719796 BE58483388989105 BE71734773678845 BE20174483803342
... BE71350811436250 BE18795488614610 BE84858551943938 BE32541820604815
... BE79203018402084 BE92227724179921 BE95657847530720 BE43887359333506
... BE05869216347393 BE13716672918550 BE09501756025056 BE25343348640131
... BE24862868507486 BE12202401892242 BE05430828656001 BE13677784802295
... BE05194204714338 BE85616511999691 BE93229218900239 BE32527262674483
... BE58067461123040 BE47884396692310 BE07364188327453 BE25882933953755
... BE36743323765718 BE67225381485065 BE79188638158195 BE02873579557797
... BE75032418043799 BE80296965174776 BE93322502171512 BE03181994160471
... BE41414473568385 BE87770998218656 BE74850873948015 BE82345649175398
... BE36315143736533 BE52849176505835 BE51427288306202 BE30999252618777
... BE95100449016428 BE46137417838797 BE16140712719138 BE70559894379973
... BE22338307655430 BE21744478962171 BE71698886383884 BE14776649354707
... BE49293501559571 BE12917065059682 BE03224307739576 BE89614705334946
... BE33025520765368 BE48644301836586 BE25869200566588 BE59398443056673
... BE05700351013001 BE93089106618187 BE53872384563730 BE23239846666050
... BE63289999168582 BE37977237425707 BE88226804784990 BE62994004047044
... BE96595371107901 BE27828364925134 BE41039439267385 BE60693753310146
... BE65327184565134 BE56614221337647 BE64494692585043 BE57105155123917
... BE26284315602949 BE29591695872615 BE37959588257665 BE68987243668796
... BE45339848048869 BE55790559805518 BE38848654741923 BE86388411676445
... BE18815106158450 BE83243857077493 BE26079576344270 BE34909685347786
... BE75136602618071 BE09595989391153 BE50000780506886 BE18764513455327
... BE25895552585494 BE06894601004584 BE98588934438901 BE22603142434371
... BE69478045723246 BE68355523201119 BE72921883510322 BE56680163678700
... BE54800210387334 BE14602195879124 BE14111760055869 BE85541535832328
... BE95454268362165 BE66224956682022 BE82747111508384 BE63097372245098
... BE39017955198559 BE54141796986685 BE02386289845249 BE33165363312439
... BE37048748480744 BE45544165675126 BE38554957819398 BE79623276881455
... BE89898236136387 BE42243901851491 BE98072909887412 BE50952449538745
... BE88119601752690 BE53064270609121 BE72120202551966 BE54073150762678
... BE58021988213001 BE29111649250074 BE50400776918739 BE41431198043381
... BE13404128585528 BE02397318558330 BE09840005458667 BE26552714094087
... BE66871155598522 BE55230546253482 BE21081301568124 BE31819267452784
... BE10602557401591 BE46110929739076 BE20492131301514 BE06175706664358
... BE87036264021027 BE40388531141018 BE10956348136305 BE14060594535639
... BE86406360676337 BE45186969681595 BE23081394358385 BE44153833882541
... BE82056316229523 BE51971075990111 BE09123506186860 BE77163131178008
... BE44094354583297 BE58515243244935 BE66527070962392 BE51455728344586
... BE14422646024996 BE95285529423235 BE79997405640022 BE79515119267402
... BE17294520366861 BE77659434470056 BE46437678380103 BE21046594839793
... BE28707245492257 BE66772298074615 BE65591592700051 BE08417691719646
... BE46994754655545 BE46994715585303 BE70198230882080 BE89786706663818
... BE83147213038781 BE20919908416561 BE52084284322416 BE77062925956471
... BE69926866200847 BE39971912183483 BE46152658693264 BE81995169644874
... BE60541297944643 BE81613962507635 BE75396642852203 BE39331066339247
... BE68091978629654 BE10201865242785 BE29955478077488 BE17642254548354
... BE78928625319741 BE76481174443473 BE88340025424235 BE39782462763107
... BE42030326767364 BE40507723623384 BE95378722370630 BE32565632366187
... BE36584235476583 BE46097919593318 BE91461112302870 BE28205672095343
... BE44195998043428 BE21861150713782 BE34066558619079 BE97495686032841
... BE24085603932842 BE12765411817734 BE71455656640371 BE06549670794137
... BE75287509344139 BE14676252110030 BE04451717299984 BE03246394845507
... BE52172466880408 BE28716937619446 BE67463379563939 BE26163669550654
... BE74247004641616 BE93792919154943 BE44476843358644 BE32260741612622
... BE15221821872539 BE20963639239289 BE55251585703250 BE16605977455807
... BE24087255453678 BE03547117460087 BE09256629686327 BE54447710192224
... BE50492124412325 BE75667145882210 BE87564352495593 BE15419133819199
... BE13832959147250 BE90566844867665 BE65829378217102 BE66603677788801
... BE54701121538409 BE64342015916077 BE23494486625099 BE59284163752638
... BE26972406338383 BE10475640265055 BE25014023940616 BE88208060378114
... BE40489011420411 BE03101360800721 BE60875447503698 BE74686132563967
... BE24247974947484 BE45664007197781 BE41890855214900 BE22692494338615
... BE65276303082923 BE39366664679647 BE14787967528107 BE38852937497563
... BE72731712721122 BE70295115332576 BE13736906734624 BE14707219749097
... BE38564130991640 BE32670485850228 BE87280359733495 BE09783848971732
... BE31687252082772 BE68438161760971 BE70173662148713 BE03082374528031
... BE83893940225618 BE06136453008356 BE19585174098553 BE80974324128873
... BE38868357557105 BE61037173495435 BE37691929441482 BE64299048243953
... BE32337470091678 BE43413671573901 BE58146083846226 BE50340320986150
... BE71333767856183 BE75309211783572 BE08409636850510 BE13695968136436
... BE35656234032345 BE94450163311804 BE34650063295980 BE42745684004821
... BE56453349648222 BE70288702006565 BE14131045229112 BE51529158472405
... BE53810212784367 BE91780121995904 BE97703964669812 BE69549508879827
... BE53501790349945 BE42536309052607 BE30020533437478 BE05514289019738
... BE32235606513230 BE91338863643153 BE57638748841340 BE03932968735390
... BE71038767281885 BE49745523130389 BE49284415413983 BE90358697467620
... BE27618825406018 BE11497546600798 BE89240330084926 BE02740696876932
... BE03164454824947 BE40316262308548 BE98597531902854 BE55540934033907
... BE02915819629526 BE86106703272689 BE85475487383557 BE23136532831387
... BE11804397580221 BE34829322365739 BE11458761645789 BE29130173930513
... BE29574484390883 BE19820541587867 BE73845165949471 BE11825210576117
... BE25499641614403 BE80564793249460 BE43437215784441 BE54138698693486
... BE61436161739340 BE27801411207953 BE55305014805004 BE13191398912855
... BE61189107260898 BE93312467480966 BE60440817758176 BE10863073287154
... BE59216818569552 BE07450905772904 BE85395276000288 BE57235129398589
... BE21523605221490 BE39687726524954 BE02785384795847 BE94282431297379
... BE37708963319221 BE20576597943736 BE58957930189048 BE89739470612521
... BE63988170278167 BE83796298703120 BE82296300383494 BE28129593595051
... BE24484557233273 BE69427657108123 BE57042159287068 BE83179882282306
... BE93341283519480 BE90729206219167 BE81269555020876 BE40766343569548
... BE53678632834066 BE93585347895031 BE69292012914179 BE49897120789494
... BE51985746520177 BE03895667683762 BE02408418329440 BE03609766419892
... BE78080963195984 BE68666652227996 BE54040081116636 BE35181410464917
... BE42606619499794 BE16513989595769 BE88647320882684 BE11272937316106
... BE38724890005186 BE07626747483096 BE02474894818162 BE53196039300567
... BE63988451926882 BE75182156579758 BE48386991558038 BE33952339454042
... BE56185415568252 BE34313568988032 BE03904423255096 BE77579995841458
... BE09071520707006 BE83207930496659 BE49318050525114 BE31056852421027
... BE51182515387458 BE42236944879280 BE50276834170512 BE58541970254007
... BE93271169674318 BE23136649204033 BE02851794180163 BE80778447523029
... BE21275778332339 BE51053181131687 BE56387983248355 BE38309089046355
... BE29002716425310 BE76365878771595 BE80911864086850 BE91136216227692
... BE08672781704375 BE34729906921209 BE85756866442049 BE48366731576602
... BE49519182254756 BE68035028622967 BE78277046396890 BE93837490942257
... BE66433528981837 BE86666893930914 BE15312933330011 BE43946329477713
... BE08594672453793 BE21433591412817 BE70467754184830 BE04374291204979
... BE64538515733632 BE87947327527963 BE51913563902956 BE04850007988759
... BE68155990881682 BE47473384527061 BE15906972704910 BE15944156398857
... BE14728111576893 BE71145331534847 BE92574219826420 BE92566945319180
... BE31013335466615 BE44090129621033 BE26226728108576 BE10231014323427
... BE68940179202254 BE56847196211284 BE55357359022868 BE23405282438663
... BE67824364770999 BE30009950029281 BE91377237232123 BE66386114502726
... BE60333519814092 BE44176520441100 BE67441017594540 BE27848533546247
... BE08833950119516 BE93975044034606 BE23645988190863 BE47472613381523
... BE54252608974116 BE87374084517534 BE42565800454106 BE98187154105383
... BE33050012842933 BE75116844443922 BE04699886831159 BE60044624208237
... BE20643226085265 BE32639360377999 BE93362702274071 BE19836703824285
... BE52073997617334 BE29024174274005 BE81349294644257 BE28732607237193
... BE55672277141733 BE36285935021940 BE54708460513595 BE34184510926957
... BE88170644701295 BE61709688753174 BE78965942728382 BE03956159662133
... BE35034870042820 BE67215399792312 BE90719088285064 BE51256194236566
... BE13167534988472 BE39947591745478 BE70077503814027 BE86161552353369
... BE86034767894306 BE74516318801696 BE94921930044512 BE28894110309803
... BE71054229463968 BE96536461902198 BE33518631756667 BE55668105864313
... """.split()

>>> def print_if_good(iban):
...    acct = Account(partner=partner, iban=iban, primary=True)
...    try:
...        acct.full_clean()
...        print(acct.iban)
...    except ValidationError:
...        pass
>>> for i in ibans:
...    print_if_good(i)

OVH being blacklisted by UCEPROTECT

Tue, 25 Jul 2023 00:00:00 +0000

OVH being blacklisted by UCEPROTECT¶

Tuesday, July 25, 2023

Our main mail server, saffre-rumma.net, hosted at OVH, is currently blacklisted by UCEPROTECTL3. Which (according to mxtoolbox.com) means that our IP address “falls into a poor reputation range. The entire range of IP addresses is blocked because the provider (OVH in our case) also hosts spammers. UCEPROTECTL3 does not offer any form of manual request to delist. Our IP address will automatically expire from listing after some time.”

uceprotect.net says that “UCEPROTECT Level 3 is the highest possible escalation: complete Autonomus Systems (AS) get listed at Level 3 if there were too many impacts from IP’s listed in Level 1 originating from said AS within the last 7 days. OVH has 4,197,632 IP addresses and 609 of these have been level 1 listed abusers within the last 7 days.”

I was seriously considering to move away from OVH to another provider. But then I contacted them and submitted support request ticket 8121647 with the following text:

uceprotect.net recommends us to “send a complaint to your provider and request they fix this problem immediately. Think about this: You pay them so that you can use the Internet without problems; if they are ignoring your complaint or claiming they can’t do anything, you should consider changing your provider. There are currently about 105,000 providers worldwide, but only a few hundred make it to get listed into UCEPROTECT-Level 3.”

OVH answered to my support ticket as follows:

Since January 2021, UCE Protect has placed our ASN (AS16276) on their blacklist like many others.

You can see this by clicking on the following link: http://www.uceprotect.net/en/l3charts.php

Our Trust and Safety legal abuse team is unable to contact UCE Protect to rectify the situation. Indeed, any attempt to communicate with them requires a paid subscription, which OVHcloud does not wish to take out.

As a result, we have no idea why our ASN has been blacklisted.

To date, we have found that no e-mail provider uses UCE Protect in their spam filters.

If this affects you, we recommend the following:

1. Prioritize an IPv6 configuration for sending e-mail, as UCE Protect does not blacklist e-mails sent via IPv6. All our OVHcloud services come with at least one IPv6 address that you can configure. All major e-mail providers now support IPv6.

Below you’ll find documentation to help you configure your IPv6: https://help.ovhcloud.com/csm/fr-vps-configuring-ipv6?id=kb_article_view&sysparm_article=KB0047576

2. Ask the receiving party to contact their e-mail provider to stop using UCE Protect in their anti-spam filters.

We invite you to consult the following guide for best practices in sending your e-mails: https://docs.ovh.com/fr/dedicated/optimiser-envoi-emails/

OVHcloud is committed to an open and reliable Cloud. Thanks to the cooperation of users like you, we’ll be able to continually improve our service in the future.

Other observations:

https://www.mail-tester.com/test-4bf413oij gives our email server a score of 9.50/10
A discussion on reddit.com comes to a similar conclusion.
We are not blacklisted on Spamhaus, only on UCEPROTECT.

There are many other people who witness that uceprotect is a dubious scammer. For example Arno Wetzel or Programmer Bear

Related tickets:

#3936 (167.114.229.225 has been blacklisted UCEPROTECTL3) (167.114.229.225 was the IP address of lino-framework.org from 2019-10-12) to 2021-03-14.
#5088 (Client host 51.68.71.43 blocked by cbl.mailcore.net in reply to RCPT TO command)

Conclusion¶

My conclusion is that UCEPROTECT lives from asking Internet providers to pay money for being whitelisted. And OVH refuses to give in to this blackmailing. I honor OVH’s civil courage.

I can witness that OVH does monitor their IP addresses and blocks them successfully when they start spamming. One of our servers once got hacked and was being misused for spamming, and OVH blocked our server within minutes, informed us about the situation and provided an easy method for unblocking our IP address after having fixed our problem.

Only a minority of email providers still trust in the UCEPROTECT blacklist.

The following email providers still trust UCEPROTECT and thus I cannot send any email to direct or indirect users of these:

MailCore (Denmark), the mail service provider behind addresses @hot.ee and @online.ee
one.com (One.com Group AB in Sweden)
hotmail.com (Microsoft) (Example error message: “host hotmail-com.olc.protection.outlook.com[104.47.56.161] said: 550 5.7.1 Unfortunately, messages from [51.68.71.43] weren’t sent. Please contact your Internet service provider since part of their network is on our block list (S3150). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [CO1NAM11FT107.eop-nam11.prod.protection.outlook.com 2024-01-28T19:45:45.245Z 08DC1F653F790B69] (in reply to MAIL FROM command)”

Last updated 2024-08-15

Friday, July 21, 2023

Fri, 21 Jul 2023 00:00:00 +0000

Friday, July 21, 2023¶

I had the following traceback during getlino startsite:

(pilv) root@pilv:/home/luc# getlino startsite amici alpha
Running as root.
This is getlino version 23.7.0 running on Debian GNU/Linux 11 (bullseye) (debian bullseye).
Preparing to reinstall amici to /usr/local/lino/lino_local/alpha
Shared virtualenv []:
Okay to reinstall amici to /usr/local/lino/lino_local/alpha? [y or n] Yes
Overwrite existing file /usr/local/lino/lino_local/alpha/manage.py ? [y or n] Yes
Overwrite existing file /usr/local/lino/lino_local/alpha/make_snapshot.sh ? [y or n] Yes
Overwrite existing file /usr/local/lino/lino_local/alpha/asgi.py ? [y or n] Yes
Installing 6 Python packages...
run . /usr/local/lino/lino_local/alpha/env/bin/activate && pip install -q --upgrade daphne mysqlclient lino lino-amici lino-xl lino-react [y or n] Yes
. /usr/local/lino/lino_local/alpha/env/bin/activate && pip install -q --upgrade daphne mysqlclient lino lino-amici lino-xl lino-react
Overwrite existing file /etc/supervisor/conf.d/alpha-asgi.conf ? [y or n] Yes
Overwrite existing file /etc/nginx/sites-available/alpha.conf ? [y or n] Yes
run . /usr/local/lino/lino_local/alpha/env/bin/activate && python manage.py install --noinput [y or n] No
run . /usr/local/lino/lino_local/alpha/env/bin/activate && python manage.py collectstatic --noinput [y or n] Yes
. /usr/local/lino/lino_local/alpha/env/bin/activate && python manage.py collectstatic --noinput
Run worker process ...
Started manage.py collectstatic --noinput (using lino_local.alpha.settings) --> PID 1963049

0 static files copied to '/usr/local/lino/lino_local/alpha/env/static_root', 4007 unmodified.
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/alpha/manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lino/lino_local/alpha/env/lib/python3.9/site-packages/lino/core/management/__init__.py", line 144, in execute_from_command_line
    stop_utility(argv)
  File "/usr/local/lino/lino_local/alpha/env/lib/python3.9/site-packages/lino/core/management/__init__.py", line 121, in stop_utility
    sock = get_socket()
  File "/usr/local/lino/lino_local/alpha/env/lib/python3.9/site-packages/lino/core/management/__init__.py", line 55, in get_socket
    sock.connect(str(worker_sock_file))
FileNotFoundError: [Errno 2] No such file or directory
Done manage.py collectstatic --noinput (PID 1963049)
Error: . /usr/local/lino/lino_local/alpha/env/bin/activate && python manage.py collectstatic --noinput ended with return code 1

This getlino was running in a sudo su session. Afterwards I tried to reproduce the error in a normal session and it didn’t fail again:

luc@pilv:~$ go alpha
luc@pilv:/usr/local/lino/lino_local/alpha$ a
(env) luc@pilv:/usr/local/lino/lino_local/alpha$ pm collectstatic
Started manage.py collectstatic (using lino_local.alpha.settings) --> PID 1963126

You have requested to collect static files at the destination
location as specified in your settings:

    /usr/local/lino/lino_local/alpha/env/static_root

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

0 static files copied to '/usr/local/lino/lino_local/alpha/env/static_root', 4007 unmodified.
Done manage.py collectstatic (PID 1963126)

Unfortunately the message “FileNotFoundError: [Errno 2] No such file or directory” is not very helpful. We would want to know the name of the file it was trying to connect to. So I changed lino.core.management.get_socket() to be more verbose in the future.

Wednesday, July 19, 2023

Wed, 19 Jul 2023 00:00:00 +0000

Wednesday, July 19, 2023¶

Le planificateur quotidien en page d’accueil reprend tous les RV de la journée dans la case « Avant-midi » et répète les dates pm dans la case « Après-midi ».
Il n’y a plus d’exportation de listes excel ou pdf possible ni d’impressions.
Dans l’onglet « Mise à l’emploi » il faudrait , dans le module « Mise à l’emploi Art60§77 », que la convention active liée apparaisse (une colonne avant remarque ?). –> Fixed after next pull.
Lorsqu’on souhaite inscrire un participant via EnrolmentsByActivity, on a le message : ‘str’ object has no attribute ‘objects’. –> Fixed after next pull.

Thursday, July 13, 2023

Thu, 13 Jul 2023 00:00:00 +0000

Thursday, July 13, 2023¶

I pushed my work for #5039 although the actual issue is not fixed. Because I want to test everything “on the field” on two production sites (jane and froinde). And because #5039 has been there before, it is not a release blocker.

I completely rewrote the lino.modlib.memo.truncate_comment() function and moved doctests to a separate document: Truncating HTML texts.

The remark field in lino_xl.lib.polls.AnswerRemark was also being truncated by truncate_comment(), with max_p_len being -1. I suspect that this was mixing up sanitizing and truncating. The remark of a poll answer needs to be sanitized, not truncated. Sanitizing is done using bleach in Lino.

Wednesday, July 12, 2023

Wed, 12 Jul 2023 00:00:00 +0000

Wednesday, July 12, 2023¶

I had written the following comment in Jane:

Indeed links should work because the server_uri is given as the base href of the email in the html header: <html><head><base href=”https://jane.mylino.net” target=”_blank”></head><body>

Ouch! This caused Jane to become unusable for a few hours of research time. What was happening? Lino now faithfully showed that <base href> tag in the Jane dashboard as part of the RecentComments story. Which caused every link to open a new window. But links in a Lino website with React look like this:

javascript:window.App.runAction({ "actorId": "tickets.Tickets", "an": "detail", "rp": null, "status": { "record_id": 5038 } })

They won’t work in an empty window. But the <base href> tag was instructing my browser to do exactly this: open a new target window for every link.

I ran the following script to find the offending comment:

from lino.api.shell import *
for msg in comments.Comment.objects.filter(body__contains='<base'):
    print(msg.id)

Output was a single line:

I wrote another script to fix the problem:

from lino.api.shell import *
msg = comments.Comment.objects.get(id=10922)
msg.body = msg.body.replace("<head", "&lt;head")
msg.body = msg.body.replace("<base", "&lt;base")
msg.save()

The only problem was that it didn’t fix the problem…

>>> from lino_book.projects.noi1e.startup import *

But why doesn’t the body_short_preview field get adapted? To understand this, I used the following doctest on my machine:

>>> msg = comments.Comment()

>>> msg.body = """<p>foo <html><head><base href="bar" target="_blank"></head><body></p><p>baz</p>"""
>>> msg.save()
>>> msg.body_short_preview    # (1)
'foo \n\nbaz\n\n'
>>> msg.body_full_preview
'<p>foo <html><head><base href="bar" target="_blank"/></head><body></body></html></p><p>baz</p>'

>>> msg.body = msg.body.replace("<head", "&lt;head")
>>> msg.body = msg.body.replace("<base", "&lt;base")
>>> msg.save()
>>> msg.body_short_preview    # (2)
'foo <head><base href="bar" target="_blank">\n\nbaz\n\n'
>>> msg.body_full_preview
'<p>foo <html>&lt;head&gt;&lt;base href="bar" target="_blank"&gt;<body></body></html></p><p>baz</p>'

>>> msg.delete()  # tidy up

Notes

(1) Where has the end tag (“</head>”) gone? Answer: the memo parser removed it when parsing the text via BeautifulSoup:

>>> txt = "foo &lt;head>bar</head> baz"
>>> soup = BeautifulSoup(txt, 'html.parser')
>>> str(soup)
'foo &lt;head&gt;bar baz'

(2) Why does the full “<head>” reappear in the short preview? Because during parse it was escaped and therefore wasn’t recognized as a HTML tag. And because the short preview (currently, wrongly) contains the rendered un-escaped HTML.

The wrong behaviour is in truncate_comment()

>>> from lino.modlib.memo.mixins import truncate_comment
>>> settings.SITE.plugins.memo.short_preview_length
300
>>> txt = "<p>foo &lt;head>bar</head> baz</p>"
>>> settings.SITE.plugins.memo.parser.parse(txt)
'<p>foo &lt;head&gt;bar baz</p>'

>>> truncate_comment(txt)
'foo <head>bar baz\n\n'

The short and long preview field are expected to contain safe HTML content, and bleach is responsible for removing any unsafe content. But truncate_comment() resolves html entities and therefore potentially converts safe html into unsafe html.

>>> truncate_comment("<p>foo &lt;head>bar</head> baz</p>")
'foo <head>bar baz\n\n'

Another topic¶

Note that lino.modlib.notify.Message.send_summary_emails() makes a special action request with permalink_uris set to True when rendering the notification body:

ar = rt.login(renderer=dd.plugins.memo.front_end.renderer, permalink_uris=True)

Old stuff:

>>> msg = comments.Comment(body="foo <head>bar</head> baz")
>>> msg.save()
>>> msg.body
'foo <head>bar</head> baz'
>>> msg.body_short_preview
'foo <head>bar</head> baz'

>>> msg.body = msg.body.replace("<head", "&lt;head")
>>> msg.save()
>>> msg.body
'foo &lt;head>bar</head> baz'
>>> msg.body_short_preview
'foo &lt;head&gt;bar baz'

>>> txt = "<p>A <b>bold</b> and <i>italic</i> thing."
>>> soup = BeautifulSoup(txt, "html.parser")
>>> list(soup.descendants)
[<p>A <b>bold</b> and <i>italic</i> thing.</p>, 'A ', <b>bold</b>, 'bold', ' and ', <i>italic</i>, 'italic', ' thing.']
>>> soup.p.name
'p'
>>> soup.p.text
'A bold and italic thing.'
>>> soup.p.string

>>> soup.b.text
'bold'
>>> soup.b.string
'bold'

>>> list(soup.p.strings)
['A ', 'bold', ' and ', 'italic', ' thing.']
>>> list(soup.strings)
['A ', 'bold', ' and ', 'italic', ' thing.']

>>> list(soup.p.children)
['A ', <b>bold</b>, ' and ', <i>italic</i>, ' thing.']
>>> list(soup.children)
[<p>A <b>bold</b> and <i>italic</i> thing.</p>]

>>> [c.name for c in soup.p.children]
[None, 'b', None, 'i', None]
>>> [c.name for c in soup.children]
['p']

Modifying the content:

>>> str(soup.b)
'<b>bold</b>'
>>> soup.b.string = soup.b.string[:2]
>>> str(soup.b)
'<b>bo</b>'
>>> soup.b.string = 'very bold'
>>> str(soup.b)
'<b>very bold</b>'
>>> str(soup)
'<p>A <b>very bold</b> and <i>italic</i> thing.</p>'

>>> from lino.modlib.memo.mixins import truncate_comment as tc

>>> tc("<p>A <b>bold</b> and <i>italic</i> thing.")
'A <b>bold</b> and <i>italic</i> thing.\n\n'

>>> tc("<p>A plain paragraph with more than 20 characters.</p>", 20)
'A plain paragraph wi...'

Monday, July 10, 2023

Mon, 10 Jul 2023 00:00:00 +0000

Monday, July 10, 2023¶

I pushed my work of the last weeks for #4551. Because tomorrow I will move back from Belgium to Estonia and I wanted to have these things published before. I admit that there are still 5 failures in the book, and presto isn’t yet adapted at all. But neither avanti nor amici are affected by these changes, and in Jane it would be okay to have some surprises.

2023-07-10 22:31 in lino: 9e546e5 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:31 in xl: 978eed1 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:31 in noi: 62e4f1b (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:32 in noi: 09913e0 (Merge branch ‘master’ of gitlab.com:lino-framework/noi)
2023-07-10 22:32 in voga: ed75fd6 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:32 in avanti: 387e8d7 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:33 in tera: 537415a (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:33 in book: 2ebef90 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:34 in welfare: 0b9c891 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:35 in ext6: c7bff07 (https://luc.lino-framework.org/blog/2023/0710.html)
2023-07-10 22:39 in getlino: 1cc0e5c (Optimize healthcheck.sh)

TODO:

Fix the remaining failures in book
Fix the test suites in welfare, presto, etc

Saturday, July 1, 2023

Sat, 01 Jul 2023 00:00:00 +0000

Saturday, July 1, 2023¶

Wednesday, June 21, 2023

Wed, 21 Jun 2023 00:00:00 +0000

Wednesday, June 21, 2023¶

What to do when use_linod but supervisor is down

snapshot/restore.py:21: RemovedInDjango50Warning: The django.utils.timezone.utc alias is deprecated. Please update your code to use datetime.timezone.utc instead.

On Jane I saw:

Loading 484412 objects (part 1 of 10) to table linod_job...

In other words, the linod.Job table is quite big. That’s because linod creates a new row for every little job.

Monday, June 19, 2023

Mon, 19 Jun 2023 00:00:00 +0000

Monday, June 19, 2023¶

The changes for #4551 will replace the “event pass” approach, which is used in Lino Voga where we collect “events” in the volatile InvoicingInfo object. This approach will be replaced by a new journal of “delivery notes”. In Voga these delivery notes are internal: the customer usually don’t need them. Lino can generate them more or less automatically.

About the ProvisionFiller mixin. A provision filler is an invoice generator that creates invoices when a given provision has reached a given minimal quantity. For example:

Voga : the participant buys an entry pass for 10 sessions, and when 8 of these sessions have been used up, the customer receive an invoice in order to buy a next entry pass. This is called continuous enrolment.
Noi : the customer buys a time credit of 5 hours, and when this credit is used up, they receive an invoice that invites them to buy new time credit.

Saturday, June 17, 2023

Sat, 17 Jun 2023 00:00:00 +0000

Saturday, June 17, 2023¶

I found and fixed a bug in lino.utils.quantities : Lino sometimes produced nonsense when working with negative durations.

I replaced “cooperation” and “co-operation” in the docs by “collaboration” because I was irritated by that fact that I still can’t decide which spelling I prefer. And also after reading this article.

Wednesday, June 7, 2023

Wed, 07 Jun 2023 00:00:00 +0000

Wednesday, June 7, 2023¶

(The following is now maintained in The storage plugin)

I am still meditating for #4551 (A first example of warehouse management). Continued from Saturday, May 27, 2023.

A provision (Vorrat) is when a given quantity of a given product is in a given storage state for a given partner.

A storage state is a name given to a certain type of provision. Some typical storage states:

01	auf Lager	in stock
02	bestellt vom Kunden	ordered by customer
03	bestellt bei Lieferant	ordered from provider
04	ausgeliehen	rented out
05	beschädigt	damaged
06	in Reparatur	under repair
07	gekauft	purchased

In Lino Noi we would use only one storage state “purchased”, and two transfer rules:

A sales invoice (SLS) credits “purchased”
A service report (SRV) debits “purchased”

And there would be only one product with storage management enabled, and that product would be named “Time credit”. Maybe there will be multiple products in order to differentiate between “temporary” (befristet) and “timeless” (unbefristet) time credit.

A subscription agreement must somehow specify the “required provision”, which means that the customer must always have a minimum provision of time credit. For example they buy 10 hours, which are being used up with each working session, and when the time credit gets used up they want to buy new credit. This can be done by Subscription.get_wanted_invoice_items. For each subscription that has such a “required provision”, it would check whether the partner still has enough provision. If not, it would generate an invoice to buy new time credit.

To specify this, we could simply add three new fields:

“product” : a pointer to the “Time credit” product
a minimum quantity (e.g. 5 hours)
an order quantity (e.g. 10 hours)

This reminds what the Tariff model and the InvoicingInfo class do until now. And indeed the storage plugin will partly replace these classes.

We have two basic approaches for generating invoices:

get_invoicing_periodicity() or “period-based invoicing” : The enrolment has a start_date and an end_date, and the “subscription periodicity” gives renew_unit, renew_every and renew_before. Period-based invoicing is for subscriptions, with a monthly or yearly fee.
get_invoicing_pass_type() or “occurence-based invoicing” : Lino needs a number_of_events, a min_asset and max_asset. This is used e.g. in Voga when a customer buys a “multi-pass card” for 10 sessions of an activity.

I am not yet sure where to store these information. Maybe we just want to add a Tariff.product field. The current flatrates (Tariff) do not currently point to another product.

qty	minimum
12	4
5	1
8	2

Side note. How to handle quantity-related prices? For example the unit price of an hour would be 90 € while 5 hours would cost only 400 € and 10 hours only 750 €. –> This is not related to the storage plugin and should be addressed by “price rules” or “complex products”.

Monday, June 5, 2023

Mon, 05 Jun 2023 00:00:00 +0000

Monday, June 5, 2023¶

I am trying to use the webcam of my notebook with fswebcam:

$ fswebcam --list-controls
Available Controls        Current Value   Range
------------------        -------------   -----
Brightness                127 (49%)       0 - 255
Contrast                  32 (12%)        0 - 255
Saturation                64 (64%)        0 - 100
Hue                       0 (50%)         -180 - 180
White Balance, Automatic  True            True | False
Gamma                     120 (50%)       90 - 150
Power Line Frequency      50 Hz           Disabled | 50 Hz | 60 Hz
White Balance Temperature 4000 (32%)      2800 - 6500
Sharpness                 2               0 - 7
Backlight Compensation    1               0 - 2
Exposure Time, Absolute   156 (12%)       4 - 1250
Exposure, Dynamic Framerate False           True | False

Saturday, May 27, 2023

Sat, 27 May 2023 00:00:00 +0000

Saturday, May 27, 2023¶

(Just some draft thoughts. Continued on Wednesday, June 7, 2023.)

Some storage states:

01	auf Lager	in stock
02	bestellt vom Kunden	reserved for customer
03	bestellt bei Lieferant	ordered from provider
04	ausgeliehen	rented out
05	beschädigt	damaged
06	in Reparatur	under repair
07	gekauft	purchased

In Lino Noi we would have only on storage state “purchased”, and two transfer rules:

A sales invoice (SLS) debits 02 and credits 06
A service report (SRV) voucher debits 06

In a bicycle shop

Monday, May 15, 2023

Mon, 15 May 2023 00:00:00 +0000

Monday, May 15, 2023¶

First use case will be Lino Noi where customers would buy services (increasing their stock of purchased services), and we would introduce a new journal “Service reports”, which will decrease the stock.

Until now both operations were expressed in money and within a single journal. A same sales invoice could potentially contain the renewal of a yearly subscription and a series of services built from working sessions. The time credit was actually expressed in money. Now the time credit will be expressed in hours. 1

We now have two invoicing areas:

Sales invoices is for selling yearly subscriptions and (optionally) contingents of time credit.
Service reports have no effect on accounting. They don’t generate ledger movements but storage movements.

About time slots

Wed, 03 May 2023 00:00:00 +0000

About time slots¶

Wednesday, May 3, 2023

We currently have two similar models courses.Slot and calview.PlannerRow. The former is subclassed in medico. I have the feeling that none of these is what we really need. Today I collected some ideas.

User documentation¶

Using time slots means that appointments are scheduled so they fit into a “timetable”. Instead of specifying start and end time, users just specify the “slot”. For example the slot “third hour” would mean “from 10:05 to 10:50”. The advantage of using time slots is to simplify planning and minimize “holes”.

A typical usage example are schools. These “slot systems” differ between schools. For a given school you can have different slot systems.

A real-world example¶

For example, in Vigala vocational school they have five slot systems, three for the “regular school” and two for the “session school”.

Regular school / Monday:

1	10:25 - 11:10
2	11:15 - 12:00
3	12:10 - 12:55
4	12:55 - 13:40
5	13:50 - 14:35
6	14:45 - 15:30
7	15:40 - 16:25

Regular school / Tuesday to Thursday:

1	08:30 - 09:15
2	09:25 - 10:10
3	10:20 - 11:05
4	11:15 - 12:00
5	12:10 - 12:55
6	12:55 - 13:40
7	13:50 - 14:35
8	14:45 - 15:30
9	15:40 - 16:25

Regular school / Friday

1	08:00 - 08:45
2	08:55 - 09:40
3	09:50 - 10:35
4	10:45 - 11:30

Session school / Monday - Satuday

1	08:30 - 09:15
2	09:25 - 10:10
3	10:20 - 11:05
4	11:15 - 12:00
5	12:10 - 12:55
6	12:55 - 13:50
7	13:50 - 14:35
8	14:45 - 15:30
9	15:40 - 16:25
9	16:30 - 18:00
10	18:10 - 19:40

Session school / Friday

1	08:00 - 08:45
2	08:55 - 09:40
3	09:50 - 10:35
4	10:45 - 11:30
5	11:40 - 12:25
6	12:25 - 13:10
7	13:20 - 14:05
8	14:05 - 14:50
9	15:00 - 15:45
10	15:45 - 16:30
11	16:40 - 17:25
12	17:25 - 18:10

>>> rt.show(cal.SlotSystems)
== ================
RE Regular / Monday
RF Regular / Friday
RT Regular / Other
SF Session / Friday
SO Session / Other
== ================

>>> rt.show(cal.SlotSelectors)
= =======
R Regular
S Session
= =======

::

class Slot:

id = CharField(primary=True) slot_system = SlotSystems.field() number = IntegerField() start_time end_time

def clean(self):: self.id = self.slot_system.value + str(self.number) super().clean()

>>> rt.show(cal.Slots)
==== ===== =====
ID   Start End
==== ===== =====
RE1  10:25 11:10
...
SF12 17:25 18:10
==== ===== =====

Slot rules¶

It’s even more complex: the “Friday” systems are not used only on Fridays, they are additionally used on every day before a holiday. And the “Monday” system is used additionally on every first day after a holiday.

Add a model cal.SlotRule:

class SlotRule(Sequenced):
  weekday = Weekdays.field()
  day, month, year = IntegerField()
  // Dynamically add one field ``XYZ_slots = SlotSystems.field()``
  // for each ``XYZ`` in ``SlotSelectors.get_list_items()``

  @classmethod
  def get_slot_system(cls, slot_sel, date):
      weekday = Weekdays.get_from_date(date)
      for rule in cls.objects.all():
        if rule.selector in (None, slot_sel):
          if rule.weekday in (None, weekday):
            if rule.year in (None, date.year):
              ...

>>> rt.show(cal.SlotRules)
======= ==== ===== === ======= =======
Weekday Year Month Day Regular Session
======= ==== ===== === ======= =======
                4   30  RF      SF
               12   24  RF      SF
1                       RM      SO
2                       RO      SO
3                       RO      SO
4                       RO      SO
5                       RF      SF
======= ==== ===== === ======= =======

The cal.Event model would get a new field slot = Slots.field()

def slot_choices():
slot_sel = self. slot_sys =

class Event(...):

  def get_slot_system(self, date, selector)

Some optimizations in calview

Tue, 02 May 2023 00:00:00 +0000

Some optimizations in calview¶

Tuesday, May 2, 2023

I pushed a few days of work for #4960. The book and welfare test suites passed, one failure in presto intentionally left open because it has to do with Actor.get_title().

Fixed bugs: In weekly view, the day numbers in the column headers are now clickable and switch to daily view. In monthly view, the week numbers are now clickable and switch to weekly view. The title of the slave table in the details of the three calendar views is now empty (instead of repeating the title of the detail window). In Presto where we have more than one planner, the daily/weekly buttons went to the default Calendar planner instead of staying in the Workers planner.

I reported a bug and a feature request in the React front end: #4958 and #4959

TODO:

Filtering doesn’t seem to work in the calendar views.
Document difference between title and label of an actor
Should Actor.get_title() return None when both title and label are None?

Sunday, April 30, 2023

Sun, 30 Apr 2023 00:00:00 +0000

Sunday, April 30, 2023¶

I am trying to understand why the Monthly and Daily views say “None in February 2017” and “None on Wed 15/02/2017” while the weekly view says correctly “Planner in Week 7 / 2017 (13 February)”.

Theoretically it should work by specifying the right details_of_master_template and title attributes.

I can currently imagine two possible causes:

some side effect caused by the complex class hierarchy,
react does not call get_title

Presto contacts :

>>> from lino import startup
>>> startup('lino_presto.projects.presto1.settings')
>>> from lino.api.doctest import *

>>> print(calview.DailyView.title)
None
>>> print(calview.DailyView.label)
Daily view

>>> ar = rt.login("robin").spawn(calview.DailyView, master_instance=0)
>>> print(ar.get_title())
Daily view

>>> rt.show(calview.Planners)
========= ========= ========== ===================== ==================== ===================
 value     name      text       Monthly view          Weekly view          Daily view
--------- --------- ---------- --------------------- -------------------- -------------------
 default   default   Calendar   calview.MonthlyView   calview.WeeklyView   calview.DailyView
========= ========= ========== ===================== ==================== ===================

Friday, April 28, 2023

Fri, 28 Apr 2023 00:00:00 +0000

Friday, April 28, 2023¶

The action request on a slave table didn’t always report correctly about invalid master type. I added more examples in More chooser examples. Fixing this caused internal optimizations in lino.core.requests and lino.core.tablerequest.

I generalized the lino.utils.MissingRow idea. Until now it was used only by lino.core.choicelists.PointingChoice, now it is also used when the master instance could not be found.

I restored the translatable texts for DurationUnits and Recurrences, which had been lost when we recently moved these classes from lino_xl.lib.cal to lino.modlib.system

The lino.modlib.linod.BackgroundTask now logs using DEBUG instead of INFO. Because it causes 3 messages every 10 seconds.

I moved the demo fixture for noi1e and noi1r demo projects from lino_book.projects.noi1e to lino_noi.lib.noi. Because we want it also on a demo server.

While working on this, I also happened to optimize content in the Developer Guide.

Wednesday, April 26, 2023

Wed, 26 Apr 2023 00:00:00 +0000

Wednesday, April 26, 2023¶

Jane memory usage after 115 days uptime.

$ uptime
11:26:58 up 115 days, 20:14,  1 user,  load average: 0,21, 0,85, 0,75

$ ps aux --sort '%mem'
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root     2719655  3.5  6.8 330456 136116 ?       Sl   11:22   0:05 python manage.py runworker linod_jane
root     2719649  5.4  7.5 199064 150212 ?       S    11:22   0:07 /usr/local/lino/lino_local/jane/env/bin/python
root     2719648  5.5  7.8 499776 154892 ?       Sl   11:22   0:08 /usr/local/lino/lino_local/jane/env/bin/python
root     2719661  5.2  7.8 351156 156072 ?       Sl   11:22   0:07 python manage.py runworker linod_welcht1
root     2719662  5.1  7.9 352700 158088 ?       Sl   11:22   0:07 python manage.py runworker linod_weleup1
root     2719652  8.7  8.5 219104 169104 ?       S    11:22   0:12 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root     2719653  9.0  8.8 447240 175752 ?       Sl   11:22   0:12 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root     2719654  9.0  8.9 298716 178220 ?       S    11:22   0:13 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root     2719650  9.0  9.1 449840 180396 ?       Sl   11:22   0:13 /usr/local/lino/lino_local/welcht1/env/bin/pyth

$ sudo reboot now

$ ps aux --sort '%mem'
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root         947  7.6  7.0 330580 138912 ?       Sl   11:28   0:04 python manage.py runworker linod_jane
root         940 12.6  8.0 351708 158636 ?       Sl   11:28   0:06 /usr/local/lino/lino_local/jane/env/bin/python
root         939 12.5  8.0 499500 159100 ?       Sl   11:28   0:06 /usr/local/lino/lino_local/jane/env/bin/python
root         948 11.5  8.0 351156 160008 ?       Sl   11:28   0:06 python manage.py runworker linod_welcht1
root         949 11.5  8.1 352692 162228 ?       Sl   11:28   0:06 python manage.py runworker linod_weleup1
root         942 20.1  8.6 219112 171032 ?       S    11:28   0:10 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root         941 20.2  8.6 219112 171072 ?       S    11:28   0:10 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root         943 20.5  8.7 220376 172740 ?       S    11:28   0:11 /usr/local/lino/lino_local/welcht1/env/bin/pyth
root         946 20.5  8.7 220640 173180 ?       S    11:28   0:11 /usr/local/lino/lino_local/welcht1/env/bin/pyth

Tuesday, April 25, 2023

Tue, 25 Apr 2023 00:00:00 +0000

Tuesday, April 25, 2023¶

I published my changes for #4940.

Lino CMS (the cms1 demo) no longer has the calendar plugin. Websites with a public calendar do not require CMS functionality, but they do require the courses plugin. Lino Voga now demonstrates this use case. The roger demo is not yet convincing but we are advancing towards this goal.

Side effects in lino_xl.lib.cal: Event had a method get_default_table(), which unconditionally returned OneEvent. This was in contradiction with the fact that OneEvent had use_as_default_table set to False. I removed the method.

These changes had extensive side effects in lino.modlib.publisher. For example, the Publishable mixin has a new method is_public(), which returns False by default. For a blog entry it would return True or False depending on the state. Most fundamental change in publisher is the renderer, which now forwards things to the site’s editing_frontend… For more details you must consult the source code :-/

List of commits:

2023-04-25 21:47 in lino: a80f1c3
2023-04-25 21:48 in xl: 24672fe
2023-04-25 21:48 in noi: 778f334
2023-04-25 21:49 in voga: 804b803
2023-04-25 21:49 in care: 1fe6368
2023-04-25 21:49 in avanti: 6955bc3
2023-04-25 21:49 in tera: a332666
2023-04-25 21:49 in book: dca42df
2023-04-25 21:49 in welfare: ddfd8e4
2023-04-25 21:49 in presto: 2b85ff3
2023-04-25 21:49 in mentori: 8697685
2023-04-25 21:49 in shop: 112532e
2023-04-25 21:50 in react: 2becc37
2023-04-25 21:50 in amici: e42562d
2023-04-25 21:50 in algus: f038cbf
2023-04-25 21:50 in cms: f849afa
2023-04-25 21:50 in medico: f870664

Watching out for a third developer

Sun, 23 Apr 2023 00:00:00 +0000

Watching out for a third developer¶

Sunday, April 23, 2023

It seems that I am realizing something important: Lino doesn’t currently need a better leader than me (“a man to bring it to a next level”), it “only” needs a third developer, a colleague for Sharif.

Lino has actually multiple potential customers here in Belgium, we just don’t have any convincing final product to show them. We have no convincing final product because the last 20% of the work are the most time-consuming and because until now I prefer to do them together with the customer. During the last months I have spoken to many “almost-customers” and I feel able to imagine their needs.

In other words, I am changing my “strategy”. It’s actually not a strategy, not even a “plan”, I prefer to call it my “prayers”. One of the advantages of being a small team is to not requiring “clear” plans and strategies.

The new master plan includes finding a way to raise funds for a year of salary for a second developer like Sharif.

The entrance test for candidates is easy: just read https://dev.lino-framework.org and contact me when you reached the end of the “Getting started” section. Most people never reach the first checkpoint because they loose interest or have other skills and talents.

Wednesday, April 19, 2023

Wed, 19 Apr 2023 00:00:00 +0000

Wednesday, April 19, 2023¶

We have a boolean field named “private” on three models: comment, lino_xl.lib.tickets.Ticket and lino_xl.lib.tickets.Site.

Actually this field should better be labelled “confidential”.

The Publishable mixin should get a new method is_public(), which would return False by default. For a blog entry it would return True or False depending on the state.

Wednesday, April 5, 2023

Wed, 05 Apr 2023 00:00:00 +0000

Wednesday, April 5, 2023¶

Sites with multiple front ends¶

I removed Plugin.force_url_prefix because it was unused.

Specifying Site.default_ui is now “the old way”. The attribute default_ui on the Kernel (which was the Plugin instance of the plugin named by Site.default_ui) is no replaced by an attribute web_front_ends.

To use the new feature, you can now set the web_front_ends on the Site, which in that case must be a list of tuples having two items (prefix, modname) each. For example a site with react as default and extjs as alternative front end would have:

web_front_ends = [
  (None, "lino_react.react"),
  ('ext', "lino.modlib.extjs")]

Or in cms1 we have publisher as default and react as the “admin” front end:

web_front_ends = [
  (None, "lino.modlib.publisher"),
  ('admin', "lino_react.react")]

Tuesday, March 28, 2023

Tue, 28 Mar 2023 00:00:00 +0000

Tuesday, March 28, 2023¶

The bootstrap3/base.html template has a language selector and I think that we actually want to to use this logic for publisher as well. And page.pub.html is based on this template, but overrides the header block. The bootstrap language selector uses lino.core.requests.BaseRequest.get_request_url, which currently doesn’t work for a publisher page. Solution was to write a Renderer for publisher, and reimplement the publisher views.

I tried to find a better place for the hash_router and permalink_uris attributes of lino.core.requests.BaseRequest.

hash_router is used in lino.core.renderer.HtmlRenderer.get_permalink() to insert the “#” sign before the URL. This information is constant for the react front end and shouldn’t be stored on each incoming request.
permalink_uris is used in lino.core.renderer.HtmlRenderer.row_action_button() lino.core.renderer.JsRenderer.obj2url()

The hash_router attribute was used only by the publisher view, which I reimplemented. So it can go away.

But lino.core.requests.BaseRequest.permalink_uris stays.

Wednesday, March 22, 2023

Wed, 22 Mar 2023 00:00:00 +0000

Wednesday, March 22, 2023¶

I plan to start coding for #4783. There will be three database models and a choicelist:

class JobRule(Sequenced, RecurrenceSet):
    name
    procedure = Procedures.field()

class Job(Started):
    rule = ForeignKey(JobRule)

class Procedures(dd.Choicelist):
    item_class = Procedure

class Procedure(dd.Choice):
    func : the callable to run
    every_unit = "sec"
    every_value = 10
    start_time

The schedule_daily and schedule_daily decorators will now add a Procedure to the Procedures choicelist. We will replace the every and at keywords, which come from the schedule API, by the RecurrenceSet syntax. For a Procedure these values are now just suggestions that lead to a default JobRule generted by the std fixture.

But halt! I realized that this is a rather invisible feature! Lino actually needs visible features. For example the warehouse plugin. Or the CMS. So I will let #4783 wait for another time.

Cannot join existing household

Fri, 17 Mar 2023 00:00:00 +0000

Cannot join existing household¶

Friday, March 17, 2023

Already quite some time we have several failures in the book doctests caused by a difference in an Ajax action call. For example docs/specs/households.rst. The common thing was that one key in that Ajax call, the base_params, had been removed. And I had no idea whether that was good or bad.

Now I discovered ticket #4905, which shows that the test suite was right: it had detected a regression. The problem is that we didn’t care, and now we will need more time to find back the cause. Sharif, do you remember to have changed something in the inner code that might have caused this?

Meanwhile I had been working on a series of “minor” changes. For example I changed the way we add intersphinx mappings to synodal docs. And finally removed the dependency from rstgen to atelier

I now pushed them all to their repositories because I am waiting for Sharif’s feedback.

2023-03-17 05:24 in synodal: 91d791d (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:25 in lino: 05c44f5 (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:25 in ug: 867ae0c (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:25 in xl: 213d2a7 (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:26 in noi: 6b51145 (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:26 in cosi: ab342f7 (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:27 in book: ee81eff (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:27 in welfare: d1b662d (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:27 in welcht: 5334afd (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:28 in getlino: 9a7b474 (https://luc.lino-framework.org/blog/2023/0317.html)
2023-03-17 05:28 in blog: 3ee2022 (https://luc.lino-framework.org/blog/2023/0317.html)

Tuesday, March 14, 2023

Tue, 14 Mar 2023 00:00:00 +0000

Tuesday, March 14, 2023¶

Monday, March 13, 2023

Mon, 13 Mar 2023 00:00:00 +0000

Monday, March 13, 2023¶

I worked on #4898 (Change “site operator” to “site owner”, add a field site_maintainer and render them both in the local help pages.)

Changes:

New database model help.SiteContacts

Side effects - #4902 - Removed the apcdocs directory

Side effects:

contacts.Person.as_paragraph() no longer inserts a   between the salutation and the title & name

TODO:

adapt test suites
write documentation
translations
Define SiteContact.as_paragraph() and SiteContact.__str__()
Add link the help pages in publisher main page.
Several test cases still failing

failing test cases:

docs/dev/site_internals.rst
docs/projects/noi1r.rst
docs/specs/polls.rst
docs/tested/dashboard_sql.rst  # there is one group membership more in noi1r
docs/specs/react/linoweb.rst
docs/specs/migrate.rst
docs/specs/jinja.rst
docs/specs/households.rst

Monday, February 27, 2023

Mon, 27 Feb 2023 00:00:00 +0000

Monday, February 27, 2023¶

I reviewed Sharif’s work. Looks good! I am looking forward to get a presentation. I responded with two minor changes (in four commits): (1) We still need to support Python 3.7, so I removed a type hint in lino. (2) And then the output of lino_xl.lib.cal.Calendars in cal : Calendar functionality was:

>>> rt.show(cal.Calendars)
...
==== ============= ================== ================== ============= =======
 ID   Designation   Designation (de)   Designation (fr)   Description   color
---- ------------- ------------------ ------------------ ------------- -------
 1    General       Allgemein          Général                          1
                                                                        **1**
==== ============= ================== ================== ============= =======

Sharif’s work had added a column “Modified”, right after the ID. But we don’t want this field to be so visible! The code needs to be fixed, not the test case! This is an example where the test suite protects us from an unwanted change in the user interface.

Here are the four commits:

2023-02-27 04:03 in lino: 4f61f1f (remove type hint to support Python before 3.8)
2023-02-27 04:33 in xl: b44bb78 (add explicit column_names to cal.Calendars)
2023-02-27 04:35 in xl: f895af5 (oops add explicit column_names to cal.Calendars)
2023-02-27 04:36 in book: 87f8517 (adapt after adding explicit column_names to cal.Calendars)

Sunday, February 19, 2023

Sun, 19 Feb 2023 00:00:00 +0000

Sunday, February 19, 2023¶

The book test suite still has 7 failures because when I was waiting for my local test suite to finish I discovered Fields with an input mask and accidentally started to work on this and pushed an untested last-minute change: the lino_react.react plugin no longer depends on lino.modlib.memo.

Saturday, February 18, 2023

Sat, 18 Feb 2023 00:00:00 +0000

Saturday, February 18, 2023¶

The Django docs (How the documentation is organized) mention four top-level entry points (“Tutorials”, “Topic guides”, “Reference guides” and “How-to guides”), but it seems that I don’t fully agree with that model.

“Guides” or “Getting started”
“Topics” or “FAQ”
“Reference” or “Index”

“Tutorials” and “Recipes” are documents types, not entry points.

Wednesday, February 15, 2023

Wed, 15 Feb 2023 00:00:00 +0000

Wednesday, February 15, 2023¶

I upgraded Jane. No big problem but a few smal ones.

pm collectstatic failed because of three trivial regressions I introduced when I moved the build_site_cache() from kernel to lino.core.site.Site.

These failures would probably have been noticed by the getlino test suite, but I didn’t look at it before releasing to pypi. And looking at it now, it is failing because of yet another reason, equally a trivial one. I wish we had somebody who monitors the test suites on GitLab!

I also upgraded froinde, which runs on an older server under Python 3.7. Seems that the type hints we recently started to use aren’t yet supported there

Traceback (most recent call last):
  ...
  File ".../env/lib/python3.7/site-packages/lino/modlib/memo/parser.py", line 341, in <module>
    def split_name_rest(s: str) -> tuple[str]:
TypeError: 'type' object is not subscriptable

File ".../env/lib/python3.7/site-packages/lino_xl/lib/cal/utils.py", line 188, in <module>
  def parse_rrule(rr: str) -> tuple[Any, int, Union[int, datetime.datetime, datetime.date], dict]:

Tuesday, February 14, 2023

Tue, 14 Feb 2023 00:00:00 +0000

Tuesday, February 14, 2023¶

Today I did an avalanche of changes in the Developer Guide. Changes in lf, ug, cg and book.

To “log in” or to “sign in”?¶

To “log in” or to “sign in”? What should we say? A controversial question! Google and Amazon say “Sign in”, Facebook and Wikipedia say “Log in”.

I remember that many years ago (before 2017, I couldn’t find the commit), I changed “Log in” to “Sign in” (and renamed LogIn to lino.modlib.users.SignIn).

It seems that “log in” is more popular than “sign in”. Wikipedia says that logging in (or logging on, signing in, or signing on) “is the process by which an individual gains access to a computer system by identifying and authenticating themselves”.

English teacher Dr. Jill Robbins, when asked about the differences between “sign in” and “log in”, explains that “unlike “log in,” “sign in” is common outside of the computer world.” She also explains where it comes from: “When a user logs in, the computer system keeps a record, or “log,” of that user’s session on the system.” (voanews.com)

I still think that saying “sign in” is more correct than saying “log in”. It means to tell our server who you are, basta. You don’t ask it to also keep a log of your visit. Maybe you are even reluctant to having the site log down your visit. In many countries the site owner must ask your permission (“May we use cookies?”) before doing so.

Saturday, January 28, 2023

Sat, 28 Jan 2023 00:00:00 +0000

Saturday, January 28, 2023¶

As a follow-up to YOUR COMPENSATION FROM CAROL.., Hannes added yesterday the following option to our postfix main.cf file:

smtpd_client_restrictions = reject_unknown_reverse_client_hostname

This morning when I tried to send an email via FairEmail on my phone, the server answered:

Invalid Addresses 450 4.7.1 Client host rejected: cannot find your reverse hostname, [109.130.103.89] To be meditated.

And here is relevant part in the mail.log file:

Jan 28 05:59:08 saffre-rumma postfix/submission/smtpd[7246]: connect from unknown[109.130.103.89]
Jan 28 05:59:09 saffre-rumma postfix/submission/smtpd[7246]: Anonymous TLS connection established from unknown[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
Jan 28 05:59:10 saffre-rumma postfix/submission/smtpd[7246]: NOQUEUE: reject: RCPT from unknown[109.130.103.89]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [109.130.103.89]; from=<luc@saffre-rumma.net> to=<luc@saffre-rumma.net> proto=ESMTP helo=<dummy.faircode.eu>
Jan 28 05:59:10 saffre-rumma postfix/submission/smtpd[7246]: disconnect from unknown[109.130.103.89] ehlo=2 starttls=1 auth=1 mail=1 rcpt=0/1 rset=1 noop=1 quit=1 commands=8/9

A bit later I tried to send via Thunderbird from my notebook, using the same WLAN as my phone, and there it worked. And the mail.log says:

Jan 28 06:49:52 saffre-rumma postfix/submission/smtpd[32661]: connect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]
Jan 28 06:49:52 saffre-rumma postfix/submission/smtpd[32661]: Anonymous TLS connection established from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)
Jan 28 06:49:52 saffre-rumma postfix/submission/smtpd[32661]: 6B5252155A: client=89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89], sasl_method=PLAIN, sasl_username=luc
Jan 28 06:49:52 saffre-rumma postfix/cleanup[577]: 6B5252155A: message-id=<244735bc-b3cb-067a-4c17-67e693db4372@saffre-rumma.net>
Jan 28 06:49:52 saffre-rumma opendkim[627]: 6B5252155A: DKIM-Signature field added (s=mail, d=saffre-rumma.net)
Jan 28 06:49:52 saffre-rumma postfix/qmgr[30080]: 6B5252155A: from=<luc@saffre-rumma.net>, size=50016, nrcpt=1 (queue active)
Jan 28 06:49:52 saffre-rumma postfix/submission/smtpd[32661]: disconnect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=8
Jan 28 06:49:55 saffre-rumma postfix/smtp[579]: 6B5252155A: to=<sharifmehedi24@outlook.com>, relay=outlook-com.olc.protection.outlook.com[104.47.51.225]:25, delay=2.9, delays=0.15/0.02/0.08/2.7, dsn=2.6.0, status=sent (250 2.6.0 <244735bc-b3cb-067a-4c17-67e693db4372@saffre-rumma.net> [InternalId=14177687056028, Hostname=SI2PR04MB5919.apcprd04.prod.outlook.com] 59623 bytes in 0.545, 106.767 KB/sec Queued mail for delivery -> 250 2.1.5)
Jan 28 06:49:55 saffre-rumma postfix/qmgr[30080]: 6B5252155A: removed

After checking an advanced setting “Use local IP address instead of host name” in FairEmail, I was able to send and the mail.log says:

Jan 28 07:37:21 saffre-rumma postfix/submission/smtpd[24706]: connect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]
Jan 28 07:37:21 saffre-rumma postfix/submission/smtpd[24706]: Anonymous TLS connection established from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
Jan 28 07:37:21 saffre-rumma postfix/submission/smtpd[24706]: disconnect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89] ehlo=2 starttls=1 auth=1 noop=1 quit=1 commands=6

A bit later I tried to send another email from TB, and it failed, TB saying:

Sending of the message failed. An error occurred while sending mail. The mail server responded: Client host rejected: cannot find your reverse hostname, [109.130.103.89]. Please check the message recipient “sharifmehedi24@outlook.com” and try again.

And the mail.log said:

Jan 28 07:51:21 saffre-rumma postfix/submission/smtpd[31470]: connect from unknown[109.130.103.89]
Jan 28 07:51:21 saffre-rumma postfix/submission/smtpd[31470]: Anonymous TLS connection established from unknown[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
Jan 28 07:51:21 saffre-rumma postfix/submission/smtpd[31470]: NOQUEUE: reject: RCPT from unknown[109.130.103.89]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [109.130.103.89]; from=<luc@saffre-rumma.net> to=<sharifmehedi24@outlook.com> proto=ESMTP helo=<[192.168.1.42]>
Jan 28 07:51:21 saffre-rumma postfix/submission/smtpd[31470]: lost connection after RCPT from unknown[109.130.103.89]
Jan 28 07:51:21 saffre-rumma postfix/submission/smtpd[31470]: disconnect from unknown[109.130.103.89] ehlo=2 starttls=1 auth=1 mail=1 rcpt=0/1 commands=5/6

I tried several more times to send my email, and after a few attempts it worked:

Jan 28 07:54:00 saffre-rumma postfix/submission/smtpd[905]: connect from unknown[109.130.103.89]
Jan 28 07:54:00 saffre-rumma postfix/submission/smtpd[905]: Anonymous TLS connection established from unknown[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)
Jan 28 07:54:00 saffre-rumma postfix/submission/smtpd[905]: NOQUEUE: reject: RCPT from unknown[109.130.103.89]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [109.130.103.89]; from=<luc@saffre-rumma.net> to=<sharifmehedi24@outlook.com> proto=ESMTP helo=<[192.168.1.42]>
Jan 28 07:54:00 saffre-rumma postfix/submission/smtpd[905]: lost connection after RCPT from unknown[109.130.103.89]
Jan 28 07:54:00 saffre-rumma postfix/submission/smtpd[905]: disconnect from unknown[109.130.103.89] ehlo=2 starttls=1 auth=1 mail=1 rcpt=0/1 commands=5/6

Jan 28 07:54:20 saffre-rumma postfix/submission/smtpd[905]: connect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]
Jan 28 07:54:20 saffre-rumma postfix/submission/smtpd[905]: Anonymous TLS connection established from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)
Jan 28 07:54:21 saffre-rumma postfix/submission/smtpd[905]: 0F73621666: client=89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89], sasl_method=PLAIN, sasl_username=luc
Jan 28 07:54:21 saffre-rumma postfix/cleanup[1083]: 0F73621666: message-id=<37ff8af4-686e-1ad2-6d59-b65a21feb8af@saffre-rumma.net>
Jan 28 07:54:21 saffre-rumma opendkim[627]: 0F73621666: DKIM-Signature field added (s=mail, d=saffre-rumma.net)
Jan 28 07:54:21 saffre-rumma postfix/qmgr[30080]: 0F73621666: from=<luc@saffre-rumma.net>, size=45170, nrcpt=1 (queue active)
Jan 28 07:54:21 saffre-rumma postfix/submission/smtpd[905]: disconnect from 89.103-130-109.adsl-dyn.isp.belgacom.be[109.130.103.89] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=8
Jan 28 07:54:24 saffre-rumma postfix/smtp[1085]: 0F73621666: to=<sharifmehedi24@outlook.com>, relay=outlook-com.olc.protection.outlook.com[104.47.2.33]:25, delay=3.6, delays=0.16/0.02/0.11/3.3, dsn=2.6.0, status=sent (250 2.6.0 <37ff8af4-686e-1ad2-6d59-b65a21feb8af@saffre-rumma.net> [InternalId=14482629735081, Hostname=PUZPR04MB6295.apcprd04.prod.outlook.com] 54786 bytes in 0.549, 97.388 KB/sec Queued mail for delivery -> 250 2.1.5)
Jan 28 07:54:24 saffre-rumma postfix/qmgr[30080]: 0F73621666: removed

I unchecked the “Use local IP address instead of host name” setting in FairEmail and still was able to send.

IOW the WLAN connection of the place where I am seems to connect using different methods, depending on some random condition.

Thursday, January 26, 2023

Thu, 26 Jan 2023 00:00:00 +0000

Thursday, January 26, 2023¶

I made an upgrade to Jane because I want to make the next service report using the subscriptions.

The upgrade worked very well, Lino behaved correctly. I made a few local bugfixes after the update (i.e. I made them both locally on the production server and in my code):

2023-01-26 02:49 in xl: c77c41c (local bugfixes after update jane)
2023-01-26 02:50 in noi: e359847 (local bugfixes after update jane)

I then wanted to make the next service report for a guinea pig customer.

First there was no “Generate invoices” entry in the Sales menu. I marked two journals as having an invoicing_area. This should have changed the menu, but it didn’t. I guess this is because users preferences are stored in memory. After restarting supervisor the menu was updated.

I still didn’t get Lino to suggest a service report. Something is still missing in the configuration. To be continued next time.

Wednesday, January 25, 2023

Wed, 25 Jan 2023 00:00:00 +0000

Wednesday, January 25, 2023¶

Uff, I finally published a full week of my work!

Insert images into arbitrary places using memo commands¶

Moved the Mention model from comments to memo.

The lino.modlib.memo plugin now creates a special BaseRequest instance that is used when parsing memo commands.

It is special in that its user is an instance of AnonymousUser with user_type admin

with admin rights. This is because we want to store the result of memo commands in the preview fields of lino.modlib.memo.Previewable. When a non-admin user saves a blog entry that contains references to other database objects, then the author maybe has no permission to see the detail of these objects, but other users have.

The new model mixin lino.modlib.memo.MemoReferrable eliminates quite some boilerplate code.

In rstgen.sphinxconf.sigal_image.line2html() I renamed the format “couple” to “duo”. And I fixed a bug that caused duo and trio to be floating (wrapped by their surrounding text).

Data migration

in your restore.py uncomment the line that loads comments_mention.py. The lino.modlib.memo.Mention objects will be populated by pm checkdata -f.

En passant

I removed the Site.max_file_size setting because it was useless: Lino would never get it because the server would make a disk overflow.

Documentation about Synodalsoft¶

I also have been working in the documentation during the last days. I started the website for https:://www.synodalsoft.net

The synodal package is far from being perfectly, but it is already useful. Soon we can remove the dependency of rstgen towards atelier.

More changes¶

Later I also removed Modified from Mention because it is not needed.

YOUR COMPENSATION FROM CAROL..

Tue, 24 Jan 2023 00:00:00 +0000

YOUR COMPENSATION FROM CAROL..¶

Tuesday, January 24, 2023

My yesterday blog entry does not explain why we received a spam mail from our own mail server three days ago.

Here are the relevant headers of that mail:

Return-Path: <cw4826878@gmail.com>
X-Original-To: info@saffre-rumma.net
Delivered-To: info@saffre-rumma.net
Received: by saffre-rumma.net (mail.saffre-rumma.net, from userid 1008)
      id 42C9C21701; Sat, 21 Jan 2023 23:03:45 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=saffre-rumma.net;
      s=mail; t=1674342225;
      bh=4zpM8MrO4ODlv4EnmHUvvh8W56Pgo7H7YgK6tHvZTRM=;
      h=Reply-To:From:To:Subject:Date:From;
      b=zaIcGPvwuxyr07J/O9nSnJJNJZ3CoRV13hHwf/nhBTvFUpt22dY8CKY+0CA9bPZ2v
       ZX/NfcJmhqoKcU0jng1cb9FCdQjFj/QVm8q+iM5HIq98+P562ybG/ryw+SOlSKBT13
       x9ccwA8En7eJY+sgTGG1JFr07iZNQWCoazkLeCe7dM0JGsFIJ6H0x0YEDhFB9CqeuG
       XsXm3/obh+R63rEhBu/C7UqvG1WSLMY+FNifIYHjLnTtcKKLpkpZoZj5C4VBDU9sL+
       TPZTnXpBl9BEYfrfG5DyQh1VXrUcCZSIPcPYc3Y8epEMKmS8NKuCJ4ATtkZ51pfse7
       j8kyVX8hxSVOw==
Received-SPF: Softfail (mailfrom) identity=mailfrom; client-ip=45.137.22.186; helo=hosted-by.rootlayer.net; envelope-from=cw4826878@gmail.com; receiver=<UNKNOWN>
Received: from hosted-by.rootlayer.net (unknown [45.137.22.186])
      by saffre-rumma.net (mail.saffre-rumma.net) with ESMTP id 224AF21672
      for <info@saffre-rumma.net>; Sat, 21 Jan 2023 23:03:42 +0000 (UTC)
Reply-To: philipisclement@gmail.com
From: CAROL@saffre-rumma.net, SMITH@saffre-rumma.net,
      cw4826878@gmail.com
To: info@saffre-rumma.net
Subject: YOUR COMPENSATION FROM CAROL..
Date: 21 Jan 2023 23:03:41 +0100
Message-ID: <20230121230341.2BAFFC26FBE2B2BA@gmail.com>

Here is the relevant part of our mail.log file:

Jan 21 23:03:42 saffre-rumma postfix/smtpd[16463]: warning: hostname hosted-by.rootlayer.net does not resolve to address 45.137.22.186
Jan 21 23:03:42 saffre-rumma postfix/smtpd[16463]: connect from unknown[45.137.22.186]
Jan 21 23:03:43 saffre-rumma policyd-spf[16467]: prepend Received-SPF: Softfail (mailfrom) identity=mailfrom; client-ip=45.137.22.186; helo=hosted-by.rootlayer.net; envelope-from=cw4826878@gmail.com; receiver=<UNKNOWN>
Jan 21 23:03:43 saffre-rumma postfix/smtpd[16463]: 224AF21672: client=unknown[45.137.22.186]
Jan 21 23:03:43 saffre-rumma postfix/cleanup[16471]: 224AF21672: message-id=<20230121230341.2BAFFC26FBE2B2BA@gmail.com>
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: [45.137.22.186] [45.137.22.186] not internal
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: not authenticated
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: no signature data
Jan 21 23:03:43 saffre-rumma postfix/qmgr[28134]: 224AF21672: from=<cw4826878@gmail.com>, size=1938, nrcpt=1 (queue active)
Jan 21 23:03:43 saffre-rumma postfix/smtpd[16463]: disconnect from unknown[45.137.22.186] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Jan 21 23:03:43 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused
Jan 21 23:03:43 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused
Jan 21 23:03:44 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#2 of 3): Connection refused
Jan 21 23:03:44 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#3 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#3 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connection attempt to spamd aborted after 3 retries
Jan 21 23:03:45 saffre-rumma postfix/pipe[16472]: 224AF21672: to=<info@saffre-rumma.net>, relay=spamassassin, delay=2.7, delays=0.6/0.02/0/2, dsn=2.0.0, status=sent (delivered via spamassassin service)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 224AF21672: removed
Jan 21 23:03:45 saffre-rumma postfix/pickup[17597]: 42C9C21701: uid=1008 from=<cw4826878@gmail.com>
Jan 21 23:03:45 saffre-rumma postfix/cleanup[16471]: 42C9C21701: message-id=<20230121230341.2BAFFC26FBE2B2BA@gmail.com>
Jan 21 23:03:45 saffre-rumma opendkim[627]: 42C9C21701: DKIM-Signature field added (s=mail, d=saffre-rumma.net)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 42C9C21701: from=<cw4826878@gmail.com>, size=2104, nrcpt=1 (queue active)
Jan 21 23:03:45 saffre-rumma postfix/local[16519]: 42C9C21701: to=<luc@saffre-rumma.net>, orig_to=<info@saffre-rumma.net>, relay=local, delay=0.04, delays=0.02/0.02/0/0, dsn=2.0.0, status=sent (delivered to maildir)
Jan 21 23:03:45 saffre-rumma postfix/local[16519]: 42C9C21701: to=<hannes@saffre-rumma.net>, orig_to=<info@saffre-rumma.net>, relay=local, delay=0.04, delays=0.02/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to maildir)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 42C9C21701: removed

Why our server was sending spam

Mon, 23 Jan 2023 00:00:00 +0000

Why our server was sending spam¶

Monday, January 23, 2023

Our server SR has been blocked once more by OVH (the data center) because it was sending spam. This was the third time since January 8. Each time we thought that we had identified and fixed the issue. The first two times we were obviously wrong. The spammer was obviously the same because there are again 30000 mails in our mailq, and they all claim to come from mail10.lwspanel.com

The first time, two weeks ago, I thought somebody had sniffed out my password because I had been using Thunderbird from a relatively insecure network. The second time we suspected dnswl.org to be somehow collaborate with spammers.

Today Hannes and I worked together on this because two brains are more powerful in parallel than in series. And now I think I know what happened: the spammer used the account of one of our team members. I guess that the spammer used brute force: we don’t use fail2ban the username are publicly known, so they just needed to try patiently enough.

The /var/log/mail.log shows lines like these:

Jan 23 15:13:09 saffre-rumma postfix/submission/smtpd[22461]: B8D82268DF: client=unknown[172.245.249.148], sasl_method=LOGIN, sasl_username=xxxxx

We changed the password of that user to a secure one. We removed the mails that were still waiting in the mail queue. Then we told OVH to remove their block. I am confident that this was the explanation.

Hannes isn’t yet convinced that dnswl.org are innocent.

Sunday, January 22, 2023

Sun, 22 Jan 2023 00:00:00 +0000

Sunday, January 22, 2023¶

Here is the /var/log/mail.log.1 of postfix on SR:

Jan 21 23:03:42 saffre-rumma postfix/smtpd[16463]: warning: hostname hosted-by.rootlayer.net does not resolve to address 45.137.22.186
Jan 21 23:03:42 saffre-rumma postfix/smtpd[16463]: connect from unknown[45.137.22.186]
Jan 21 23:03:43 saffre-rumma policyd-spf[16467]: prepend Received-SPF: Softfail (mailfrom) identity=mailfrom; client-ip=45.137.22.186; helo=hosted-by.rootlayer.net; envelope-from=cw4826878@gmail.com; receiver=<UNKNOWN>
Jan 21 23:03:43 saffre-rumma postfix/smtpd[16463]: 224AF21672: client=unknown[45.137.22.186]
Jan 21 23:03:43 saffre-rumma postfix/cleanup[16471]: 224AF21672: message-id=<20230121230341.2BAFFC26FBE2B2BA@gmail.com>
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: [45.137.22.186] [45.137.22.186] not internal
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: not authenticated
Jan 21 23:03:43 saffre-rumma opendkim[627]: 224AF21672: no signature data
Jan 21 23:03:43 saffre-rumma postfix/qmgr[28134]: 224AF21672: from=<cw4826878@gmail.com>, size=1938, nrcpt=1 (queue active)
Jan 21 23:03:43 saffre-rumma postfix/smtpd[16463]: disconnect from unknown[45.137.22.186] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Jan 21 23:03:43 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused
Jan 21 23:03:43 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused
Jan 21 23:03:44 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#2 of 3): Connection refused
Jan 21 23:03:44 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connect to spamd on ::1 failed, retrying (#3 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connect to spamd on 127.0.0.1 failed, retrying (#3 of 3): Connection refused
Jan 21 23:03:45 saffre-rumma spamc[16474]: connection attempt to spamd aborted after 3 retries
Jan 21 23:03:45 saffre-rumma postfix/pipe[16472]: 224AF21672: to=<info@saffre-rumma.net>, relay=spamassassin, delay=2.7, delays=0.6/0.02/0/2, dsn=2.0.0, status=sent (delivered via spamassassin service)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 224AF21672: removed
Jan 21 23:03:45 saffre-rumma postfix/pickup[17597]: 42C9C21701: uid=1008 from=<cw4826878@gmail.com>
Jan 21 23:03:45 saffre-rumma postfix/cleanup[16471]: 42C9C21701: message-id=<20230121230341.2BAFFC26FBE2B2BA@gmail.com>
Jan 21 23:03:45 saffre-rumma opendkim[627]: 42C9C21701: DKIM-Signature field added (s=mail, d=saffre-rumma.net)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 42C9C21701: from=<cw4826878@gmail.com>, size=2104, nrcpt=1 (queue active)
Jan 21 23:03:45 saffre-rumma postfix/local[16519]: 42C9C21701: to=<luc@saffre-rumma.net>, orig_to=<info@saffre-rumma.net>, relay=local, delay=0.04, delays=0.02/0.02/0/0, dsn=2.0.0, status=sent (delivered to maildir)
Jan 21 23:03:45 saffre-rumma postfix/local[16519]: 42C9C21701: to=<hannes@saffre-rumma.net>, orig_to=<info@saffre-rumma.net>, relay=local, delay=0.04, delays=0.02/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to maildir)
Jan 21 23:03:45 saffre-rumma postfix/qmgr[28134]: 42C9C21701: removed

Oops, our mail server is sending spam!

Mon, 09 Jan 2023 00:00:00 +0000

Oops, our mail server is sending spam!¶

Monday, January 9, 2023

I am currently using a Wi-Fi network that still uses WEP security, which is known to be insecure.

A few days after I started to use this network, our SR server started to send spam and hence was blocked by OVH to protect their network.

It had received this spam (190000 mails) from mail10.lwspanel.com, a server that runs Roundcube. The mail10.lwspanel.com itself is probably innocent, but one of their users is a spammer.

I guess that the spammer spoofed my password on SR, and that’s why SR accepted these mails.

I wonder whether there is a connection method that can’t be spoofed even on a low-security network.

Security settings in Thunderbird:

Connection security: “None” or “STARTTLS” (Port 110) or “SSL/TLS” (Port 995)
Authentication method is “Normal password” or “Encrypted password” or “Kerberos / GSSAPI” or “NTLM” or “OAuth2”

Both STARTTLS nor SSL/TLS need my password (they fail after I changed my password on the server). But I won’t tell Thunderbird my new password until I feel secure again.

Our server SR works only when authentication method is “Normal password”. Can Postfix also do “Encrypted password” or “OAuth2”?

Sunday, January 8, 2023

Sun, 08 Jan 2023 00:00:00 +0000

Sunday, January 8, 2023¶

Start using subscriptions on Jane¶

Regarding #4784: Yes, subscriptions will be referrable. Our subscriptions are long-term agreements and we refer to them in our daily work using a nickname. There is usually one subscription per customer, but some customers have multiple subscriptions. On the other hand they are vouchers in a journal and have a number because they are certifiable and legally valid documents, but they don’t create any ledger movements. Our customers don’t pay for signing an agreement with us (they pay every year for the invoice that gets generated by the agreement). For the (few) customers who want a service report, the service reports will now be automatically generated as monthly “invoices”.

I updated TicketOrderChecker so that I can run it after the next release to Jane. I will have to delete the current subscriptions before running it (e.g. by commenting out the execfile subscriptions_subscription line in the restore.py file).

Spontaneous optimization en passant: the plugins subscriptions and tickets unite now with the working plugin into the “Working time” menu group.

In lino_xl.lib.tickets, ActiveTickets and all other data tables on Ticket no longer show the overview field but the detail_link of every ticket.

The simple text representation of Ticket no longer includes the button_text of the ticket’s state.

Welcome to our team, Fahmida¶

From 10:30 to 11:30 Sharif and I had a first Jitsi meeting with Fahmida.

Fahmida lives in Bangladesh, like Sharif. She has learned theory about software development at university during 4 years. Now she wants to meet reality. She plans to get started as a volunteer contributor.

We decided that we will have a stand-up meeting every day at 9:00 UTC (which is 10:00 in Belgium and 15:00 in Bangladesh).

The meetings will be in our virtual office room on Jitsi:

https://meet.jit.si/synodalsoft

Quiz of the day¶

I’d like the test runner to give me a list of the failing test cases. Something like the following:

docs/apps/cms/db.rst
docs/commands/dump2py.rst
docs/dev/action_parameters.rst
docs/dev/actors.rst
docs/dev/admin_main.rst
docs/plugins/subscriptions.rst
docs/specs/delayed_values.rst
docs/specs/cosi5/index.rst
docs/apps/noi/db.rst
docs/apps/noi/users.rst

Hint: doctests run using atelier.test.DocTestCase.

Availability and presences planning¶

I created ticket #4786.

Each worker should be able to (optionally) register into Jane their “presences” or “availability” already in advance. When I go online and am “available” (for talk or for chat or email), I would click the play button, similar to what we do when working on a ticket. These sessions are kind of a “higher” level: they remain active for the whole working session.

We will probably also use this in Lino Presto.

Lino daemon and async processes¶

The lino.modlib.linod.consumers.LinodConsumer had a method send_push(), which looked suspect and seems unused. I renamed it to unused_send_push() to see whether something breaks.

I found a more or less satisfying answer to yesterday’s expert quiz: The demo2 fixture of lino.modlib.comments was setting use_multiprocessing to False in order to disable notifications. But this site attribute is no longer used, we now want to set use_linod to False.

MarkupResemblesLocatorWarning¶

I got a MarkupResemblesLocatorWarning and finally found out why it came and how to get rid of it.

The warning:

.../python3.10/site-packages/bs4/__init__.py:436: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup.
warnings.warn(

Why it comes: when you ask BeautifulSoup to parse a text like “See [url http://www.example.com].”, they estimate that your markup “resembles a filename closely enough to justify a warning”.

How to get rid of it:

from bs4 import MarkupResemblesLocatorWarning
import warnings
warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning)

Saturday, January 7, 2023

Sat, 07 Jan 2023 00:00:00 +0000

Saturday, January 7, 2023¶

En passant¶

Yesterday and today, while trying to understand what’s happening, I did a series of en passant changes.

A fake dump2py¶

When upgrading a production site

I wrote a new option --simulate for the pm dump2py command and optimized the code that causes the models contenttypes.ContentType, sessions.Session and lino.modlib.checkdata.Message to never be dumped.

There were still references to https://docs.djangoproject.com/en/6.0 and https://docs.djangoproject.com/en/6.0 in the docs. I replaced them by https://docs.djangoproject.com/en/6.0

I started a new section Django-admin commands in the “Reference” section of the Developer Guide. The first commands to benefit from this reorganization are pm dump2py, pm run and pm prep.

Inactive plugins¶

I recently wrote a solution for ticket #4780, but now I optimized my own approach. The lino.core.plugin.Plugin class has two new methods deactivate() and is_active(). When a plugin is inactive, its database models exist but otherwise almost everything, including its actors, becomes non-existent. The tinymce and extensible plugins now deactivate themselves in their on_init() method when the default_ui is not extjs. As a side effect, the main menu is now built using lino.core.site.Site.sorted_plugins.

Event loop is closed¶

When I run pm prep in noi1r, there is a long series of –non-fatal– tracebacks that have been there for some time:

Task exception was never retrieved
future: <Task finished name='Task-1548' coro=<Connection.disconnect() done, defined at /home/luc/virtualenvs/dev/lib/python3.10/site-packages/redis/asyncio/connection.py:687> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "/home/luc/virtualenvs/dev/lib/python3.10/site-packages/redis/asyncio/connection.py", line 696, in disconnect
    self._writer.close()  # type: ignore[union-attr]
  File "/usr/lib/python3.10/asyncio/streams.py", line 338, in close
    return self._transport.close()
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 698, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 750, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Question to async experts: what’s happening?

Friday, January 6, 2023

Fri, 06 Jan 2023 00:00:00 +0000

Friday, January 6, 2023¶

Contracts vanishing during restore?¶

One of our customers will switch their preview site to production. Theoretically this means simply to run their initdb_from_prod.sh a last time and then tell their system administrator to change the DNS entry of their Lino server to point to the new machine.

But then surprise: something is still going wrong with the migration because there are only 1811 contracts in isip.Contract on the preview while there are 1859 on the production server! And the initdb_from_prod.sh script runs without error message!

After running initdb_from_prod.sh there are only 1811 and not 1858 rows:

>>> from lino.api.shell import *
>>> print(isip.Contract.objects.count())
1811

The restore.py (which is called from initdb_from_prod.sh) logs the following output to stdout when running:

Execute file isip_contract.py ...
Loading 1811 objects to table isip_contract...
...
Execute file uploads_upload.py ...
Loading 5585 objects to table uploads_upload...

Here is how the isip_contract.py file looks like:

# -*- coding: UTF-8 -*-
logger.info("Loading 1858 objects to table isip_contract...")
# fields: id, signer1, signer2, user, printed_by, client, language, applies_from, applies_until, date_decided, date_issued, user_asd, exam_policy, ending, date_ended, type, study_type, stages, goals, duties_asd, duties_dsbe, duties_pcsw, duties_person, user_dsbe
loader.save(create_isip_contract(u'1',u'124',u'125',u'30',None,u'140',u'fr',None,None,None,None,u'4',u'2',None,None,u'1',None,None,None,None,None,None,None,u'30'))
loader.save(create_isip_contract(u'3',u'124',u'125',u'30',u'46',u'138',u'fr',date(2014,9,25),date(2015,9,24),None,None,None,u'2',None,date(2015,9,24),u'1',None,u'',u'',None,None,None,None,u'30'))
...
loader.save(create_isip_contract(u'1908',u'124',u'125',u'67',None,u'13364',u'',date(2022,12,22),date(2023,11,30),None,None,u'74',u'4',None,date(2023,11,30),u'3',None,None,None,None,None,None,None,None))

loader.flush_deferred_objects()

The reason was stupid: the (modified) restore.py took the snapshot files from another directory from October 2022.

async logging seems buggy¶

I also made the following observation: the lino.log file contains the messages emitted by restore.py are in a kind of random ordering.

202301-07 03:25:48 INFO [restore2preview 2273819 139926527686464] : Execute file isip_contract.py … 202301-07 04:27:12 INFO [isip_contract 2274609 140001715451712] : Loading 1811 objects to table isip_contract…

We are on a production server with asgi and daphne, so logging happens via an async consumer. How does a restore.py script (invoked using a pm run command) connect to the logger consumer? What if supervisor is not running? What if it is running but has not been restarted after the pull.sh?

I stopped the supervisor service and set use_linod to False, and then restarted the migration. Now the lino.log file is at least consistent with what I saw on screen:

202301-07 03:25:48 INFO [restore2preview 2273819 139926527686464] : Execute file isip_contract.py ...
202301-07 03:25:48 INFO [isip_contract 2273819 139926527686464] : Loading 1811 objects to table isip_contract...
...
202301-07 03:28:14 INFO [uploads_upload 2273819 139926527686464] : Loading 5585 objects to table uploads_upload...
202301-07 03:28:38 INFO [restore2preview 2273819 139926527686464] : Execute file users_authority.py ...

But setting use_linod to False when running a manual Django-admin command is of course not a satisfying workaround. I’d like to have a Jitsi session with Sharif and try to understand what’s happening.

Monday, October 31, 2022

Mon, 31 Oct 2022 00:00:00 +0000

Monday, October 31, 2022¶

Example of the magic things you can do with Lino migrations.

In an amici database there was a series of ca 250 blog entries that had been imported from TIM. They are rather comments than blog entries (were never thought to get published). I simply changed the create_blogs_entry() function in the restore.py so that it converts them to comments.

Here is the create_blogs_entry() function as it was before:

def create_blogs_entry(id, user_id, owner_type_id, owner_id, body, body_short_preview, body_full_preview, title, pub_date, pub_time, entry_type_id, language):
    owner_type_id = new_content_type_id(owner_type_id)
    kw = dict()
    kw.update(id=id)
    kw.update(user_id=user_id)
    kw.update(owner_type_id=owner_type_id)
    kw.update(owner_id=owner_id)
    kw.update(body=body)
    kw.update(body_short_preview=body_short_preview)
    kw.update(body_full_preview=body_full_preview)
    kw.update(title=title)
    kw.update(pub_date=pub_date)
    kw.update(pub_time=pub_time)
    kw.update(entry_type_id=entry_type_id)
    kw.update(language=language)
    return blogs_Entry(**kw)

Here is what I changed it into:

def create_blogs_entry(id, user_id, owner_type_id, owner_id, body, body_short_preview, body_full_preview, title, pub_date, pub_time, entry_type_id, language):
    owner_type_id = new_content_type_id(owner_type_id)
    kw = dict()
    #kw.update(id=id)
    kw.update(user_id=user_id)
    kw.update(owner_type_id=owner_type_id)
    kw.update(owner_id=owner_id)
    kw.update(body=title+". "+body)
    #kw.update(body_short_preview=body_short_preview)
    #kw.update(body_full_preview=body_full_preview)
    #kw.update(title=title)
    if pub_time is None:
        pub_time = time(12,0,0)
    kw.update(created=make_aware(datetime.combine(pub_date, pub_time)))
    #kw.update(entry_type_id=entry_type_id)
    #kw.update(language=language)
    return comments_Comment(**kw)

I also had to change the ordering of the execfile statements in the main() function because otherwise the automatically generated id fields of the former blog entries would clash with ids of existing comments:

execfile("comments_reaction.py", *args)
execfile("blogs_entry.py", *args)

After the migration we need to run pm checkdata -pf in order to fill the body preview fields.

Monday, October 17, 2022

Mon, 17 Oct 2022 00:00:00 +0000

Monday, October 17, 2022¶

Thursday, October 13, 2022

Thu, 13 Oct 2022 00:00:00 +0000

Thursday, October 13, 2022¶

I started wondering what they actually wanted to say when they asked to keep the position number unchanged for each client. I guess that they are talking about lino_xl.lib.reception.WaitingVisitors.

It was a bit irritating to see that the visitors in our demo have been waiting there for more than 8 years. So I accidentally wrote a solution for the old naturaltime problem: when you want the output of a naturaltime string to respect Site.the_demo_date (to make it testable, but also to avoid the irritating value mentioned above), then you can now simply replace your import:

from django.contrib.humanize.templatetags.humanize import naturaltime

by this:

from lino.utils.format_dates import naturaltime

>>> import lino
>>> # lino.startup('lino_book.projects.min1.settings')
>>> lino.startup('lino_welfare.projects.mathieu.settings.demo')
>>> from lino.api.doctest import *

>>> from datetime import date, datetime, timezone
>>> from django.contrib.humanize.templatetags.humanize import naturaltime as hnaturaltime
>>> from lino_xl.lib.reception.models import naturaltime

>>> print(datetime.now())
2022-10-12 23:14:...

>>> t1 = datetime(2022,10,12,22,20,0)
>>> print(hnaturaltime(t1))
54 minutes ago

>>> print(settings.SITE.the_demo_date)
2014-05-22
>>> print(hnaturaltime(t1))
il y a une heure

>>> print(naturaltime(t1))
dans 8 années, 4 mois

Tuesday, October 11, 2022

Tue, 11 Oct 2022 00:00:00 +0000

Tuesday, October 11, 2022¶

Here is something we didn’t yet need: update the “othermodel” pointed to by a ForeignKey. Something like this:

dd.update_field(Convention, "company", othermodel="jobs.JobProvider")

But that doesn’t work. I updated the docstring of dd.update_field().

The best solution (IMHO) was to split up the lino_xl.lib.contacts.ContactRelated mixin into an AbstractContactRelated, which does the same as ContactRelated except defining the company field.

A side effect of this change is that lino.utils.diag.Analyzer.show_database_structure() gives a series of false alarms: every model based on ContactRelated

>>> import lino
>>> lino.startup('lino_welfare.projects.mathieu.settings.demo')
>>> from lino.api.doctest import *

>>> art60.Convention._meta.get_field("company")
<django.db.models.fields.related.ForeignKey: company>

>>> art60.Convention.objects.first().company.__class__
<class 'lino_welfare.modlib.jobs.models.JobProvider'>

>>> art61.Contract.objects.first().company.__class__
<class 'lino_welfare.modlib.jobs.models.Employer'>

Sunday, October 2, 2022

Sun, 02 Oct 2022 00:00:00 +0000

Sunday, October 2, 2022¶

Working on #4436.

>>> from lino import startup
>>> startup('lino_welfare.projects.mathieu.settings.demo')
>>> from lino.api.doctest import *

I had started (locally) a new lino_welcht.lib.jobs plugin as a copy of lino_welfare.modlib.jobs, with the only change (so far) that the is_social field was now on the job provider, no longer on the job type. My plan was that this might replace its parent in the future. En passant I also had moved all data tables to a separate module ui.py.

But then I realized that copying the whole jobs plugin is not needed. I made a new plugin lino_welfare.modlib.art60.

With the new art60 contracts we have multiple subcontracts per contract. An art60 contract is between the center and a client, the center is the legal employer and makes an “agreement” with one of their partners (called the “utilizing services”) to take that particular client for that particular period and that particular job. Usually there’s one “agreement” per “contract”. But reality can differ from theory, there can be “movements”.

Centers who don’t want this new feature just don’t install lino_welfare.modlib.art60 and continue to use the lino_welfare.modlib.jobs.Contract as before. Their verbose_name is the same:

>>> show_model_translations(jobs.Contract, art60.Contract, languages="en fr de")
...
+----------------+-------------------------+--------------------------+-----------------------+
| Name           | en                      | fr                       | de                    |
+================+=========================+==========================+=======================+
| jobs.Contract  | Art60§7 job supplyment  | Mise à l'emploi art60§7  | Art.60§7-Konvention   |
|                | Art60§7 job supplyments | Mises à l'emploi art60§7 | Art.60§7-Konventionen |
+----------------+-------------------------+--------------------------+-----------------------+
| art60.Contract | Art60§7 job supplyment  | Mise à l'emploi art60§7  | Art.60§7-Konvention   |
|                | Art60§7 job supplyments | Mises à l'emploi art60§7 | Art.60§7-Konventionen |
+----------------+-------------------------+--------------------------+-----------------------+

Actually art60 even removes jobs.Contract, above test was made with an intermediate version.

>>> show_field_translations(jobs.Contract, languages="en fr de")
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   |   | Field            | en                                       | fr                                       | de                                       |
+===+===+==================+==========================================+==========================================+==========================================+
| x | x | id               | ID                                       | ID                                       | ID                                       |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | signer1          | Secretary                                | Secrétaire                               | Sekretär                                 |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | signer2          | President                                | Président                                | Präsident                                |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | user             | Author : The person who created this     | Auteur : La personne qui a créé ce       | Autor : Die Person, die diesen Vertrag   |
|   |   |                  | contract.                                | contrat.                                 | erstellt hat.                            |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | company          | Organization : Pointer to Company.       | Organisation : Pointer to Company.       | Organisation : Pointer to Company.       |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | contact_person   | Represented by : Pointer to Person.      | Represented by : Pointer to Person.      | Represented by : Pointer to Person.      |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | contact_role     | Represented as : The optional Role of    | Represented as : The optional Role of    | Represented as : The optional Role of    |
|   |   |                  | the contact_person within company.       | the contact_person within company.       | the contact_person within company.       |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | printed_by       | Printed : ForeignKey to the Excerpt      | Imprimé : ForeignKey to the Excerpt      | Gedruckt : ForeignKey to the Excerpt     |
|   |   |                  | which certifies this instance.           | which certifies this instance.           | which certifies this instance.           |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | client           | Client : The client for whom this        | Bénéficiaire : The client for whom this  | Klient : The client for whom this        |
|   |   |                  | contract is done.                        | contract is done.                        | contract is done.                        |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | language         | Language : The language of this          | Langue : The language of this contract.  | Sprache : The language of this contract. |
|   |   |                  | contract. Default value is the client’s  | Default value is the client’s language.  | Default value is the client’s language.  |
|   |   |                  | language.                                |                                          |                                          |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | applies_from     | applies from : The start date of the     | Début de contrat : The start date of the | Laufzeit von : The start date of the     |
|   |   |                  | contract.                                | contract.                                | contract.                                |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | applies_until    | applies until : The planned end date of  | Fin prévue : The planned end date of     | Laufzeit bis : The planned end date of   |
|   |   |                  | this contract.                           | this contract.                           | this contract.                           |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | date_decided     | date decided : When the contract was     | Décidé le : When the contract was        | Beschlossen am : When the contract was   |
|   |   |                  | ratified by the responsible board.       | ratified by the responsible board.       | ratified by the responsible board.       |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | date_issued      | date issued : When the contract was      | Date de signature : When the contract    | Ausgestellt am : When the contract was   |
|   |   |                  | issued to the client and signed by them. | was issued to the client and signed by   | issued to the client and signed by them. |
|   |   |                  |                                          | them.                                    |                                          |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | user_asd         | Responsible (GSS) : The responsible      | Titulaire (SSG) : La personne            | Verantwortlicher (ASD) : Die             |
|   |   |                  | person at the general social service.    | responsable au service social général.   | verantwortliche Person im allgemeinen    |
|   |   |                  |                                          |                                          | Sozialdienst.                            |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | exam_policy      | Examination Policy                       | Régime d'évaluation                      | Auswertungsstrategie                     |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | ending           | Reason of termination : The reason of    | Motif d'arrêt : The reason of prematured | Beendigungsgrund : The reason of         |
|   |   |                  | prematured ending.  Pointer to           | ending.  Pointer to ContractEnding       | prematured ending.  Pointer to           |
|   |   |                  | ContractEnding                           |                                          | ContractEnding                           |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | date_ended       | date ended : The date when this contract | Date de fin : The date when this         | Enddatum : The date when this contract   |
|   |   |                  | was effectively ended. This field is set | contract was effectively ended. This     | was effectively ended. This field is set |
|   |   |                  | to the same value as applies_until.      | field is set to the same value as        | to the same value as applies_until.      |
|   |   |                  |                                          | applies_until.                           |                                          |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| x |   | duration         | duration (days)                          | Jours ONSS                               | Dauer (Arbeitstage)                      |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | reference_person | reference person                         | persone de référence                     | Referenzperson                           |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | responsibilities | responsibilities                         | responsabilités                          | Aufgabenbereich                          |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | remark           | Remark                                   | Remarque                                 | Bemerkung                                |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| ? |   | type             | Type : The type of this contract.        | Type : The type of this contract.        | Art : The type of this contract. Pointer |
|   |   |                  | Pointer to a subclass of                 | Pointer to a subclass of                 | to a subclass of ContractTypeBase.       |
|   |   |                  | ContractTypeBase.                        | ContractTypeBase.                        |                                          |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| ? |   | job              | Job                                      | Poste de travail                         | Stelle                                   |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | regime           | Work Regime                              | Régime de travail                        | Arbeitsregime                            |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | schedule         | Work Schedule                            | Horaire                                  | Stundenplan                              |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | hourly_rate      | hourly rate                              | coût horaire                             | Stundensatz                              |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+
|   | x | refund_rate      | refund rate                              | tarif de remboursement                   | Rückzahlung                              |
+---+---+------------------+------------------------------------------+------------------------------------------+------------------------------------------+

Review the hierarchy of JobSupplyment, ContractPartnerBase, ContractBase mixins.

>>> print(' '.join([f.name for f in isip.ContractBase._meta.get_fields()]))
...
signer1 signer2 user printed_by client language applies_from applies_until date_decided date_issued user_asd exam_policy ending date_ended

>>> print(' '.join([f.name for f in isip.ContractPartnerBase._meta.get_fields()]))
company contact_person contact_role

>>> print(' '.join([f.name for f in jobs.JobSupplyment._meta.get_fields(include_parents=False)]))
signer1 signer2 user printed_by client language applies_from applies_until date_decided date_issued user_asd exam_policy ending date_ended duration reference_person responsibilities remark

Internal fork of a plugin

Fri, 30 Sep 2022 00:00:00 +0000

Internal fork of a plugin¶

Friday, September 30, 2022

I started a new plugin lino_welcht.lib.jobs as a copy of lino_welfare.modlib.jobs.

I wrote a new utility function lino.api.doctest.show_field_translations().

>>> from lino import startup
>>> startup('lino_welfare.projects.mathieu.settings.demo')
>>> from lino.api.doctest import *

>>> show_field_translations(jobs.Contract, "user user_asd", languages="en fr de")
+----------+---------------------------------------+----------------------------------------+----------------------------------------+
| Field    | en                                    | fr                                     | de                                     |
+==========+=======================================+========================================+========================================+
| user     | Author : The person who created this  | Auteur : La personne qui a créé ce     | Autor : Die Person, die diesen Vertrag |
|          | contract.                             | contrat.                               | erstellt hat.                          |
+----------+---------------------------------------+----------------------------------------+----------------------------------------+
| user_asd | Responsible (GSS) : The responsible   | Titulaire (SSG) : La personne          | Verantwortlicher (ASD) : Die           |
|          | person at the general social service. | responsable au service social général. | verantwortliche Person im allgemeinen  |
|          |                                       |                                        | Sozialdienst.                          |
+----------+---------------------------------------+----------------------------------------+----------------------------------------+

>>> show_field_translations(jobs.JobProvider, "is_social", languages="en fr de")
=========== ================ ================== ================
 Field       en               fr                 de
----------- ---------------- ------------------ ----------------
 is_social   Social economy   Économie sociale   Sozialökonomie
=========== ================ ================== ================

My third time at HackerX

Thu, 29 Sep 2022 00:00:00 +0000

My third time at HackerX¶

Thursday, September 29, 2022

Yesterday I went once more to HackerX, an event where developers can enter for free because they are actually what is being sold. The customers are companies in search of new human resources. I probably don’t fit into their standard product pattern but they invited me. Nice. See also May 2022 and May 2021.

This time there was an employer for whom I would agree to work: Eesti Energia, the public energy company in Estonia. I had only a few minutes of talk with them. I started by saying “So you are the ones who burn down Estonia’s forests. What a shame!” They seemed to be used to that one, they even added oil shale mining, and I admitted “it’s true that the real culprit is the EU who tells everybody that burning wood is a good thing.” I also managed to explain them that I am not for sale without Lino, how Lino might be useful for them and how I might help them to use Lino. All kind of projects where multiple users need to collect, verify, process or query some data.

All other employers were private corporations; I didn’t even try to talk to them. Being an employee means that you give your time and energy away for some higher purpose. The purpose of Eesti Energia is to provide electricity to everybody in Estonia. That’s a purpose I can imagine to work for. But the purpose of every private corporation is to maximize the profit of its investors. I’ll never belong to such a purpose! I explained this several times to other developers. Most of them were younger than me.

But as expected, I met several interesting people among the other developers. There was Kristjan, who might give me a free crash course on the Progress/OpenEdge SQL query tool. There was that guy from Brasil who loves Estonia because you can walk around in the city at 10pm with your cell phone in your hand and nobody will steal it. There was the guy who surprised me because he came from Estonia but didn’t speak Estonian. The Russian-speaking Estonians are a similar to the German-speaking Belgians in that they are a bridge between two cultures. There was Aleksandr, another Russian-speaking Estonian, who didn’t know that there is a country named Belgium, despite the fact that their geography teacher made them cite the NATO members by heart.

Wednesday, September 28, 2022

Wed, 28 Sep 2022 00:00:00 +0000

Wednesday, September 28, 2022¶

Jane seems to be getting slow again. I have this impression. Uptime is 35 days. I will reboot the server now and see whether she gets quick again. But I’d like to have a measurement for the responsiveness.

I ran ps aux --sort '%mem' before and after.

Before:

root     11709  0.0  2.2 948280 45584 ?        Ssl  sept 27   0:04 /usr/lib/snapd/snapd
mysql    32406  0.0  6.5 1728188 130132 ?      Ssl  sept 13  13:16 /usr/sbin/mysqld
www-data   948  0.0  6.8 325680 135672 ?       S    sept 22   0:27 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/welcht1/nginx/uwsgi.ini --ignore-sigpipe
www-data   951  0.0  6.9 328920 138924 ?       S    sept 22   0:27 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/weleup1/nginx/uwsgi.ini --ignore-sigpipe
www-data  1229  0.0  7.1 327852 142060 ?       S    sept 22   0:00 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/welcht1/nginx/uwsgi.ini --ignore-sigpipe
www-data  1230  0.0  7.2 330668 144860 ?       S    sept 22   0:00 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/weleup1/nginx/uwsgi.ini --ignore-sigpipe
www-data  1231  0.0  7.2 330668 144864 ?       S    sept 22   0:00 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/weleup1/nginx/uwsgi.ini --ignore-sigpipe
root       946  0.2  7.6 491892 153156 ?       Sl   sept 22  21:35 python manage.py runworker linod_jane
root       944  0.1  9.9 1103500 198172 ?      Sl   sept 22  12:49 /usr/local/lino/lino_local/jane/env/bin/python3 /usr/local/lino/lino_local/jane/env/bin/daphne -u /
root       945  0.1 10.4 1114180 208516 ?      Sl   sept 22  16:07 /usr/local/lino/lino_local/jane/env/bin/python3 /usr/local/lino/lino_local/jane/env/bin/daphne -u /

After:

postgres   812  0.0  1.3 211424 27116 ?        S    16:07   0:00 /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/
root       765  7.0  2.0 874484 41500 ?        Ssl  16:07   0:00 /usr/lib/snapd/snapd
root       743 19.2  2.7 102284 55056 ?        Rs   16:07   0:02 /snap/certbot/2344/bin/python3 -s /snap/certbot/2344/bin/certbot renew -q
root       974  1.1  3.7 422612 74568 ?        Sl   16:07   0:00 /usr/lib/libreoffice/program/soffice.bin --accept=socket,host=127.0.0.1,port=8100;urp; --nologo --hea
mysql      947  3.1  4.8 1721016 96604 ?       Ssl  16:07   0:00 /usr/sbin/mysqld
www-data   922 22.1  4.8 238196 97456 ?        R    16:07   0:02 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/welcht1/nginx/uwsgi.ini --ignore-sigpipe
www-data   924 22.6  4.9 238980 98224 ?        R    16:07   0:02 /usr/bin/uwsgi --ini /usr/local/lino/lino_local/weleup1/nginx/uwsgi.ini --ignore-sigpipe
root       917 22.7  5.2 146076 104584 ?       R    16:07   0:02 /usr/local/lino/lino_local/jane/env/bin/python3 /usr/local/lino/lino_local/jane/env/bin/daphne -u /ru
root       918 22.6  5.2 146076 105068 ?       R    16:07   0:02 /usr/local/lino/lino_local/jane/env/bin/python3 /usr/local/lino/lino_local/jane/env/bin/daphne -u /ru
root       919 22.9  5.8 186292 116644 ?       R    16:07   0:02 python manage.py runworker linod_jane
root       920 22.5  5.9 187368 118224 ?       R    16:07   0:02 python manage.py initiate_linod

As we know, our Lino processess use a bit less memory now then before.

Saturday, September 17, 2022

Sat, 17 Sep 2022 00:00:00 +0000

Saturday, September 17, 2022¶

In lino.utils.code I changed codefiles() function to search the file system for source files. So the analyze_rst() now gives useful results.

Seems that the text “optional arguments:” generated by the --help of optparse has been changed to “options:” between Python 3.9 and Python 3.10. Causes two failures. https://gitlab.com/lino-framework/book/-/jobs/3045154005

Which IM client to use?

Fri, 16 Sep 2022 00:00:00 +0000

Which IM client to use?¶

Friday, September 16, 2022

Sharif and I have been using Discord for quite some time. Because it simply works. Jitsi didn’t always work well. And then Jitsi has no history at all of our chat messages. We know that Discord is non-free. It is a compromise. Now it seems that screen-sharing doesn’t work any more on Discord with my new computer. Rather than investing time into a non-free software, I thought that we should give Signal a try.

Thursday, September 15, 2022

Thu, 15 Sep 2022 00:00:00 +0000

Thursday, September 15, 2022¶

This is the first blog entry written on my new notebook computer, a Lenovo Yoga.

inv clean -b bd in my blog takes now 1 real minute instead of 3, inv prep in book takes now 3m52 instead of 15m48, that’s 3 to 4 times faster.

Atom editor¶

I used Ubuntu’s snap to install “snappyatom”. But then uninstalled it and downloaded the deb installer from Atom project instead. Don’t know what snap wanted to make snappier, and the snappy variant didn’t find my existing atom config files.

Oops! Then I saw that Atom is being discontinued! What a pity!

Yes, it was predictable. Atom is a competitor for Microsoft Visual Studio. Atom was being developed mainly by GitHub. GitHub was bought by Microsoft. It’s a typical Microsoft strategy for eliminating their competitors: they buy them. And even a free software product is vulnerable to this when it is being developed by a single legal person.

Let’s try Eric. Looks nice, but it keeps saying:

Could not download the requested file from https://eric-ide.python-projects.org/plugins7/repository.xml.

Error: TLS initialization failed

Probably related to the following warnings in the console:

qt.tlsbackend.ossl: Incompatible version of OpenSSL (built with OpenSSL 1.x, runtime version is >= 3.x)
qt.network.ssl: The backend "cert-only" does not support QSslKey
qt.network.ssl: Active TLS backend does not support key creation

$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Nah. Sorry, Eric, I’ll stay with Atom right now. I guess that anyway somebody will continue at least its maintenance.

Yet another warning:

python-tools was unable to find your machine’s python executable.

Please try set the path in package settings and then restart atom.

If the issue persists please post an issue on https://github.com/michaelaquilina/python-tools/issues/new

Error: spawn python ENOENT

Solution: Edit ‣ Preferences ‣ Packages

VirtualBox¶

On my new computer I said:

$ sudo apt install virtualbox

When I then started VirtualBox on my .vdi file, it said:

Kernel driver not installed (rc=-1908)

The VirtualBox Linux kernel driver is either not loaded or not set up correctly. Please reinstall virtualbox-dkms package and load the kernel module by executing

‘modprobe vboxdrv’

as root.

If your system has EFI Secure Boot enabled you may also need to sign the kernel modules (vboxdrv, vboxnetflt, vboxnetadp, vboxpci) before you can load them. Please see your Linux system’s documentation for more information.

where: suplibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) - The support driver is not installed. On linux, open returned ENOENT.

Solution:

$ sudo apt reinstall virtualbox-dkms

This told me that my “system has UEFI Secure Boot enabled” and prompted me to enter a password. It didn’t reboot automatically, but my guts told me to say:

$ sudo reboot now

And during the boot process I had a little menu inviting me to “register” some “modification key”.

Next problem:

The virtual machine execution ran into a non-fatal problem as described below. We suggest that you take appropriate action to prevent the problem from recurring.

The image file ‘/usr/share/virtualbox/VBoxGuestAdditions.iso’ is inaccessible and is being ignored. Please select a different image file for the virtual DVD drive..

Solution:

$ sudo apt install virtualbox-guest-additions-iso

A last remark, it took me some time to re-understand the following informative message:

The virtual machine window will be now switched to Scale mode. You can go back to windowed mode at any time by pressing Host+C.

Note that the Host key is currently defined as Right Ctrl.

Note that the main menu bar is hidden in scaled mode. You can access it by pressing Host+Home

I checked the “Don’t show this again” checkbox. And later I disabled the “Host key” in the VirtualBox preferences. I use the Right Ctrl key quite often, so it was disturbing to have it not operate the usual way.

Sphinx¶

Could not import extension sphinxcontrib.yt (exception: No module named ‘sphinxcontrib.yt’)

Wednesday, September 7, 2022

Wed, 07 Sep 2022 00:00:00 +0000

Wednesday, September 7, 2022¶

I tried once more to upgrade my Tuxedo notebook from Ubuntu 18.04 to something newer. I ran do-release-upgrade multiple times after disabling all the custom PPAs. Still no success. Tuxedo themselves say quite clearly that I must rather backup everything and then reinstall a new Linux.

$ sudo apt update
[sudo] password for luc:
Get:1 http://ppa.launchpad.net/libreoffice/ppa/ubuntu bionic InRelease [20,8 kB]
Hit:2 https://installer.id.ee/media/ubuntu bionic InRelease
Get:3 https://download.docker.com/linux/ubuntu bionic InRelease [64,4 kB]
Get:4 https://updates.signal.org/desktop/apt xenial InRelease [2 315 B]
Get:5 https://linux.teamviewer.com/deb stable InRelease [11,9 kB]
Get:6 http://oibaf.tuxedocomputers.com/ubuntu bionic InRelease [3 197 B]
Get:7 http://deb.tuxedocomputers.com/ubuntu bionic InRelease [3 187 B]
Get:8 https://packages.riot.im/debian default InRelease [2 892 B]
Hit:9 http://graphics.tuxedocomputers.com/ubuntu bionic InRelease
Get:10 https://packages.microsoft.com/repos/ms-teams stable InRelease [17,5 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security InRelease [88,7 kB]
Hit:12 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:13 https://updates.signal.org/desktop/apt xenial/main amd64 Packages [13,8 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88,7 kB]
Get:15 http://ppa.launchpad.net/libreoffice/ppa/ubuntu bionic/main amd64 Packages [33,7 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74,6 kB]
Get:17 http://ppa.launchpad.net/libreoffice/ppa/ubuntu bionic/main i386 Packages [33,7 kB]
Get:18 http://deb.tuxedocomputers.com/ubuntu bionic/main amd64 Packages [23,3 kB]
Get:19 http://deb.tuxedocomputers.com/ubuntu bionic/main i386 Packages [11,6 kB]
Get:20 https://packages.riot.im/debian default/main amd64 Packages [959 B]
Get:21 https://packages.microsoft.com/repos/ms-teams stable/main amd64 Packages [9 361 B]
Get:22 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2 389 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2 729 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1 233 kB]
Get:25 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [414 kB]
Get:26 http://security.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [55,2 kB]
Get:27 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [884 kB]
Get:28 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [122 kB]
Get:29 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1 035 kB]
Get:30 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1 533 kB]
Get:31 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1 228 kB]
Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [503 kB]
Get:33 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [282 kB]
Get:34 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [61,1 kB]
Get:35 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2 464 B]
Get:36 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [296 kB]
Get:37 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [913 kB]
Get:38 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [126 kB]
Get:39 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1 842 kB]
Get:40 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1 625 kB]
Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [399 kB]
Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [302 kB]
Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2 468 B]
Get:44 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [9 268 B]
Fetched 18,5 MB in 6s (3 175 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
62 packages can be upgraded. Run 'apt list --upgradable' to see them.

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-hwe-5.4-headers-5.4.0-109 linux-hwe-5.4-headers-5.4.0-110 linux-hwe-5.4-headers-5.4.0-113 linux-hwe-5.4-headers-5.4.0-117 linux-hwe-5.4-headers-5.4.0-120
  linux-hwe-5.4-headers-5.4.0-121
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  linux-headers-5.4.0-125-generic linux-hwe-5.4-headers-5.4.0-125 linux-image-5.4.0-125-generic linux-modules-5.4.0-125-generic linux-modules-extra-5.4.0-125-generic
The following packages have been kept back:
  open-eid
The following packages will be upgraded:
  curl element-desktop fonts-opensymbol gir1.2-notify-0.7 libcurl3-gnutls libcurl4 liblibreoffice-java libnotify-bin libnotify4 libreoffice libreoffice-base
  libreoffice-base-core libreoffice-base-drivers libreoffice-calc libreoffice-common libreoffice-core libreoffice-draw libreoffice-gnome libreoffice-gtk3
  libreoffice-help-common libreoffice-help-de libreoffice-help-en-gb libreoffice-help-en-us libreoffice-impress libreoffice-java-common libreoffice-l10n-de
  libreoffice-l10n-en-gb libreoffice-l10n-en-za libreoffice-math libreoffice-nlpsolver libreoffice-report-builder libreoffice-report-builder-bin
  libreoffice-script-provider-bsh libreoffice-script-provider-js libreoffice-script-provider-python libreoffice-sdbc-firebird libreoffice-sdbc-hsqldb libreoffice-sdbc-mysql
  libreoffice-sdbc-postgresql libreoffice-style-colibre libreoffice-style-elementary libreoffice-style-tango libreoffice-style-yaru libreoffice-wiki-publisher
  libreoffice-writer libuno-cppu3 libuno-cppuhelpergcc3-3 libuno-purpenvhelpergcc3-3 libuno-sal3 libuno-salhelpergcc3-3 libunoloader-java linux-generic-hwe-18.04
  linux-headers-generic-hwe-18.04 linux-image-generic-hwe-18.04 linux-libc-dev python3-uno signal-desktop tzdata uno-libs-private ure ure-java
61 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
8 standard security updates
Need to get 413 MB of archives.
After this operation, 368 MB of additional disk space will be used.
Do you want to continue? [Y/n]
...
update-initramfs: Generating /boot/initrd.img-5.4.0-125-generic
I: The initramfs will attempt to resume from /dev/sda3
I: (UUID=d8772344-2a32-4995-8575-f47bfe8da338)
I: Set the RESUME variable to override this.
/etc/kernel/postinst.d/zz-update-grub:

sudo apt remove xserver-xorg-video-nouveau
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libbsh-java libel-api-java libhsqldb1.8.0-java libjsp-api-java libservlet-api-java libservlet3.1-java libwebsocket-api-java libxatracker2 libxvmc1
  xserver-xorg-video-amdgpu xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel xserver-xorg-video-qxl xserver-xorg-video-radeon
  xserver-xorg-video-vesa xserver-xorg-video-vmware
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  xserver-xorg-video-all xserver-xorg-video-nouveau
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 328 kB disk space will be freed.
Do you want to continue? [Y/n]

Checking for installed snaps

Calculating snap size requirements
No candidate ver:  linux-image-5.4.0-56-generic
No candidate ver:  linux-modules-5.4.0-56-generic
No candidate ver:  linux-modules-extra-5.4.0-56-generic
No candidate ver:  megasync

Tuesday, September 6, 2022

Tue, 06 Sep 2022 00:00:00 +0000

Tuesday, September 6, 2022¶

#4642 is an example of something that would be difficult if Lino were maintained by a bigger team: modifying the name of a database model.

This would have caused endless discussions. It’s true that by “just doing it” I stumbled into a few surprises. But the advantage is that I didn’t even need to bother the other developers, they will just see the result and feel intuitively that things are more clear now. Unless I am wrong, which is still possible, but at least my change has done already a big series of conceptual tests when I handled all the surprises.

Another reason why such a change would be difficult: no customer would ever pay for fixing them.

One of the surprises was:

checkdata.Message.owner_type: (fields.E304) Reverse accessor 'ContentType.message_set' for 'checkdata.Message.owner_type' clashes with reverse accessor for 'notify.Message.owner_type'.
      HINT: Add or change a related_name argument to the definition for 'checkdata.Message.owner_type' or 'notify.Message.owner_type'.
notify.Message.owner_type: (fields.E304) Reverse accessor 'ContentType.message_set' for 'notify.Message.owner_type' clashes with reverse accessor for 'checkdata.Message.owner_type'.
      HINT: Add or change a related_name argument to the definition for 'notify.Message.owner_type' or 'checkdata.Message.owner_type'.

I had to modify lino.modlib.gfks.Controllable so that it includes the appname into the related_name.

Write a Lino for real estate agents (brokers)

Fri, 02 Sep 2022 00:00:00 +0000

Write a Lino for real estate agents (brokers)¶

Friday, September 2, 2022

I had a meeting with a friend who works as a professional broker. In Estonia there are ca 3 major platforms for real estate offers, but none of them is free. A Lino application would make sense, but only at international level and only long-term. No immediate investors at the horizon.

I created #4637 and started to think about it.

Trading real estate objects is not the same as managing them. So this Lino would be for brokers (kinnisvaramaakler), not for homeowner associations (korteriühistu). OTOH there will be common vocabulary with a application for homeowner associations, and there might be another Lino for homeowner associations as well. So the name “Lino Immo” would be biased.

It would define a series of new product types, each of them would inherit from lino_xl.lib.products.Product and from lino_xl.lib.countries.AddressLocation. Journals would be rental offers, sales offers, rental contracts, sales contracts, and maybe more.

Wednesday, August 31, 2022

Wed, 31 Aug 2022 00:00:00 +0000

Wednesday, August 31, 2022¶

$ sudo apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org buster/updates InRelease [34,8 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [56,6 kB]
Hit:4 http://deb.debian.org/debian buster-backports InRelease
Get:5 http://mirror.hetzner.de/debian/packages buster InRelease [122 kB]
Hit:6 http://ftp.de.debian.org/debian buster-backports InRelease
Get:7 http://mirror.hetzner.de/debian/packages buster-updates InRelease [56,6 kB]
Hit:8 http://mirror.hetzner.de/debian/packages buster-backports InRelease
Get:9 http://mirror.hetzner.de/debian/security buster/updates InRelease [34,8 kB]
Reading package lists... Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.6' to '10.12'
E: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://security.debian.org buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://deb.debian.org/debian buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
N: Repository 'http://mirror.hetzner.de/debian/packages buster InRelease' changed its 'Version' value from '10.6' to '10.12'
E: Repository 'http://mirror.hetzner.de/debian/packages buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://mirror.hetzner.de/debian/packages buster-updates InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
E: Repository 'http://mirror.hetzner.de/debian/security buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

I added the following line to different files below /etc/apt/apt.conf.d:

Acquire::AllowInsecureRepositories "true";

But without success.

The solution was to simply say --allow-releaseinfo-change at the command-line. Thanks to Josh Sherman for sharing.

Wednesday, August 24, 2022

Wed, 24 Aug 2022 00:00:00 +0000

Wednesday, August 24, 2022¶

I used https://sharepic.fsfe.org/#pmpc

“Get rid of vendor lock-in in public administrations”

Monday, August 22, 2022

Mon, 22 Aug 2022 00:00:00 +0000

Monday, August 22, 2022¶

Friday, August 19, 2022

Fri, 19 Aug 2022 00:00:00 +0000

Friday, August 19, 2022¶

I sent an email to Martin Leblanc, the author and maintainer of the PyVAT package, asking him about his plans regarding pyvat.

The pyvat project has been forked 27 times (source). Most of these forks don’t add anything, except for the following ones:

https://github.com/Alex-Espressone/pyvat is 1 commit ahead, 73 commits behind iconfinder:master.
https://github.com/jbma/pyvat is 6 commits ahead, 97 commits behind iconfinder:master.
https://github.com/karpitsky/pyvat is 4 commits ahead, 85 commits behind iconfinder:master.
https://github.com/matbas-onwave/pyvat is 9 commits ahead, 3 commits behind iconfinder:master.
https://github.com/onlinecity/pyvat is 1 commit ahead, 112 commits behind iconfinder:master.
https://github.com/wp-media/pyvat is 1 commit ahead, 3 commits behind iconfinder:master.

Wednesday, August 10, 2022

Wed, 10 Aug 2022 00:00:00 +0000

Wednesday, August 10, 2022¶

Morning session¶

The inv prep command now accepts a new option --after.

En passant I worked in the documentation for atelier. For example, I can now write sentences like “The per_project command has an option --list” and it links to the right place.

Don’t mix up the directives .. command:: and .. program::, and the roles :cmd: , :command: and :option:.

program, command and option are defined by Sphinx.
command and cmd are defined by rstgen.sphinxcontrib.

About display_mode on mobile views¶

We currently have display_mode_slave and display_mode_main. I think about replacing them by a single attribute display_mode, which would be a tuple, with the following default value:

display_mode = (
  (70, constants.DISPLAY_MODE_TABLE),
  (None, constants.DISPLAY_MODE_LIST)
)

Where “70” would be the available width on screen, in “logical characters”.

The display mode would no longer depend on whether it is a slave or a main widget, but on the available width.

The whole display_mode tuple must be given to the client (in the .js file), and the client must loop over this tuple and pick the first item that fits into the available width.

The question is can we know the available width when we need it? In ExtJS we could say that it is always 80, meaning that the dynamic display mode doesn’t work in ExtJS.

Thursday, August 4, 2022

Thu, 04 Aug 2022 00:00:00 +0000

Thursday, August 4, 2022¶

aioredis.errors.ReplyError: ERR unknown command ‘BZPOPMIN’

I had to downgrade channels-redis to 2.4.* because I am still running redis-server 4.

Review operation when use_multiprocessing

Mon, 01 Aug 2022 00:00:00 +0000

Review operation when use_multiprocessing¶

Monday, August 1, 2022

The :ticket:`` issue reveals that we need to change a few fundamental things regarding how Lino runs in a multiprocess environment (on a production server). The linod command will become useless. When use_multiprocessing is True, every Lino process should spawn a server process and then connect itself to it.

Saturday, July 30, 2022

Sat, 30 Jul 2022 00:00:00 +0000

Saturday, July 30, 2022¶

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

This message in the JS console usually means that the client made an AJAX request expecting an answer in JSON, but received a response in HTML. This currently happens also when there was an internal server error (a traceback) while building the response.

The /var/log/nginx/access.log says:

90.191.153.133 - - [30/Jul/2022:05:22:34 +0200]
"GET /api/tickets/Tickets/4586?dm=grid&fmt=json&limit=15&lv=1645431581.0682056&query=&rp=weak-key-54&start=0&wt=d HTTP/1.1"
502 173 "https://jane.mylino.net/"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"

The /var/log/nginx/error.log says:

2022/07/30 05:22:34 [error] 3273#3273: *43784 upstream prematurely closed
connection while reading response header from upstream, client:
90.191.153.133, server: jane.mylino.net, request:
"GET /api/tickets/Tickets/4586?dm=grid&fmt=json&limit=15&lv=1645431581.0682056&query=&rp=weak-key-54&start=0&wt=d HTTP/1.1",
upstream: "uwsgi://unix:/usr/local/lino/lino_local/jane/nginx.sock:",
host: "jane.mylino.net",
referrer: "https://jane.mylino.net/"

The file /var/log/supervisor/jane-uwsgi-stderr---supervisor-caHb85.log says:

[pid: 2307|app: 0|req: 125/240] 90.191.153.133 () {50 vars in 1162 bytes} [Sat Jul 30 06:22:23 2022] GET /api/tickets/Tickets/4586?dm=grid&fmt=json&limit=15&lv=1645431581.0682056&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&pv&query=&rp=weak-key-1&start=0&wt=d => generated 7813 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 439 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 126/241] 90.191.153.133 () {50 vars in 935 bytes} [Sat Jul 30 06:22:24 2022] GET /values/tickets/Tickets/4586/working.SessionsByTicket => generated 49 bytes in 20 msecs (HTTP/1.1 200) 4 headers in 124 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 121/242] 90.191.153.133 () {50 vars in 931 bytes} [Sat Jul 30 06:22:24 2022] GET /values/tickets/Tickets/4586/comments.CommentsByRFC => generated 8044 bytes in 126 msecs (HTTP/1.1 200) 4 headers in 126 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 122/243] 90.191.153.133 () {50 vars in 918 bytes} [Sat Jul 30 06:22:27 2022] GET /api/main_html?fmt=json&lv=1645431581.0682056 => generated 3872 bytes in 176 msecs (HTTP/1.1 200) 4 headers in 126 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 127/245] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:27 2022] GET /dashboard/0?fmt=json&lv=1645431581.0682056 => generated 9096 bytes in 972 msecs (HTTP/1.1 200) 4 headers in 126 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 123/245] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:27 2022] GET /dashboard/1?fmt=json&lv=1645431581.0682056 => generated 10648 bytes in 796 msecs (HTTP/1.1 200) 4 headers in 127 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 124/246] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:28 2022] GET /dashboard/2?fmt=json&lv=1645431581.0682056 => generated 45242 bytes in 307 msecs (HTTP/1.1 200) 4 headers in 127 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 125/247] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:28 2022] GET /dashboard/4?fmt=json&lv=1645431581.0682056 => generated 44574 bytes in 254 msecs (HTTP/1.1 200) 4 headers in 127 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 128/248] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:28 2022] GET /dashboard/3?fmt=json&lv=1645431581.0682056 => generated 26846 bytes in 564 msecs (HTTP/1.1 200) 4 headers in 127 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 129/249] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:29 2022] GET /dashboard/6?fmt=json&lv=1645431581.0682056 => generated 31 bytes in 49 msecs (HTTP/1.1 200) 4 headers in 124 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 130/250] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:29 2022] GET /dashboard/7?fmt=json&lv=1645431581.0682056 => generated 4344 bytes in 99 msecs (HTTP/1.1 200) 4 headers in 126 bytes (1 switches on core 0)
[pid: 2307|app: 0|req: 131/252] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:29 2022] GET /dashboard/8?fmt=json&lv=1645431581.0682056 => generated 1002 bytes in 121 msecs (HTTP/1.1 200) 4 headers in 126 bytes (1 switches on core 0)
[pid: 4875|app: 0|req: 126/252] 90.191.153.133 () {50 vars in 914 bytes} [Sat Jul 30 06:22:29 2022] GET /dashboard/5?fmt=json&lv=1645431581.0682056 => generated 20501 bytes in 274 msecs (HTTP/1.1 200) 4 headers in 127 bytes (1 switches on core 0)
DAMN ! worker 1 (pid: 2307) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 5443)

The file /var/log/supervisor/linod-jane-stderr---supervisor-SToYrb.log contains a traceback:

Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/management/commands/linod.py", line 49, in handle
    schedule.run_pending()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/schedule/__init__.py", line 780, in run_pending
    default_scheduler.run_pending()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/schedule/__init__.py", line 100, in run_pending
    self._run_job(job)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/schedule/__init__.py", line 172, in _run_job
    ret = job.run()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/schedule/__init__.py", line 661, in run
    ret = self.job_func()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/models.py", line 241, in update_all_repos
    Repository.update_all_repos.run_from_code(rt.models.github.Repositories.request())
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/actions.py", line 149, in run_from_code
    super(Update_all_repos, self).run_from_code(ar, *args, **kw)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/actions.py", line 81, in run_from_code
    for commit in self.get_commits(**kw):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/actions.py", line 127, in get_commits
    for commit in super(Import_new_commits, self).get_commits(**kw):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/actions.py", line 64, in get_commits
    commit = rt.models.github.Commit.from_api(c, kw.get('repo'))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_xl/lib/github/models.py", line 207, in from_api
    c = Commit.objects.get(sha=d['sha'])
TypeError: string indices must be integers

The timestamp of this file is shown as “Jul 30 05:05” by ls -l. When I say touch tmp ; ls -l tmp, then it shows one hour earlier than my desktop clock. Because the server is in CEST time zone. This traceback is caused by the following code (in lino_xl.lib.github):

@dd.schedule_often(3600)
def update_all_repos():
    Repository.update_all_repos.run_from_code(rt.models.github.Repositories.request())

I am surprised that the lino.log contains nothing about the event. It looks as if it contains only the logging of the linod process. Yes, that’s because the uwsgi and the lino processes are two concurrent processes trying to log to a single file. See here for example. When I run reload_services.sh, the lino.log contains:

202207-30 07:09:17 INFO [kernel 4875 139817209964416] : Done uwsgi (PID 4875)
202207-30 07:09:17 INFO [kernel 5443 139817209964416] : Done uwsgi (PID 5443)
202207-30 07:09:27 INFO [kernel 6001 140241389811520] : Started /usr/local/lino/lino_local/jane/manage.py linod (using lino_local.jane.settings) --> PID 6001
202207-30 07:09:29 INFO [kernel 6000 140542291392384] : Started uwsgi (using lino_local.jane.settings) --> PID 6000
202207-30 07:09:29 INFO [linod 6001 140241389811520] : 8 scheduled jobs:
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [1] Every 300 seconds do event_notification_scheduler() (last run: [never], next run: 2022-07-30 07:14:22)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [2] Every 1 day at 20:00:00 do checksummaries() (last run: [never], next run: 2022-07-30 20:00:00)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [3] Every 1 day at 20:00:00 do checkdata() (last run: [never], next run: 2022-07-30 20:00:00)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [4] Every 10 seconds do send_pending_emails_often() (last run: [never], next run: 2022-07-30 07:09:34)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [5] Every 1 day at 20:00:00 do send_pending_emails_daily() (last run: [never], next run: 2022-07-30 20:00:00)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [6] Every 1 day at 20:00:00 do clear_seen_messages() (last run: [never], next run: 2022-07-30 20:00:00)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [7] Every 3600 seconds do update_all_repos() (last run: [never], next run: 2022-07-30 08:09:25)
202207-30 07:09:29 INFO [linod 6001 140241389811520] : [8] Every 10 seconds do get_new_mail() (last run: [never], next run: 2022-07-30 07:09:37)

About sales notes

Sat, 18 Jun 2022 00:00:00 +0000

About sales notes¶

Saturday, June 18, 2022

Sales notes

First use case: a sales note is being paid cash immediately. We don’t know anything about the customer.

Account	Debit	Credit	Partner
701000 sales of goods		100 €
570000 cash	100 €

Second case: client pays with credit card or online payment. Here we know the partner (at least his card number), and the API communication with the bank has been done during register() (otherwise the voucher would not have been registered).

Account	Debit	Credit	Partner
701000 sales of goods		100 €
400000 clients receivable	100 €		somebody
400000 clients receivable		100 €	somebody
570100 credit card payments	100 €

Third case. The payment fails. We say “Okay, take your goods and we trust you to come back tomorrow and pay in cash”. Just set payment method to “invoice” or “later”. Lino should book this:

Account	Debit	Credit	Partner
701000 sales of goods		100 €
400000 clients receivable	100 €		somebody

When the customer returns later the same day and pays cash, we register a payment without sale. This is just a sales note without any product. Lino should automatically show in the balance_before virtual field. And since there are no items, the balance_to_pay will be the same.

Account	Debit	Credit	Partner
400000 clients receivable		100 €	somebody
570000 cash payment	100 €

If the client forgets to return and to pay, we will automatically issue an invoice.

Next use case: we have an agreement with the client that they receive a monthly invoice for all sales. Two sub-cases: (a) client pays on each delivery or (b) client pays only when invoice comes.

In both cases we have already the money (or a proof that promises that we will get money), but we still need to write an invoice. The payment should not clear with the sales note but with the invoice.

When (a), we do have the cash without invoice. Invoices to issue

Sunday, May 29, 2022

Sun, 29 May 2022 00:00:00 +0000

Sunday, May 29, 2022¶

For example use_dirhtml. This option is needed by inv bd and when rendering the footer of a html page. So both rstgen and atelier need it. But we want to store (and maintain) it in only one place.

Sunday, May 22, 2022

Sun, 22 May 2022 00:00:00 +0000

Sunday, May 22, 2022¶

About site features¶

we leave the translatable text in the code. Because that makes sense. A one-sentence short desciption.
we add a convention: when you develop a feature, then you are responsible for also providing a ref name to that feature in the users guide.
Example: the third_party_authentication feature is bla blabla

The site features are needed in the About window only so that an end user, when reporting some problem, can can easily report which features they have activated on their site

A list of all available features is needed for a site operator who wants to check which features they want and which the don’t want. This list can be either in the local help pages or in the application’s user guide (or both). Anyway these are static Sphinx docs. The one-line descriptions of each feature are needed when generating such an overview page.

TODO (Sharif):

The About window should just show a one-paragraph list of the enabled site features. Only the internal names, not even clickable.

TODO (Luc):

The makedocs command can generate what is currently being shown in the about window. Maybe as a list instead of a table.

Examples: notify.remove_after is a plugin setting.

Presenting Lino to my future boss

Tue, 17 May 2022 00:00:00 +0000

Presenting Lino to my future boss¶

Tuesday, May 17, 2022

Tonight I plan to attend once more to HackerX, an event where developers can enter for free because they are actually what is being sold to the customers. The customers are companies in search of new human resources. I probably don’t fit into their standard product pattern but they did let me in. Nice.

Here is my self-presentation:

Developer skills: Linux, Python, Django, Sphinx, Git, React. Long-term experience in accounting, team management, sales agreements, website authoring.

I am patiently looking forward to meeting my future boss and employer.

With me you get a functioning community of 3 workers and 20 customers. I am the author and maintainer of the Lino framework. I am CEO and co-owner of Rumma & Ko. I am employing other developers since 2015 (see my team page). I have been working on this vision my whole life. I am not going to stop working on it.

But Rumma & Ko is willing to subdue all this to someone who does what I am less good at: bringing this community to the next level. Of course our customers and employees will have their voice in any agreement.

Lino Polly, yet another sailboat

Mon, 16 May 2022 00:00:00 +0000

Lino Polly, yet another sailboat¶

Monday, May 16, 2022

Today I answered to an online poll made by a workgroup “Empirische Politikwissenschaft” led by Dr. Jan Karem Höhne at the Political Sciences faculty of the Duisburg-Essen university.

While answering it, I thought “Hey, this is something we could easily do with Lino Polly”. I remembered Lino Polly and had a look at it. I actually like it very much. But:

The documentation about Lino Polly is incomplete and chaotic.
Currently the Polly online demo doesn’t even work
getlino : the Lino installer doesn’t even know Polly.
The project should get its own repository and documentation.
The name “Polly” is suboptimal, I’d rather suggest Lino Chiedo (“ti chiedo” in Italian means “I ask you”)
There is still a disturbing bug under the React front end: In Responses of Participant feedback » Robin Rood’s response to Participant feedback when you select an answer, the screen doesn’t update automatically.

And then: does it make sense to work on Polly? I started to wonder: Who develops such software? How much does the university pay for it? Are they satisfied with the solution? Who pays the university? Who pays Dr. Höhne for doing this?

The mentioned online poll was implemented on Unipark, a project that is operated by Tivian XI GmbH, who engages over 300 employees in 19 countries and claims to be European market leader for Enterprise Feedback Management solutions. Unipark also says that it is “QuestBack’s academic program”. QuestBack is another private corporation.

One thing is clear: Lino Polly is once again a sailing boat challenging ocean liners.

Thursday, May 12, 2022

Thu, 12 May 2022 00:00:00 +0000

Thursday, May 12, 2022¶

Optimizations for points of sale¶

Sharif and I had a brainstorming session about new optimizations to do in Lino Così for a friend who has a small shop.

Create a new data table “Sales note”. Move most of the fields in the General tab to the More tab. Only the field “payment method” and one more field “received cash” and a virtual field “cash to return”.
Possibility to quickly start a delivery note, even without specifying a customer: Configure a default partner “Miscellaneous” (to be created by the demo fixture) only when invoicing.delivery_notes_demo is True. In SDN journal, set Journal.partner to this partner.
Add a quick link [start new sale] that creates an Invoice record and then shows the detail.
Every POS (point of sale) will have its own journal of delivery notes.
Add a new field “received cash” and a virtual field “cash to return”. Used when registering cash amounts.
Add a new toolbar button for registering.
About printing: we don’t want to open a pdf document here. Each printer would be declared as a network printer.
Solution for shop that is operated via a mobile device.

Don’t start these alone:

New action to register when cash money was taken from the cash box in order to buy a coffee or to pay a provider (potentially clearing a purchase invoice)
Quick link “Daily report” prints cash in the morning, details of sales, total amount of cash in the evening. Maybe this will be a new voucher type “CashReport”, with a field “How much money has been taken out of the cash box to “

How to update the total amount after each row? New concept of “master data” : when I submit some row of an invoice, Lino could return additional data in its response, which would be some changed values in the master instance.

On the InvoiceItem model we would have the following application code:

master_data_fields = "total_incl total_base"

def get_master_data(self, ar):
  self.voucher.compute_totals()
  return super().get_master_data(ar)

The default implementation in core.model.Model would be:

def get_master_data(self, ar):
  rv = {}
  for mdf in self.master_data_fields:
    rv[mdf]  = getattr(ar.master_instance, mdf)
  return rv

When a model has master data fields, any submit to a ui change in a row of this model will also return an additional item “master_data”:

master_data: {"total_incl": "...", "total_base": ...}

Code changes¶

I moved the table definitions in lino_xl.lib.trading to a separate ui.py module. Because things were getting messy in a single models.py module.

I replaced the print_items_table class attribute on the VatProductInvoice model by a new plugin setting sales.print_items_table. This feature is used in Lino Voga.

I thought that instead of specifying invoicing.voucher_model the app dev would now have to specify invoicing.default_voucher_view, but that idea caused complexity when resolving the name.

I fixed a subtle double bug that became visible because now we a demo customer in Bangladesh, and because the sample IBANS in lino_xl.lib.sepa.fixtures.sample_ibans don’t have any IBAN in Bangladesh. In that situation, Lino tried to create a sepa.ACcount row with an empty iban field. And in that case sepa.Account.__str__() returned None. After fixing this bug, I had another problem during pm prep ({‘iban’: [‘This field cannot be null.’]}) because Lino insisted on adding a demo SEPA account for every partner who has a country.

For some reason I didn’t investigate further, the sample IBANs in lino_book.projects.cosi3 have changed, causing doctest to fail on docs/plugins/eevat.rst.

Friday, May 6, 2022

Fri, 06 May 2022 00:00:00 +0000

Friday, May 6, 2022¶

Yes, TIM was right. Lino will have something equivalent to the VNA.DBF in TIM.

I plan to call it Transfer. This will be a concrete model. It is a bit similar to ledger.Movement in that these transfers are generated when a voucher gets registered, and deleted when the voucher gets de-registered. And InvoiceGenerator will have a get_wanted_transfers(), which is similar to the get_wanted_movements() and will be defined in LedgerRegistrable as well. This LedgerRegistrable mixin should maybe move to another place because it might be useful even in a Lino that has no ledger.

An “invoiceable” represents a future row of a trade voucher.

The invoicing plan will no longer need items because that information will be just the (filtered) list of invoiceables. The only interactive element of the invoicing plan’s items were the checkboxes to select individual items. But they aren’t really needed because the user can simply delete the rows of a generated invoice before closing it.

The get_generators_for_plan() method will be replaced by something else.

The new `trading` plugin

Tue, 03 May 2022 00:00:00 +0000

The new `trading` plugin¶

Tuesday, May 3, 2022. Seems that Lino is going to have a new plugin, trading.

Use case 1¶

A small shop who sells electronic devices and related services. Some customers don’t need to pay immediately, they receive a monthly invoice. In such cases the shop owner writes a delivery note.

delivery note¶: A voucher with a series of rows, each of which means that a given quantity of a given “product” has been “delivered”.

The same customer can come multiple times, each visit expressed by a delivery note. At the end of the month we want Lino to generate all invoices. It can happen that a customer asks, at some arbitrary moment “Please make my invoice now”. And of course these deliveries should not get invoiced again at the end of the month. Or it can happen that the site operator decides to not write an invoice to a given customer and to wait another month before invoicing the delivered items.

When looking at any given product, we want to see when it has been (1) delivered and (2) invoiced to a customer. Each “open” delivery note means that a corresponding invoice is waiting to be emitted.

Use case 2¶

The shop also offers the possibility to their customers to order some device. That’s another operation, and the workflow becomes more complex:

order form¶: A voucher with a series of rows, each of which means that the business partner “orders” a given quantity of a given “product”.

An order form can include a deposit or pre-payment, which may be partly.

When looking at any given product, we want to see who ordered it. Each “open” order means that a corresponding delivery note is waiting to be emitted.

Use case 3¶

Like case 2, but we add a feature that Lino also helps with filling our own orders towards our providers. At any moment we could say “Let’s write an order form four our provider X”. This order form may cover the orders of multiple customers. It adds “incoming delivery notes” and a feature to notify customers that their order has arrived.

It can get even more complex. For example when you have serial numbers and warranty services. In that case you need to assign each incoming item to the customer to whom it is to be delivered. Or the customer comes back one year after a purchase and has a warranty issue. In that case you want to track where you bought the device, and you will need at least one new journal of vouchers “warranty issues”. Or your activity is not about electronic devices but about small objects, which you buy in big quantities and sell them after having them packed into smaller parcels.

Use case 4¶

And a last use case, e.g. in Lino Amici where you don’t speak about money. But you have a collection of books and other things, which you happen to lend to some friend. And you might want to keep track of these. It’s not about money, but about asking them back after some time.

The plugin¶

The trading plugin would be used in all these cases.

It would define a choicelist TradingStates, which describes all states of the workflow. In use case 1 this choicelist would have only one item, “delivered” or “to be invoiced”. In use case 2 another state “ordered” or “to be delivered”, etc.

And then a model TradingRule (or Transition), which describes the possible ways of getting from one state to another state. It will have the following fields:

from_state
to_state
journal

In use case 1 you would have a journal SDN (“Sales Delivery Notes”) and two rows in your TradingRule table:

journal	from_state	to_state
SDN	None	delivered
SLS	delivered	None

But that’s not all. Don’t we also want a separate model for storing the individual items that are in one of these states? Each row of a delivery note would generate a row in that table. In TIM this table was called VNA. And e.g. when you de-register a delivery note that has already been invoiced, TIM had a special warning in that case, it said something like “This will mess up your invoicing workflow because the operations of this delivery note have already been invoiced”. In Lino such a table seems not needed because we have the invoiceable field, a GFK on the invoicing.item_model.

I also need to consider the other “philosophies” of generating invoices, as used in Lino Voga or Lino Presto where they have subscription-style orders, which are a completely different way of invoicing. They are related, though. For example when a customer orders 10 times a same product, you manage to order 6 of them from your provider, deliver them to the customer, maybe write an invoice. What about the four remaining ordered items? The customer may decide to cancel them. Or their price may change.

Actually we don’t need another plugin. We just add TradingStates and TradingRule to the invoicing plugin.

Friday, April 29, 2022

Fri, 29 Apr 2022 00:00:00 +0000

Friday, April 29, 2022¶

A started a brainstorming page for a new plugin leaders (which I removed already one day later because that was a bit too quick).

Sunday, April 24, 2022

Sun, 24 Apr 2022 00:00:00 +0000

Sunday, April 24, 2022¶

I wanted to quickly document the Table views with dynamic columns feature. Took me just a few hours because I also reviewed the events : Publishing an event calendar page. And the actual documentation is still rather poor.

Thursday, March 17, 2022

Thu, 17 Mar 2022 00:00:00 +0000

Thursday, March 17, 2022¶

Brainstorming with Sharif for Lino Medico.

A patient comes to a first meeting. This is just a calendar entry. Bangladesh does not have the concept of family doctors (perearst, Hausarzt). During the meeting, the doctor may decide with the patient to start one or several treatments or therapies. A therapy is usually a course with one participant.

For example:

Doctor: “I write you a prescription. Try it. And if your problem doesn’t go away, come back.” Doctor creates a course from the calendar entry. And this entry should automatically be linked to that course. The course mainly mentions the original problem. When the patient goes home, they should get their “case” number.

::
courses.Course._meta.verbose_name = _(“Case”) # _(“Dossier”) _(“File”)

Lino should have a procedure for detecting cases that have timed out. For example after two months we can assume that the problem is fixed if the patient didn’t come back. Lino should automatically set Course.state to “timed_out” for such cases.

Follow up: Patient contacts the reception: “I need another consultation”. Reception clerk finds the patient contact and sees: yes, you have an open case. Creates another calendar entry for that case.
Examples of course lines: “general consultation” and “urgent consultation”
The generalist is a doctor who receives patients for such first-time consultations. A specialist receives cases when some generalist decides so. Every case is like a ticket in Noi: the Course.user is the first doctor, Course.teacher is the “currently responsible” doctor. The generalist can assign a case to a specialist.
The generalist can enrol the patient into a group therapy. “Every Monday they are doing gymnastics”.
The generalist can create several cases at the same time when the patient has two problems at once.
Example: emergency patient. write an order for x-ray. another doctor makes the diagnose: leg is broken. writes an order to plaster.

I will write a order for X-Ray and they will assign some doctor to continue with you.
Or “You need some therapy, I will put you to the waiting list.” This will

Monday, March 14, 2022

Mon, 14 Mar 2022 00:00:00 +0000

Monday, March 14, 2022¶

The GitLab CI for laudate is failing with message rST localisation for language 'et' not found.

Where does it come from:

$ grep -R "rST localisation" /home/luc/virtualenvs/py38/lib/python3.8/site-packages/
.../site-packages/docutils/parsers/rst/languages/__init__.py:    warn_msg = 'rST localisation for language "%s" not found.'

Yes, I remember that some time ago I fixed this locally by add a language file “et” to my local docutils installation as explained in http://docutils.sf.net/docs/howto/i18n.html

But I am not going to do this on GitLab. How to add a language to docutils?

The optimal way would be to contribute. But that would take a lot of time time, and I want a quick fix.

I see that docutils.languages.LanguageImporter has an attribute packages, which is a tuple of prefixes:

packages = ('docutils.languages.', '')

So I might add this et.py to my project directory instead of adding it to the docutils/languages directory. But if I manage to add 'rstgen.sphinxconf' to that tuple, I might get my quick workaround.

I made two PyPI releases in this direction before I realized that I would have to do the same patch for docutils.parsers.rst.languages.RstLanguageImporter.

Oh no. Enough hacking around! I will rather start contributing to docutils!

Get started:

$ cd ~/repositories
$ svn checkout https://svn.code.sf.net/p/docutils/code/trunk docutils-code
$ pip install -e docutils-code/docutils

First problem: The pip install -e says:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sphinx 4.4.0 requires docutils<0.18,>=0.14, but you have docutils 0.19b0.dev0 which is incompatible.
sphinx-rtd-theme 1.0.0 requires docutils<0.18, but you have docutils 0.19b0.dev0 which is incompatible.

A stupid dependency bug: sphinx and sphinx-rtd-theme fail to work with the development version of docutils. OMG! So I should ideally now contribute to these as well! First question is: why do they require “<0.18”? Shouldn’t they simply remove this requirement? Okay I can work around this by saying:

pip install -e --no-warn-conflicts .

But I also submitted a ticket for it: https://github.com/sphinx-doc/sphinx/issues/10264

To which Takeshi KOMIYA answered “The reason why we restrict the version of docutils is the combination not guaranteed yet. Please wait for the update (refs: #9777)”.

Okay, I see. Nothing is easy!

Sunday, March 13, 2022

Sun, 13 Mar 2022 00:00:00 +0000

Sunday, March 13, 2022¶

Just to share an emotional story:

Tuesday, February 8, 2022

Tue, 08 Feb 2022 00:00:00 +0000

Tuesday, February 8, 2022¶

I think I will reactivate #1516 in favour of #4427.

#3004 Replace “Reference tickets” by “Starred tickets”?
#4427 A “nicknames” plugin
#1516 Add a “parent” field to ticket
#2384 Kanban for Jane

#4427 has at least two disadvantages: Adds complexity. Can be replaced by a list of recently active tickets. Requires every user to maintain their nicknames.

#1516 says “I seem to remember that I had this in the ticketing system in TIM, and that I deliberately decided to drop this, because it caused problems for reporting. Until now tickets are designed to be “atomic” units for project management.” but I doubt whether problems for reporting still holds. Yes, reporting becomes more complex because the client will want to see a summary of all their top-level tickets, together with the sum of invested time (including time on child tickets), but that’s not a problem in an SQL database.

One advantage of #1516 would be that top-level tickets would be the “goals” of a Kanban board and that we can easily convert any ticket into a top-level ticket (or the opposite: convert a top-level ticket to a child of another ticket)

Thursday, January 20, 2022

Thu, 20 Jan 2022 00:00:00 +0000

Thursday, January 20, 2022¶

Migrating a Jekyll site from GitHub to GitLab¶

I worked on #4345 the https://gitlab.com/alvatal/www project (which I am migrating from https://github.com/alvatal/alvatal.github.io )

After having added my .gitlab-ci.yml file, the content gets rendered correctly at https://alvatal.gitlab.io/www/ but the news page is still empty.

Edmund helped me to discover the planet.yml file. But for some reason, this file was obviously being used on GitHub somehow “automatically”. I need to add something to my .gitlab-ci.yml file.

While searching for an answer, I stumbled into

How to setup the link that our readers can use to subscribe to our (aggregated) news feed: simple and more sophisticated.

The instructions on bluesabre.org look interesting, but it seems that Lauri had found a simpler solution.

The instructions on https://github.com/feedreader/planet.rb look even closer. But still they don’t seem to be the final solution.

Or here: https://github.com/feedreader/pluto/tree/master/pluto

I no longer believe that GitHub has this as a magic default feature. Next step is to look with Edmund in the settings of the GitHub project. Maybe there are some customization in the GitHub settings that are not stored as files in the repo.

Monday, December 27, 2021

Mon, 27 Dec 2021 00:00:00 +0000

Monday, December 27, 2021¶

How would a lino.ini file look like?

[features]
    use_third_party_authentication = true

[config]
    excerpts.responsible_user = 'jean'
    notify.use_push_api = true

Thursday, December 16, 2021

Thu, 16 Dec 2021 00:00:00 +0000

Thursday, December 16, 2021¶

pyvat.exceptions.ServerError: ServerError: MS_MAX_CONCURRENT_REQ¶

I fixed a sporadic error during cmd prep in lino_book.projects.cosi2, which occurred only when the pyvat server wasn’t available:

pyvat.check_vat_number('BE0650238114', 'BE') failed:
Traceback (most recent call last):
  File "~/work/xl/lino_xl/lib/vat/choicelists.py", line 736, in validate
    result = pyvat.check_vat_number(self.vn, self.country_code)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/pyvat/__init__.py", line 211, in check_vat_number
    return VAT_REGISTRIES[country_code].check_vat_number(vat_number,
  File "~/virtualenvs/py38/lib/python3.8/site-packages/pyvat/registries.py", line 133, in check_vat_number
    raise ServerError(fault_code)
pyvat.exceptions.ServerError: ServerError: MS_MAX_CONCURRENT_REQ

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 7, in <module>
    execute_from_command_line(sys.argv)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/__init__.py", line 425, in execute_from_command_line
    utility.execute()
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/base.py", line 373, in run_from_argv
    self.execute(*args, **cmd_options)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "~/work/lino/lino/management/commands/prep.py", line 36, in handle
    super(Command, self).handle(**options)
  File "~/work/lino/lino/management/commands/initdb.py", line 192, in handle
    call_command('loaddata', *fixtures, **options)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 78, in handle
    self.loaddata(fixture_labels)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 138, in loaddata
    self.load_label(fixture_label)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 214, in load_label
    for obj in objects:
  File "~/work/lino/lino/utils/dpy.py", line 409, in deserialize
    for o in self.deserialize_module(module, **options):
  File "~/work/lino/lino/utils/dpy.py", line 423, in deserialize_module
    for obj in objects():
  File "~/work/lino/lino/modlib/checkdata/fixtures/checkdata.py", line 13, in objects
    call_command('checkdata', fix=True)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "~/virtualenvs/py38/lib/python3.8/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "~/work/lino/lino/modlib/checkdata/management/commands/checkdata.py", line 64, in handle
    check_data(args=args, fix=options['fix'], prune=options['prune'])
  File "~/work/lino/lino/modlib/checkdata/models.py", line 299, in check_data
    todo, done = chk.update_problems(obj, False, fix)
  File "~/work/lino/lino/modlib/checkdata/choicelists.py", line 96, in update_problems
    for fixable, msg in self.get_checkdata_problems(obj, fix):
  File "~/work/xl/lino_xl/lib/vat/choicelists.py", line 924, in get_checkdata_problems
    obj.vid_manager.validate(online=True)
  File "~/work/xl/lino_xl/lib/vat/choicelists.py", line 753, in validate
    dd.logger.info("  " + ("\n  ".join(result.log_lines)))
AttributeError: Problem installing fixture '~/work/lino/lino/modlib/checkdata/fixtures/checkdata.py': 'NoneType' object has no attribute 'log_lines'

A Lino for the Synodal Church?¶

I had a crazy idea: I will write a demo project sct (for Synodal Church Teachings), which will be a prototype for a Lino site to be used by the Synodal Church.

It will probably be a Lino Noi (but after having implemented #2717). It will also use a new plugin lino_xl.lib.glossary with the following models:

class lino_xl.lib.glossary.Term¶

designation¶

The designation of the term being described.

This is a translatable field.

description¶

The generally accepted description of this term.

This is a translatable field.

class lino_xl.lib.glossary.UsageType¶

designation¶

I discovered that e.g. the documentation of the languages setting was quite broken. We now have a reference about the individual settings in Site settings reference for maintainers.

I fixed two warnings during inv bd in book after the upgrade to Django 4.

Friday, December 3, 2021

Fri, 03 Dec 2021 00:00:00 +0000

Friday, December 3, 2021¶

The following traceback is for Sharif:

Traceback (most recent call last):
  File "./wsgi.py", line 7, in <module>
    application = get_wsgi_application()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/comments/models.py", line 18, in <module>
    from lino.modlib.search.mixins import ElasticSearchable
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/search/mixins.py", line 26, in <module>
    from elasticsearch_django.models import SearchDocumentMixin
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/elasticsearch_django/models.py", line 33, in <module>
    UPDATE_STRATEGY = get_setting("update_strategy", UPDATE_STRATEGY_FULL)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/elasticsearch_django/settings.py", line 39, in get_setting
    return get_settings().get(key, default[0])
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/elasticsearch_django/settings.py", line 33, in get_settings
    return settings.SEARCH_SETTINGS["settings"]
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/conf/__init__.py", line 83, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'SEARCH_SETTINGS'

I ran pull.sh on LF. First problem:

error: Your local changes to the following files would be overwritten by merge:
      lino/modlib/help/management/commands/makehelp.py
Please commit your changes or stash them before you merge.
Aborting

(Solution: go lino and git st, git checkout .)

Second problem:

/tmp/tmp0s_puzr0/frdocs/beid.BeIdCardTypes.rst:15: WARNING: undefined label: ug.plugins.beid
/tmp/tmp0s_puzr0/frdocs/beid.ResidenceTypes.rst:16: WARNING: undefined label: ug.plugins.beid
/tmp/tmp0s_puzr0/frdocs/notes.AllNotes.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.EventTypes.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.MyNotes.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NoteTypes.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.Notes.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NotesByCompany.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NotesByEventType.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NotesByOwner.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NotesByPerson.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.NotesByType.rst:17: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/notes.SpecialTypes.rst:15: WARNING: undefined label: ug.plugins.notes
/tmp/tmp0s_puzr0/frdocs/outbox.Attachments.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.AttachmentsByController.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.AttachmentsByMail.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.Mails.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.MailsByController.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.MailsByUser.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.MyOutbox.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.RecipientTypes.rst:16: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.Recipients.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.RecipientsByMail.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/outbox.SentByPartner.rst:17: WARNING: undefined label: ug.plugins.outbox
/tmp/tmp0s_puzr0/frdocs/rooms.BookingStates.rst:15: WARNING: undefined label: ug.plugins.rooms
/tmp/tmp0s_puzr0/frdocs/rooms.Bookings.rst:17: WARNING: undefined label: ug.plugins.rooms
/tmp/tmp0s_puzr0/frdocs/rooms.BookingsByCompany.rst:17: WARNING: undefined label: ug.plugins.rooms
generating indices... done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in French (code: fr)... done
dumping object inventory... done
build succeeded, 27 warnings.

The HTML pages are in ../../../usr/local/lino/lino_local/voga1e/media/cache/help/fr.
5 lino*.js files have been built in 12.596930027008057 seconds.

Solution: add dummy headings for the missing refs to Plugin reference for end users.

Thursday, December 2, 2021

Thu, 02 Dec 2021 00:00:00 +0000

Thursday, December 2, 2021¶

I started to review the docs about ElasticSearch in book and hg. Not finished because I was distracted by another issue: The latest Sphinx versions are more strict regarding duplicate definitions, which is good, but I had a few of them and needed to dive into the docs in order to decide how to resolve them.

I can now refer to use_elasticsearch and search.elasticsearch_url, which refers to the new “Local settings reference” page (Site settings reference for maintainers) in the Lino Hosting Guide.

I also started to boldly edit Sharif’s first drafts in search : Add search functionality.

Saturday, November 27, 2021

Sat, 27 Nov 2021 00:00:00 +0000

Saturday, November 27, 2021¶

rST localisation for language “et” not found

Fri, 19 Nov 2021 00:00:00 +0000

rST localisation for language “et” not found¶

Friday, November 19, 2021

Today I upgraded docutils after some time. And afterwards I had the following warnings when building a website in Estonian:

/mysite/some/arbitrary/file.rst:: WARNING: rST localisation for language "et" not found.

This was because docutils now supports i18n. To fix the problem at least for me, I said cp en.py et.py in two source code directories below docutils:

…/lib/python3.8/site-packages/docutils/languages
…/lib/python3.8/site-packages/docutils/parsers/rst/languages

Together on our way

Thu, 28 Oct 2021 00:00:00 +0000

Together on our way¶

Thursday, October 28, 2021

Congratulations to Hannes and Sharif: They managed to collaborate completely without my help and get Jane back after an accident during Upgrade. This is an important little success in the history of Lino: a application developer and a server administrator collaborating to maintain a Lino site. Lino is getting bigger than me.

This is just one of many small signs I see almost every day and which tell me: we are on the right way. Lino is in a kind of cocoon state. This blog is getting less used as I delegate coding to Sharif whenever possible and as Jane becomes our main communication channel. We are together on our way as a team. We are ready for the next Lino project without being in a hurry. We use spare time for doing creative research work.

Saturday, September 25, 2021

Sat, 25 Sep 2021 00:00:00 +0000

Saturday, September 25, 2021¶

Lino Corsa¶

I started the Lino Corsa project, an application for managing races, i.e. sport events where people compete against each other in order to see who’s the best.

This was my first usage of the getlino startproject.

Go to your projects root directory:

$ cd ~/projects

Run getlino startproject:

$ getlino startproject -c corsa

Install the new Python package:

$ pip install -e corsa/
Obtaining file:///home/luc/work/corsa
...
Installing collected packages: lino-corsa
  Running setup.py develop for lino-corsa
Successfully installed lino-corsa-0.2.0

Add the project to your ~/.atelier/config.py file.

Decide whether you want to copy functionality from existing projects.

For example, we started Lino Corsa with a copy the courses plugin from Lino Voga:

$ cp -a ~/projects/voga/lino_voga/lib/courses lino_corsa/lib

And then manually edited the copied source code.

Edit the user_types.py file of your new project.

In our example we add the following to our user_types.py file:

from lino_xl.lib.courses.roles import CoursesUser
from lino.modlib.office.roles import OfficeUser

class SiteUser(SiteUser, CoursesUser, OfficeUser):
    pass

class SiteStaff(SiteStaff, CoursesUser, OfficeUser):
    pass

class SiteAdmin(SiteAdmin, CoursesUser, OfficeUser):
    pass

Summary: we import the roles of the plugins, then tell Lino which user type inherits from which role.

Go to the demo project:

$ cd ~/projects/corsa/lino_corsa/projects/corsa1
$ pm prep

If you changed anything, you are likely to get error messages here.

Wednesday, September 8, 2021

Wed, 08 Sep 2021 00:00:00 +0000

Wednesday, September 8, 2021¶

202109-08 12:40:15 ERROR [log 29322 140314402891648] : Internal Server Error: /api/comments/RecentComments
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_react/react/views.py", line 169, in get
    for row in ar.sliced_data_iterator]
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_react/react/views.py", line 169, in <listcomp>
    for row in ar.sliced_data_iterator]
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_react/react/views.py", line 166, in serialize
    row, fields=fields, card_title=ar.get_card_title(row))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/requests.py", line 1118, in get_card_title
    return self.actor.get_card_title(self, elem)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/comments/ui.py", line 209, in get_card_title
    return ar.actor.get_comment_header(obj, ar)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/comments/ui.py", line 137, in get_comment_header
    if sar.has_permission():
AttributeError: 'TableRequest' object has no attribute 'has_permission'

Friday, September 3, 2021

Fri, 03 Sep 2021 00:00:00 +0000

Friday, September 3, 2021¶

I am basically done with #2406. One “little detail” is still missing: the result ;-) The button to export the intra-community statement works and produces a validated XML file, but the numbers are still wrong. I need to understand what is wrong with the SQL statement generated by lino_xl.lib.vat.VatDeclaration.intracom_statement_iterator(). Here is an indented example without the quote chars for readability:

SELECT DISTINCT
  contacts_partner.id, contacts_partner.email,
  contacts_partner.language, contacts_partner.url, contacts_partner.phone,
  contacts_partner.gsm, contacts_partner.fax, contacts_partner.country_id,
  contacts_partner.city_id, contacts_partner.zip_code, contacts_partner.region_id,
  contacts_partner.addr1, contacts_partner.street_prefix, contacts_partner.street,
  contacts_partner.street_no, contacts_partner.street_box, contacts_partner.addr2,
  contacts_partner.vat_regime, contacts_partner.vat_id, contacts_partner.prefix,
  contacts_partner.name, contacts_partner.remarks,
  contacts_partner.payment_term_id, contacts_partner.purchase_account_id,
  CAST(SUM(vat_vataccountinvoice.total_base) AS NUMERIC) AS vataccountinvoice_base,
  CAST(SUM(sales_vatproductinvoice.total_base) AS NUMERIC) AS vatproductinvoice_base
FROM contacts_partner
LEFT OUTER JOIN vat_vataccountinvoice ON (contacts_partner.id = vat_vataccountinvoice.partner_id)
LEFT OUTER JOIN sales_vatproductinvoice ON (contacts_partner.id = sales_vatproductinvoice.partner_id)
LEFT OUTER JOIN vat_vataccountinvoice T4 ON (contacts_partner.id = T4.partner_id)
LEFT OUTER JOIN ledger_voucher ON (T4.voucher_ptr_id = ledger_voucher.id)
LEFT OUTER JOIN ledger_journal ON (ledger_voucher.journal_id = ledger_journal.id)
LEFT OUTER JOIN sales_vatproductinvoice T8 ON (contacts_partner.id = T8.partner_id)
LEFT OUTER JOIN ledger_voucher T9 ON (T8.voucher_ptr_id = T9.id)
LEFT OUTER JOIN ledger_journal T11 ON (T9.journal_id = T11.id)
GROUP BY contacts_partner.id,
  contacts_partner.email, contacts_partner.language, contacts_partner.url,
  contacts_partner.phone, contacts_partner.gsm, contacts_partner.fax,
  contacts_partner.country_id, contacts_partner.city_id,
  contacts_partner.zip_code, contacts_partner.region_id, contacts_partner.addr1,
  contacts_partner.street_prefix, contacts_partner.street,
  contacts_partner.street_no, contacts_partner.street_box, contacts_partner.addr2,
  contacts_partner.vat_regime, contacts_partner.vat_id, contacts_partner.prefix,
  contacts_partner.name, contacts_partner.remarks,
  contacts_partner.payment_term_id, contacts_partner.purchase_account_id,
  ledger_voucher.accounting_period_id, ledger_journal.preliminary,
  ledger_journal.trade_type, T4.vat_regime, T9.accounting_period_id,
  T11.preliminary, T11.trade_type, T8.vat_regime
  HAVING (
      (ledger_voucher.accounting_period_id = 2
      AND NOT ledger_journal.preliminary
      AND ledger_journal.trade_type = S
      AND T4.vat_regime IN (40, 30, 31)
      AND CAST(SUM(vat_vataccountinvoice.total_base) AS NUMERIC) >= 0)
    OR
      (T9.accounting_period_id = 2
      AND NOT T11.preliminary
      AND T11.trade_type = S
      AND T8.vat_regime IN (40, 30, 31)
      AND CAST(SUM(sales_vatproductinvoice.total_base) AS NUMERIC) >= 0))
  ORDER BY contacts_partner.vat_id ASC

Thursday, September 2, 2021

Thu, 02 Sep 2021 00:00:00 +0000

Thursday, September 2, 2021¶

Today I worked on #2406. The intra-community statement is the most important XML file to generate because this statement can contain quite much data. Lino will also be able to generate the VAT declaration, but that’s less urgent because end users more easily agree to manually enter a few numbers.

First thing was to move the XSD file for SEPA payment orders from etgen to lino_xl.lib.finan and to generalize it.

The last thing for today is to design the queryset to be used in the intra-community statement. Here is how far I got for now:

import lino

lino.startup('lino_book.projects.cosi1.settings.demo')
from django.utils import translation
from lino.api.doctest import *
from django.db.models import Q

from django.db.models import Sum
from lino_xl.lib.ledger.choicelists import TradeTypes
from lino_xl.lib.vat.choicelists import VatRegimes, VatAreas

intracom_regimes = set(
    [r for r in VatRegimes.get_list_items() if r.vat_area == VatAreas.eu])
ap = rt.models.ledger.AccountingPeriod.objects.get(ref="2014-02")
qs = rt.models.contacts.Partner.objects.all()
qs = qs.filter(vat_id__isnull=False)
# qs = qs.exclude(vat_id__startswith="BE")
qs = qs.order_by('vat_id')
qs = qs.annotate(vataccountinvoice_base=Sum(
    'vat_vataccountinvoice_set_by_partner__total_base'))
flt1 = Q(
    vat_vataccountinvoice_set_by_partner__journal__trade_type=TradeTypes.sales,
    vat_vataccountinvoice_set_by_partner__accounting_period=ap,
    vat_vataccountinvoice_set_by_partner__vat_regime__in=intracom_regimes,
    vataccountinvoice_base__gte=0)
if dd.is_installed("sales"):
    qs = qs.annotate(vatproductinvoice_base=Sum(
        'sales_vatproductinvoice_set_by_partner__total_base'))
    flt2 = Q(
        sales_vatproductinvoice_set_by_partner__journal__trade_type=TradeTypes.
        sales,
        sales_vatproductinvoice_set_by_partner__accounting_period=ap,
        sales_vatproductinvoice_set_by_partner__vat_regime__in=intracom_regimes,
        vatproductinvoice_base__gte=0)
    qs = qs.filter(flt1 | flt2)
else:
    qs = qs.filter(flt1)
qs = qs.distinct()

for p in qs:
    print(p, p.vat_id, p.vatproductinvoice_base)
    # print(p, p.vat_id, p.vatproductinvoice_base, p.vataccountinvoice_base)

Friday, August 27, 2021

Fri, 27 Aug 2021 00:00:00 +0000

Friday, August 27, 2021¶

I am working on compound ticket #4302 for Lino Così. This page is my progress summary.

WAITING:

Remove the “payment method” field. We don’t use it in Così. –> Not sure whether we want this. Think more. Maybe we can use the payment method also in a simple Così.

DONE:

Add phones plugin to Cosi
“Introductive text” should be printed before the table of items. And the field label is not translated to German.
Remove the VAT class “vehicles” in bevat and bevats
Show the VAT rate in the designations of VAT classes
intra-community sales invoice were not booked correctly in bevat
Enter and register a purchase invoice doesn’t work when the accounting period doesn’t exist. (Not reproducible. I guess that this was a misdiagnosis)
The __str__() of an unregistered invoice used to say “jnl.ref #id”, but that’s irritating. Note the hide_editable_number feature: We usually don’t want to see the number of a voucher in an editable state because that number may change. Now the __str__() of an unregistered invoice shows (Sales invoice #1) instead of (VKR #1), which was indeed dangerously similar to VKR 1/2021. And it’s true that not only the number might change but also the journal.
The icon of the toggle_state action is not rendered in ExtJS

TODO:

Adapt the payment date when voucher_date changes. Must wait for #4305.
Export the list of intracom customers (intra-community statement) to XML (the well-known #2406).
Write a data checker to report duplicate or missing invoice numbers
When hitting Insert in a grid view of invoice items, the new item should get inserted before the currently selected row. Requires a change in the framework.

Wednesday, August 18, 2021

Wed, 18 Aug 2021 00:00:00 +0000

Wednesday, August 18, 2021¶

The fields vat_id and vat_regime are now implemented by the new model mixin lino_xl.lib.vat.VatSubjectable (instead of being injected). And lino_xl.lib.contacts.Partner inherits from lino_xl.lib.vat.VatSubjectable.

No functional change, but Sharif can now extend the full_clean() of lino_xl.lib.vat.VatSubjectable so that it validates the VAT id of a partner.

I am not yet sure how to document all this…

Tuesday, August 17, 2021

Tue, 17 Aug 2021 00:00:00 +0000

Tuesday, August 17, 2021¶

I reviewed the user guide about Purchases and slightly optimized the behaviour when entering partly deducible VAT: when you manually change the vat_class of an invoice item, then Lino now re-computes all amounts of that item, based on the invoices total amount and the new VAT class.

There was no insert_layout in sheet.Items, making it impossible to create a new item using the Insert key.

Renamed VAT class “Operations without VAT” to “Without VAT”. And the translations for de and fr were missing.

Friday, August 6, 2021

Fri, 06 Aug 2021 00:00:00 +0000

Friday, August 6, 2021¶

Link to other ticket in a comment doesn’t work¶

Observation about #4278: When a comment has a link to ticket (e.g. “#2614”), then that link is broken when shown on ticket detail, but it works when shown on the dashboard.

It works in the dashboard (RecentComments):

href='javascript:window.App.runAction({ "actorId": "tickets.Tickets", "an":
"detail", "rp": "dashboard-1", "status": { "base_params": {  }, "param_values":
{ "end_date": null, "observed_event": null, "observed_eventHidden": null,
"reply_to": null, "reply_toHidden": null, "start_date": null, "user": null,
"userHidden": null }, "record_id": 2614 } })'

It’s broken in the detail of a ticket (CommentsByRFC):

href='javascript:window.App.runAction({ "actorId": "tickets.Tickets", "an":
"detail", "rp": null, "status": { "base_params": { "mk": 2239, "mt": 37 },
"param_values": { "end_date": null, "observed_event": null,
"observed_eventHidden": null, "reply_to": "Blank", "reply_toHidden":
"&lt;BLANK&gt;", "start_date": null, "user": null, "userHidden": null },
"record_id": 2614 } })'

Thursday, August 5, 2021

Thu, 05 Aug 2021 00:00:00 +0000

Thursday, August 5, 2021¶

Yesterday, after several days of trying to understand why my changes for #3301 broke Presto, I explained the situation to Sharif, and then finally understood the last missing piece and got Lino Presto fixed. Thanks to Sharif for listening.

Now I committed and started to review the docs: invoicing : Generating invoices The invoicing plugin The orders plugin

TODO: Manual tests in Noi. Make a template for printing an SLA. Write more docs.

Tuesday, August 3, 2021

Tue, 03 Aug 2021 00:00:00 +0000

Tuesday, August 3, 2021¶

Continued from yesterday.

Trying to understood why presto is still failing. It’s because I tried to optimize things regarding the differentiation between the “partner” and the invoice recipient. Seems that Presto is just the only demo having examples of invoice recipients different from partner, and therefore the only use case where this optimization caused a failure.

I tried to get rid of this difference, but this means that the invoice recipients always receive one invoice per partner, possibly having multiple invoices sent by a single plan. We cannot remove the feature of group invoices. For example an orphan house having 10 children attend to therapies, they want to see the costs separated per child, but they do not want to get 10 invoices every month.

TODO: Lino Presto is still broken. Manual tests in Noi. Make a template for printing an SLA. Write more docs.

Monday, August 2, 2021

Mon, 02 Aug 2021 00:00:00 +0000

Monday, August 2, 2021¶

I committed about 2 weeks of work on #3301. The main changes are in lino_xl.lib.invoicing and lino_xl.lib.orders (both Lino Extensions Library) and the first user is Lino Noi where we can now write service level agreements (periodic invoice generators), so that we can use Lino to write our yearly invoices.

TODO: Lino Presto is still broken. Manual tests in Noi. Make a template for printing an SLA. Write more docs.

Intermezzo: Oops, there is a little problem on GitLab <https://gitlab.com/lino-framework/book/-/jobs/1469319484>`___: already :cmd:`inv prep fails with undefined symbol: pango_context_set_round_glyph_positions. After doing pip install -U weasyprint cffi on my machine, I have it also here. Other people have reported and documented the issue: Weasyprint 53 needs a newer version of Pango, which is not yet available on stable debian. So as a works-for-me, I simply add “weasyprint<52” to Lino’s install_requires.

Tuesday, July 27, 2021

Tue, 27 Jul 2021 00:00:00 +0000

Tuesday, July 27, 2021¶

The lino_xl.lib.orders.InvoicesByOrder. now uses lino_xl.lib.invoicing.InvoiceGenerator.get_invoiceable_partner(). We want this table to show all invoices having the same invoice recipient as this current order. Until now it was used only in Lino Presto where the lino.core.site.Site.project_model is a subclass of lino_xl.lib.contacts.Partner. But in Lino Noi it’s different.

Monday, July 26, 2021

Mon, 26 Jul 2021 00:00:00 +0000

Monday, July 26, 2021¶

I reviewed my decision of two days ago to rename InvoiceGenerator.get_invoiceable_product() to InvoiceGenerator.get_invoicing_product() because this actually is supposed to return the product that will be invoiced. When the InvoiceGenerator.get_invoiceable_product() method returns None, this generally means to not generate any invoice item. With the nuance that it may return None when the product is decided later by an overridden InvoiceGenerator.get_wanted_invoice_items() method.

The InvoiceGenerator.get_wanted_invoice_items() method has currently three implementations:

the standard implementation defines the default behaviour, which is that every generator produces one invoice item.
Lino Presto overrides the default behaviour to add extra invoice items (one per order item). You can configure orders where the calendar entries themselves are free (work time doesn’t count) but only these order items are being invoiced.
Lino Tera overrides it to introduce an even more customized approach. It collects the individual invoicing events per calendar entry type and only then decides which products to actually invoice. Because the price of a calendar entry in Tera depends on the partner’s price factors and on the calendar entry type.

Besides having a customized approach of grouping and pricing, Lino Tera also features a way where therapists may use a simplified cash book in which they simply type received money per calendar entry. (It might make sense to generalize this feature by moving it from tera to xl)

The InvoiceGenerator.get_wanted_invoice_items() method is called from two places:

in Plan.fill_plan() to create an invoicing suggestion. The invoice and its items are created in memory just to compute the sums and then thrown away again.
in Item.create_invoice(), i.e. to actually create the invoice for an invoicing suggestion.

Seems that I now know how to name the Tariff model: it is an event pass type. When a product has an event pass type, this means that buying one unit of the product gives you the right to consume a given number of invoicing events.

So I rename InvoiceGenerator.get_invoiceable_tariff() to InvoiceGenerator.get_invoicing_pass_type(). If this returns None, Lino simply assumes that every invoicing event generates one invoice item.

After rediscovering above things, I now turn back to Lino Noi: Something is missing for invoicing the yearly membership fees. But what?

Saturday, July 24, 2021

Sat, 24 Jul 2021 00:00:00 +0000

Saturday, July 24, 2021¶

I worked on #3301. While generating service reports was almost done and is probably just a question of configuration, the new feature will be to have Lino Noi automatically generate yearly (or monthly) invoices for membership fees. Something very subtle is missing there, or just not well documented. I will now commit a set of changes that are work in process. I kind of get prepared for the big coup.

Changes for now:

Changed the label for StartInvoicing and subclasses Create invoices to Generate invoices.

New plugin attribute Plugin.order_model to generalize what Lino Tera, Lino Voga and Lino Presto did until now in three plugin extensions, which were suspiciously similar. The invoicing plugins in voga, tera and presto are no longer used.

New plugin attribute Plugin.three_demo_areas to formulate the only difference between the plugin extensions: In Voga the demo fixture creates only one invoicing area while other applications have three invoicing areas.

I removed two unused method definitions get_invoiceable_amount() in tera and voga.

I optimized some translations, which had an unexpected side effect: quite some time ago I moved the roger variant of voga from voga to book. And only now, after having done inv mm in voga, the translations of the texts in this variant disappeared.

TODO:

replace get_invoiceable_product() altogether by get_invoicing_tariff()?
rename “flatrate” (the verbose name for Tariff) to “invoicing rule”.
implement the new InvoiceGenerator methods on orders.Order

Friday, July 23, 2021

Fri, 23 Jul 2021 00:00:00 +0000

Friday, July 23, 2021¶

Thursday, July 22, 2021

Thu, 22 Jul 2021 00:00:00 +0000

Thursday, July 22, 2021¶

Changes in the lino_xl.lib.invoicing model.

To Plan.today field of an invoicing plan is now read-only. That’s because Lino needs it to make sure that the database never contains more than one invoicing plan per user.

New plugin attribute Plugin.order_model is “orders.Order” for Lino Presto. Until now we had some special logic for orders in presto. But this special logic is actually generic (and optional): we need it in noi as well.

I converted docs about lino_voga.lib.courses to prosa.

I started to review the docs about lino_xl.lib.invoicing. Until now we had no difference between invoice generator and invoiceable, but maybe we want to separate them.

TODO:

Extend the lino_xl.lib.orders plugin: an order is not only an EventGenerator but also an InvoiceGenerator.

Saturday, July 17, 2021

Sat, 17 Jul 2021 00:00:00 +0000

Saturday, July 17, 2021¶

Sharif and I started to work on a script get_cities.py below lino_xl.lib.countries. Not finished. Idea is to replace commondata by data taken from wikidata.

Friday, July 16, 2021

Fri, 16 Jul 2021 00:00:00 +0000

Friday, July 16, 2021¶

I eliminated a pseudo-feature of rstgen.sphinxconf.interproject.configure() : calling it without specifying a list of projects to include (the prjspec argument) was adding all local atelier projects. I created this feature for generating my blog. But now I removed the feature and added explicit intersphinx dependencies to my blog’s conf.py instead.

Friday, July 9, 2021

Fri, 09 Jul 2021 00:00:00 +0000

Friday, July 9, 2021¶

I committed my recent work on makehelp (#4172). I started to understand a few things. Time to write some documentation.

makehelp now generates only the actor pages (and an index page), IOW a page per actor but no more pages per model and per plugin. That’s because actor pages are the only ones to be opened by the help button. Each actor page should directly refer to the relevant places in the reference documentation and the user guide.

Note that reference documentation is a new glossary term. We should say reference documentation instead of developer guide because this kind of documentation is actually not only for developers, they are also for key users. These “reference docs” are currently integrated into the developer guide, but this might change.

I moved the logic for loading help texts at startup into a separate class lino.core.help.HelpTextsLoader. The HTL is instantiated temporarily by the lino.core.kernel.Kernel.kernel_startup(). The makedocs command instantiates it another time because it also uses the first paragraph of actor class descriptions, which the kernel does not store. All this is still experimental.

I moved all the help functionality, including the OpenHelpWindow action to a new plugin lino.modlib.help and make it an optional feature. I renamed makedocs to makehelp.

To make my work more visible we should start to update the demo sites more often, and to run makedocs each time there. Here it gets related to the #2480 ticket: it would be cool to have an impressive series of Lino demo sites on Heroku, each with a usable user guide….

My goal for this time is that when clicking on the ? button, Lino opens a page that is actually useful for an end user. Or more precisely a page that has a chance to be useful, when documentation is written accordingly. Because that’s another pair of shoes: defining the right conventions and habits of where to write what. This is currently still a big mess, and reviewing all docs is a big work. But I am advancing…

TODO: Test whether translation actually works.

Thursday, July 8, 2021

Thu, 08 Jul 2021 00:00:00 +0000

Thursday, July 8, 2021¶

I am working on makehelp (#4172).

Wednesday, July 7, 2021

Wed, 07 Jul 2021 00:00:00 +0000

Wednesday, July 7, 2021¶

I worked on #4245 (Lino deleted the guests from my event) .

The problem happens only when can_edit_guests_manually() returns False. It seems that can_edit_guests_manually() should not base its decision on the state of the event. But when then do we have EntryState.fill_guests

In amici1, log in as robin, create an activity, check a weekday, hit update_events: Lino says “No automatic calendar entries because no entry type is configured.”

The std fixture of lino_xl.lib.cal now sets the event type “Meeting” as SiteConfig.default_event_type.

lino_xl.lib.Course.update_cal_event_type() now returns SiteConfig.default_event_type if the activity has no line (or if the line has no event_type).

I replaced the can_edit_guests_manually() of lino_xl.lib.cal.Event by a new database field lino_xl.lib.cal.Event.guests_edited. This is an invisible field, it is similar to lino_xl.lib.vat.VatDocument.items_edited

Manual tests

In Lino Amici, try to make Lino “accidentally” delete my manually entered guests.
In Lino Avanti: create a course with 2 enrolments. have it generate some lessons. check that the enrolments have become guests. manually edit the guest list in one event: add an extra guest. Add one new enrolment and remove one enrolment from the course, then run update_guests on the course. Check whether the guests of the first event have remained unchanged while the second event has received the new guest.

The manual tests pass. I run the test suite, which will need some changes because there is a new database field.

Tuesday, July 6, 2021

Tue, 06 Jul 2021 00:00:00 +0000

Tuesday, July 6, 2021¶

I fixed the first issue observed Sunday, July 4, 2021. Explanation: when the date of a suggested event gets modified by the user, Lino aligns all following events accordingly. Change: From now on Lino also verifies that such a date change remains within its limits. You cannot go back in time. When lesson 1 was on a given date, you cannot make lesson 2 happen before lesson 1. I changed the algorithm and internal API for analyzing the existing events.

I found and fixed another bug: create an activity without enrolments, let it generate some events, pick one of the events, manually add a guest, mark the event as took_place. Then try to delete the activity. Lino simply did this. Now it says “Cannot delete X because it has user-modified calendar entries.”

Internal change: I renamed update_cal_rset() to get_recurrence_set().

I extended the documentation about Automatic calendar entries.

I added more test cases in lino_book/projects/voga2/tests/test_faggio.py

Sunday, July 4, 2021

Sun, 04 Jul 2021 00:00:00 +0000

Sunday, July 4, 2021¶

Today I upgraded my Lino Amici production site and then played around in order to manually test the new version on real data.

I stumbled into an irritating behaviour of lino_xl.lib.cal.EventGenerator. I have a long-running weekly activity that started years ago. Until now I didn’t use my Lino Amici very much, most events are in state suggested only some of the events are actually confirmed (i.e. state is took_place). The weekday of the event recently changed from Tuesday to Monday. Now I’d like to start confirming these events more seriously. I’d like Lino to forget all the old suggestions and to keep only the confirmed old events. And to generate a new series. I change start_date to 2021-06-28, change the weekday from Tue to Mon, then hit the flash button. Events are now on Monday, yes, but they start already in December 2020. I said start_date 2021-06-28. Continued Tuesday, July 6, 2021.

And then a behaviour that is not only disturbing but destructive: I had manually entered the guests of an event (the activity of which had no enrolments). Editing the guests was possible only by setting the event state to took_place. That’s okay as a limitation. I then saw that these manually entered guests had the state “invited”. I switched the state back to draft, believing that switching back to took_place will set them to present (because force_guests is True) But poof! Setting the event to draft deleted all my presences. Without warning. We must forgive Lino because there are situations where we actually want it to delete suggested guests. But this was not such a situation. To be continued.

Saturday, July 3, 2021

Sat, 03 Jul 2021 00:00:00 +0000

Saturday, July 3, 2021¶

Medico:

one new concept will be “availabilities”.

Every Physician configures a list of availabilities: “I am available on Mondays and Wednesdays from 8 to 12 in hospital X room Y”. And then the every physician has a system of “time slots” where they can accept appointments. Lino uses this table to generate suggested calendar entries. Availability inherits from cal.EventGenerator.

We will need to extend EventGenerator because atm it supports only one calendar entry per day.

A patient says “I need to see a cardiologist”. Lino would show a list of available calendar events that do not yet have a guest (more precisely which are in state “Available”).

We also need the receeption plugin

Tuesday, June 29, 2021

Tue, 29 Jun 2021 00:00:00 +0000

Tuesday, June 29, 2021¶

Lino Medico¶

Sharif and I had a brainstorming for a new Lino, Lino Medico, that might be useful in Bangladesh where there is no national health insurance system.

Inspiration from Estonia:

Concepts that might be useful in Lino Medico

calendar : National eBooking system : Booking, changing and cancelling the specialist visit time
personal medical history
medical data the health care service providers have submitted about you.
incapacity certificates
physician
prescriptions for pharmaceuticals
prescriptions for medical devices

Existing plugins that might be useful:

cal, calview
contacts : physicians and patients
clients
products (pharmaceuticals)
tickets, comments : patient history
courses (therapies)

Wednesday, June 23, 2021

Wed, 23 Jun 2021 00:00:00 +0000

Wednesday, June 23, 2021¶

Slave tables and delayed values¶

I fixed #4222, which was caused by my recent changes #4046.

I started a new topic guide about Delayed values, which also contains a test case that (probably) would have covered this issue.

A slave table can be a valid data element of a detail layout, and it is named foos.Foos (i.e. with a dot). And also the corresponding store field will be named foos.Foos (not foos_Foos), i.e. the names of store fields don’t need to be valid Python attribute names.

Diátaxis workshop¶

Documentation should be structured under the following four top-level entries:

Tutorials (aka “Get started”) should be: clear, have a concise outcome, …
How-to guides (Recipes) should be reproducible, cheat sheet
Reference (Technical specs, description of what exists) should be correct, complete, excludes instruction and explanation, answers to facts (not the needs of the reader)
Explanation (aka “Topics”, “Background”, “Discussion”) offer context, establishes connections, the bigger picture, history

This differs e.g. from established systems like Darwin Information Typing Architechture.

Each top-level index might be divided by “areas of usage”. Think like a shop owner: put the interesting and tempting topics at the front.

Q: Where to put the different audiences of readers? A: Consider them as being completely different manuals. E.g. end users, developers, contributors, training providers, hosters… one manual for each of them

Monday, June 21, 2021

Mon, 21 Jun 2021 00:00:00 +0000

Monday, June 21, 2021¶

Lino Così failed to define a user type named user. Which caused passwd to fail when called with the -c option.

Friday, June 18, 2021

Fri, 18 Jun 2021 00:00:00 +0000

Friday, June 18, 2021¶

I started a topic page in the developer guide: About data elements, atomizers & Co.

Actor._handle_class TableHandle

get_handle_name() is still used in lino_welfare.modlib.debts, where we want to have different variants of table columns for a slave table. I wrote this solution before I discovered the recipe for doing this using a choicelist holding actors. We should replace this by a choicelist of actors.

Thursday, June 17, 2021

Thu, 17 Jun 2021 00:00:00 +0000

Thursday, June 17, 2021¶

An overview of existing Lino applications¶

Here is my plan for the demo session for Natasa.

I switched mathieu and avanti1 to demo_date 20210617 and ran pm prep. In mathieu this caused a unique constraint failure, so I optimized welfare.modlib.isip.ContractBase.suggest_cal_guests() to remove duplicate partners from its suggestions.

In mathieu I also changed languages to “en fr de nl”.

mathieu (Lino Welfare)

Sign in as theresia (reception clerk) and receive a client, e.g. Guido Radermacher.
Sign in as a social worker and meet the client
workshops and enrolments
aids and aids confirmations

avanti1 (ref:avanti)

noi1r (ref:noi)

roger (ref:voga)

tera1 (ref:tera)

amici1 (ref:amici)

Wednesday, June 16, 2021

Wed, 16 Jun 2021 00:00:00 +0000

Wednesday, June 16, 2021¶

insipid ticket #50¶

With Matthias’ help I fixed insipid ticket #50.

The request’s session was deleted before the request completed¶

After clicking the submit button of the insert window of a comment (POST to /api/comments/CommentsByRFC) I got once more a red notification message (saying something I didn’t manage to read).

The Lino log says:

202106-16 06:26:24 INFO [kernel 7902 140385269999488] : run_action luc comments.CommentsByRFC.submit_insert Ticket #4213 ('#4213 (☐ Font size becomes bigger when I tap on link)') []
202106-16 06:26:25 WARNING [log 7902 140385269999488] : The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example.: /api/comments/CommentsByRFC
Traceback (most recent call last):
  File ".../env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 87, in save
    obj.save(force_insert=must_create, force_update=not must_create, using=using)
  File ".../env/lib/python3.7/site-packages/django/db/models/base.py", line 727, in save
    force_update=force_update, update_fields=update_fields)
  File ".../env/lib/python3.7/site-packages/django/db/models/base.py", line 765, in save_base
    force_update, using, update_fields,
  File ".../env/lib/python3.7/site-packages/django/db/models/base.py", line 848, in _save_table
    raise DatabaseError("Forced update did not affect any rows.")
django.db.utils.DatabaseError: Forced update did not affect any rows.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../env/lib/python3.7/site-packages/django/contrib/sessions/middleware.py", line 61, in process_response
    request.session.save()
  File ".../env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 94, in save
    raise UpdateError
django.contrib.sessions.backends.base.UpdateError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../env/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File ".../env/lib/python3.7/site-packages/django/utils/deprecation.py", line 119, in __call__
    response = self.process_response(request, response)
  File ".../env/lib/python3.7/site-packages/django/contrib/sessions/middleware.py", line 64, in process_response
    "The request's session was deleted before the "
django.contrib.sessions.exceptions.SessionInterrupted: The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example.

In this thread on stackoverflow they suggest miscellaneous explanations, but none of them seems to apply here. Jane uses the database for storing sessions, and I am quite sure that I didn’t log out in some other tab.

>>> from django.conf import settings
>>> settings.SESSION_ENGINE
'django.contrib.sessions.backends.db'

::

$ pip freeze | grep Django Django==3.2.4

$ pip install -U mysqlclient Django Requirement already satisfied: mysqlclient in ./env/lib/python3.7/site-packages (2.0.3) Requirement already satisfied: Django in ./env/lib/python3.7/site-packages (3.2.4) Requirement already satisfied: sqlparse>=0.2.2 in ./env/lib/python3.7/site-packages (from Django) (0.4.1) Requirement already satisfied: pytz in ./env/lib/python3.7/site-packages (from Django) (2021.1) Requirement already satisfied: asgiref<4,>=3.3.2 in ./env/lib/python3.7/site-packages (from Django) (3.3.4) Requirement already satisfied: typing-extensions in ./env/lib/python3.7/site-packages (from asgiref<4,>=3.3.2->Django) (3.7.4.3)

Actually the first problem in above traceback is DatabaseError: Forced update did not affect any rows, when something tries to save the current session.

The following line of the traceback looks interesting:

File ".../python3.7/site-packages/django/utils/deprecation.py", line 119, in __call__
  response = self.process_response(request, response)

That code uses asyncio, so I wonder whether the problems might come because the notify plugin uses asgiref.sync.async_to_sync. But hasn’t it been doing that already earlier? That question remains open for my next session.

I create a new comment “new comment for testing” and click “Create”. Sometimes it works and sometimes it doesn’t work. When it doesn’t work, then it actually did create the comment, but

The problem seems to have disappeared after setting lino.core.site.Site.use_multiprocessing to False.

Just as a reminder, here are the new features Lino on which we are working

push api
use multiprocessing when notifying others
dashboard layouts
delayed values
sync google api for calendar and contacts
optimizations of react front end

New DashboardLayouts make things slow

Tue, 15 Jun 2021 00:00:00 +0000

New DashboardLayouts make things slow¶

Tuesday, June 15, 2021

In order to understand the cause of #4207, I wrote a testing page Exploring SQL activity of new dashboard book/docs/apps/noi/sql_dashboard.rst, which activates SQL logging, then renders the dashboard, and then shows the number of database queries that were executed.

New feature for tested documents: lino.core.requests.BaseRequest.show() now supports showing a detail window. Side effects: lino.core.site.Site.get_main_html() now takes an ar instead of a request. New method lino.modlib.jinja.Plugin.render_from_ar(). New method lino.core.renderer.HtmlRenderer.show_detail(). The output of lino.api.doctest.show_dashboard() now uses the tex renderer instead of the site’s default renderer.

Note that all this does not yet even try to fix the #4207 issue. My plan is to use these new features on the production site where I plan to pm run the following script:

from lino.api.doctest import *
reset_sql_queries()
rt.login('robin').show('system.Dashboard')
show_sql_summary()

Sunday, June 13, 2021

Sun, 13 Jun 2021 00:00:00 +0000

Sunday, June 13, 2021¶

The new dashboard in Jane is still slow, and if often goes into “502 Bad Gateway” mode. This time I triggered 502 by deleting a ticket.

I consult Analyzing RAM usage and then compare Jane with another production site that is not slow.

Output of ps -e -orss=,args= |awk ‘{print $1 “ “ $2 }’| awk ‘{tot[$2]+=$1;count[$2]++} END {for (i in tot) {print tot[i],i,count[i]}}’ | sort -n

On Jane:

...
22784 postgres: 6
24336 /usr/lib/libreoffice/program/soffice.bin 1
29204 smtpd 3
37668 /lib/systemd/systemd-journald 1
114352 /usr/sbin/mysqld 1
130680 python 1
964544 /usr/bin/uwsgi 7

On the other production site:

...
71908 /usr/lib/libreoffice/program/soffice.bin 1
121340 /usr/sbin/mysqld 1
136868 python 1
1289172 /usr/bin/uwsgi 11

Output of top (and then M) on Jane:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
www-data  20   0  342784 149880   5144 S   0,0   7,5   1:07.29 uwsgi
www-data  20   0  339304 145692   4488 S   0,0   7,3   1:41.30 uwsgi
www-data  20   0  332804 138796   3872 S   0,0   7,0   0:00.46 uwsgi
www-data  20   0  330812 137752   4820 S   0,0   6,9   0:15.93 uwsgi
www-data  20   0  329576 135692   3872 S   0,0   6,8   0:00.56 uwsgi
www-data  20   0  327696 134952   5000 S   0,0   6,8   0:16.70 uwsgi
www-data  20   0  284960 130680   7872 S   0,0   6,6   0:29.15 python
www-data  20   0  315052 121780   4868 S   0,0   6,1   0:11.89 uwsgi
mysql     20   0 1727408 114352   1960 S   0,0   5,7   2:34.27 mysqld
...

Nothing suspicious there.

But here. The /var/log/mysql/error.log says:

2021-06-13  2:58:27 613 [Warning] Aborted connection 613 to db: 'xyz' user:
'abcd' host: 'cdef' (Got timeout reading communication packets)

Whenever a communication error occurs, MySQL increments the status counter for either Aborted_clients or Aborted_connects. Muhammad Irfan

My feeling says that we can exclude Aborted_connects (failed attempts to connect to the MySQL server), so this log entry informs about a case of Aborted_clients, IOW a connection that was aborted because the client died without closing the connection properly.

The client died without closing the connection properly?

Irfan: “To be honest, aborted connection errors are not easy to diagnose.”

I should now enable the general query log of the MySQL server

https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_general_log

https://dev.mysql.com/doc/refman/5.6/en/query-log.html

As it seems I can activate the general MySQL query log simply by adding a SET to the OPTIONS of my DATABASES setting.

Something like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        ...
        'OPTIONS': {
           "init_command": "SET default_storage_engine=MyISAM; SET log_output=FILE; SET general_log=ON; SET general_log_file=xyz-general.log"
        }
}

   "init_command": "SET default_storage_engine=MyISAM;
   SET log_output=FILE;
   SET general_log=ON;
   SET general_log_file=jane-general.log"

If you specify no name for the general query log file, the default name is host_name.log. The server creates the file in the data directory unless an absolute path name is given to specify a different directory.

What is the “data directory” in MySQL?

https://dev.mysql.com/doc/refman/5.7/en/data-directory.html

Short answer: For a database named “xyz” it is /var/lib/mysql/xyz.

So after restarting Lino, my log file should be in /var/lib/mysql/xyz/xyz-general.log

Only problem is that it doesn’t work. After almost an hour of trying, I still didn’t manage to activate the MySQL general query log (at runtime, i.e. without changing the server config).

The /etc/mysql/mariadb.conf.d/50-server.cnf config file says this:

# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file       = /var/log/mysql/mysql.log
#general_log            = 1

So I finally uncommented these two lines in the /etc/mysql/mariadb.conf.d/50-server.cnf and said sudo service mysql restart.

Now I have a /var/log/mysql/mysql.log file.

Every ten seconds it says:

210613 19:55:12          467 Query    SELECT COUNT(*) AS `__count` FROM `notify_message` LEFT OUTER JOIN `users_user` ON (`notify_message`.`user_id` = `users_user`.`id`) WHERE (`notify_message`.`sent` IS NULL AND NOT (`users_user`.`email` = '' AND `users_user`.`email` IS NOT NULL) AND `notify_message`.`mail_mode` = 'often')
210613 19:55:14          467 Query    SELECT `django_mailbox_mailbox`.`id`, `django_mailbox_mailbox`.`name`, `django_mailbox_mailbox`.`uri`, `django_mailbox_mailbox`.`from_email`, `django_mailbox_mailbox`.`active`, `django_mailbox_mailbox`.`last_polling` FROM `django_mailbox_mailbox` WHERE `django_mailbox_mailbox`.`active`
                       467 Query      UPDATE `django_mailbox_mailbox` SET `last_polling` = '2021-06-13 17:55:14.871730' WHERE `django_mailbox_mailbox`.`id` = 1
210613 19:55:22          467 Query    SELECT COUNT(*) AS `__count` FROM `notify_message` LEFT OUTER JOIN `users_user` ON (`notify_message`.`user_id` = `users_user`.`id`) WHERE (`notify_message`.`sent` IS NULL AND NOT (`users_user`.`email` = '' AND `users_user`.`email` IS NOT NULL) AND `notify_message`.`mail_mode` = 'often')
210613 19:55:24          467 Query    SELECT `django_mailbox_mailbox`.`id`, `django_mailbox_mailbox`.`name`, `django_mailbox_mailbox`.`uri`, `django_mailbox_mailbox`.`from_email`, `django_mailbox_mailbox`.`active`, `django_mailbox_mailbox`.`last_polling` FROM `django_mailbox_mailbox` WHERE `django_mailbox_mailbox`.`active`
                             467 Query        UPDATE `django_mailbox_mailbox` SET `last_polling` = '2021-06-13 17:55:24.889516' WHERE `django_mailbox_mailbox`.`id` = 1

Yes, okay, that’s how to have MySQL log everything. But it’s not useful for our problem. Because people are working on other Lino sites on the same server.

I will rather try Exploring SQL activity in Lino Noi.

Thursday, June 10, 2021

Thu, 10 Jun 2021 00:00:00 +0000

Thursday, June 10, 2021¶

Sharif and I had a brainstorming about what we can do to improve Lino’s calendar functionality.

TODO:

The lino_xl.lib.calview plugin still has some quite visible issues :

Fix the minimum column width: when rendering the columns for the weekdays, the calendar view switches to quickly into the mode “There is not enough width to display all columns, so I must add a horizontal wrapper”
The navigation_panel should be collapsible.
The navigation buttons are kind of duplicated. But which one to remove? User can skip by day and by month, but not by year.

Google API:

rename googleapi_people to google and have it collaborate also with cal and albums.
Have lino_xl.lib.contacts.Person and lino_xl.lib.cal.Event inherit from GooglePeopleSynchronized.
GooglePeopleSynchronized to a separate mixins.py module and add a test if dd.is_installed('google'): so that the mixin can be used also when the plugin is not installed (and doesn’t add anything in that case).

DONE:

Rename GooglePeople to GoogleSynchronized google_resourceName becomes a class attribute. For each resource we will have a subclass: GooglePeopleSynchronized GoogleCalendarSynchronized GoogleImagesSynchronized

Wednesday, June 9, 2021

Wed, 09 Jun 2021 00:00:00 +0000

Wednesday, June 9, 2021¶

“Commands out of sync; you can’t run this command now”¶

In the detail view of a ticket, after clicking on the mark_ready icon, I saw a red notification, and the lino.log says:

202106-09 12:14:19 INFO [kernel 23479 140427458520960] :
  run_action luc tickets.ActiveTickets.mark_ready None [Ticket #4194 ("#4194 (⚒ on jane The insert window doesn't close after submitting)")]
202106-09 12:14:19 WARNING [log 23479 140427458520960] :
  The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example.: /api/tickets/ActiveTickets/4194
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
    self.connection.autocommit(autocommit)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/MySQLdb/connections.py", line 244, in autocommit_mysql.connection.autocommit(self, on)
MySQLdb._exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 87, in save
    obj.save(force_insert=must_create, force_update=not must_create, using=using)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/transaction.py", line 290, in __exit__
    connection.set_autocommit(True)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 415, in set_autocommit
    self._set_autocommit(autocommit)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
    self.connection.autocommit(autocommit)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 272, in _set_autocommit
    self.connection.autocommit(autocommit)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/MySQLdb/connections.py", line 244, in autocommit
    _mysql.connection.autocommit(self, on)
django.db.utils.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/sessions/middleware.py", line 61, in process_response
    request.session.save()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 94, in save
    raise UpdateError
django.contrib.sessions.backends.base.UpdateError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/utils/deprecation.py", line 119, in __call__
    response = self.process_response(request, response)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/sessions/middleware.py", line 64, in process_response
    "The request's session was deleted before the "
django.contrib.sessions.exceptions.SessionInterrupted: The request's session was deleted before the request completed. The user may have logged out in a concurrent request, for example.

I guess that this is another symptom of #4195. And I have now idea what might be the reason. And it is not reproducible. We can only collect such tracebacks when they occur and try to get it in a reproducible way.

I did pip install -U for Django and mysqlclient:

Django upgrades from 3.2 to 3.2.4
mysqlclient remains 2.0.3

AttributeError: ‘str’ object has no attribute ‘tzinfo’¶

And after submitting a comment about this, I encountered another problem:

202106-09 12:49:46 INFO [kernel 23993 140541549119360] : run_action luc comments.CommentsByRFC.submit_insert Ticket #4195 ('#4195 (🗪 Lost connection to MySQL server during query)') []
202106-09 12:49:46 ERROR [ajax 23993 140541549119360] : AjaxExceptionResponse AttributeError: 'str' object has no attribute 'tzinfo'
in request POST /api/comments/CommentsByRFC (data: <QueryDict: {
  'an': ['submit_insert'],
  'body': ['<p>Jane is back now, with Django upgraded from 3.2 to 3.2.4. mysqlclient was already at the newest version.</p>'],
  'disabled_fields': ['[object Object]'...)
TRACEBACK:
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_react/react/views.py", line 109, in post
    return settings.SITE.kernel.run_action(ar)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/kernel.py", line 797, in run_action
    a.run_from_ui(ar)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/actions.py", line 1115, in run_from_ui
    self.save_new_instance(ar, elem)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/actions.py", line 1045, in save_new_instance
    ar.set_response(rows=[ar.ah.store.row2list(ar, elem)])
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 1303, in row2list
    fld.value2list(ar, v, l, row)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 254, in value2list
    value, text = self.get_value_text(ar, v, row)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 270, in get_value_text
    return (v, ch.get_text_for_value(v, obj))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/utils/choosers.py", line 424, in get_text_for_value
    return m(value)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/gfks/fields.py", line 54, in fk_display
    return str(ct.get_object_for_this_type(pk=value))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/contenttypes/models.py", line 175, in get_object_for_this_type
    return self.model_class()._base_manager.using(self._state.db).get(**kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 68, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
    value = converter(value, expression, connection)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/mysql/operations.py", line 313, in convert_datetimefield_value
    value = timezone.make_aware(value, self.connection.timezone)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/utils/timezone.py", line 239, in make_aware
    return timezone.localize(value, is_dst=is_dst)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/pytz/__init__.py", line 244, in localize
    if dt.tzinfo is not None:
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino_react/react/views.py", line 109, in post
    return settings.SITE.kernel.run_action(ar)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/kernel.py", line 797, in run_action
    a.run_from_ui(ar)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/actions.py", line 1115, in run_from_ui
    self.save_new_instance(ar, elem)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/actions.py", line 1045, in save_new_instance
    ar.set_response(rows=[ar.ah.store.row2list(ar, elem)])
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 1303, in row2list
    fld.value2list(ar, v, l, row)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 254, in value2list
    value, text = self.get_value_text(ar, v, row)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/core/store.py", line 270, in get_value_text
    return (v, ch.get_text_for_value(v, obj))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/utils/choosers.py", line 424, in get_text_for_value
    return m(value)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/modlib/gfks/fields.py", line 54, in fk_display
    return str(ct.get_object_for_this_type(pk=value))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/contrib/contenttypes/models.py", line 175, in get_object_for_this_type
    return self.model_class()._base_manager.using(self._state.db).get(**kwargs)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/query.py", line 68, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
    value = converter(value, expression, connection)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/backends/mysql/operations.py", line 313, in convert_datetimefield_value
    value = timezone.make_aware(value, self.connection.timezone)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/utils/timezone.py", line 239, in make_aware
    return timezone.localize(value, is_dst=is_dst)
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/pytz/__init__.py", line 244, in localize
    if dt.tzinfo is not None:
AttributeError: 'str' object has no attribute 'tzinfo'

I saw that we have some broken GFKs in Jane: pm show gfks.BrokenGFKs produces a list of them.

Lino automatically adds lino.utils.ajax.AjaxExceptionResponse to the MIDDLEWARE_CLASSES setting. On a production site this caused error notification emails with a huge subject because the traceback was also printed in the subject of the message. I locally disabled this middleware on Jane because I don’t remember why it’s needed at all.

I also met again with the issue that after submitting an insert window, Lino had a 504 Gateway Time-out and therefore returned an empty response to a perfectly reasonable POST request. The client then complains SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data.

Tuesday, June 8, 2021

Tue, 08 Jun 2021 00:00:00 +0000

Tuesday, June 8, 2021¶

The UI issue (window doesn’t close after POST) is fixed, but the MySQL connection still gets lost after a POST. IOW I can submit a first comment, but the second comment receives an empty response to the POST request.

Here is a second comment just to test whether submitting an insert window still causes the MySQL connection to get lost.

Saturday, June 5, 2021

Sat, 05 Jun 2021 00:00:00 +0000

Saturday, June 5, 2021¶

Fixing #4185¶

Yesterday Sharif and I had a session where we discovered issue #4185. I had no idea at this moment and gave this to Sharif. Later he commented:

Basically they all have the same: “type”: “Father(Mother)” in the href. Even if the type were correct, it wouldn’t still work because react weren’t acknowledging these values. …

That comment made me tilt. I commented:

Ha! I understood (maybe !) : we need to call BaseRequest.clear_cached_status() before rendering each button. Read the docs and explore. Maybe you understand. Ask questions if it isn’t clear.

Now I found time to verify it myself. Indeed this was the explanation of the issue. Fixing the issue itself was less than two minutes of work.

But this little bug has been living there for more than a year. And the test suite didn’t complain. That’s the real problem. And another problem is that Sharif had no chance to understand what I was talking about, at least not in reasonable time.

So before changing any code, I wrote a test case that reproduces the problem: Reproduce #4185.

En passant I converted the developer docs about lino_xl.lib.humanlinks from autodoc to prosa. And I fixed a cosmetic bug, a missing space: Lino said “Father(Mother)” instead of “Father (Mother)”.

TODO: write end-user documentation.

Here are my code changes:

2021-06-05 05:36 in book: 5ad1ece (fixing #4185 (reproduce the issue))
2021-06-05 05:42 in xl: cd03f93 (fix #4185)
2021-06-05 05:42 in book: 4594e20 (fixing #4185 (reproduce the issue))
2021-06-05 06:35 in xl: 8a9dd7a (convert humanlinks docs to prosa)
2021-06-05 06:36 in book: d0d387d (convert humanlinks docs to prosa)

Here is how this test was failing before I fixed the issue:

$ doctest docs/specs/humanlinks.rst
**********************************************************************
File "docs/specs/humanlinks.rst", line 88, in humanlinks.rst
Failed example:
    print(links[8])  #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
Expected:
    <a href='javascript:Lino.humanlinks.LinksByHuman.insert.run(...
    "type": "Adopted son(Adopted daughter)"...">Adopted son</a>
Got:
    <a href='javascript:Lino.humanlinks.LinksByHuman.insert.run(null,{ "base_params": { "parent": 218, "type": "01" }, "data_record": { "data": { "child": null, "childHidden": null, "disabled_fields": {  }, "parent": "Mr Lars Braun", "parentHidden": 218, "type": "Father(Mother)", "typeHidden": "01" }, "phantom": true, "title": "New Personal Link" }, "record_id": null })' style="text-decoration:none" title="Open a dialog window to insert a new Personal Link.">Adopted son</a>
**********************************************************************
1 items had failures:
   1 of  18 in humanlinks.rst
***Test Failed*** 1 failures.

Dockerizing Lino¶

Hannes and I had a session about dockerizing Lino.

We wrote a new dockerfile to deploy a running Lino noi server:

$ docker build -t lino -f Dockerfiles/deploy .

debian_updated : this is to avoid installing lots of debian packages again and again when testing. If I haven’t working getlino for a while it can make sense to build a fresh image:

$ go getlino
$ docker build --no-cache -t debian_updated -f Dockerfiles/debian_updated .
...
Successfully tagged debian_updated:latest

Here is where we stopped:

$ docker build -t lino -f Dockerfiles/deploy .
...
Successfully installed BTrees-4.9.1 ZConfig-3.6.0 appy-1.0.7 bleach-3.3.0 docopt-0.6.2 num2words-0.5.10 persistent-4.7.0 transaction-3.0.1 zc.lockfile-2.0 zodb-5.6.0 zodbpickle-2.0.0
mysql -u root  -e "create user 'mysite1'@'localhost' identified by '4zz4o1ga5xA';"
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Error: mysql -u root  -e "create user 'mysite1'@'localhost' identified by '4zz4o1ga5xA';" ended with return code 1
The command '/bin/sh -c getlino startsite --batch noi mysite1' returned a non-zero code: 1

Upgrade on Jane¶

We had a strange issue after upgrading Jane:

TypeError: t.state.site_data.choicelists[‘system.DashboardLayouts’] is undefined

The issue vanished after replacing one this by window.App.

makedocs failed because one template (conf.tpl.py) is not included in the sdist.

Friday, June 4, 2021

Fri, 04 Jun 2021 00:00:00 +0000

Friday, June 4, 2021¶

review the quicklink api for dashboard¶

This was yet another avalanche of long-awaited code cleanup.

This change in the quicklink definition API was needed because the new DashboardLayouts need a static list of the quicklinks, i.e. which does not depend on the user type. Only when rendering them, Lino checks for view permission. Because the new Dashboard is an actor, and the Site’s quicklinks are in the toolbar of this actor.

New property Site.quicklinks. New method Site.get_quicklink_items().

Methods Site.setup_quicklinks() and Site.get_quicklinks() have a changed API.

Also Plugin.get_quicklinks() and Plugin.setup_quicklinks().

List of the touched repositories:

Project	Status	URL
lino	master!	https://gitlab.com/lino-framework/lino
xl	master!	https://gitlab.com/lino-framework/xl
noi	master!	https://gitlab.com/lino-framework/noi
avanti	master!	https://gitlab.com/lino-framework/avanti
book	master!	https://gitlab.com/lino-framework/book
welfare	master!	https://gitlab.com/lino-framework/welfare
welcht	master!	https://gitlab.com/lino-framework/welcht
presto	master!	https://gitlab.com/lino-framework/presto
shop	master!	https://gitlab.com/lino-framework/shop
amici	master!	https://gitlab.com/lino-framework/amici
algus	master!	https://gitlab.com/lino-framework/algus
blog	master!	https://github.com/lsaffre/blog

Upgrade to Sphinx 4¶

In eid I had the following problem:

Extension error (sphinx.ext.extlinks):
Handler <function setup_link_roles at 0x7f381b1c1048> for event 'builder-inited' threw an exception (exception: 'NoneType' object has no attribute 'replace')

This was because rstgen.sphinxconf.default_conf didn’t test for an empty srcref_url_template(). Now it does:

if prj.main_package:
    root_mod, tpl = srcref_url_template(prj.main_package)
    if tpl:
        extlinks.update(srcref=(tpl, ''))

A second problem was this:

from sphinx.roles import menusel_role
ImportError: cannot import name 'menusel_role'

from sphinx.roles import MenuSelection as menusel_role
ImportError: cannot import name 'menusel_role'

A third problem was in Lino Welfare:

docs/specs/plugins.rst:5: WARNING: duplicated entry found in toctree: specs/clients

Wednesday, June 2, 2021

Wed, 02 Jun 2021 00:00:00 +0000

Wednesday, June 2, 2021¶

As a side effect of my work as a Python developer for Lino I often receive job offers. I generalized the statement I wrote Thursday, May 20, 2021 and used those thoughts in my new Hire me page.

The inv bd for getlino is failing on GitLab. I can reproduce the problem on my computer as follows.

Deactivate my ~/.atelier/config.py.

In my terminal:

go getlino
pip uninstall atelier
pip install atelier
inv clean -b bd

The conf.py file of the getlino docs is not yet converted to the “new style”. But that shouldn’t be the problem.

If calls rstgen.sphinxconf.interproject() as follows:

from rstgen.sphinxconf import interproject
interproject.configure(globals(), 'atelier')

The problem was caused by a bug in atelier.projects.Project.get_xconfig(). For projects that were loaded from a python module installed from pypi (i.e. without a tasks.py file) it ignored the hard-coded default values.

The problem appeared only now because I had recently removed the line doc_trees = ['docs'] from the __init__.py file of several main modules. I removed it because it caused useless complexity: everybody knows that the default value for doc_trees is [‘docs’] and we shouldn’t repeat this.

I released atelier 1.1.39 to PyPI

Note that the test job for getlino still fails on GitLab. This is another story. It says:

./prep.sh: line 11: docker: command not found

We simply need to also install the docker system package. We can use their convenience script <https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script>.

Tuesday, June 1, 2021

Tue, 01 Jun 2021 00:00:00 +0000

Tuesday, June 1, 2021¶

What does the following git message mean?

From gitlab.com:lino-framework/xl
 * branch              master     -> FETCH_HEAD

For example:

$ go xl
$ git pull
From gitlab.com:lino-framework/xl
 * branch              master     -> FETCH_HEAD
Already up to date.
$ go ug
$ git pull
Already up to date.

Saturday, May 29, 2021

Sat, 29 May 2021 00:00:00 +0000

Saturday, May 29, 2021¶

When detail_layout is a callable, then Lino calls it during startup and expects it to yield a list of layout descriptions. The first is the “actual” detail_layout.

Add support for having multiple items in detail_layout.

Dynamically created layouts.

Friday, May 28, 2021

Fri, 28 May 2021 00:00:00 +0000

Friday, May 28, 2021¶

I “finished” the developer documentation about notify: Notification framework and did a last code change: renamed enable_push to use_push_api.

Once more I gave in to the temptation of working a bit in the general documentation and reorganized some pages in the The Lino framework, cg and the Lino User Guide.

Sharif and I had a session where we started the OpenHelp action. Every actor gets a “Help” button, which opens a window on the site documentation generated by makehelp. The new button action appears already in the toolbar, but is not yet clickable. We added a new method Action.get_a_href_target(), which is to be called by the renderer when rendering the JS cache files. It usually returns None. When it returns a string, Lino should treat this string as a constant URL to be opened in a new window instead of any other action. Sharif will continue on this.

https://gitlab.com/lino-framework/cg/-/jobs/1299635109 : this failure (and several others of this style) is because I forgot to add the static path to atelier’s MANIFESST.in.

Atelier is now at version 1.1.38. It starts to sound ridiculous. We might change from semantic versioning to the month-based versioning like for Lino. There is one thing that makes me hesitate: I have absolutely no idea why I would ever want to use semantic versioning. That’s suspicious.

Thursday, May 27, 2021

Thu, 27 May 2021 00:00:00 +0000

Thursday, May 27, 2021¶

End-user help pages¶

I added a documentation entry about makehelp. The generation process itself seems now good, but the next step and the most exciting work here is still to do: have it generate some useful content into help pages. I am full of ideas, but find no time to realize them.

Review the notification framework¶

Sharif and I continued to work on lino.modlib.notify and #4174 (Improve serviceWorker).

I pushed my changes in lino, xl, ug and book.

This sprint on the notification framework seems to come to its end. The biggest change is that we now use the Push API instead of websockets (django-channels and awsgi) for implementing desktop notifications. The lino.modlib.notify.use_websockets plugin parameter is still there but setting it to True means that you enter unmaintained areas. lino.modlib.notify.use_websockets

The notification messages are now being created in a subprocess. Thanks to Sharif who did this fully alone. It’s only a few lines of code thanks to Python’s multiprocessing module, but you stilled need to find out how to use it.

The Subscription model now has allow_cascaded_delete for its user field.

In order to test whether ChangeNotifier works, the demo2 fixture for lino.modlib.comments now also calls after_ui_save.

In notify: Notification framework we added a snippet that shows these messages.

send_notification() now logs what it does. The subject of a push notification must be the title of the Lino site.

The lino_xl.lib.groups plugin was saying that it wants to install lino.modlib.comments, which was nonsense.

New Site parameter lino.core.site.Site.use_multiprocessing because the demo fixtures now also create notifications for the demo comments. And the subprocess didn’t execute correctly during pm prep.

TODO: Is there some way to say in pm prep that it should wait for all subprocess to finish?

TODO: After releasing to Jane, observe whether notifications are correctly done.

Using Docker¶

How to build a Docker image that allows you to ssh into it:

$ go hg
$ cd projects/ansible
$ docker build -t newserver -f Newserver ~/.ssh
$ docker run -it newserver /bin/bash

Wednesday, May 26, 2021

Wed, 26 May 2021 00:00:00 +0000

Wednesday, May 26, 2021¶

pm prep in lino_book.projects.tera1 demo project (Tera) gives “AttributeError: ‘str’ object has no attribute ‘main’”. I don’t yet understand why this happened now, but anyway set_detail_layout() is no longer needed in this case because we can now override a detail layout by using a class already in the beginning.

Miscellaneous changes:

ledger.MyMovements
makehelp
atelier: It is now possible to instantiate a atelier.projects.Project for a root directory that contains neither a tasks.py nor a setup.py file. The makehelp

Monday, May 24, 2021

Mon, 24 May 2021 00:00:00 +0000

Monday, May 24, 2021¶

Sharif and I continued to work on #3049 (Websocket connection in React for notifications).

I moved the “Writing documentation” section of the Developer Guide into a new top-level section “Documenter Guide” (Writing documentation). It contains a new page Shared documentation pages.

Sunday, May 23, 2021

Sun, 23 May 2021 00:00:00 +0000

Sunday, May 23, 2021¶

Sharif and I had a three-hours session for #3049 (Websocket connection in React for notifications). Before this, Saturday, we had a historic meeting where Hamza joined us and presented what he had done on use_websockets two years ago.

Now Sharif and I moved use_websockets and use_push_api from Site to the lino.modlib.notify plugin. They are no longer site attributes but plugin options. The lino_book.projects.chatter demo project will be the only place where

The use_push_api option means that Lino uses the Push API for sending notifications.

I installed pywebpush and ngrok (pronounced [en-grok]) on my machine. https://ngrok.com/docs#name

We had some fiddling with git because I still don’t fully understand how to work with remote branches. Some excerpts to be integrated into Git cheat sheet:

$ git branch --track remote_branch

$ git stash save
Saved working directory and index state WIP on master: 0d9947c2 fix 2 warnings term not in glossary
$ git stash pop

$ git branch -u origin/blurry blurry
Branch 'blurry' set up to track remote branch 'blurry' from 'origin'.


$ git branch -u origin/blurry blurry
error: the requested upstream branch 'origin/blurry' does not exist
hint:
hint: If you are planning on basing your work on an upstream
hint: branch that already exists at the remote, you may need to
hint: run "git fetch" to retrieve it.
hint:
hint: If you are planning to push out a new local branch that
hint: will track its remote counterpart, you may want to use
hint: "git push -u" to set the upstream config as you push.

How to fetch a remote branch and start tracking it:

git fetch git@gitlab.com:lino-framework/lino blurry:blurry

How to see whether my branches are remote:

git branch -vv

Situation: I am on master. Sharif created and pushed a new remote branch blurry. To see his work, I do:

git checkout -b blurry origin/blurry

See man git checkout.

Friday, May 21, 2021

Fri, 21 May 2021 00:00:00 +0000

Friday, May 21, 2021¶

After several weeks of heavily reviewing the Lino documentation in general, I am trying to calm down. The job is not finished, but I am tired of writing about Lino. I prefer to write free software than to write about it. But before focussing back to the real things, I want to get the CI pipelines on GitLab green again.

The documentation about Lino is currently stored in at least five repositories, and their URLs are automatically added as intersphinx targets to every doctree that uses lino.sphinxcontrib.configure():

im['lf'] = ('https://www.lino-framework.org/', None)
im['cg'] = ('https://community.lino-framework.org/', None)
im['ug'] = ('https://using.lino-framework.org/', None)
im['hg'] = ('https://hosting.lino-framework.org/', None)
im['book'] = ('https://dev.lino-framework.org/', None)

These URLs are duplicated in another file, docs/shared/include/part_of_lino.rst, which adds my blog.

The Rumma & Ko company website is not part of the Lino docs because our long-term vision is to create a About our name who governs Lino while Rumma & Ko would just be one development provider among many others.

I added a substitution definition for the button, which the React front end uses for the action that “liberates a slave”, i.e. opens a slave panel in its own data window.

Oops, the pi-button class used by the button (and all other icons from PrimeIcons) was in the linodocs.css file, which I removed some days ago. I resurrected it in a new atelier.css, which comes automatically with atelier.

voga1r demo : cannot enroll existing client in existing activity¶

Main problem is that it doesn’t work in the online demo. Not reproducible in current roger demo.

Another problem is usability: users must guess that they need to go to the Activities tab and then click the [external-link] button of the “Enrolments in Activities of Hans Altenberg (MEL) (Also Cancelled)” panel where they have an “Insert” button. The fix to this problem is probably straightforward: just say “summary” or “html” instead of “grid” for the courses.EnrolmentsByPupil slave table.

display_mode “html” is good, “summary” isn’t what they want. But the html display mode is not beautiful in React. I will create another ticket for that.

Another thing to do : Voga needs a user type “Participant”, and a participant should see the courses to which they are enrolled, should be able to apply for enrolment in a course, stop an enrolment, manage their presences, …

About pronto, mercato and ciao¶

I moved pronto from GitHub to GitLab. One of the last repos that needed to move (Moving from GitHub to GitLab). The pronto project is sleeping doesn’t contain much work, but it might wake up one day, and it is part of the known contributor repositories installed by getlino. But the mercato and ciao projects didn’t contain any significant work and I started to remove them: getlino actually never knew them. I deleted them on GitHub.

Thursday, May 20, 2021

Thu, 20 May 2021 00:00:00 +0000

Thursday, May 20, 2021¶

Lino at the HackerX event¶

Tonight I will attend HackerX, “the world’s largest invite-only networking & recruiting event for developers”. It is an online event where developers can enter for free because they are actually what is being sold to the customers. The customers are companies in search of new human resources. I probably don’t fit into their standard product pattern but they did let me in. Nice.

Here is my self-presentation:

Developer skills: Linux, Python, Django, Sphinx, Git, React. Long-term experience in accounting, team management, sales agreements, website authoring.

I am not only a Senior Developer, I am also CEO and one of the two owners of Rumma & Ko. I started employing other developers in 2015 (see my team page).

I am the author and maintainer of the Lino framework. I have been working on this vision my whole life. I am not going to stop working on it.

I attend to HackerX in the hope of finding my future boss and employer, an existing legal person who would take over our company and bring Lino to the next level.

So I am a job seeker with special conditions: you need to convince me that you believe in the Lino vision and will continue collaborating with our existing customers and employees.

(Edit: I later used above thoughts for my Hire me page.)

I explained “What is Lino?” in 5 minutes to about 15 humans. It was a good exercise and a vitalizing evening. As expected I didn’t find any partner, which just confirms that private corporations are no candidate for Lino.

Tweaking the docs about Lino¶

I am having the minor issue that inv bd writes all the pages each time again and again, even when I changed only one word in one page of my blog.

The rstgen.sphinxconf.configure() now adds the ‘sphinx.ext.autodoc’ and ‘sphinx.ext.autosummary’ extensions only when the project’s SETUP_INFO contains a ‘name’ key. It’s a good thing to avoid loading autodoc when it is not needed, but this didn’t fix my problem.

One visitor recommended to use plain black as font color. I fully disabled the linodocs.css file added by lino.sphinxcontrib.logo. If it turns out that we still want some of the features in this file, we should anyway reimplement them in a more flexible way. At the moment we have a copy of linodocs.css in every doctree.

Simulate a HTTPS server on your development machine¶

Sharif is testing desktop notifications. We installed a simulated production server on his machine, and I wrote a Howto page for this in the Contributor Guide: Simulate a HTTPS server on your development machine.

Tuesday, May 18, 2021

Tue, 18 May 2021 00:00:00 +0000

Tuesday, May 18, 2021¶

The docs are ugly on a desktop screen because the pydata theme tries to display the top-level menu items in the top navigation bar.

pydata-theme works only well when the top-level page titles are very short. I read the docs and played around in both theme configurations, but finally repented back to the insipid theme.

Sustainability of OSS Communities¶

I attended to an online webinar organized by OSOR about their document “Guidelines for Sustainable Open Source Communities in the Public Sector”.

Have a look at Joinup to discover the OSOR resources:

https://joinup.ec.europa.eu/collection/open-source-observatory-osor/knowledge-centre

The Guidelines for creating sustainable open source communities are accessible here:

https://joinup.ec.europa.eu/collection/open-source-observatory-osor/guidelines-creating-sustainable-open-source-communities

The document is based on five case studies:

CONSUL platform for participatory democracy developed in in Groningen.
Integreat. New arrivals’ integration in Germany.
Developers Italia, an online platform to discuss ongoing or future projects.
Lutece in France. A CMS developed and maintained by City of Paris.
OSKARI framework for integrating geospatial data into web sites.

Chloé Dussutour (OSOR Team) asked us to share our experience with them by providing feedback on the Guidelines:

https://joinup.ec.europa.eu/collection/open-source-observatory-osor/news/we-want-hear-your-feedback-sustainability-guidelines

I said that I am surprised that the FSFE and its Public Money Public Code campaign is not mentioned in the Guidelines. Answer: PMPC has a clear political direction, while the OSOR is a public administration and as such must remain neutral.

Which does not mean that there is no communication. For example here is a FOSDEM 2021 interview by Alexander Sander (FSFE) and Evangelos Tsavalopoulos about the European Commission Open Source Strategy:

https://www.youtube.com/watch?v=UB9Wga-AaHk

The FSFE is pleased that the EU Commission’s Open Source Strategy “recognises the benefits of Free Software and the four freedoms to use, study, share and improve”, but in the same sentence adds that “the Commission lacks concrete targets and indicators to implement the strategy.” (source).

EU Commissioner Johannes Hahn published a clear political statement on ‘Public Money? Public Code!’:

https://www.youtube.com/watch?v=h3SK96jLTSU

During the meeting I also wrote the following to the Jitsi chat of the event:

Thanks for this event. I would like to launch a call for help here. I enrolled to this webinar in the hope of finding my future boss. I am currently the CEO of a family-owned company in Estonia. We work sustainably with a dozen of customers, two developers and one system administrator. Our biggest customers are three local public administrations in Belgium. We believe that Lino would be useful for many other organizations. We did the legal steps to publish it as Free Software under the AGPL. But I am a software developer, not a manager. My vocation is to write Lino, not to build up a clear governance for its community.

https://www.lino-framework.org/vision.html.html

https://www.lino-framework.org/framework.html.html

If you feel that you should contact me, please don’t hesitate. https://www.saffre-rumma.net/team/luc/

Monday, May 17, 2021

Mon, 17 May 2021 00:00:00 +0000

Monday, May 17, 2021¶

The front page (https://www.lino-framework.org) is now multilingual.

SR is now also considered part of the Lino documentation framework. Testing linkable pages Hamza Khchine, Luc Saffre.

I changed the sphinx theme from insipid to pydata. Mostly because insipid had two disturbing oddities on a mobile phone: the sidebar menu didn’t hide away after selecting another page from the menu. And in some situations the font weight of a whole page changed when you first touched it.

Note that when I refer to Lino is a framework via intersphinx, I should get always the English version.

I did an update of the demos on The Lino framework:

pull.sh
go cosi1e ; pm prep --noinput
go noi1r ; pm prep --noinput
go voga1e ; pm prep --noinput
sudo service supervisor restart

(Before this I removed some unused repositories from the pull.sh script, which was as generated by getlino : the Lino installer. But e.g. we don’t need book, welfare and some other repositories there.)

Sharif and I had a 2,5 hours extreme programming session where we did an extensive reorganization of the site_data for #4102. That was fun!

Sunday, May 16, 2021

Sun, 16 May 2021 00:00:00 +0000

Sunday, May 16, 2021¶

The file__thumbnail in FileUsage.detail_layout doesn’t seem to work. Why?

Sharif submitted a solution, and this solution didn’t cause any test cases to break. But it would have caused a problem with a remote field to a virtual ForeignKey. I was afraid that there are uncovered cases where his change would break things. So I started to dig into it: Using remote virtual fields in a detail view. Result: yes, we do have remote fields to virtual ForeignKey fields, but until now we did never use them in any detail window. And the issue occurred only there. I verified also for Lino Welfare (added a list of all remote virtual fields in The Lino Welfare “Eupen” variant)

Minor internal changes:

lino.core.store.StoreField.value2dict() now takes the field name as an argument. The reason for this is a hack in lino.core.store.create_atomizer() where we create a StoreField instance based on the leaf field and then patch some methods of it. A StoreField actually doesn’t know its name. (TODO: shouldn’t we remove the StoreField.name attribute altogether?)

I wrote a new utility function lino.api.doctest.walk_store_fields().

I thought “Oops, the actors users.My, ledger.ItemsByVoucher, ledger.ByJournal, ledger.PrintableByJournal and sheets.EntriesByReport were not declared as abstract”.

But that was a pitfall. They don’t need to be explicitly declared abstract because Lino knows it: They are subclasses of DbTable having their model at None. I had forgotten the Actor.is_abstract() method. The problem was that lino.api.doctest.walk_store_fields() filtered on Actor.abstract instead of calling Actor.is_abstract(). lino.utils.diag.Analyzer.get_complexity_factors() now uses Actor.is_abstract()

Saturday, May 15, 2021

Sat, 15 May 2021 00:00:00 +0000

Saturday, May 15, 2021¶

Sharif and I had another video brainstorming meeting.

TODO:

Accept all kinds of media files, not only images.
Selecting a Media file in Uploads.insert should show thumbnails of the images- Define a custom chooser for FileUsage.file (the FK field pointing to existing File). Maybe add a new method Model.get_text_for_value()

Model.choice_layout = “””name thumbnail”””
A method FileUsage.get_file_display() would not be optimal. This feature is not tested and not used anywhere so far. But it is standard Django behaviour.
Define a __str__() method for FileUsage (which is used in different places, e.g. when setting display_mode of UsagesByController to summary.
The buttons to toggle between display modes should have a tooltip.
Tooltips (and other texts in the react js code) should be translatable.
The file__thumbnail in FileUsage.detail_layout doesn’t seem to work. Why?
Reactivate lino.core.site.Site.use_websockets : the basic feature is that Lino would ask the client’s browser permission for notifications. And then the user would get notified as a desktop notification. notify with use_websockets is not used anywhere, and would require a revision Note that the current use_websockets was written before Django 3. See https://docs.djangoproject.com/en/6.0/topics/async/
In Lino Shop unverified users should not have permission to place an order. But they can collect products into their basket.
When lino.modlib.users.use_verify is True and the user is not verified, then Lino should give a welcome message “You are not yet verified. Check your inbox. Or click here to re-sent a verification code.”
Add phone verification. Testing this requires a SMS sending service.

DONE:

Added a new plugin option notify.mark_seen_when_sent When this is True, Lino now marks notification messages as seen when they have been sent via email.
The default implementation of Actor.get_table_summary() now also uses Request.plain_toolbar_buttons(). The difference is that if an actor has some custom action that has show_in_plain = True, this action would also be rendered now below a summary. The currently only use case of this is the “Mark all as seen” action of lino.modlib.notify.Message
About method Model.as_paragraph() : this approach has the disadvantage that HTML is being rendered on the server, not on the client. More data flow, and more server resources.
Use cases for grid_layout: products
We added a detail layout to lino_xl.lib.albums.FileUsages.

We changed the settings.py on jane:

def get_plugin_configs(self):
    yield super(Site, self).get_plugin_configs()
    # example of local plugin settings:
    yield ('users', 'use_verify', True)
    yield ('notify', 'keep_unseen', False)
    yield ('notify', 'remove_after', 24*7)

How to restore a Borg backup to my computer¶

Wednesday, May 12, 2021

Wed, 12 May 2021 00:00:00 +0000

Wednesday, May 12, 2021¶

Some revolutions in the docs: the Developer Guide (docs of the book repository) is now published under a new subdomain https://dev.lino-framework.org/, and I moved the chapters about Lino from the Community Guide (cg) to a new repository “lf”, which is now served at www.lino-framework.org. Th cg doctree no longer explains what Lino is (this is now done in lf), it focuses on how to interact with the Lino community. There are probably still some links that I forgot to update. Let’s hope that this is yet another step towards better documentation.

At the moment all these main doctrees (lf, cg, ug, hg and book) happily do circular references to each other. My first strategy was to have a strict hierarchy (lf <– cg <– ug <– hg <– book), but I am not sure whether this is possible. And until now it doesn’t disturb very much. It can happen that you get warnings about missing references when you do changes in several doctrees at once. Or I guess that I would have a problem when I have no internet connection and then say pp inv clean -b followed by pp inv bd.

Sharif did good work for #4149. This was his first “chirurgical intervention” in the heart of Lino. Of course it caused quite some detail devils and side effects. We had some fun (re)discovering cool things. And finally, the new feature is there, and it was needed: the new Actor.hide_editing() method makes it possible to say that ProductsStaff can edit products while a normal ProductsUser can’t.

We also had some dialogue about how to organize the documentation. One conclusion maybe: we can’t remove all the docstrings in a core module like lino.core.actors. But the docstrings should be rather short and refer to a topics page.

We used this to write documentation about the new Actor.hide_editing() method. Also Actor.hide_navigator and Actor.editable.

Note that lino.core.actions.Action.hide_editing() is an instance method.

Today was chaotic: Many changes in different places. Once more I used inv ci, and the commit messages just refer to this blog page.

Tuesday, May 11, 2021

Tue, 11 May 2021 00:00:00 +0000

Tuesday, May 11, 2021¶

I optimized the docs about copyright on Legal considerations for contributors and ss.free.

We can be really thankful for Sharif. Thanks to him the React front end is again “in good hands” (as Tonis expressed it). During the last weeks Sharif has gained deep knowledge of the Lino internals, together we have very efficient extreme programming sessions, he also actively contributes issue reports to the PrimeReact community. For his reporting of issue 2008 he earned thanks from Mert Sincan who wrote “Good catch! Thanks a lot for your report!”. And last but not least, the Lino Shop project is advancing.

I start to imagine that we will soon get yet another doctree. The Developer Guide should rather be published under a subdomain like dev.lino-framework.org and the main front page at www.lino-framework.org should rather be an independant doctree that serves as entry point into the other doctrees. It would basically say what currently is said on lino.doctrees.

Monday, May 10, 2021

Mon, 10 May 2021 00:00:00 +0000

Monday, May 10, 2021¶

Actor.display_mode

Action.window_type

opens_a_window is obsolete. opens_a_window is True when window_type is not None.

T data window
P dialog window (enter the parameters of an action)
I insert window (a special dialog window of the insert action)
U unbound window (e.g. the about window) EmptyTable

For data windows we also have the display modes:

T tabular view (like a spreadsheet) D detail view (one row at a time) L list view C cards view S summary view G gallery view H plain tabular view (not editable, ‘html’)

callable_from

available display modes

hide_display_modes: = ‘tdlcsgh’ = ‘tlcsgh’

Sunday, May 9, 2021

Sun, 09 May 2021 00:00:00 +0000

Sunday, May 9, 2021¶

I started the Howto pages section of the User Guide. That made me discover #4148, which I fixed. Later I wrote new content (or moved it from the book docs) in The accounting plugin, The vat plugin and Howto pages.

The docs/shared/include/defs.rst still contained a series of URLs that referred to github instead of gitlab.

Sharif asked a good question. Example: in Lino Shop a customer can see (view) the tables of products and use the filter parameters (which will be very complex) and the feature of sorting product lists by price or something else, so we clearly want a customer to see the products.Products table and its children, but this shouldn’t mean that a customer is also allowed to edit anything in these products. To make this possible, we changed the API: Actors no longer have an attribute editable, but a method is_row_editable() (which is a class method on the actor, and the default implementation calls an instance method of same name on the model). The application developer can now specify that some model or actor generally refuses editing for a whole row based on the user’s role(s).

Note that even when is_row_editable returns False, we need to loop over the names in disabled_fields because they also contain the actions. For example we don’t want the add_to_cart action to be disabled for customers. IOW when is_row_editable() return False, lino.core.store.DisabledFieldsStoreField should simply add all fields to the set returned by the actor’s get_disabled_fields().

TODO:

# core.fields.TableRow
def is_row_editable(self, ar):
    return True

# Actor
@classmethod
def is_row_editable(cls, obj, ar):
    return obj.is_row_editable(ar)

# products.Product
def is_row_editable(self, ar):
    if not ar.get_user().user_type.has_required_roles([ProductsStaff]):
        return False
    return True

As a side effect of planning above changes, we renamed make_view_permission_handler() to make_static_permission_handler() because this expresses more clearly what it does. It is static, not necessarily only for views. Though we use it only for getting the view permission. All other permissions are “dynamic”. i.e. they get a table row for deciding whether permission is given or not.

We investigated into #4150 (When creating a Furniture, Lino shows also Thriller as category)

I created #4151 (Cannot upload new image from product detail).

I fixed two “trivial” failures in the book, both were caused by minor “quick wins” I had done recently:

The doctest docs/specs/avanti/uploads.rst failed because MyExpiringUploads.description_link now uses the __str__() method instead of doing almost exactly the same. There was a subtle difference, though. Probably not important. I just adapted the expected result.

The roger project, which installs the publisher plugin, now has one pluing more because publisher now automatically installs bootstrap3 as well. This combination is not yet used in production, and it might go away again in case we decide to deprecate publisher. Until then I just adapted the expected result.

Saturday, May 8, 2021

Sat, 08 May 2021 00:00:00 +0000

Saturday, May 8, 2021¶

Sharif and I have done two sessions for Lino Shop. It is advancing. We started with -probably- the last fundamental question: how to render products for normal customers. We tried using lino.modlib.publisher but this is not what we need for webshop.

It’s even possible that we will deprecate lino.modlib.publisher altogether. Even in lino_book.projects.cms it yields poor results. Partly because lino_book.projects.cms is just a submarine project and because it is quite crazy to believe that a CMS in Lino has a chance on the market. But the CMS project inn’t dead, it’s just sleeping. Maybe one day it will wake up with lino_xl.lib.albums instead of lino.modlib.publisher… It was broken for some trivial reason, and this wasn’t covered by the test suite. I added it to the test_demo.py file.

Wednesday, May 5, 2021

Wed, 05 May 2021 00:00:00 +0000

Wednesday, May 5, 2021¶

I worked several hours in the cg. I am still not satisfied but it’s getting better.

The term Lino developer is too general. It’s better to say application developer, contributing developer or core developer.

Note that core developer doesn’t mean somebody who cares only about the Lino core.

I replaced all occurrences of “site manager” by site manager.

For the first time in my life I did a mass commit:

$ pp -d git ci -am "replace github by gitlab in requirements.txt"
Successfully terminated `git ci -am replace github by gitlab in requirements.txt` for all projects

(This was after https://gitlab.com/lino-framework/shop/-/jobs/1236797299)

The doctest docs/dev/redirect.rst in Developer Guide is currently failing because Hannes is working on the redirect (#4133). This test as such is a good thing, but it is at the wrong place. It should not be part of the book project. The book shouldn’t break just because one of our URLs is broken. It is a form of monitoring and should run more often, and it should notify a list of responsible administrators. Is this a job for Ansible?

Tuesday, May 4, 2021

Tue, 04 May 2021 00:00:00 +0000

Tuesday, May 4, 2021¶

Monday, May 3, 2021

Mon, 03 May 2021 00:00:00 +0000

Monday, May 3, 2021¶

Online registration for Lino Shop¶

This week-end I worked, partly with Sharif, on #4143 (Review handling of the User.partner field), which was triggered by #4121, which we need for Lino Shop. But it is about online registration, a general feature on which we had previously done some work, which went to sleep with the Lino Care project. Now is the moment to wake it up. I also did #2049 (Users don’t need to be partners in Noi) because it is part of the problem. I am now sure that we want to get rid of lino_xl.lib.online.users.

I moved some functionality from lino_xl.lib.online.users to lino.modlib.users:

New plugin option use_verify and User.verify action.
The submit_insert action now always does a nice warning when a username is already taken.
The User.send_welcome_email action is no longer a window action. We don’t want any question about recipient or subject of the mail.

When the User.partner field is empty, and all three fields first_name, last_name and email are non-empty, (and if the user is verified on a site with use_verify set to True), then Lino now automatically creates a contacts.Person based on this data. This automatism does not apply to SiteAdmin users who can edit the partner field manually.

As a side effect I had to review lino_noi.lib.users.UserDetail.

One advantage of not using lino_xl.lib.online.users any more: In Lino Shop we can have a same person with two user accounts: one as a lino_shop.lib.shop.UserTypes.customer and one as a lino_shop.lib.shop.UserTypes.vendor.

These changes caused (trivial) failures in quite some doctest suites because we have now one more view (about.DateFormats), one window action less and one more field in User.

Data migration will be funny: In noi we no longer use online.users: User is no longer a subclass of Person. Might require some magic when migrating production data.

New user role `LedgerPartner`¶

In lino_xl.lib.accounting I added a new role LedgerPartner, and lino_shop.lib.shop.UserTypes.customer and lino_shop.lib.shop.UserTypes.vendor already inherit from it. Next step is to add data tables like MyInvoices and MyPayments

Configuring how to represent dates¶

I added a new field User.date_format. It is there for all applications. It is not yet being used, but lino_react is ready to implement it. And lino.modlib.extjs might follow one day.

The Hydra effect¶

Note: The Hydra effect is when you have the impression of not advancing. You cut down one head of the monster and it grows seven new ones. But Lino is like a Hydra, except that it gets more beautiful (not more dangerous) each time you fix a bug. Working on a beast like Lino makes sense only because it is free software.

The documentation framework, even taken on its own, is a full-featured Hydra. But is getting better with each iteration. Now we have a page about it: lino.doctrees.

Some bug in the React text editor¶

I created a comment and then pasted some text into it. This caused my browser page to lock up. Firefox asked me “This page is slowing down your browser. Do you want to stop it?” After stopping it, I had the following messages in the JS console:

NWS App.js:1097:15
request aborted DOMException: The operation was aborted. DataProvider.js:117:19
Use of Mutation Events is deprecated. Use MutationObserver instead. main.js line 3530 > eval:5511:24
Script terminated by timeout at:
commitMutationEffects@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:3945:214
callCallback@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:657:119
invokeGuardedCallbackDev@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:677:45
invokeGuardedCallback@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:696:126
commitRootImpl@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:3908:49
unstable_runWithPriority@webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465:16
runWithPriority$1@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2063:1059
commitRoot@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:3877:526
performSyncWorkOnRoot@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:3753:107
flushSyncCallbackQueueImpl/<@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2069:188
unstable_runWithPriority@webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465:16
runWithPriority$1@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2063:1059
flushSyncCallbackQueueImpl@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2069:95
flushSyncCallbackQueue@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2068:339
scheduleUpdateOnFiber@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:3631:20
enqueueSetState@webpack:///./node_modules/react-dom/cjs/react-dom.development.js?:2195:489
Component.prototype.setState@webpack:///./node_modules/react/cjs/react.development.js?:358:20
onTextChange@webpack:///./lino_react/react/components/TextFieldElement.js?:120:12
componentDidMount/</<@webpack:///./node_modules/primereact/components/editor/Editor.js?:282:27
emit@webpack:///./node_modules/quill/dist/quill.js?:11207:33
emit@webpack:///./node_modules/quill/dist/quill.js?:2526:105
modify@webpack:///./node_modules/quill/dist/quill.js?:2089:45
updateContents@webpack:///./node_modules/quill/dist/quill.js?:1959:27
onPaste/<@webpack:///./node_modules/quill/dist/quill.js?:11622:28
main.js line 3818 > eval:3945:214
The above error occurred in the <Tooltip> component:

Tooltip@webpack:///./node_modules/primereact/components/tooltip/Tooltip.js?:291:20

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://reactjs.org/link/error-boundaries to learn more about error boundaries. react-dom.development.js:3308:9
    React 10
    unstable_runWithPriority webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465
    React 4
    unstable_runWithPriority webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465
    React 6
    onTextChange webpack:///./lino_react/react/components/TextFieldElement.js?:120
    componentDidMount webpack:///./node_modules/primereact/components/editor/Editor.js?:282
    emit webpack:///./node_modules/quill/dist/quill.js?:11207
    emit webpack:///./node_modules/quill/dist/quill.js?:2526
    modify webpack:///./node_modules/quill/dist/quill.js?:2089
    updateContents webpack:///./node_modules/quill/dist/quill.js?:1959
    onPaste webpack:///./node_modules/quill/dist/quill.js?:11622
Uncaught Error: An error was thrown inside one of your components, but React doesn't know what it was. This is likely due to browser flakiness. React does its best to preserve the "Pause on exceptions" behavior of the DevTools, which requires some DEV-mode only tricks. It's possible that these don't work in your browser. Try triggering the error in production mode, or switching to a modern browser. If you suspect that this is actually an issue with React, please file an issue.
    React 3
    unstable_runWithPriority webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465
    React 4
    unstable_runWithPriority webpack:///./node_modules/scheduler/cjs/scheduler.development.js?:465
    React 6
    onTextChange webpack:///./lino_react/react/components/TextFieldElement.js?:120
    componentDidMount webpack:///./node_modules/primereact/components/editor/Editor.js?:282
    emit webpack:///./node_modules/quill/dist/quill.js?:11207
    emit webpack:///./node_modules/quill/dist/quill.js?:2526
    modify webpack:///./node_modules/quill/dist/quill.js?:2089
    updateContents webpack:///./node_modules/quill/dist/quill.js?:1959
    onPaste webpack:///./node_modules/quill/dist/quill.js?:11622
main.js line 3818 > eval:678:7

But I was not yet able to systematically reproduce this problem.

Testing online registration ¶

For testing online registration feature, we now have a new function lino.core.auth.utils.activate_social_auth_testing(). These “public secrets” are used for testing social authentication on a localhost. Until now they were available only in noi1e (and noi1r which imports the settings from noi1e). But now they are used also for lino_shop.projects.shop1

I started to write some content for Lino Shop User’s Guide.

Database migration on Jane¶

Today I did two releases to Jane. The first went very smoothly, despite my expectations. I commented:

Wow, despite the fact that I wrote it myself, Lino will always surprise me. It did the data migration without any need to manually change anything in the restore.py file. I am surprised.

And this was only the first surprise. It got even more surprising. I then noticed that the lino.mixins.polymorphic.Polymorphic.mti_navigator field was still there on the Person detail. Explanation: for some reason my inv sdist release in Lino Noi hadn’t worked. PyPI still returned the old version of noi. And only for noi, not for lino, xl and react.

It is interesting to see that the old noi version had been working well together with the new lino and xl.

After releasing noi to pypi for good and running the pull.sh script again, my restore.py finally gave the expected problem:

django.core.exceptions.ValidationError: ['A Person cannot be parent for a User']

My first guess is to simply not create any Partner row for the existing users. Because this probably reflects

def create_users_user(person_ptr_id, modified, created, start_date, end_date, password, last_login, username, user_typ$
    kw = dict()
    kw.update(modified=modified)
    kw.update(created=created)
    kw.update(start_date=start_date)
    kw.update(end_date=end_date)
    kw.update(password=password)
    kw.update(last_login=last_login)
    kw.update(username=username)
    kw.update(user_type=user_type)
    kw.update(initials=initials)
    kw.update(id=person_ptr_id)  # ADDED
    # REMOVED kw.update(partner_id=partner_id)
    kw.update(verification_code=verification_code)
    kw.update(time_zone=time_zone)
    kw.update(date_format=date_format)
    # REMOVED kw.update(callme_mode=callme_mode)
    # REMOVED kw.update(user_state=user_state)
    kw.update(access_class=access_class)
    kw.update(event_type_id=event_type_id)
    kw.update(open_session_on_new_ticket=open_session_on_new_ticket)
    kw.update(notify_myself=notify_myself)
    kw.update(mail_mode=mail_mode)
    kw.update(github_username=github_username)
    # REMOVED return create_mti_child(contacts_Person, person_ptr_id, users_User, **kw)
    return users_User(**kw)  # ADDED

Loading 23880 objects to table changes_change...
Deferred Change #11704 ('#11704') : {'user': ['User instance with id 324 does not exist.']}
{'user': ['User instance with id 324 does not exist.']}
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/utils/dpy.py", line 159, in try_save
    obj.full_clean()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/base.py", line 1238, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'user': ['User instance with id 324 does not exist.']}
Deferred Change #14221 ('#14221') : {'user': ['User instance with id 328 does not exist.']}
Deferred Change #15241 ('#15241') : {'user': ['User instance with id 333 does not exist.']}

Deferred Comment #1446 ('Comment #1446') : {'user': ['User instance with id 324 does not exist.']}
{'user': ['User instance with id 324 does not exist.']}
Traceback (most recent call last):
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/lino/utils/dpy.py", line 159, in try_save
    obj.full_clean()
  File "/usr/local/lino/lino_local/jane/env/lib/python3.7/site-packages/django/db/models/base.py", line 1238, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'user': ['User instance with id 324 does not exist.']}

::: Trying to save 2773 deferred objects.

kw.update(id=person_ptr_id) # new

324 : kader 328 : thierry

Sunday, May 2, 2021

Sun, 02 May 2021 00:00:00 +0000

Sunday, May 2, 2021¶

Search for text .partner in voga/**/fixtures/*.py.

Saturday, May 1, 2021

Sat, 01 May 2021 00:00:00 +0000

Saturday, May 1, 2021¶

Friday, April 30, 2021

Fri, 30 Apr 2021 00:00:00 +0000

Friday, April 30, 2021¶

I found yet another missing piece: we must differentiate between the Community Guide (cg) and the User Guide (ug). I now created the latter. It is hosted on https://using.lino-framework.org and the source code is on https://gitlab.com/lino-framework/ug.

The Plugin reference for end users section has already four chapters! And I didn’t write these now, I copied them from Reference and then removed on both sides the partns that were at the wrong place. I have the impression that everything is much more readable now. My first steps seem to confirm this move.

The cg and the Lino User Guide are similar in that they both are docs-only repositories (they don’t define any Python package) and are translated. They differ because of the target audience: cg is for people who do not yet use Lino (future site operators, hosting providers or application developers) while the Lino User Guide is for end users needing help.

Where to put the parameter panel¶

In ExtJS the parameter panel is an “inline” panel that is “docked” into the window together with the main element. The end user can then expand or collapse it. In React it was a popup window. That was also nice.

We added a new Actor attribute lino.core.actors.Actor.params_panel_pos.

Thursday, April 29, 2021

Thu, 29 Apr 2021 00:00:00 +0000

Thursday, April 29, 2021¶

Improving the documentation about Lino¶

I continued my dive into the documentation framework.

My next avalanche of changes is rather in the docs themselves. That’s a good sign: the documentation framework starts to meet my expectations, I can start to care about the content.

We need a common standard for structuring the docs and reduce redundant explanations.

For example when I write application-specific end-user docs, I often use words like detail window or grid window. And I don’t want to explain these words over and over again.

I moved the “User Guide” section from book/docs to cg/docs where they are called “Using Lino”. That was needed because cg is for non-technical end-user documentation that will be translated. EDIT: the next morning I continued this move, see Friday, April 30, 2021.

I have a feeling that the title of the book doctree will change from “The Lino Book” to “Lino developer docs”, and the cg will rather get a title like “The Lino Book”.

Isn’t it a pity that we cannot use the demo projects of the book for running doctests in other repositories as well? Answer after some trying: no. For example in The users plugin it seems to me that it would make sense to add tested code snippets.

The fundamental difference between book and cg is their audience. book is meant for developers. We do not want to maintain translations for it.

About intersphinx dependencies. The book (quite often) refers to something that is explained in the cg. But at the moment we still have references in the other direction as well (cg referring to book). This is not good because it can cause a deadlock. The end-user docs should indeed really be implementation-agnostic. But Lino should have Help buttons that open links to the technical reference.

Testing whether references work: lino.modlib.printing.PrintableContext.this.

Release to Jane¶

Sharif and I did an upgrade on Jane.

We found and fixed a little bug: lino_react did not yet automatically get installed via install.

Note that mysqlclient is not yet being automatically installed by install. Only getlino knows the python package to be used for a given Django dbengine.

Wednesday, April 28, 2021

Wed, 28 Apr 2021 00:00:00 +0000

Wednesday, April 28, 2021¶

I changed the URL of the German and French doctrees about welfare. Until now they were:

https://de.welfare.lino-framework.org
https://fr.welfare.lino-framework.org

New URLs are:

https://welfare.lino-framework.org/de
https://welfare.lino-framework.org/fr

TODO: Adapt the nginx config files on LF for subdomains de.welfare and fr.welfare to add a rewrite rule so that requests to the old URL redirect to the new URL. Adapt the Redirected historic URLs page so that it verifies that the new content is being served.

Tuesday, April 27, 2021

Tue, 27 Apr 2021 00:00:00 +0000

Tuesday, April 27, 2021¶

I continued with my review of the “documentation network” (see Monday, April 26, 2021).

Today’s most visible result is a new documentation page List of known Lino applications, which is automatically generated when I run inv bd in the cg because this repository uses the new make_docs_command setting, which calls the new command getlino list.

I reviewed the description, url, author and author_email fields in all setup_info.py files.

In every repository I changed author to “Rumma & Ko Ltd” and author_email to “info@lino-framework.org”.

I have changes in almost 40 repositories. And tthey all have the same commit message “https://luc.lino-framework.org/blog/2021/0427.html”. (I don’t recommend my exotic practice of referring to a blog entry in a commit message… but today I would have taken half an hour more for these commits without it).

Monday, April 26, 2021

Mon, 26 Apr 2021 00:00:00 +0000

Monday, April 26, 2021¶

I did more changes in the “documentation network”. My concrete goal for today is that the docs about the applications we declare “ready to use” and for which we offer hosting service would become “good enough”. Where “good enough” means that a potential customer who found our page Lino for all page would get a reasonable answer to their obvious question “What can these applications do? Where are the functional specifications?”.

A part of this project was the question whether we should move the specs of the privileged apps from the book to their individual docs. Answer (after several hours of research) is “no”. A step forwards today was that we no longer need to define the public_url of each doctree at two different places. It is now given by the intersphinx_urls setting.

I moved the support for selectable_languages from lino to atelier, including the default_conf.py and the templates files.

Another “result” is that I now know: Sphinx does not have a built-in way to share include files across multiple doctrees. For bigger systems these things would be regulated using a content delivery network. As a little step forward, I now wrote a file update_shared_docs.sh, which does for noi what I did manually until now.

A third result of today is the Redirected historic URLs page, which should make sure that historic URLs are correctly redirected.

Sunday, April 25, 2021

Sun, 25 Apr 2021 00:00:00 +0000

Sunday, April 25, 2021¶

The docs about Lino Tera were still missing after the fire at OVH. Now I restored them, but they are multilingual (en, de), and I did not want to have the German docs under their own domain name. Now they are using the same method as for www.saffre-rumma.net, where the German doctree is just a subdirectory of the English one.

Testing whether intersphinx_urls work: Lino Tera and Lino Tera Benutzerhandbuch.

Tera wasn’t the only project with URL problems. Several other projects were not yet being served on LF. For some other projects, e.g. presto, I decided to not serve the presto docs (because I didn’t want to add yet another subdomain on LF for a project that doesn’t yet have impressive docs) on LF and to do it as with shop and mentori: have them hosted only on GitLab pages. Same for react. That’s why I also moved these repositories from GitHub to GitLab. And two others as well. Contributors must update the remotes of their clones, as explained in Moving from GitHub to GitLab.

presto is the first project with multilingual docs that is rendered only on GitLab pages.

Friday, April 23, 2021

Fri, 23 Apr 2021 00:00:00 +0000

Friday, April 23, 2021¶

The new `BabelPreviewable` model mixin¶

For the Lino Shop project we realized that the description fields of categories and products should be previewable. Since they are also babel fields, I wrote a new mixin lino.modlib.memo.BabelPreviewable.

First use case is lino_xl.lib.products, where Product and Category now inherit from it.

In lino.modlib.memo.Previewable I renamed the fields short_preview to body_short_preview and long_preview to body_long_preview, and started to prepare the case that we want to use this mixin for models where the previewable_field is not 'body'.

But I don’t yet use this possibility here. I prefer to change the field name for the description of products and categories from “description” to “body”. Making the name of the “source” field configurable would make things uselessly complicated.

To test whether everything works, we should add, to the body of some product or category, some memo command that renders translatable text.

How should a read-only RichTextField render its content? Should it “execute” the html tags (showing their result), or should it rather “show” them?

I think I made up my mind about read-only RichText fields: let’s remove the dangerouslySetInnerHTML option again. A read-only RichTextField widget should show the “real” source text, not a preview. When we want to see the preview, we have the body_short_preview and body_full_preview fields of the Previewable mixin.

I started to play with the new feature in Lino Shop. There is now a third panel in the product detail. One book now has a body with rich text.

About quicklinks and the dashboard layout¶

Another question is my little spontaneous change yesterday to have the products plugin automatically add quick links for every product type. It causes several unit tests to fail because –as expected– other applications now also have these quick links. This makes me think again. It seems clear that not every application wants these quick links. OTOH it’s as with the [My settings] quick link, which is now a standard: a standard is nice to have, but we want to customize it in different ways. For example I am not sure whether the Lino Welfare users (or their system admins) will be happy with this new [My settings] quick link. Anyway there should be a way to to customize the quick links. The default list should be customizable per application. And then they should –at least for some applications– be customizable per user as well. Quick links become a bit like dashboard items. And the dashboard has some open tickets:

#4046 (Configurable dashboard cards)
#3337 (React-Grid-Layout)

I currently imagine that we introduce a setting dashboard_layout, or more precisely a choicelist DashboardLayouts where users can select from an list of dashboard layouts. The dashboard would become like a detail view where the Site object itself acts as the table row.

Friday, April 16, 2021

Fri, 16 Apr 2021 00:00:00 +0000

Friday, April 16, 2021¶

I finished The mystery of the failing welfare test suite and pushed my work to master. Actually there are still minor things to fix, but I can do this sporadically now. This was for me a week in what I would call the Maigret mode. It’s a mode of working that makes sense only with free software.

Today I learned that Tuxedo recommend to not upgrade Ubuntu from 18.04 to 20.04 on their computers. Wow. That was unexpected. It’s the explanation for the message I have been getting each time I tried to upgrade:

Could not calculate the upgrade. An unresolvable problem occurred while calculating the upgrade. This was likely caused by: Unofficial software packages not provided by Ubuntu. Please use the tool ‘ppa-purge’ from the ppa-purge package to remove software from a Launchpad PPA and try the upgrade again.

Actually that’s okay for me.

The mystery of the failing welfare test suite

Thu, 15 Apr 2021 00:00:00 +0000

The mystery of the failing welfare test suite¶

Thursday, April 15, 2021. Continued from The Mystery of the failing Welfare test suite (continued).

Ticket #4096 (The mystery of the failing welfare test suite) is like finding a function when you are given only its derivative. Or like a detective novel where miss Marple or inspector Maigret try to find out what happened and who is the culprit.

Why does this take so long? Because it’s complex and because every little change needs an pm prep before you can test its effect.

Should I explain the problem to Tonis or Sharif or Hamza? I start to consider asking them. But I am afraid that the mere introduction would take an hour. I’d need to explain terms like aid granting and ISIP. And then the difference between The Châtelet variant of Lino Welfare and The Eupen variant of Lino Welfare. Hamza has the best chances to get into it because he has been working on Lino Welfare. Look at the code of lino_welfare.modlib.welfare.fixtures.demo or lino_welcht.lib.courses.fixtures.demo. There are quite some places where we loop over some existing data and generate another set of data. We use a lot of cyclers. Now imagine what happened with Django 3.2 where the default ordering of lino_welfare.modlib.pcsw.Client changed from id to name.

Is it required to maintain these tests? Can’t we simply throw them away? – Of course it’s required! These tests are the main reason why our customers want Lino and not some industry solution. These tests are the only way to make sure that things don’t change in an unexpected way.

Can’t we reduce the complexity of demo data? – I don’t see how we could do this. We need demo data to test the miscellaneous functions.

I inspected the following fixtures for code that might have changed:

lino_xl.lib.statbel.countries.fixtures.demo
lino_welfare.modlib.contacts.fixtures.demo
lino_welfare.modlib.users.fixtures.demo
lino_welfare.modlib.cal.fixtures.demo : nothing new
lino_welfare.modlib.welfare.fixtures.demo
lino_welcht.lib.cv.fixtures.demo : nothing new
lino_welfare.modlib.integ.fixtures.demo : immersion
lino_welfare.modlib.active_job_search.fixtures.demo : nothing new
lino_welcht.lib.courses.fixtures.demo
lino_welfare.modlib.newcomers.fixtures.demo
lino_welfare.modlib.households.fixtures.demo
lino_xl.lib.humanlinks.fixtures.demo
lino_welfare.modlib.debts.fixtures.demo
lino_welfare.modlib.notes.fixtures.demo
lino_welfare.modlib.aids.fixtures.demo
lino_welfare.modlib.polls.fixtures.demo : some new models
lino_welfare.modlib.esf.fixtures.demo
lino_welcht.lib.pcsw.fixtures.demo2

My general procedure:

Add code snippets to sorting2.rst
go mathieu
pip install -U Django==3.1.8
pm prep
doctest sorting2.rst
pip install -U Django
pm prep
doctest sorting2.rst

I committed my work, which has only one tiny functional change, the Meta.sorting for Client. All other changes are just code cleanup, a side effect of my research work.

The welfare test suite now has 13 failures, but –unlike last week– they are now the same failures with Django 3.1.8 and with Django 3.2:

Loaded 153 doctests from docs
...........................................................................................................F..............F......F......F....FFFFF.FFF...F.
======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/esf.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/esf.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/esf.rst", line 309, in esf.rst
Failed example:
    rt.show(esf.SummariesByClient, obj, language="en")
Expected:
    ====== ======= ======= ======= ====== ====== ====== ====== ====== ====== ====== ====== ========
     Year   S.Inf   E.Ind   E.For   SIS    Tst    Info   Mob    Rem    AN!    MSP    CyE    60§7
    ------ ------- ------- ------- ------ ------ ------ ------ ------ ------ ------ ------ --------
     2012   0:00    0:00    0:00    0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   478:48
     2013   0:00    3:00    0:00    0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00
     2014   0:00    0:00    0:00    1:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00
    ====== ======= ======= ======= ====== ====== ====== ====== ====== ====== ====== ====== ========
    <BLANKLINE>
Got:
    ====== ======= ======= ======= ====== ====== ====== ====== ====== ====== ====== ====== ======
     Year   S.Inf   E.Ind   E.For   SIS    Tst    Info   Mob    Rem    AN!    MSP    CyE    60§7
    ------ ------- ------- ------- ------ ------ ------ ------ ------ ------ ------ ------ ------
     2012   0:00    0:00    0:00    0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00
     2013   0:00    9:00    0:00    0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00
     2014   0:00    11:00   0:00    0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00   0:00
    ====== ======= ======= ======= ====== ====== ====== ====== ====== ====== ====== ====== ======
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of  18 in esf.rst
***Test Failed*** 1 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/tasks.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/tasks.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/tasks.rst", line 39, in tasks.rst
Failed example:
    rt.login('hubert').show(cal.MyTasks)
Expected:
    ========== ============ ============================= ============================= ==========================
     Priorité   Date début   Description brève             Workflow                      Bénéficiaire
    ---------- ------------ ----------------------------- ----------------------------- --------------------------
     Normale    12/06/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   RADERMACHER Hedi (161)
     Normale    27/05/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   RADERMACHER Edgard (157)
    ========== ============ ============================= ============================= ==========================
    <BLANKLINE>
Got:
    Aucun enregistrement
**********************************************************************
File "docs/specs/tasks.rst", line 51, in tasks.rst
Failed example:
    rt.login('alicia').show(cal.MyTasks)
Expected:
    ========== ============ ============================= ============================= ======================
     Priorité   Date début   Description brève             Workflow                      Bénéficiaire
    ---------- ------------ ----------------------------- ----------------------------- ----------------------
     Normale    21/05/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   DA VINCI David (165)
    ========== ============ ============================= ============================= ======================
    <BLANKLINE>
Got:
    Aucun enregistrement
**********************************************************************
File "docs/specs/tasks.rst", line 65, in tasks.rst
Failed example:
    rt.login('alicia').show(cal.MyTasks, param_values=pv)
Expected:
    ========== ============ ============================= ============================= ============================
     Priorité   Date début   Description brève             Workflow                      Bénéficiaire
    ---------- ------------ ----------------------------- ----------------------------- ----------------------------
     Normale    30/03/2015   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   DA VINCI David (165)
     Normale    05/01/2015   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   RADERMACHER Fritz (158)
     Normale    16/12/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   MEESSEN Melissa (147)
     Normale    22/11/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   KAIVERS Karl (141)
     Normale    07/10/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   ENGELS Edgar (129)
     Normale    24/09/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   DUBOIS Robin (179)
     Normale    02/08/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   VAN VEEN Vincent (166)
     Normale    30/06/2014   Permis de travail expire le   **☐ à faire** → [☑] [���] [⚠]   DOBBELSTEIN Dorothée (124)
     Normale    21/05/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   DA VINCI David (165)
    ========== ============ ============================= ============================= ============================
    <BLANKLINE>
Got:
    ========== ============ ============================= ============================= =============================
     Priorité   Date début   Description brève             Workflow                      Bénéficiaire
    ---------- ------------ ----------------------------- ----------------------------- -----------------------------
     Normale    20/01/2015   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   KELLER Karl (178)
     Normale    26/12/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   DA VINCI David (165)
     Normale    15/12/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   HILGERS Hildegard (133)
     Normale    01/12/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   RADERMACHER Christian (155)
     Normale    12/11/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   MALMENDIER Marc (146)
     Normale    23/09/2014   Projet termine dans un mois   **☐ à faire** → [☑] [☒] [⚠]   EMONTS Daniel (128)
     Normale    30/06/2014   Permis de travail expire le   **☐ à faire** → [☑] [☒] [⚠]   DOBBELSTEIN Dorothée (124)
    ========== ============ ============================= ============================= =============================
    <BLANKLINE>
**********************************************************************
1 items had failures:
   3 of   9 in tasks.rst
***Test Failed*** 3 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/reception/index.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/reception/index.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/reception/index.rst", line 379, in index.rst
Failed example:
    pk
Expected:
    <django.db.models.fields.BigAutoField: id>
Got:
    <django.db.models.fields.AutoField: id>
**********************************************************************
1 items had failures:
   1 of  93 in index.rst
***Test Failed*** 1 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/topics/sorting.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/topics/sorting.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/topics/sorting.rst", line 914, in sorting.rst
Failed example:
    for o in jobs.Candidature.objects.all():
       print("{o.id} {o}".format(o=o))  #doctest: +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -71,4 +71,4 @@
     71 Kandidatur von Alfons AUSDEMWALD
     72 Kandidatur von Laurent BASTIAENSEN
    -73 Kandidatur von Charlotte COLLARD
    +73 Kandidatur von Bernd BRECHT
     74 Kandidatur von Marc CHANTRAINE
**********************************************************************
1 items had failures:
   1 of  38 in sorting.rst
***Test Failed*** 1 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/chatelet.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/chatelet.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/chatelet.rst", line 145, in chatelet.rst
Failed example:
    print(analyzer.show_db_overview()) #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF -SKIP
Differences (unified diff with -expected +actual):
    @@ -8,9 +8,9 @@
      aids.AidType                   aids.AidTypes                   23        11
      aids.Category                  aids.Categories                 5         3
    - aids.Granting                  aids.Grantings                  12        55
    - aids.IncomeConfirmation        aids.IncomeConfirmations        17        54
    + aids.Granting                  aids.Grantings                  12        54
    + aids.IncomeConfirmation        aids.IncomeConfirmations        17        52
      aids.RefundConfirmation        aids.RefundConfirmations        18        12
      aids.SimpleConfirmation        aids.SimpleConfirmations        15        19
    - art61.Contract                 art61.Contracts                 32        7
    + art61.Contract                 art61.Contracts                 32        6
      art61.ContractType             art61.ContractTypes             10        1
      badges.Award                   badges.Awards                   6         0
    @@ -18,9 +18,9 @@
      boards.Board                   boards.Boards                   7         3
      boards.Member                  boards.Members                  4         0
    - cal.Calendar                   cal.Calendars                   7         ...
    - cal.Event                      cal.OneEvent                    24        548
    + cal.Calendar                   cal.Calendars                   7         11
    + cal.Event                      cal.OneEvent                    24        535
      cal.EventPolicy                cal.EventPolicies               21        6
      cal.EventType                  cal.EventTypes                  27        13
    - cal.Guest                      cal.Guests                      9         588
    + cal.Guest                      cal.Guests                      9         614
      cal.GuestRole                  cal.GuestRoles                  6         4
      cal.RecurrentEvent             cal.RecurrentEvents             23        15
    @@ -28,5 +28,5 @@
      cal.Room                       cal.Rooms                       10        0
      cal.Subscription               cal.Subscriptions               4         9
    - cal.Task                       cal.Tasks                       20        30
    + cal.Task                       cal.Tasks                       20        26
      calview.DailyPlannerRow        calview.DailyPlannerRows        8         2
      cbss.IdentifyPersonRequest     cbss.IdentifyPersonRequests     21        5
    @@ -60,5 +60,5 @@
      cv.Experience                  cv.Experiences                  18        30
      cv.Function                    cv.Functions                    7         4
    - cv.LanguageKnowledge           cv.LanguageKnowledges           10        114
    + cv.LanguageKnowledge           cv.LanguageKnowledges           10        112
      cv.Obstacle                    cv.Obstacles                    6         20
      cv.ObstacleType                cv.ObstacleTypes                5         4
    @@ -80,5 +80,5 @@
      debts.Group                    debts.Groups                    8         8
      esf.ClientSummary              esf.Summaries                   24        189
    - excerpts.Excerpt               excerpts.Excerpts               12        78
    + excerpts.Excerpt               excerpts.Excerpts               12        79
      excerpts.ExcerptType           excerpts.ExcerptTypes           18        19
      gfks.HelpText                  gfks.HelpTexts                  4         5
    @@ -87,14 +87,14 @@
      households.Type                households.Types                5         6
      humanlinks.Link                humanlinks.Links                4         59
    - immersion.Contract             immersion.Contracts             25        6
    + immersion.Contract             immersion.Contracts             25        8
      immersion.ContractType         immersion.ContractTypes         9         3
      immersion.Goal                 immersion.Goals                 5         4
    - isip.Contract                  isip.Contracts                  24        30
    + isip.Contract                  isip.Contracts                  24        29
      isip.ContractEnding            isip.ContractEndings            6         4
    - isip.ContractPartner           isip.ContractPartners           6         35
    + isip.ContractPartner           isip.ContractPartners           6         33
      isip.ContractType              isip.ContractTypes              11        5
      isip.ExamPolicy                isip.ExamPolicies               21        6
      jobs.Candidature               jobs.Candidatures               10        74
    - jobs.Contract                  jobs.Contracts                  28        13
    + jobs.Contract                  jobs.Contracts                  28        14
      jobs.ContractType              jobs.ContractTypes              10        5
      jobs.Job                       jobs.Jobs                       10        8
    @@ -131,5 +131,5 @@
      polls.Question                 polls.Questions                 9         38
      polls.Response                 polls.Responses                 7         6
    - sessions.Session               users.Sessions                  3         ...
    + sessions.Session               users.Sessions                  3         1
      system.SiteConfig              system.SiteConfigs              29        1
      tinymce.TextFieldTemplate      tinymce.TextFieldTemplates      5         2
**********************************************************************
File "docs/specs/welcht/chatelet.rst", line 745, in chatelet.rst
Failed example:
    rt.show(integ.UsersWithClients) #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ====================== ============ =========== ======== ========= ========= =================== ====================== ========
     Intervenant            Évaluation   Formation   Search   Travail   Standby   Dossiers complèts   Bénéficiaires actifs   Total
    ---------------------- ------------ ----------- -------- --------- --------- ------------------- ---------------------- --------
     Alicia Allmanns        **1**        **1**                          **1**     **3**               **3**                  **7**
     Hubert Huppertz        **1**        **3**       **4**    **2**     **1**     **11**              **11**                 **19**
     Mélanie Mélard         **2**                    **2**    **4**     **3**     **11**              **11**                 **18**
     **Total (3 lignes)**   **4**        **4**       **6**    **6**     **5**     **25**              **25**                 **44**
    ====================== ============ =========== ======== ========= ========= =================== ====================== ========
    <BLANKLINE>
Got:
    ====================== ============ =========== ======== ========= ========= =================== ====================== ========
     Intervenant            Évaluation   Formation   Search   Travail   Standby   Dossiers complèts   Bénéficiaires actifs   Total
    ---------------------- ------------ ----------- -------- --------- --------- ------------------- ---------------------- --------
     Alicia Allmanns        **1**                             **1**     **1**     **3**               **3**                  **7**
     Hubert Huppertz        **4**        **4**       **6**                        **14**              **14**                 **17**
     Mélanie Mélard         **1**        **2**       **2**    **3**     **2**     **10**              **10**                 **19**
     **Total (3 lignes)**   **6**        **6**       **8**    **4**     **3**     **27**              **27**                 **43**
    ====================== ============ =========== ======== ========= ========= =================== ====================== ========
    <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/chatelet.rst", line 857, in chatelet.rst
Failed example:
    walk_menu_items('romain') #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -12,5 +12,5 @@
     - Bureau --> Mes problèmes de données : 0
     - Calendrier --> Mes rendez-vous : 7
    -- Calendrier --> Rendez-vous dépassés : 62
    +- Calendrier --> Rendez-vous dépassés : 58
     - Calendrier --> Mes rendez-vous à confirmer : 4
     - Calendrier --> Mes tâches : 1
    @@ -19,5 +19,5 @@
     - Calendrier --> Mes rendez-vous dépassés : 1
     - Réception --> Bénéficiaires : 30
    -- Réception --> Rendez-vous aujourd'hui : 4
    +- Réception --> Rendez-vous aujourd'hui : 5
     - Réception --> Salle d'attente : 8
     - Réception --> Visiteurs occupés : 4
    @@ -65,5 +65,5 @@
     - Configuration --> Bureau --> Types d'événements : 11
     - Configuration --> Bureau --> Mes Text field templates : 1
    -- Configuration --> Calendrier --> Calendriers : ...
    +- Configuration --> Calendrier --> Calendriers : 12
     - Configuration --> Calendrier --> Locaux : 1
     - Configuration --> Calendrier --> Évènements periodiques : 16
    @@ -127,5 +127,5 @@
     - Explorateur --> Système --> Tests de données : 15
     - Explorateur --> Système --> Messages de problème : 10
    -- Explorateur --> Bureau --> Extraits : 78
    +- Explorateur --> Bureau --> Extraits : 79
     - Explorateur --> Bureau --> Fichiers téléchargés : 6
     - Explorateur --> Bureau --> Domaines de téléchargement : 2
    @@ -134,7 +134,7 @@
     - Explorateur --> Bureau --> Observations : 112
     - Explorateur --> Bureau --> Text field templates : 3
    -- Explorateur --> Calendrier --> Entrées calendrier : 343
    -- Explorateur --> Calendrier --> Tâches : 31
    -- Explorateur --> Calendrier --> Présences : 589
    +- Explorateur --> Calendrier --> Entrées calendrier : 338
    +- Explorateur --> Calendrier --> Tâches : 27
    +- Explorateur --> Calendrier --> Présences : 615
     - Explorateur --> Calendrier --> Abonnements : 10
     - Explorateur --> Calendrier --> Entry states : 5
    @@ -156,6 +156,6 @@
     - Explorateur --> CPAS --> Etats bénéficiaires : 4
     - Explorateur --> CPAS --> Types de carte eID : 11
    -- Explorateur --> CPAS --> Octrois d'aide : 56
    -- Explorateur --> CPAS --> Certificats de revenu : 55
    +- Explorateur --> CPAS --> Octrois d'aide : 55
    +- Explorateur --> CPAS --> Certificats de revenu : 53
     - Explorateur --> CPAS --> Refund confirmations : 13
     - Explorateur --> CPAS --> Confirmations simple : 20
    @@ -163,18 +163,18 @@
     - Explorateur --> CPAS --> Types de contact connus : 2
     - Explorateur --> CPAS --> Interventions : 91
    -- Explorateur --> Parcours --> Connaissances de langue : 115
    +- Explorateur --> Parcours --> Connaissances de langue : 113
     - Explorateur --> Parcours --> Formations : 21
     - Explorateur --> Parcours --> Études : 23
     - Explorateur --> Parcours --> Expériences professionnelles : 31
    -- Explorateur --> Parcours --> Connaissances de langue : 115
    +- Explorateur --> Parcours --> Connaissances de langue : 113
     - Explorateur --> Parcours --> Compétences professionnelles : 1
     - Explorateur --> Parcours --> Compétences sociales : 1
     - Explorateur --> Parcours --> Freins : 21
    -- Explorateur --> Intégration --> PIISs : 31
    -- Explorateur --> Intégration --> Mises à l'emploi art60§7 : 14
    +- Explorateur --> Intégration --> PIISs : 30
    +- Explorateur --> Intégration --> Mises à l'emploi art60§7 : 15
     - Explorateur --> Intégration --> Candidatures : 75
    -- Explorateur --> Intégration --> Services utilisateurs : 36
    -- Explorateur --> Intégration --> Mises à l'emploi art61 : 8
    -- Explorateur --> Intégration --> Stages d'immersion : 7
    +- Explorateur --> Intégration --> Services utilisateurs : 34
    +- Explorateur --> Intégration --> Mises à l'emploi art61 : 7
    +- Explorateur --> Intégration --> Stages d'immersion : 9
     - Explorateur --> Intégration --> Preuves de recherche : 11
     - Explorateur --> Intégration --> Fiches FSE : 189
    @@ -192,4 +192,4 @@
     - Explorateur --> Questionnaires --> Choix de réponse : 89
     - Explorateur --> Questionnaires --> Answer Remarks : 1
    -- Site --> Sessions utilisateurs : ...
    +- Site --> Sessions utilisateurs : 2
     <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/chatelet.rst", line 1067, in chatelet.rst
Failed example:
    for actor in debts.Actor.objects.all():
        n = actor.get_first_meeting()
        if n is not None:
            print(msg.format(actor.budget.id, dd.fdl(n.date), n.user))
Expected:
    Budget 4 : First meeting on 22 July 2013 with user nicolas
Got nothing
**********************************************************************
File "docs/specs/welcht/chatelet.rst", line 1081, in chatelet.rst
Failed example:
    for actor in budget.get_actors():
        print(actor.get_first_meeting_text())
Expected:
    None
    First meeting on 22 July 2013 with nicolas
    None
Got:
    None
    None
    None
**********************************************************************
1 items had failures:
   5 of  23 in chatelet.rst
***Test Failed*** 5 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/clients_chatelet.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/clients_chatelet.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/clients_chatelet.rst", line 142, in clients_chatelet.rst
Failed example:
    ses.show(pcsw.CoachedClients, column_names="name_column", param_values=pv)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ==========================
     Name
    --------------------------
     EVERS Eberhart (127)
     KELLER Karl (178)
     MALMENDIER Marc (146)
     MEESSEN Melissa (147)
     RADERMACHER Alfons (153)
     DA VINCI David (165)
     VAN VEEN Vincent (166)
    ==========================
    <BLANKLINE>
Got:
    ==========================
     Name
    --------------------------
     DUBOIS Robin (179)
     JACOBS Jacqueline (137)
     JONAS Josef (139)
     RADERMACHER Alfons (153)
     RADERMACHER Guido (159)
     RADERMACHER Hedi (161)
     DA VINCI David (165)
    ==========================
    <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/clients_chatelet.rst", line 161, in clients_chatelet.rst
Failed example:
    ses.show(cv.TrainingsByPerson, obj, column_names="type start_date end_date")
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ================ ============ ============
     Education Type   Start date   End date
    ---------------- ------------ ------------
     Alpha            13/03/2011   13/03/2012
    ================ ============ ============
    <BLANKLINE>
Got:
    ================ ============ ============
     Education Type   Start date   End date
    ---------------- ------------ ------------
     Alpha            01/03/2011   01/05/2011
    ================ ============ ============
    <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/clients_chatelet.rst", line 180, in clients_chatelet.rst
Failed example:
    ses.show(cv.ExperiencesByPerson, obj, column_names="start_date end_date")
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ============ ============
     Start date   End date
    ------------ ------------
     11/03/2011   11/03/2012
    ============ ============
    <BLANKLINE>
Got:
    ============ ============
     Start date   End date
    ------------ ------------
     27/02/2011   27/03/2011
    ============ ============
    <BLANKLINE>
**********************************************************************
1 items had failures:
   3 of  22 in clients_chatelet.rst
***Test Failed*** 3 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/courses.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/courses.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/courses.rst", line 89, in courses.rst
Failed example:
    rt.show(rt.models.cal.GuestsByEvent, event)
Expected:
    ===================== ========= ============= ========
     Partner               Role      Workflow      Remark
    --------------------- --------- ------------- --------
     Bastiaensen Laurent   Visitor   **Invited**
     Denon Denis           Visitor   **Invited**
     Dericum Daniel        Visitor   **Invited**
     Emonts-Gast Erna      Visitor   **Invited**
     Faymonville Luc       Visitor   **Invited**
     Gernegroß Germaine    Visitor   **Invited**
     Jacobs Jacqueline     Visitor   **Invited**
     Jonas Josef           Visitor   **Invited**
     Kaivers Karl          Visitor   **Invited**
     Laschet Laura         Visitor   **Invited**
     Radermacher Hedi      Visitor   **Invited**
    ===================== ========= ============= ========
    <BLANKLINE>
Got:
    =================================== ========= ============= ========
     Partner                             Role      Workflow      Remark
    ----------------------------------- --------- ------------- --------
     Bastiaensen Laurent                 Visitor   **Invited**
     Dobbelstein Dorothée                Visitor   **Invited**
     Dobbelstein-Demeulenaere Dorothée   Visitor   **Invited**
     Emonts Erich                        Visitor   **Invited**
     Gernegroß Germaine                  Visitor   **Invited**
     Jacobs Jacqueline                   Visitor   **Invited**
     Johnen Johann                       Visitor   **Invited**
     Laschet Laura                       Visitor   **Invited**
     Meessen Melissa                     Visitor   **Invited**
     Radermacher Christian               Visitor   **Invited**
     Vandenmeulenbos Marie-Louise        Visitor   **Invited**
    =================================== ========= ============= ========
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of  23 in courses.rst
***Test Failed*** 1 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/cv2.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/cv2.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/cv2.rst", line 92, in cv2.rst
Failed example:
    rt.show(cv.Obstacles)
Expected:
    ==== ======================= ========== ===================== ============= ============
     ID   Personne                Remarque   Type                  Détecté par   Date
    ---- ----------------------- ---------- --------------------- ------------- ------------
     1    M. Luc FAYMONVILLE                 Alcohol                             22/05/2014
     2    M. Gregory GROTECLAES              Santé                               22/05/2014
     3    Mme Hildegard HILGERS              Dettes                              22/05/2014
     4    Mme Jacqueline JACOBS              Problèmes familiers                 22/05/2014
     5    M. Jérôme JEANÉMART                Alcohol                             22/05/2014
     6    M. Luc FAYMONVILLE                 Santé                               22/05/2014
     7    M. Gregory GROTECLAES              Dettes                              22/05/2014
     8    Mme Hildegard HILGERS              Problèmes familiers                 22/05/2014
     9    Mme Jacqueline JACOBS              Alcohol                             22/05/2014
     10   M. Jérôme JEANÉMART                Santé                               22/05/2014
     11   M. Luc FAYMONVILLE                 Dettes                              22/05/2014
     12   M. Gregory GROTECLAES              Problèmes familiers                 22/05/2014
     13   Mme Hildegard HILGERS              Alcohol                             22/05/2014
     14   Mme Jacqueline JACOBS              Santé                               22/05/2014
     15   M. Jérôme JEANÉMART                Dettes                              22/05/2014
     16   M. Luc FAYMONVILLE                 Problèmes familiers                 22/05/2014
     17   M. Gregory GROTECLAES              Alcohol                             22/05/2014
     18   Mme Hildegard HILGERS              Santé                               22/05/2014
     19   Mme Jacqueline JACOBS              Dettes                              22/05/2014
     20   M. Jérôme JEANÉMART                Problèmes familiers                 22/05/2014
    ==== ======================= ========== ===================== ============= ============
    <BLANKLINE>
Got:
    ==== ==================== ========== ===================== ============= ============
     ID   Personne             Remarque   Type                  Détecté par   Date
    ---- -------------------- ---------- --------------------- ------------- ------------
     1    M. Josef JONAS                  Alcohol                             22/05/2014
     2    M. Karl KAIVERS                 Santé                               22/05/2014
     3    M. Guido LAMBERTZ               Dettes                              22/05/2014
     4    Mme Line LAZARUS                Problèmes familiers                 22/05/2014
     5    M. Marc MALMENDIER              Alcohol                             22/05/2014
     6    M. Josef JONAS                  Santé                               22/05/2014
     7    M. Karl KAIVERS                 Dettes                              22/05/2014
     8    M. Guido LAMBERTZ               Problèmes familiers                 22/05/2014
     9    Mme Line LAZARUS                Alcohol                             22/05/2014
     10   M. Marc MALMENDIER              Santé                               22/05/2014
     11   M. Josef JONAS                  Dettes                              22/05/2014
     12   M. Karl KAIVERS                 Problèmes familiers                 22/05/2014
     13   M. Guido LAMBERTZ               Alcohol                             22/05/2014
     14   Mme Line LAZARUS                Santé                               22/05/2014
     15   M. Marc MALMENDIER              Dettes                              22/05/2014
     16   M. Josef JONAS                  Problèmes familiers                 22/05/2014
     17   M. Karl KAIVERS                 Alcohol                             22/05/2014
     18   M. Guido LAMBERTZ               Santé                               22/05/2014
     19   Mme Line LAZARUS                Dettes                              22/05/2014
     20   M. Marc MALMENDIER              Problèmes familiers                 22/05/2014
    ==== ==================== ========== ===================== ============= ============
    <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/cv2.rst", line 120, in cv2.rst
Failed example:
    rt.login('robin').show(cv.ObstaclesByPerson, hildegard)
Expected:
    ===================== ============= ============ ==========
     Type                  Détecté par   Date         Remarque
    --------------------- ------------- ------------ ----------
     Dettes                              22/05/2014
     Problèmes familiers                 22/05/2014
     Alcohol                             22/05/2014
     Santé                               22/05/2014
    ===================== ============= ============ ==========
    <BLANKLINE>
Got:
    Aucun enregistrement
**********************************************************************
1 items had failures:
   2 of  22 in cv2.rst
***Test Failed*** 2 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/immersion.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/immersion.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/immersion.rst", line 52, in immersion.rst
Failed example:
    rt.show(immersion.Contracts)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -1,11 +1,13 @@
    -==== ========================= ===================== ================== ============ ================= ===========================
    - ID   Bénéficiaire              Organisation          Début de contrat   Fin prévue   Titulaire (ISP)   Type de stage d'immersion
    ----- ------------------------- --------------------- ------------------ ------------ ----------------- ---------------------------
    - 1    EMONTS Daniel (128)       Bäckerei Ausdemwald   29/10/2012         21/02/2014   Alicia Allmanns   MISIP
    - 2    EMONTS Daniel (128)       Bäckerei Mießen       22/02/2014         31/12/2014   Caroline Carnol   Mise en situation interne
    - 3    HILGERS Hildegard (133)   Rumma & Ko OÜ         03/12/2012         11/10/2013   Alicia Allmanns   Stage d'immersion
    - 4    LAZARUS Line (144)        Bäckerei Schmitz      07/01/2013         06/02/2013   Alicia Allmanns   MISIP
    - 5    RADERMACHER Guido (159)   Rumma & Ko OÜ         11/02/2013         06/06/2014   Alicia Allmanns   Mise en situation interne
    - 6    RADERMACHER Guido (159)   Bäckerei Ausdemwald   07/06/2014         15/04/2015   Mélanie Mélard    Stage d'immersion
    -==== ========================= ===================== ================== ============ ================= ===========================
    +==== ========================== ======================== ================== ============ ================= ===========================
    + ID   Bénéficiaire               Organisation             Début de contrat   Fin prévue   Titulaire (ISP)   Type de stage d'immersion
    +---- -------------------------- ------------------------ ------------------ ------------ ----------------- ---------------------------
    + 1    ENGELS Edgar (129)         Rumma & Ko OÜ            29/10/2012         21/02/2014   Alicia Allmanns   MISIP
    + 2    ENGELS Edgar (129)         Bäckerei Ausdemwald      22/02/2014         31/12/2014   Mélanie Mélard    Mise en situation interne
    + 3    LAMBERTZ Guido (142)       Belgisches Rotes Kreuz   03/12/2012         28/03/2014   Alicia Allmanns   Stage d'immersion
    + 4    RADERMACHER Edgard (157)   Bäckerei Ausdemwald      07/01/2013         02/05/2014   Alicia Allmanns   MISIP
    + 5    RADERMACHER Edgard (157)   Bäckerei Mießen          03/05/2014         02/06/2014   Mélanie Mélard    Mise en situation interne
    + 6    RADERMACHER Edgard (157)   Bäckerei Schmitz         03/06/2014         11/04/2015   Mélanie Mélard    Stage d'immersion
    + 7    RADERMECKER Rik (173)      Bäckerei Ausdemwald      11/02/2013         20/12/2013   Mélanie Mélard    MISIP
    + 8    RADERMECKER Rik (173)      Bäckerei Mießen          21/12/2013         15/04/2015   Hubert Huppertz   Mise en situation interne
    +==== ========================== ======================== ================== ============ ================= ===========================
     <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/immersion.rst", line 74, in immersion.rst
Failed example:
    print(str(obj.company))
Expected:
    Rumma & Ko OÜ
Got:
    Belgisches Rotes Kreuz
**********************************************************************
1 items had failures:
   2 of  10 in immersion.rst
***Test Failed*** 2 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/integ.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/integ.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/integ.rst", line 123, in integ.rst
Failed example:
    ses.show(integ.UsersWithClients)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ==================== ============ =========== ======== ========= ========= ================= ================ ========
     Coach                Évaluation   Formation   Search   Travail   Standby   Primary clients   Active clients   Total
    -------------------- ------------ ----------- -------- --------- --------- ----------------- ---------------- --------
     Alicia Allmanns      **1**        **1**                          **1**     **3**             **3**            **7**
     Hubert Huppertz      **1**        **3**       **4**    **2**     **1**     **11**            **11**           **19**
     Mélanie Mélard       **2**                    **2**    **4**     **3**     **11**            **11**           **18**
     **Total (3 rows)**   **4**        **4**       **6**    **6**     **5**     **25**            **25**           **44**
    ==================== ============ =========== ======== ========= ========= ================= ================ ========
    <BLANKLINE>
Got:
    ==================== ============ =========== ======== ========= ========= ================= ================ ========
     Coach                Évaluation   Formation   Search   Travail   Standby   Primary clients   Active clients   Total
    -------------------- ------------ ----------- -------- --------- --------- ----------------- ---------------- --------
     Alicia Allmanns      **1**                             **1**     **1**     **3**             **3**            **7**
     Hubert Huppertz      **4**        **4**       **6**                        **14**            **14**           **17**
     Mélanie Mélard       **1**        **2**       **2**    **3**     **2**     **10**            **10**           **19**
     **Total (3 rows)**   **6**        **6**       **8**    **4**     **3**     **27**            **27**           **43**
    ==================== ============ =========== ======== ========= ========= ================= ================ ========
    <BLANKLINE>
**********************************************************************
File "docs/specs/welcht/integ.rst", line 147, in integ.rst
Failed example:
    ses.show(integ.ActivityReport, stripped=True)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -17,8 +17,8 @@
      Coach                Évaluation   Formation   Search   Travail   Standby   Primary clients   Active clients   Total
     -------------------- ------------ ----------- -------- --------- --------- ----------------- ---------------- --------
    - Alicia Allmanns      **1**        **1**                          **1**     **3**             **3**            **7**
    - Hubert Huppertz      **1**        **3**       **4**    **2**     **1**     **11**            **11**           **19**
    - Mélanie Mélard       **2**                    **2**    **4**     **3**     **11**            **11**           **18**
    - **Total (3 rows)**   **4**        **4**       **6**    **6**     **5**     **25**            **25**           **44**
    + Alicia Allmanns      **1**                             **1**     **1**     **3**             **3**            **7**
    + Hubert Huppertz      **4**        **4**       **6**                        **14**            **14**           **17**
    + Mélanie Mélard       **1**        **2**       **2**    **3**     **2**     **10**            **10**           **19**
    + **Total (3 rows)**   **6**        **6**       **8**    **4**     **3**     **27**            **27**           **43**
     ==================== ============ =========== ======== ========= ========= ================= ================ ========
     <BLANKLINE>
**********************************************************************
1 items had failures:
   2 of  30 in integ.rst
***Test Failed*** 2 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/isip_chatelet.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/isip_chatelet.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/isip_chatelet.rst", line 23, in isip_chatelet.rst
Failed example:
    rt.show(isip.Contracts)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE -REPORT_UDIFF
Expected:
    ==== ============== ============ ============================ ================= =====================
     ID   applies from   date ended   Client                       Author            Contract Type
    ---- -------------- ------------ ---------------------------- ----------------- ---------------------
     1    29/09/2012     07/08/2013   AUSDEMWALD Alfons (116)      Hubert Huppertz   VSE Ausbildung
     2    08/08/2013     01/12/2014   AUSDEMWALD Alfons (116)      Mélanie Mélard    VSE Arbeitssuche
     3    09/10/2012     17/08/2013   COLLARD Charlotte (118)      Alicia Allmanns   VSE Lehre
     4    19/10/2012     11/02/2014   DOBBELSTEIN Dorothée (124)   Alicia Allmanns   VSE Vollzeitstudium
     5    12/02/2014     14/03/2014   DOBBELSTEIN Dorothée (124)   Caroline Carnol   VSE Sprachkurs
     6    15/03/2014     21/01/2015   DOBBELSTEIN Dorothée (124)   Caroline Carnol   VSE Ausbildung
     7    03/11/2012     26/02/2014   EMONTS-GAST Erna (152)       Alicia Allmanns   VSE Arbeitssuche
     8    13/11/2012     13/12/2012   EVERS Eberhart (127)         Alicia Allmanns   VSE Lehre
     9    23/11/2012     01/10/2013   FAYMONVILLE Luc (130*)       Mélanie Mélard    VSE Vollzeitstudium
     10   02/10/2013     25/01/2015   FAYMONVILLE Luc (130*)       Hubert Huppertz   VSE Sprachkurs
     11   08/12/2012     02/04/2014   JACOBS Jacqueline (137)      Alicia Allmanns   VSE Ausbildung
     12   03/04/2014     03/05/2014   JACOBS Jacqueline (137)      Mélanie Mélard    VSE Arbeitssuche
     13   04/05/2014     12/03/2015   JACOBS Jacqueline (137)      Mélanie Mélard    VSE Lehre
     14   18/12/2012     12/04/2014   JONAS Josef (139)            Hubert Huppertz   VSE Vollzeitstudium
     15   13/04/2014     19/02/2015   JONAS Josef (139)            Hubert Huppertz   VSE Sprachkurs
     16   28/12/2012     22/04/2014   KELLER Karl (178)            Alicia Allmanns   VSE Ausbildung
     17   12/01/2013     20/11/2013   MALMENDIER Marc (146)        Mélanie Mélard    VSE Arbeitssuche
     18   21/11/2013     16/03/2015   MALMENDIER Marc (146)        Hubert Huppertz   VSE Lehre
     19   22/01/2013     30/11/2013   RADERMACHER Alfons (153)     Alicia Allmanns   VSE Vollzeitstudium
     20   01/02/2013     27/05/2014   RADERMACHER Edgard (157)     Alicia Allmanns   VSE Sprachkurs
     21   28/05/2014     27/06/2014   RADERMACHER Edgard (157)     Hubert Huppertz   VSE Ausbildung
     22   28/06/2014     06/05/2015   RADERMACHER Edgard (157)     Hubert Huppertz   VSE Arbeitssuche
     23   16/02/2013     11/06/2014   RADERMACHER Hedi (161)       Alicia Allmanns   VSE Lehre
     24   12/06/2014     12/07/2014   RADERMACHER Hedi (161)       Hubert Huppertz   VSE Vollzeitstudium
     25   13/07/2014     21/05/2015   RADERMACHER Hedi (161)       Hubert Huppertz   VSE Sprachkurs
     26   26/02/2013     21/06/2014   DA VINCI David (165)         Alicia Allmanns   VSE Ausbildung
     27   22/06/2014     30/04/2015   DA VINCI David (165)         Alicia Allmanns   VSE Arbeitssuche
     28   08/03/2013     01/07/2014   ÖSTGES Otto (168)            Mélanie Mélard    VSE Lehre
     29   02/07/2014     01/08/2014   ÖSTGES Otto (168)            Hubert Huppertz   VSE Vollzeitstudium
     30   02/08/2014     10/06/2015   ÖSTGES Otto (168)            Hubert Huppertz   VSE Sprachkurs
    ==== ============== ============ ============================ ================= =====================
    <BLANKLINE>
Got:
    ==== ============== ============ ============================ ================= =====================
     ID   applies from   date ended   Client                       Author            Contract Type
    ---- -------------- ------------ ---------------------------- ----------------- ---------------------
     1    29/09/2012     07/08/2013   AUSDEMWALD Alfons (116)      Hubert Huppertz   VSE Ausbildung
     2    08/08/2013     01/12/2014   AUSDEMWALD Alfons (116)      Mélanie Mélard    VSE Arbeitssuche
     3    09/10/2012     17/08/2013   DOBBELSTEIN Dorothée (124)   Alicia Allmanns   VSE Lehre
     4    19/10/2012     11/02/2014   EVERS Eberhart (127)         Alicia Allmanns   VSE Vollzeitstudium
     5    12/02/2014     14/03/2014   EVERS Eberhart (127)         Caroline Carnol   VSE Sprachkurs
     6    15/03/2014     21/01/2015   EVERS Eberhart (127)         Caroline Carnol   VSE Ausbildung
     7    03/11/2012     26/02/2014   FAYMONVILLE Luc (130*)       Alicia Allmanns   VSE Arbeitssuche
     8    27/02/2014     29/03/2014   FAYMONVILLE Luc (130*)       Hubert Huppertz   VSE Lehre
     9    30/03/2014     05/02/2015   FAYMONVILLE Luc (130*)       Hubert Huppertz   VSE Vollzeitstudium
     10   13/11/2012     08/03/2014   HILGERS Hildegard (133)      Alicia Allmanns   VSE Sprachkurs
     11   09/03/2014     15/01/2015   HILGERS Hildegard (133)      Alicia Allmanns   VSE Ausbildung
     12   23/11/2012     18/03/2014   JONAS Josef (139)            Mélanie Mélard    VSE Arbeitssuche
     13   19/03/2014     18/04/2014   JONAS Josef (139)            Hubert Huppertz   VSE Lehre
     14   19/04/2014     25/02/2015   JONAS Josef (139)            Hubert Huppertz   VSE Vollzeitstudium
     15   08/12/2012     16/10/2013   LAZARUS Line (144)           Alicia Allmanns   VSE Sprachkurs
     16   17/10/2013     09/02/2015   LAZARUS Line (144)           Mélanie Mélard    VSE Ausbildung
     17   18/12/2012     17/01/2013   MEESSEN Melissa (147)        Mélanie Mélard    VSE Arbeitssuche
     18   18/01/2013     26/11/2013   MEESSEN Melissa (147)        Mélanie Mélard    VSE Lehre
     19   27/11/2013     22/03/2015   MEESSEN Melissa (147)        Mélanie Mélard    VSE Vollzeitstudium
     20   28/12/2012     05/11/2013   RADERMACHER Alfons (153)     Alicia Allmanns   VSE Sprachkurs
     21   12/01/2013     07/05/2014   RADERMACHER Fritz (158)      Alicia Allmanns   VSE Ausbildung
     22   22/01/2013     30/11/2013   RADERMACHER Hedi (161)       Alicia Allmanns   VSE Arbeitssuche
     23   01/12/2013     26/03/2015   RADERMACHER Hedi (161)       Mélanie Mélard    VSE Lehre
     24   01/02/2013     03/03/2013   VAN VEEN Vincent (166)       Alicia Allmanns   VSE Vollzeitstudium
     25   16/02/2013     25/12/2013   BRECHT Bernd (177)           Alicia Allmanns   VSE Sprachkurs
     26   26/12/2013     20/04/2015   BRECHT Bernd (177)           Hubert Huppertz   VSE Ausbildung
     27   26/02/2013     28/03/2013   DUBOIS Robin (179)           Alicia Allmanns   VSE Arbeitssuche
     28   08/03/2013     14/01/2014   JEANÉMART Jérôme (181)       Mélanie Mélard    VSE Lehre
     29   15/01/2014     10/05/2015   JEANÉMART Jérôme (181)       Hubert Huppertz   VSE Vollzeitstudium
    ==== ============== ============ ============================ ================= =====================
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of   8 in isip_chatelet.rst
***Test Failed*** 1 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/welcht/misc.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/welcht/misc.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/welcht/misc.rst", line 129, in misc.rst
Failed example:
    obj.owner
Expected:
    Contract #2 ("Stage d'immersion#2 (Daniel EMONTS)")
Got:
    Contract #2 ("Stage d'immersion#2 (Edgar ENGELS)")
**********************************************************************
File "docs/specs/welcht/misc.rst", line 181, in misc.rst
Failed example:
    demo_get('hubert', 'api/integ/Clients', json_fields, 19, **kw)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest misc.rst[39]>", line 1, in <module>
        demo_get('hubert', 'api/integ/Clients', json_fields, 19, **kw)
      File "/home/luc/work/lino/lino/api/doctest.py", line 157, in demo_get
        raise Exception(msg)
    Exception: /api/integ/Clients?fmt=json&limit=10&start=0 got 17 rows instead of [19]
**********************************************************************
1 items had failures:
   2 of  58 in misc.rst
***Test Failed*** 2 failures.

-----

======================================================================
FAIL: atelier.test.DocTestCase (docs/specs/weleup/eupen.rst)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/luc/work/atelier/atelier/test.py", line 44, in func
    self.run_subprocess(args)
  File "/home/luc/work/rstgen/rstgen/utils.py", line 230, in run_subprocess
    self.fail(msg)
AssertionError: /home/luc/virtualenvs/py3/bin/python -m doctest docs/specs/weleup/eupen.rst ({}) returned 1:
-----
**********************************************************************
File "docs/specs/weleup/eupen.rst", line 1181, in eupen.rst
Failed example:
    walk_menu_items('rolf', severe=False)
    #doctest: +ELLIPSIS +NORMALIZE_WHITESPACE +REPORT_UDIFF
Differences (unified diff with -expected +actual):
    @@ -12,5 +12,5 @@
     - Büro --> Meine Datenkontrollliste : 0
     - Kalender --> Meine Termine : 6
    -- Kalender --> Überfällige Termine : 25
    +- Kalender --> Überfällige Termine : 284
     - Kalender --> Meine unbestätigten Termine : 4
     - Kalender --> Meine Aufgaben : 1
    @@ -70,5 +70,5 @@
     - Konfigurierung --> Büro --> Ereignisarten : 11
     - Konfigurierung --> Büro --> Meine Einfügetexte : 1
    -- Konfigurierung --> Kalender --> Kalenderliste : ...
    +- Konfigurierung --> Kalender --> Kalenderliste : 12
     - Konfigurierung --> Kalender --> Räume : 1
     - Konfigurierung --> Kalender --> Regelmäßige Ereignisse : 16
    @@ -140,5 +140,5 @@
     - Explorer --> Büro --> Ereignisse/Notizen : 112
     - Explorer --> Büro --> Einfügetexte : 3
    -- Explorer --> Kalender --> Kalendereinträge : 311
    +- Explorer --> Kalender --> Kalendereinträge : 588
     - Explorer --> Kalender --> Aufgaben : 37
     - Explorer --> Kalender --> Anwesenheiten : 629
    @@ -198,4 +198,4 @@
     - Explorer --> Schuldnerberatung --> Budgets : 15
     - Explorer --> Schuldnerberatung --> Einträge : 717
    -- Site --> Benutzersitzungen : ...
    +- Site --> Benutzersitzungen : 21
     <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of  37 in eupen.rst
***Test Failed*** 1 failures.

-----

----------------------------------------------------------------------
Ran 155 tests in 799.894s

FAILED (failures=13)

My next step will be to adapt those doctests and accept the fact that demo data has changed because “some fixture(s) somewhere” had been generating data using the pcsw.Client rows with undefined ordering, and that ordering was not exactly the same as now (which is by id).

The demo fixture for lino_welfare.modlib.integ was doing the following

af = settings.SITE.demo_date(-600 + i * 5)

I changed this to:

af = settings.SITE.demo_date(-600 + i * 7)

because otherwise we were having no tasks to see in the MyTasks table in Calendar tasks.

Yes, it was so difficult to find the reason because the doctests aren’t granular. The demo database covers many things at once. One little change can cause an avalanche of doctest snippets that need to get adapted. That can be frustrating. But it’s part of the game with doctests, and it often brings interesting new insights.

For example the demo fixture for lino_welfare.modlib.debts now adds a Notes.first_meeting note and a calendar entry for the meeting of the social agent and the beneficiary. Because the snippet in The first meeting of a budget was failing. Until now it had happened to find a budget with an actor for whom there happened to be a get_first_meeting, but this was actually a pure coincidence.

Luc's developer blog

Saturday, May 30, 2026

Saturday, May 30, 2026¶

Thursday, May 28, 2026

Thursday, May 28, 2026¶

Thursday, May 14, 2026

Thursday, May 14, 2026¶

Explanation for mail_queue_enter: create file maildrop/870989.992: Read-only file system

Explanation for mail_queue_enter: create file maildrop/870989.992: Read-only file system¶

Saturday, May 2, 2026

Saturday, May 2, 2026¶

Thursday, April 30, 2026

Thursday, April 30, 2026¶

mail_queue_enter: create file maildrop/870989.992: Read-only file system

mail_queue_enter: create file maildrop/870989.992: Read-only file system¶

Monday, April 27, 2026

Monday, April 27, 2026¶

Wednesday, April 22, 2026

Wednesday, April 22, 2026¶

Monday, April 13, 2026

Monday, April 13, 2026¶

Friday, April 10, 2026

Friday, April 10, 2026¶

Saturday, March 21, 2026

Saturday, March 21, 2026¶

Monday, March 9, 2026

Monday, March 9, 2026¶

Friday, February 20, 2026

Friday, February 20, 2026¶

Sunday, January 18, 2026

Sunday, January 18, 2026¶

Wednesday, January 14, 2026

Wednesday, January 14, 2026¶

The request_handler() method¶

Sunday, January 4, 2026

Sunday, January 4, 2026¶

Waiting for a miracle

Waiting for a miracle¶

Tuesday, November 25, 2025

Tuesday, November 25, 2025¶

Insert a local table of contents into a HTML page

Insert a local table of contents into a HTML page¶

Wonderful world

Wonderful world¶

Sunday, September 14, 2025

Sunday, September 14, 2025¶

Monday, August 18, 2025

Monday, August 18, 2025¶

Friday, August 1, 2025

Friday, August 1, 2025¶

Saturday, July 5, 2025

Saturday, July 5, 2025¶

BeautifulSoup and lxml

BeautifulSoup and lxml¶

Thursday, June 26, 2025

Thursday, June 26, 2025¶

Monday, June 23, 2025

Monday, June 23, 2025¶

Friday, June 20, 2025

Friday, June 20, 2025¶

Tuesday, June 3, 2025

Tuesday, June 3, 2025¶

Monday, May 19, 2025

Monday, May 19, 2025¶

Puppeteer¶

SalesRule becomes TradeRule¶

I tried lilyponddist

I tried lilyponddist¶

Tuesday, April 22, 2025

Tuesday, April 22, 2025¶

Friday, April 18, 2025

Friday, April 18, 2025¶

Thursday, April 17, 2025

Thursday, April 17, 2025¶

Monday, April 14, 2025

Monday, April 14, 2025¶

Friday, April 11, 2025

Friday, April 11, 2025¶

Python 3.13 breaks sending of emails

Python 3.13 breaks sending of emails¶

The `request_handler()` method¶

Should we stop using `from .ui import *` ?¶