Friday, December 2, 2016

Two deployments at the same time

My mum always tells me to never run two deployments at the same time, at least not when you have only 15 minutes. Yesterday I did exactly this.

Now I noticed that the deployment on our own Lino Noi database had a problem:

  • votes.Vote (1048, “Column ‘created’ cannot be null”) (58 object(s) with primary key 41, 73, 100, 96, 97, 98, 99, 58, 59, 86, 87, 39, 102, 53, 25, 64, 56

So my starred tickets had not been migrated correctly. I decided to re-run the restore.py now because the few changes I did after this upgrade are less work than restoring the votes manually.

I did not see this problem yesterday because of an old logging problem: the warnings emitted are written to lino.log but not to the console. This problem had arrived with some Django upgrade. I now found the reason and fixed it: I must remove the ‘require_debug_true’ filter from the ‘console’ handler in Django’s default logging config. I have no idea why Django has such a strange behaviour as default… #1295.

This fix for #1295 probably caused the following warnings to no longer pass unseen:

.../django/db/models/fields/__init__.py:1393: RuntimeWarning: DateTimeField Ticket.created received a naive datetime (2015-02-22 00:00:00) while time zone support is active.

Cannot update unresolved field <class 'lino_xl.lib.excerpts.models.Excerpt'>.company

This package should not be accessible on Python 3

A mysterious error happens:

$ go voga
$ python setup.py test -s tests.DocsTests.test_voga
running test
running egg_info
writing requirements to lino_voga.egg-info/requires.txt
writing lino_voga.egg-info/PKG-INFO
writing top-level names to lino_voga.egg-info/top_level.txt
writing dependency_links to lino_voga.egg-info/dependency_links.txt
reading manifest file 'lino_voga.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'lino_voga.egg-info/SOURCES.txt'
running build_ext
test_voga (tests.DocsTests) ... FAIL

======================================================================
FAIL: test_voga (tests.DocsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/dell1tb/work/voga/tests/__init__.py", line 76, in test_voga
    return self.run_simple_doctests('docs/specs/voga.rst')
  File "/media/dell1tb/work/atelier/atelier/test.py", line 109, in run_simple_doctests
    self.run_subprocess(args, **kw)
  File "/media/dell1tb/work/atelier/atelier/test.py", line 92, in run_subprocess
    self.fail(msg)
AssertionError: /media/dell1tb/virtualenvs/py27/bin/python -m atelier.doctest_utf8 docs/specs/voga.rst ({}) returned 1:
-----
**********************************************************************
File "/media/dell1tb/work/voga/docs/specs/voga.rst", line 53, in voga.rst
Failed example:
    rv = ses.run(obj.do_print)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest voga.rst[19]>", line 1, in <module>
        rv = ses.run(obj.do_print)
      File "/media/dell1tb/work/lino/lino/core/requests.py", line 631, in run
        return thing.run_from_session(self, *args, **kw)
      File "/media/dell1tb/work/lino/lino/core/utils.py", line 669, in run_from_session
        self.bound_action.action.run_from_code(ar)
      File "/media/dell1tb/work/lino/lino/core/actions.py", line 613, in run_from_code
        self.run_from_ui(ar, *args, **kw)
      File "/media/dell1tb/work/xl/lino_xl/lib/excerpts/models.py", line 340, in run_from_ui
        ex.do_print.run_from_ui(ar, **kw)
      File "/media/dell1tb/work/lino/lino/core/utils.py", line 659, in run_from_ui
        self.bound_action.action.run_from_ui(ar)
      File "/media/dell1tb/work/lino/lino/modlib/printing/actions.py", line 158, in run_from_ui
        obj.build_target(ar)
      File "/media/dell1tb/work/lino/lino/modlib/printing/mixins.py", line 209, in build_target
        t = bm.build(ar, elem.__class__.do_print, elem)
      File "/media/dell1tb/work/lino/lino/modlib/printing/choicelists.py", line 245, in build
        return self.simple_build(ar, elem, tplfile, target)
      File "/media/dell1tb/work/xl/lino_xl/lib/appypod/choicelists.py", line 59, in simple_build
        **settings.SITE.appy_params).run()
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/appy/pod/renderer.py", line 428, in run
        self.finalize()
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/appy/pod/renderer.py", line 565, in finalize
        raise PodError(CONVERT_ERROR % output)
    PodError: An error occurred during the conversion. Traceback (most recent call last):
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/appy/pod/converter.py", line 20, in <module>
        import sys, os, os.path, time, signal
      File "/usr/lib/python3.5/signal.py", line 3, in <module>
        from functools import wraps as _wraps
      File "/usr/lib/python3.5/functools.py", line 21, in <module>
        from collections import namedtuple
      File "/usr/lib/python3.5/collections/__init__.py", line 32, in <module>
        from reprlib import recursive_repr as _recursive_repr
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
        raise ImportError('This package should not be accessible on Python 3. '
    ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
    Error in sys.excepthook:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
        if not enabled():
      File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
        import re
      File "/usr/lib/python3.5/re.py", line 335, in <module>
        import copyreg
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/copyreg/__init__.py", line 7, in <module>
        raise ImportError('This package should not be accessible on Python 3. '
    ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

    Original exception was:
    Traceback (most recent call last):
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/appy/pod/converter.py", line 20, in <module>
        import sys, os, os.path, time, signal
      File "/usr/lib/python3.5/signal.py", line 3, in <module>
        from functools import wraps as _wraps
      File "/usr/lib/python3.5/functools.py", line 21, in <module>
        from collections import namedtuple
      File "/usr/lib/python3.5/collections/__init__.py", line 32, in <module>
        from reprlib import recursive_repr as _recursive_repr
      File "/media/dell1tb/virtualenvs/py27/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
        raise ImportError('This package should not be accessible on Python 3. '
    ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

**********************************************************************
1 items had failures:
   7 of  29 in voga.rst
***Test Failed*** 7 failures.

-----

----------------------------------------------------------------------
Ran 1 test in 3.834s

FAILED (failures=1)

The problem seems to be caused by my good old enemy future, especially when I read things like this:

I removed the following lines from most places:

from future import standard_library
standard_library.install_aliases()

To be honest, I must admit that above changes did not help, and that the newstr trick of future is AFAICS still the only solution for getting our doctests to pass on both Py 2 and 3.

Note also that I did not run the full test suite under Python 3 (I stopped when it had a problem with appy).

There is a workaround: when I test that same file using py.test, then it passes:

$ py.test -k test_voga
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: /media/dell1tb/work/voga, inifile: pytest.ini
plugins: cov-2.4.0
collected 18 items

tests/__init__.py .

============================= 17 tests deselected ==============================
=================== 1 passed, 17 deselected in 4.89 seconds ====================

Adapting the tests

The changes in output behaviour during test due to the console output being visible at places where it wasn’t before, the “This package should not be accessible on Python 3” problem, caused me another 3 hours over midnight session for adapting the tests suites…