3 Commits (22454bd1e9668a3094cd059cb52856972bbd40c8)

Author	SHA1	Message	Date
Michael Prokop	983d11a11a	MT#57855 venv deployments: use venv instead of virtualenv ... When invoking `virtualenv --python=python3 virtualenv` to deploy the virtual environment, installing the python-debian wheel, and then re-executing `virtualenv --python=python3 virtualenv` once again, we might end up without the virtualenv/lib/python*/site-packages/debian directory, resulting in: | Traceback (most recent call last): | File "<string>", line 1, in <module> | ModuleNotFoundError: No module named 'debian' This is a bug in setuptools >=51.3.3-1, see https://bugs.debian.org/1041091 for the details, so currently affecting all systems with Debian/bookworm. By excluding installation of setuptools (via --no-setuptools), we would be able to work around this bootstrapping problem itself, though then installation of django-jsonify (which we depend on) fails with: | ModuleNotFoundError: No module named 'setuptools' Let's take this as a chance to switch from virtualenv to the more lightweight venv, quoting https://docs.python.org/3/library/venv.html: | Changed in version 3.5: The use of venv is now recommended for creating virtual environments. Change-Id: Iad06bf8c49abd96441ab04bb9b7e3917acd1ee59	3 years ago
Michael Prokop	1e9aa77d53	TT#19059 Port to Python3, update to latest Django LTS version + based on Debian/stretch ... Updates: * Debian jessie->stretch (to use current stable Debian version) * Python 2.7->3.5 (since Python v2 won't be supported with the next Django LTS version any longer and we want to be prepared) * Improve uwsgi configuration for production usage while at it (thanks Florian Apolloner for feedback and suggestions) Necessary changes: * Debian: - we need to use uwsgi-plugin-python3 for Python3 support - replace python-dev with python3-dev for Python3 support and python with python3 accordingly - get rid of __pycache__ directories to avoid inclusion in resulting Debian package * uwsgi: - if we hardcode python34 (for jessie) or python35 (for stretch) then we can't run uwsgi on all supported systems, so instead try `plugin = python3` and rely on update-alternatives mechanism of uwsgi-plugin-python3 Debian package to get it right for, if that shouldn't work as needed then we can look into /usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate or shipping repoapi.ini via puppet instead * docker: - python-distribute is not needed (anymore) - adjust package names for Python3 usage, support jessie and stretch at the same time - note: just replacing 'stretch' with 'jessie' in t/Dockerfile is enough to get a working repoapi docker image based on jessie * Python3: - assertItemsEqual became assertCountEqual - mocking: `__builtin__.open` became `builtins.open`, see https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth - we need six >=1.9.0 for raise_from astraction in py3 Otherwise we're failing on Debian/jessie with: | AttributeError: 'module' object has no attribute 'raise_from' See https://github.com/benjaminp/six/blob/master/CHANGES - assert_has_calls changes, fixes: | AssertionError: Calls not found. | Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'), | call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')] | Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'), | call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')] See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls - urllib: - `from urlparse import urlparse` became `from urllib.parse import urlparse` - `urllib.quote` became `urllib.parse.quote` - configparser: - `from ConfigParser import ...` became `from configparser import ...` * Django: - 'from views import ...' became 'from .views import ...' - the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...." nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/ * celery: - to use latest celery version 4 django-celery needs to be replaced by django-celery-beat + django-celery-results (otherwise we're running into https://github.com/celery/django-celery/issues/497), see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html - adjust systemd unit files to use ExecStart | ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...] instead of | ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...] and use `--db=/var/lib/repoapi/flower` instead of `--db=/var/lib/repoapi/flower.db` to avoid ending up with a file named flower.db.db Notes for migration (assuming root permissions): - get rid of existing flower database (not containing relevant data): mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old - re-deploy virtual environment from scratch: mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old cd /usr/share/repoapi && make deploy - service restarts (in case of virtual env changes): systemctl restart nginx systemctl restart repoapi-beat.service systemctl restart repoapi-flower.service systemctl restart repoapi-worker.service systemctl restart uwsgi - usage instructions for celery + django stuff: /var/lib/repoapi/venv_prod/bin/celery flower --help source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod" - create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/): source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod" Thanks: Victor Seva for assistance with porting Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c	9 years ago
Victor Seva	e7bdd7505f	MT#12487 add deploy environment and configs for uwsgi ... http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html#emperor-mode Change-Id: I9c00f30c93fe2ad3fe8d37c98bef3616e0ae7fd6	11 years ago