While working on the support for triggering $debian_release/master
branches (see previous commit) I noticed a bunch of missing or unclear
instructions as well as incomplete setup/configuration:
* Makefile: add missing `venv_dev` target to populate development
environment
* extend and clarify usage instructions in README.md
* dev environment: avoid having to manually configure ALLOWED_HOSTS,
instead identify hostname and listen on that (useful e.g.
when running the environment inside docker and then accessing
it via the docker IP like http://172.17.0.3:8000/release_panel/)
* Dockerfile: add python3-virtualenv + virtualenv as they are
needed for the venv handling, also install sqlite3 for
handling the sqlite database manually
* pin pylint to version 1.9.3[1]
[1] We need to pin pylint to version 1.9.3, which is the last
working version for us. Newer version of pylint (2.0.0 and up to
2.1.1) fail for us with:
| Executing django_jenkins.tasks.run_pylint...
| Traceback (most recent call last):
| File "./manage.py", line 10, in <module>
| execute_from_command_line(sys.argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
| utility.execute()
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute
| self.fetch_command(subcommand).run_from_argv(self.argv)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
| self.execute(*args, **cmd_options)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
| output = self.handle(*args, **options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle
| coverage.save(tested_locations, options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run
| lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False)
| TypeError: __init__() got an unexpected keyword argument 'exit'
This needs further investigation, though independent of this change.
It didn't hit us so far as our repoapi docker image isn't rebuilt
regularly and all previous builds, until this commit - which refreshes
the Dockerfile and we therefore get newer python modules - use older
python modules, including an older pylint version.
Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
* update dependencies due postgresql
* fake files needed to deploy at debian
* add new needed config values at server.ini file
Change-Id: I02699c721ac79a0328da07a4c2e3a2cb1451d403
* add 'shell' rule to Makefile for prod
> Traceback (most recent call last):
> File "/var/lib/repoapi/venv_prod/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
> R = retval = fun(*args, **kwargs)
> File "/var/lib/repoapi/venv_prod/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
> return self.run(*args, **kwargs)
> File "/usr/share/repoapi/repoapi/celery.py", line 35, in jbi_parse_hotfix
> app.send_task('hotfix_released', args=[jbi_id, path])
> File "/var/lib/repoapi/venv_prod/local/lib/python2.7/site-packages/celery/app/base.py", line 354, in send_task
> reply_to=reply_to or self.oid, **options
> File "/var/lib/repoapi/venv_prod/local/lib/python2.7/site-packages/celery/app/amqp.py", line 255, in publish_task
> raise ValueError('task args must be a list or tuple')
> ValueError: task args must be a list or tuple
Change-Id: Id1ccea3665a8face98e31608e2cd5a30acb4eaea
* Dockerfile: add screen and refresh due new requirements
* create settings/test.py just for tests
- celery in memory ( no need of server )
- files in RESULTS
* repoapi/tasks.py for celery tasks
- download console.txt and job.json from jenkins
when a JenkinsBuildInfo is created
* repoapi.ini:
- fire worker from uwsgi
http://uwsgi-docs.readthedocs.io/en/latest/AttachingDaemons.html
Change-Id: Ib23c45194878a6fdbbe547058013e39516ea2d17
- migrate tag field from 32 to 64 chars max
- add make migrate rule to deploy changes on db
- add panel project-view
Checking just one project per review
- panel release-view upgraded to deal with uuids
no fixed project names
- update API to support dynamic discover of uuids and jobs by release
Change-Id: I0e64b3b341744ec326ee4e123901a1fdcf220baa
- force uwsgi to use python 2.7 ( django 1.8 requirement )
- set /var/lib/repoapi as destination for venv_prod
- debian/rules set VAR_DIR in order not to install venv_prod
Change-Id: I900087558268896798d3d9fe0957c5d733c6346b
- fixing prod settings
- use settings option with manage.py
- create ./venv_prod/run dir for www-data socket.
make run should be run under sudo
- tweak uwsgi.ini
Change-Id: I321d6da3b17a1240ed536fc106cdc91dacbd7fe7
Michael Prokop
Victor Seva