After switching repoapi to Python3 we noticed that workfront
issues received a wrong target release set by repoapi:
| Updated Target Release to b'mr5.5.1'
The underlying issue is that meta-release-helper's output
is considered a bytes object, while we need a string object.
Demonstration of what's going on, using just the relevant
(though minimized) repoapi code:
| % cat demo.py
| import subprocess
|
| def executeAndReturnOutput(command, env=None):
| proc = subprocess.Popen(command, stdout=subprocess.PIPE,
| stderr=subprocess.PIPE, env=env)
| stdoutdata, stderrdata = proc.communicate()
| return proc.returncode, stdoutdata, stderrdata
|
| def get_next_release(branch):
| command = [
| "/usr/bin/meta-release-helper",
| "--next-release",
| branch
| ]
| res = executeAndReturnOutput(command)
| if res[0] != 0:
| return None
| val = res[1].rstrip()
| if len(val) > 0:
| return val
| else:
| return None
|
| release = get_next_release("master")
| print(type(release))
|
| % python2 demo.py
| <type 'str'>
| % python3 demo.py
| <class 'bytes'>
NOTE regarding tests: inside our tests (test_utils.py) the call
to meta-release-helper gets mocked/patched, so it's set to a
string object already (which differs from what we get in actual
usage/production). That's why this issue was hidden so far.
NOTE for later debugging: if repoapi source gets changed
don't forget to restart uwsgi service (`systemctl restart uwsgi.service`)
and find its logging output in file `/var/log/uwsgi/app/repoapi.log`
(which is NOT same es `journalctl -u uwsgi.service`!).
Change-Id: I0f797ed3f7ae49c963c2f3c07fd601221e1dcd6c
In Python3 all strings are considered Unicode by default. The
unicode type from Python 2 is called str in Python 3, and str
becomes bytes. In Python3, there aren’t any automatic conversions
between str and bytes though. The json.loads in Python versions
>3 and <3.6 expects a string representation and not bytes. As we
receive bytes we need to decode() it, as this always returns str,
just as needed.
Fixes:
| File "./release_dashboard/views/build.py", line 59, in hotfix_build
| json_data = json.loads(request.body)
| File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
| s.__class__.__name__))
| TypeError: the JSON object must be str, not 'bytes'
See https://docs.djangoproject.com/en/1.11/topics/python3/ and
https://vinta.ws/code/notes-of-porting-to-python-3-for-a-django-project.html
for related documentation.
Change-Id: I3984c5fd1f79b94680e680c8da64dd0ac111cbe9
Lots of changes, time for a new release, since it's a switch
from Python2 to Python3, updating Django and its components
and switch from Debian jessie to stretch let's bump the minor
version number.
Change-Id: Id400901510e34f6c90e8eea538fe0f24acb23dd0
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
* using https://github.com/manosim/django-rest-framework-api-key
for Permissions. Will generate an Auth KEY for jenkins to be able
to trigger build from there using build REST API
* store the info needed to trigger builds for a set of projects for
a release
- build is a list of projects to be built with branch or tag as orig
and release as destination for a debian distribution
* added release_uuid to JenkinsJobInfo model
- release_uuid is the way to group project builds for a release
- tag works for group of jobs for a project
Change-Id: Iebeaa54308c3e569a167f55d98c184b52248af8a
* we need to use reference ( docker API it doesn't support tag name )
* fix tag date (value in the first occurrence not last)
Change-Id: I2fa2f7bedaca4cdeeb3a3a2d4a3f4a661d15fad1
* provide info of actual images and tags
* refresh docker images/tag via release_dashboard
* keep that info in db
* fix Dockerfile documentation
* cleanup thanks to flake8
Change-Id: I743482da9b4d50f7b1832cad324882f3a49cbcf0
* no need to have files twice
* djangorestframework==3.4 so we keep stable compatibility
- fix small issue with Serializer classes for that version
Change-Id: Ib0f04def39b41adf8cc0a700fe096e405b620cb0
* update dependencies due postgresql
* fake files needed to deploy at debian
* add new needed config values at server.ini file
Change-Id: I02699c721ac79a0328da07a4c2e3a2cb1451d403
* don't trigger get_jbi_files for hosts not in JBI_ALLOWED_HOSTS
- prod => ['jenkins.mgm.sipwise.com']
- dev => ['jenkins-dev.mgm.sipwise.com']
Change-Id: I14bf397a54ea5dcad9ec64a6660c4f6e86a387b6
In order to be able to deploy that file with different values
depending on the server so we can use -dev.mgm services on
repoapi-dev.mgm
Change-Id: I2f5546c4f664707250ab89f29e71d4092c848890
Alexander Lutay
Michael Prokop
Sergii Kipot
Victor Seva