TT#19059 Port to Python3, update to latest Django LTS version + based on Debian/stretch

Updates:

* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
  next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
  (thanks Florian Apolloner for feedback and suggestions)

Necessary changes:

* Debian:
  - we need to use uwsgi-plugin-python3 for Python3 support
  - replace python-dev with python3-dev for Python3 support
    and python with python3 accordingly
  - get rid of __pycache__ directories to avoid inclusion in
    resulting Debian package

* uwsgi:
  - if we hardcode python34 (for jessie) or python35 (for stretch)
    then we can't run uwsgi on all supported systems, so instead
    try `plugin = python3` and rely on update-alternatives mechanism
    of uwsgi-plugin-python3 Debian package to get it right for,
    if that shouldn't work as needed then we can look into
    /usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
    or shipping repoapi.ini via puppet instead

* docker:
  - python-distribute is not needed (anymore)
  - adjust package names for Python3 usage, support jessie
    and stretch at the same time
  - note: just replacing 'stretch' with 'jessie' in t/Dockerfile
    is enough to get a working repoapi docker image based on jessie

* Python3:
  - assertItemsEqual became assertCountEqual
  - mocking: `__builtin__.open` became `builtins.open`, see
    https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
  - we need six >=1.9.0 for raise_from astraction in py3
    Otherwise we're failing on Debian/jessie with:
    | AttributeError: 'module' object has no attribute 'raise_from'
    See https://github.com/benjaminp/six/blob/master/CHANGES
  - assert_has_calls changes, fixes:
    | AssertionError: Calls not found.
    | Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
    |  call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
    | Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
    |  call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
    See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
  - urllib:
    - `from urlparse import urlparse` became `from urllib.parse import urlparse`
    - `urllib.quote` became `urllib.parse.quote`
  - configparser:
    - `from ConfigParser import ...` became `from configparser import ...`

* Django:
  - 'from views import ...' became 'from .views import ...'
  - the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
    nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/

* celery:
  - to use latest celery version 4 django-celery needs to be replaced
    by django-celery-beat + django-celery-results
    (otherwise we're running into https://github.com/celery/django-celery/issues/497),
    see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
  - adjust systemd unit files to use ExecStart
    | ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
    instead of
    | ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
    and use `--db=/var/lib/repoapi/flower` instead of
    `--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
    named flower.db.db

Notes for migration (assuming root permissions):

  - get rid of existing flower database (not containing relevant data):
    mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old

  - re-deploy virtual environment from scratch:
    mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
    cd /usr/share/repoapi && make deploy

  - service restarts (in case of virtual env changes):
    systemctl restart nginx
    systemctl restart repoapi-beat.service
    systemctl restart repoapi-flower.service
    systemctl restart repoapi-worker.service
    systemctl restart uwsgi

  - usage instructions for celery + django stuff:
    /var/lib/repoapi/venv_prod/bin/celery flower --help
    source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"

  - create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
    source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"

Thanks: Victor Seva for assistance with porting

Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
changes/46/14446/19
Michael Prokop 8 years ago
parent 6330e15a11
commit 1e9aa77d53

@ -7,9 +7,9 @@ all:
.ONESHELL:
SHELL = /bin/bash
venv_prod: requirements/prod.txt
virtualenv --python=python2.7 $(VAR_DIR)/venv_prod
virtualenv --python=python3 $(VAR_DIR)/venv_prod
source $(VAR_DIR)/venv_prod/bin/activate && \
pip install -r ./requirements/prod.txt | tee install.log
pip3 install -r ./requirements/prod.txt | tee install.log
###################################
test:
@ -67,4 +67,4 @@ clean:
distclean dist-clean: clean
rm -rf venv*
.PHONY: all test clean dist-clean
.PHONY: all test clean dist-clean

@ -24,5 +24,5 @@ class BuildReleaseTestCase(TestCase):
def test_projects_list(self):
build = BuildRelease.objects.get(
uuid="dbe569f7-eab6-4532-a6d1-d31fb559649b")
self.assertItemsEqual(build.projects_list,
self.assertCountEqual(build.projects_list,
['kamailio', 'lua-ngcp-kamailio', 'ngcp-panel'])

@ -85,4 +85,4 @@ class TestRest(APIAuthenticatedTestCase):
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
data_all['start_date'] = response.data['start_date']
data_all['id'] = response.data['id']
self.assertItemsEqual(response.data, data_all)
self.assertCountEqual(response.data, data_all)

@ -34,23 +34,23 @@ def trigger_build(project, release_uuid, trigger_release=None,
params = {
'base': settings.JENKINS_URL,
'job': project,
'token': urllib.quote(settings.JENKINS_TOKEN),
'cause': urllib.quote(trigger_release),
'token': urllib.parse.quote(settings.JENKINS_TOKEN),
'cause': urllib.parse.quote(trigger_release),
'branch': 'none',
'tag': 'none',
'release': urllib.quote(trigger_release),
'distribution': urllib.quote(trigger_distribution),
'release': urllib.parse.quote(trigger_release),
'distribution': urllib.parse.quote(trigger_distribution),
'uuid': uuid.uuid4(),
'release_uuid': release_uuid,
}
if trigger_branch_or_tag.startswith("tag/"):
tag = trigger_branch_or_tag.split("tag/")[1]
params['tag'] = urllib.quote(tag)
params['tag'] = urllib.parse.quote(tag)
elif trigger_branch_or_tag.startswith("branch/"):
branch = trigger_branch_or_tag.split("branch/")[1]
params['branch'] = urllib.quote(branch)
params['branch'] = urllib.parse.quote(branch)
else:
params['branch'] = urllib.quote(trigger_branch_or_tag)
params['branch'] = urllib.parse.quote(trigger_branch_or_tag)
url = project_url.format(**params)
if settings.DEBUG:

8
debian/control vendored

@ -5,7 +5,7 @@ Build-Depends:
debhelper (>= 9~),
dh-systemd (>= 1.5),
libpq-dev,
python-dev,
python3-dev,
virtualenv,
Standards-Version: 3.9.8
Section: python
@ -19,11 +19,11 @@ Depends:
libpq-dev,
make,
postgresql,
python,
python-dev,
python3,
python3-dev,
sipwise-repos-scripts-helpers,
uwsgi,
uwsgi-plugin-python,
uwsgi-plugin-python3,
virtualenv,
${misc:Depends},
Description: REST API webapp

@ -9,7 +9,7 @@ Group=www-data
Environment=DJANGO_SETTINGS_MODULE=repoapi.settings.prod
PIDFile=/var/lib/repoapi/celery-beat.pid
WorkingDirectory=/usr/share/repoapi
ExecStart=/var/lib/repoapi/venv_prod/bin/python ./manage.py celery beat \
ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi beat \
--schedule=/var/lib/repoapi/celery-beat.schedule \
--pidfile=/var/lib/repoapi/celery-beat.pid --loglevel=INFO

@ -8,9 +8,9 @@ User=www-data
Group=www-data
Environment=DJANGO_SETTINGS_MODULE=repoapi.settings.prod
WorkingDirectory=/usr/share/repoapi
ExecStart=/var/lib/repoapi/venv_prod/bin/python ./manage.py celery flower \
--db=/var/lib/repoapi/flower.db --persistent --xheaders \
ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi flower \
--db=/var/lib/repoapi/flower --persistent --xheaders \
--url_prefix=flower --unix_socket=/var/lib/repoapi/celery-flower.socket
[Install]
WantedBy=multi-user.target
WantedBy=multi-user.target

@ -9,8 +9,8 @@ Group=www-data
Environment=DJANGO_SETTINGS_MODULE=repoapi.settings.prod
PIDFile=/var/lib/repoapi/celery-worker.pid
WorkingDirectory=/usr/share/repoapi
ExecStart=/var/lib/repoapi/venv_prod/bin/python ./manage.py celery worker \
ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi worker \
--pidfile=/var/lib/repoapi/celery-worker.pid --loglevel=INFO
[Install]
WantedBy=multi-user.target
WantedBy=multi-user.target

1
debian/rules vendored

@ -22,6 +22,7 @@ override_dh_auto_build:
override_dh_auto_install:
VAR_DIR=$(shell pwd) make deploy
find . -name '__pycache__' -type d -print0 | xargs --no-run-if-empty -0 rm -r
make clean
override_dh_fixperms:

@ -6,7 +6,7 @@
tasks:
- name: install Debian package requirements
apt: name=libyaml-dev state=present
apt: name=python2.7-dev state=present
apt: name=python3-dev state=present
apt: name=python-virtualenv state=present
- hosts: all

@ -54,10 +54,10 @@ class TestHotfixReleased(BaseTest):
}
return defaults
@patch('__builtin__.open', mock_open(read_data=debian_changelog))
@patch('builtins.open', mock_open(read_data=debian_changelog))
def test_parse_changelog(self):
ids, changelog = utils.parse_changelog("/tmp/fake.txt")
self.assertItemsEqual(ids, ["345", "123"])
self.assertCountEqual(ids, ["345", "123"])
self.assertEquals(changelog.full_version, "3.8.7.4+0~mr3.8.7.4")
self.assertEquals(changelog.package, "ngcp-fake")
@ -69,7 +69,7 @@ class TestHotfixReleased(BaseTest):
val = utils.get_target_release("3.8.7.4-1")
self.assertIsNone(val)
@patch('__builtin__.open', mock_open(read_data=debian_changelog))
@patch('builtins.open', mock_open(read_data=debian_changelog))
@patch('repoapi.utils.dlfile')
@patch('repoapi.utils.workfront_set_release_target')
@patch('repoapi.utils.workfront_note_send')
@ -97,6 +97,7 @@ class TestHotfixReleased(BaseTest):
self.assertEquals(gri.count(), 1)
msg = "hotfix %s.git %s triggered" % (projectname, version)
calls.append(call("123", msg))
wns.assert_has_calls(calls)
wns.assert_has_calls(calls, any_order=True)
wsrt.assert_has_calls(
[call("345", "mr3.8.7.4"), call("123", "mr3.8.7.4")])
[call("345", "mr3.8.7.4"), call("123", "mr3.8.7.4")],
any_order=True)

@ -1,4 +1,4 @@
#!/usr/bin/env python
#!/usr/bin/env python3
import os
import sys

@ -66,26 +66,26 @@ class ProjectTestCase(TestCase):
proj = Project.objects.create(name="fake", json_tags=FILTERED_TAGS)
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.tags, list)
self.assertItemsEqual(proj.tags, ["mr2.0.0", "mr1.0.0", ])
self.assertCountEqual(proj.tags, ["mr2.0.0", "mr1.0.0", ])
def test_tags_null(self):
proj = Project.objects.create(name="fake")
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.tags, list)
self.assertItemsEqual(proj.tags, [])
self.assertCountEqual(proj.tags, [])
def test_branches(self):
proj = Project.objects.create(name="fake",
json_branches=FILTERED_BRANCHES)
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.branches, list)
self.assertItemsEqual(proj.branches, ["vseva/1789", "master"])
self.assertCountEqual(proj.branches, ["vseva/1789", "master"])
def test_branches_null(self):
proj = Project.objects.create(name="fake")
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.branches, list)
self.assertItemsEqual(proj.branches, [])
self.assertCountEqual(proj.branches, [])
def test_filtered_json(self):
res = Project._get_filtered_json(GERRIT_REST_TAGS)
@ -116,17 +116,17 @@ class ProjectTestCase(TestCase):
proj = Project.objects.create(name="fake")
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.tags, list)
self.assertItemsEqual(proj.tags, [])
self.assertCountEqual(proj.tags, [])
proj.tags = GERRIT_REST_TAGS
self.assertItemsEqual(proj.tags, ["mr2.0.0", "mr1.0.0"])
self.assertCountEqual(proj.tags, ["mr2.0.0", "mr1.0.0"])
def test_branches_set(self):
proj = Project.objects.create(name="fake")
self.assertEquals(proj.name, "fake")
self.assertIsInstance(proj.branches, list)
self.assertItemsEqual(proj.branches, [])
self.assertCountEqual(proj.branches, [])
proj.branches = GERRIT_REST_BRANCHES
self.assertItemsEqual(proj.branches, ["master", "vseva/1789"])
self.assertCountEqual(proj.branches, ["master", "vseva/1789"])
def test_branches_mrXX(self):
tmp = [
@ -138,7 +138,7 @@ class ProjectTestCase(TestCase):
proj = Project.objects.create(name="fake",
json_branches=tmp)
self.assertEquals(proj.name, "fake")
self.assertItemsEqual(proj.branches_mrXX(),
self.assertCountEqual(proj.branches_mrXX(),
["mr0.1", ])
def test_branches_mrXXX(self):
@ -151,5 +151,5 @@ class ProjectTestCase(TestCase):
proj = Project.objects.create(name="fake",
json_branches=tmp)
self.assertEquals(proj.name, "fake")
self.assertItemsEqual(proj.branches_mrXXX(),
self.assertCountEqual(proj.branches_mrXXX(),
["mr0.1.1", ])

@ -28,13 +28,13 @@ class DockerImageTestCase(TestCase):
def test_create(self):
image = diobj.create(
name='fake-jessie', project=self.proj)
self.assertItemsEqual(self.proj.dockerimage_set.all(),
self.assertCountEqual(self.proj.dockerimage_set.all(),
[image, ])
def test_remove_image(self):
image = diobj.create(
name='fake-jessie', project=self.proj)
self.assertItemsEqual(self.proj.dockerimage_set.all(),
self.assertCountEqual(self.proj.dockerimage_set.all(),
[image, ])
image.delete()
self.assertTrue(Project.objects.filter(name="fake").exists())
@ -42,7 +42,7 @@ class DockerImageTestCase(TestCase):
def test_remove_project(self):
image = diobj.create(
name='fake-jessie', project=self.proj)
self.assertItemsEqual(self.proj.dockerimage_set.all(), [image, ])
self.assertCountEqual(self.proj.dockerimage_set.all(), [image, ])
self.proj.delete()
self.assertFalse(Project.objects.filter(name="fake").exists())
self.assertFalse(diobj.filter(name="fake").exists())
@ -50,24 +50,24 @@ class DockerImageTestCase(TestCase):
def test_filter_images(self):
images = ['fake-jessie', 'other', 'ngcp-fake', 'fake-more']
images_ok = ['fake-jessie', 'ngcp-fake', 'fake-more']
self.assertItemsEqual(
self.assertCountEqual(
self.proj.filter_docker_images(images), images_ok)
def test_image_tags(self):
image = diobj.create(
name='fake-jessie', project=self.proj)
self.assertItemsEqual(image.tags, [])
self.assertCountEqual(image.tags, [])
DockerTag.objects.create(
name='latest',
image=image,
manifests='{}')
self.assertItemsEqual(image.tags, ['latest', ])
self.assertCountEqual(image.tags, ['latest', ])
DockerTag.objects.create(
name='mr5.4',
image=image,
manifests='{}',
reference='whatever')
self.assertItemsEqual(image.tags, ['latest', 'mr5.4'])
self.assertCountEqual(image.tags, ['latest', 'mr5.4'])
class DockerImageTest2Case(TestCase):
@ -83,20 +83,20 @@ class DockerImageTest2Case(TestCase):
]
def test_images_with_tags(self):
self.assertItemsEqual(
self.assertCountEqual(
diobj.images_with_tags(),
self.images_with_tags)
def test_project_images_with_tags(self):
self.assertItemsEqual(
self.assertCountEqual(
diobj.images_with_tags('data-hal'),
[diobj.get(name='data-hal-jessie'), ])
self.assertItemsEqual(
self.assertCountEqual(
diobj.images_with_tags('ngcp-panel'),
[diobj.get(name='ngcp-panel-selenium'),
diobj.get(name='ngcp-panel-tests-rest-api-jessie'),
diobj.get(name='ngcp-panel-tests-selenium-jessie'), ])
self.assertItemsEqual(diobj.images_with_tags('libtcap'), [])
self.assertCountEqual(diobj.images_with_tags('libtcap'), [])
def test_date(self):
tag = DockerTag.objects.get(

@ -72,7 +72,7 @@ class TasksDockerTestCase(TestCase):
self.assertEquals(proj.name, "data-hal")
image = DockerImage.objects.create(
name='data-hal-jessie', project=proj)
self.assertItemsEqual(proj.dockerimage_set.all(), [image, ])
self.assertCountEqual(proj.dockerimage_set.all(), [image, ])
result = tasks.docker_fetch_info.delay('data-hal-jessie')
self.assertTrue(result.successful())
image = DockerImage.objects.get(name='data-hal-jessie')
@ -85,7 +85,7 @@ class TasksDockerTestCase(TestCase):
call("data-hal-jessie/manifests/latest"),
]
gdmi.assert_has_calls(calls)
self.assertItemsEqual(image.tags, ["I3a899", "latest"])
self.assertCountEqual(image.tags, ["I3a899", "latest"])
@patch('release_dashboard.utils.docker.get_docker_manifests_info',
side_effect=fake_manifest)
@ -106,7 +106,7 @@ class TasksDockerTestCase(TestCase):
call("data-hal-jessie/manifests/latest"),
]
gdmi.assert_has_calls(calls)
self.assertItemsEqual(image.tags, ["I3a899", "latest"])
self.assertCountEqual(image.tags, ["I3a899", "latest"])
@patch('release_dashboard.utils.docker.get_docker_manifests_info',
side_effect=fake_manifest)
@ -118,9 +118,9 @@ class TasksDockerTestCase(TestCase):
proj = Project.objects.get(name="data-hal")
images = [DockerImage.objects.get(name='data-hal-jessie'),
DockerImage.objects.get(name='data-hal-selenium-jessie')]
self.assertItemsEqual(proj.dockerimage_set.all(), images)
self.assertItemsEqual(images[0].tags, ["I3a899", "latest"])
self.assertItemsEqual(images[1].tags, ["If53a9", "latest"])
self.assertCountEqual(proj.dockerimage_set.all(), images)
self.assertCountEqual(images[0].tags, ["I3a899", "latest"])
self.assertCountEqual(images[1].tags, ["If53a9", "latest"])
calls = [
call("_catalog"),
call("data-hal-jessie/tags/list"),

@ -53,7 +53,7 @@ class UtilsDockerTestCase(TestCase):
@patch('release_dashboard.utils.docker.get_docker_info')
def test_get_docker_repositories(self, gdi):
gdi.return_value = DOCKER_REST_CATALOG
self.assertItemsEqual(
self.assertCountEqual(
docker.get_docker_repositories(),
['fake-jessie',
'fake-selenium-jessie',
@ -64,7 +64,7 @@ class UtilsDockerTestCase(TestCase):
@patch('release_dashboard.utils.docker.get_docker_info',
side_effect=fake_tag)
def test_get_docker_tags(self, gdi):
self.assertItemsEqual(
self.assertCountEqual(
docker.get_docker_tags('fake-jessie'),
["I3a899b8945688c2ef3a4be6ba6c4c1d4cbf6d548",
"latest"])
@ -76,7 +76,7 @@ class UtilsDockerTestCase(TestCase):
@patch('release_dashboard.utils.docker.get_docker_info',
side_effect=fake_tag)
def test_get_docker_tags_empty(self, gdi):
self.assertItemsEqual(docker.get_docker_tags('other'), [])
self.assertCountEqual(docker.get_docker_tags('other'), [])
calls = [
call("other/tags/list"),
]

@ -14,7 +14,7 @@
# with this program. If not, see <http://www.gnu.org/licenses/>.
from django.conf.urls import url
from views import build, docker
from .views import build, docker
urlpatterns = [
url(r'^$', build.index, name='index'),

@ -46,11 +46,11 @@ def trigger_hotfix(project, branch, push="yes"):
flow_uuid = uuid.uuid4()
params = {
"base": settings.JENKINS_URL,
'token': urllib.quote(settings.JENKINS_TOKEN),
'action': urllib.quote("--hotfix"),
'branch': urllib.quote(branch),
'project': urllib.quote(project),
'push': urllib.quote(push),
'token': urllib.parse.quote(settings.JENKINS_TOKEN),
'action': urllib.parse.quote("--hotfix"),
'branch': urllib.parse.quote(branch),
'project': urllib.parse.quote(project),
'push': urllib.parse.quote(push),
'uuid': flow_uuid,
}
@ -74,22 +74,22 @@ def trigger_build(project, trigger_release=None,
params = {
'base': settings.JENKINS_URL,
'job': project,
'token': urllib.quote(settings.JENKINS_TOKEN),
'cause': urllib.quote(trigger_release),
'token': urllib.parse.quote(settings.JENKINS_TOKEN),
'cause': urllib.parse.quote(trigger_release),
'branch': 'none',
'tag': 'none',
'release': urllib.quote(trigger_release),
'distribution': urllib.quote(trigger_distribution),
'release': urllib.parse.quote(trigger_release),
'distribution': urllib.parse.quote(trigger_distribution),
'uuid': flow_uuid,
}
if trigger_branch_or_tag.startswith("tag/"):
tag = trigger_branch_or_tag.split("tag/")[1]
params['tag'] = urllib.quote(tag)
params['tag'] = urllib.parse.quote(tag)
elif trigger_branch_or_tag.startswith("branch/"):
branch = trigger_branch_or_tag.split("branch/")[1]
params['branch'] = urllib.quote(branch)
params['branch'] = urllib.parse.quote(branch)
else:
params['branch'] = urllib.quote(trigger_branch_or_tag)
params['branch'] = urllib.parse.quote(trigger_branch_or_tag)
url = project_url.format(**params)
if settings.DEBUG:

@ -35,9 +35,9 @@ def trigger_docker_build(project, branch):
branch = branch.split("branch/")[1]
params = {
'base': settings.JENKINS_URL,
'token': urllib.quote(settings.JENKINS_TOKEN),
'token': urllib.parse.quote(settings.JENKINS_TOKEN),
'project': project,
'branch': urllib.quote(branch),
'branch': urllib.parse.quote(branch),
}
url = docker_url.format(**params)

@ -1,8 +1,8 @@
[uwsgi]
# Django-related settings
# django 1.8 needs python >= 2.7
plugin = python27
# django 1.11 needs python >= 2.7, though it's the latest LTS release to support python2 at all
plugin = python3
# the base directory (full path)
chdir = /usr/share/repoapi
# Django's wsgi file
@ -17,3 +17,13 @@ workers = 10
# Better process names
auto-procname = true
procname-prefix-spaced = [%n]
# improve behavior esp with systemd
die-on-term = true
# improve runtime behavior
enable-threads = true
thunder-lock = true
# avoid plugin autoloading problems (cmdline argument magic)
autoload = false

@ -21,7 +21,7 @@ from django.db import models
from django.forms.models import model_to_dict
from django.conf import settings
from repoapi.tasks import get_jbi_files
from urlparse import urlparse
from urllib.parse import urlparse
logger = logging.getLogger(__name__)
workfront_re = re.compile(r"TT#(\d+)")

@ -39,9 +39,9 @@ INSTALLED_APPS = [
'rest_framework',
'rest_framework_api_key',
'rest_framework_swagger',
'django_extensions',
'django_assets',
'djcelery',
'django_celery_results',
'django_extensions',
'jsonify',
]
@ -61,7 +61,9 @@ ROOT_URLCONF = 'repoapi.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'DIRS': [
'repoapi/templates',
],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
@ -103,10 +105,6 @@ STATICFILES_FINDERS = (
STATIC_ROOT = os.path.join(BASE_DIR, 'static_media/')
TEMPLATE_DIRS = (
'repoapi/templates',
)
REST_FRAMEWORK = {
'PAGE_SIZE': 10,
'DEFAULT_FILTER_BACKENDS': (
@ -149,7 +147,7 @@ JENKINS_TOKEN = "sipwise_jenkins_ci"
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_RESULT_BACKEND = 'django-db'
HOTFIX_ARTIFACT = 'debian_changelog.txt'

@ -15,7 +15,7 @@
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
from ConfigParser import RawConfigParser
from configparser import RawConfigParser
# pylint: disable=W0401,W0614
from .common import *
from celery.schedules import crontab

@ -56,7 +56,7 @@ class TestJBICelery(BaseTest):
}
return defaults
@patch('__builtin__.open', mock_open(read_data=artifacts_json))
@patch('builtins.open', mock_open(read_data=artifacts_json))
@patch('repoapi.utils.dlfile')
def test_jbi_path_creation(self, dlfile):
param = self.get_defaults()
@ -66,7 +66,7 @@ class TestJBICelery(BaseTest):
jbi.jobname, str(jbi.buildnumber))
self.assertTrue(os.path.isdir(base_path), base_path)
@patch('__builtin__.open', mock_open(read_data=artifacts_json))
@patch('builtins.open', mock_open(read_data=artifacts_json))
@patch('repoapi.utils.dlfile')
def test_jbi_console(self, dlfile):
param = self.get_defaults()
@ -91,7 +91,7 @@ class TestJBICelery(BaseTest):
path = os.path.join(artifact_base_path, 'builddeps.list')
self.assertNotIn(call(url, path), dlfile.call_args_list)
@patch('__builtin__.open', mock_open(read_data=artifacts_json))
@patch('builtins.open', mock_open(read_data=artifacts_json))
@patch('repoapi.utils.dlfile')
def test_jbi_buildinfo(self, dlfile):
param = self.get_defaults()
@ -115,7 +115,7 @@ class TestJBICelery(BaseTest):
path = os.path.join(artifact_base_path, 'builddeps.list')
self.assertNotIn(call(url, path), dlfile.call_args_list)
@patch('__builtin__.open', mock_open(read_data=artifacts_json))
@patch('builtins.open', mock_open(read_data=artifacts_json))
@patch('repoapi.utils.dlfile')
def test_jbi_artifact(self, dlfile):
param = self.get_defaults()
@ -133,7 +133,7 @@ class TestJBICelery(BaseTest):
path = os.path.join(artifact_base_path, 'builddeps.list')
dlfile.assert_any_call(url, path)
@patch('__builtin__.open', mock_open(read_data=artifacts_json))
@patch('builtins.open', mock_open(read_data=artifacts_json))
@patch('repoapi.utils.dlfile')
def test_jbi_envVars(self, dlfile):
param = self.get_defaults()

@ -24,12 +24,12 @@ class JBIQueriesTestCase(BaseTest):
def test_releases(self):
releases = JenkinsBuildInfo.objects.releases()
self.assertItemsEqual(releases, ['mr3.1-fake', ])
self.assertCountEqual(releases, ['mr3.1-fake', ])
def test_release_projects(self):
projects = ['fake', ]
check = JenkinsBuildInfo.objects.release_projects('mr3.1-fake')
self.assertItemsEqual(check, projects)
self.assertCountEqual(check, projects)
def test_release_project_uuids(self):
projects = ['fake', ]
@ -40,18 +40,18 @@ class JBIQueriesTestCase(BaseTest):
for project in projects:
uuids[project] = JenkinsBuildInfo.objects.release_project_uuids(
'mr3.1-fake', project)
self.assertItemsEqual(uuids_ok[project], uuids[project])
self.assertCountEqual(uuids_ok[project], uuids[project])
def test_jobs_by_uuid(self):
jobs = JenkinsBuildInfo.objects.jobs_by_uuid(
'mr3.1-fake', 'fake', 'UUID0')
self.assertItemsEqual(
self.assertCountEqual(
JenkinsBuildInfo.objects.filter(param_release='mr3.1-fake',
projectname='fake',
tag='UUID0'), jobs)
jobs = JenkinsBuildInfo.objects.jobs_by_uuid(
'mr3.1-fake', 'fake', 'UUID1')
self.assertItemsEqual(
self.assertCountEqual(
JenkinsBuildInfo.objects.filter(param_release='mr3.1-fake',
projectname='fake',
tag='UUID1'), jobs)

@ -23,15 +23,15 @@ class WorkfrontNoteTestCase(BaseTest):
def test_getID(self):
res = WorkfrontNoteInfo.getIds("jojo TT#0891 whatever")
self.assertItemsEqual(res, ['0891'])
self.assertCountEqual(res, ['0891'])
def test_getID_multiple(self):
res = WorkfrontNoteInfo.getIds("jojo TT#0891 whatever TT#0001")
self.assertItemsEqual(res, ['0891', '0001'])
self.assertCountEqual(res, ['0891', '0001'])
def test_getID_multiple_duplicate(self):
res = WorkfrontNoteInfo.getIds("jojo TT#0891 whatever TT#0001 TT#0891")
self.assertItemsEqual(res, ['0891', '0001'])
self.assertCountEqual(res, ['0891', '0001'])
def test_getCommit(self):
res = WorkfrontNoteInfo.getCommit("1234567 TT#67676 whatever")
@ -269,7 +269,7 @@ class WorkfrontNoteTestCase(BaseTest):
param['param_branch'],
settings.GITWEB_URL.format("kamailio", "7fg4567"))
gnr.assert_called_once_with("stretch/master")
self.assertItemsEqual(wsrt.mock_calls, [])
self.assertCountEqual(wsrt.mock_calls, [])
wsrt.assert_not_called()
wns.assert_called_once_with("0001", msg)

@ -19,7 +19,7 @@ import logging
import os
import shutil
import subprocess
import urllib2
import urllib3
from django.conf import settings
logger = logging.getLogger(__name__)
@ -43,16 +43,16 @@ def dlfile(url, path):
if settings.DEBUG:
logger.info("I would call %s", url)
else:
remote_file = urllib2.urlopen(url)
remote_file = urllib3.urlopen(url)
logger.debug("url:[%s]", url)
with open(path, "wb") as local_file:
shutil.copyfileobj(remote_file, local_file)
def openurl(url):
req = urllib2.Request(url)
req = urllib3.Request(url)
logger.debug("url:[%s]", url)
response = urllib2.urlopen(req)
response = urllib3.urlopen(req)
if response.code > 199 and response.code < 300:
logger.debug("OK[%d] url: %s", url, response.code)
return True

@ -1,16 +1,17 @@
Django==1.9
Django==1.11
django-extensions
yuicompressor
django-assets
djangorestframework==3.4
djangorestframework
drfapikey
django-rest-swagger
markdown
django-filter
six==1.10
six>=1.9
webassets
celery<4.0
django-celery
celery
django-celery-beat
django-celery-results
flower
django-jsonify
python-debian

@ -1,19 +1,19 @@
# DOCKER_NAME=repoapi-jessie
FROM docker.mgm.sipwise.com/sipwise-jessie:latest
# DOCKER_NAME=repoapi-stretch
FROM docker.mgm.sipwise.com/sipwise-stretch:latest
# Important! Update this no-op ENV variable when this Dockerfile
# is updated with the current date. It will force refresh of all
# of the base images and things like `apt-get update` won't be using
# old cached versions when the Dockerfile is built.
ENV REFRESHED_AT 2017-06-28
ENV REFRESHED_AT 2017-07-21
RUN apt-get update
RUN apt-get install --assume-yes python2.7 python2.7-dev \
python-distribute python-pip git screen
RUN apt-get install --assume-yes python3 python3-dev \
python3-pip git screen
# Get pip to download and install requirements:
ADD requirements/*.txt /tmp/
RUN pip install -r /tmp/test.txt
RUN pip3 install -r /tmp/test.txt
RUN echo './t/testrunner' >>/root/.bash_history
@ -25,23 +25,23 @@ WORKDIR /code/
# Instructions for usage
# ----------------------
# When you want to build the base image from scratch (jump to the next section if you don't want to build yourself!):
# % docker build --tag="repoapi-jessie" -f t/Dockerfile .
# % docker run --rm -i -t -v $(pwd):/code:rw repoapi-jessie:latest bash
# % docker build --tag="repoapi-stretch" -f t/Dockerfile .
# % docker run --rm -i -t -v $(pwd):/code:rw repoapi-stretch:latest bash
#
# Use the existing docker image:
# % docker pull docker.mgm.sipwise.com/repoapi-jessie
# % docker run --rm -i -t -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-jessie:latest bash
# % docker pull docker.mgm.sipwise.com/repoapi-stretch
# % docker run --rm -i -t -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-stretch:latest bash
#
# Inside docker (the command is in history, just press UP button):
# ./t/testrunner
#
# Run django inside docker:
# % pip install -r requirements/dev.txt && make run_dev
# % pip3 install -r requirements/dev.txt && make run_dev
#
# We need a working rabbit server, so in another terminal:
# % docker run --rm --hostname repoapi-rabbit --name repoapi-rabbit rabbitmq:3
# and link both containers:
# % docker run --rm -i -t --link repoapi-rabbit:rabbit -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-jessie:latest bash
# % docker run --rm -i -t --link repoapi-rabbit:rabbit -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-stretch:latest bash
#
# use screen to get a working worker in the background:
# % make worker_dev

Loading…
Cancel
Save