From a8daaaf2469c551021038d695d726b41f04deb58 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 26 Oct 2018 15:51:38 +0200 Subject: [PATCH] TT#46500 Improve development environment While working on the support for triggering $debian_release/master branches (see previous commit) I noticed a bunch of missing or unclear instructions as well as incomplete setup/configuration: * Makefile: add missing `venv_dev` target to populate development environment * extend and clarify usage instructions in README.md * dev environment: avoid having to manually configure ALLOWED_HOSTS, instead identify hostname and listen on that (useful e.g. when running the environment inside docker and then accessing it via the docker IP like http://172.17.0.3:8000/release_panel/) * Dockerfile: add python3-virtualenv + virtualenv as they are needed for the venv handling, also install sqlite3 for handling the sqlite database manually * pin pylint to version 1.9.3[1] [1] We need to pin pylint to version 1.9.3, which is the last working version for us. Newer version of pylint (2.0.0 and up to 2.1.1) fail for us with: | Executing django_jenkins.tasks.run_pylint... | Traceback (most recent call last): | File "./manage.py", line 10, in | execute_from_command_line(sys.argv) | File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line | utility.execute() | File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute | self.fetch_command(subcommand).run_from_argv(self.argv) | File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv | super(Command, self).run_from_argv(argv) | File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv | super(Command, self).run_from_argv(argv) | File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv | self.execute(*args, **cmd_options) | File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute | output = self.handle(*args, **options) | File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle | coverage.save(tested_locations, options) | File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run | lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False) | TypeError: __init__() got an unexpected keyword argument 'exit' This needs further investigation, though independent of this change. It didn't hit us so far as our repoapi docker image isn't rebuilt regularly and all previous builds, until this commit - which refreshes the Dockerfile and we therefore get newer python modules - use older python modules, including an older pylint version. Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1 --- Makefile | 5 +++++ README.md | 46 ++++++++++++++++++++++++++++++----------- repoapi/settings/dev.py | 8 +++++++ requirements/test.txt | 2 +- t/Dockerfile | 4 ++-- 5 files changed, 50 insertions(+), 15 deletions(-) diff --git a/Makefile b/Makefile index 5b6d4fa..7abd22b 100644 --- a/Makefile +++ b/Makefile @@ -10,6 +10,11 @@ venv_prod: requirements/prod.txt virtualenv --python=python3 $(VAR_DIR)/venv_prod source $(VAR_DIR)/venv_prod/bin/activate && \ pip3 install -r ./requirements/prod.txt | tee install.log + +venv_dev: requirements/dev.txt + virtualenv --python=python3 $(VAR_DIR)/venv_dev + source $(VAR_DIR)/venv_dev/bin/activate && \ + pip3 install -r ./requirements/dev.txt | tee install.log ################################### test: diff --git a/README.md b/README.md index 0655b23..10ff200 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,55 @@ repoapi -=========== - -interface to manage our debian repositories metadata +======= +interface to manage our Debian repositories metadata. go away! This is on pre-alpha^4 development stage. +Run docker containers +--------------------- + + $ docker run --rm --hostname repoapi-rabbit --name repoapi-rabbit rabbitmq:3 + $ docker run --rm -i -t --link repoapi-rabbit:rabbit -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-stretch:latest bash + +Prepare development environment +=============================== -Devel environment -================= -$ make venv_dev -$ source ./venv_dev/bin/activate +Inside the repoapi-stretch container run: + + $ export VAR_DIR=/tmp/repoapi + $ make venv_dev + $ source ${VAR_DIR}/venv_dev/bin/activate Create DB ========= -(repoapi)$ ./manage.py migrate + +To ensure `db.sqlite3` exists as needed: + + (repoapi)$ ./manage.py migrate --settings="repoapi.settings.dev" Create superuser ================ -(repoapi)$ ./manage.py createsuperuser + + (repoapi)$ ./manage.py createsuperuser --settings="repoapi.settings.dev" Run test server ================ -(repoapi)$ ./manage.py runserver_plus + +If you want to run it on a specific IP, use: + + (repoapi)$ IP=172.17.0.3 # adjust as needed + (repoapi)$ ./manage.py runserver_plus $IP:8000 --settings="repoapi.settings.dev" + +or just: + + (repoapi)$ make run_dev Tests ===== -(repoapi)$ ./manage.py test + + (repoapi)$ ./manage.py test Reports ======= -(repoapi)$ ./manage.py jenkins + + (repoapi)$ ./manage.py jenkins diff --git a/repoapi/settings/dev.py b/repoapi/settings/dev.py index ebf0667..f1e70f9 100644 --- a/repoapi/settings/dev.py +++ b/repoapi/settings/dev.py @@ -19,6 +19,9 @@ import os # pylint: disable=W0401,W0614,C0413 from .test import * +# avoid having to hardcode an IP address +from socket import gethostname, gethostbyname + LOGGING['loggers']['release_dashboard']['level'] = \ os.getenv('DJANGO_LOG_LEVEL', 'DEBUG') @@ -30,3 +33,8 @@ BROKER_BACKEND = 'amqp' CELERY_ALWAYS_EAGER = False BROKER_URL = 'amqp://guest:guest@rabbit' JBI_BASEDIR = os.path.join(BASE_DIR, 'jbi_files') + +# Enable access when not accessing from localhost: +ALLOWED_HOSTS = [gethostname(), gethostbyname(gethostname()), ] +# or to manually override: +# ALLOWED_HOSTS = ['172.17.0.3'] diff --git a/requirements/test.txt b/requirements/test.txt index 302659e..47ca3b3 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -2,6 +2,6 @@ django-jenkins==0.19 flake8==2.6 pep8 -pylint +pylint==1.9.3 coverage mock diff --git a/t/Dockerfile b/t/Dockerfile index 87867b1..c935fdb 100644 --- a/t/Dockerfile +++ b/t/Dockerfile @@ -5,11 +5,11 @@ FROM docker.mgm.sipwise.com/sipwise-stretch:latest # is updated with the current date. It will force refresh of all # of the base images and things like `apt-get update` won't be using # old cached versions when the Dockerfile is built. -ENV REFRESHED_AT 2017-07-21 +ENV REFRESHED_AT 2018-10-26 RUN apt-get update RUN apt-get install --assume-yes python3 python3-dev \ - python3-pip git screen + python3-pip python3-virtualenv virtualenv sqlite3 git screen # Get pip to download and install requirements: ADD requirements/*.txt /tmp/