TT#46500 Improve development environment

While working on the support for triggering $debian_release/master
branches (see previous commit) I noticed a bunch of missing or unclear
instructions as well as incomplete setup/configuration:

* Makefile: add missing `venv_dev` target to populate development
  environment
* extend and clarify usage instructions in README.md
* dev environment: avoid having to manually configure ALLOWED_HOSTS,
  instead identify hostname and listen on that (useful e.g.
  when running the environment inside docker and then accessing
  it via the docker IP like http://172.17.0.3:8000/release_panel/)
* Dockerfile: add python3-virtualenv + virtualenv as they are
  needed for the venv handling, also install sqlite3 for
  handling the sqlite database manually
* pin pylint to version 1.9.3[1]

[1] We need to pin pylint to version 1.9.3, which is the last
working version for us. Newer version of pylint (2.0.0 and up to
2.1.1) fail for us with:

| Executing django_jenkins.tasks.run_pylint...
| Traceback (most recent call last):
|   File "./manage.py", line 10, in <module>
|     execute_from_command_line(sys.argv)
|   File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
|     utility.execute()
|   File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute
|     self.fetch_command(subcommand).run_from_argv(self.argv)
|   File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv
|     super(Command, self).run_from_argv(argv)
|   File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv
|     super(Command, self).run_from_argv(argv)
|   File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
|     self.execute(*args, **cmd_options)
|   File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
|     output = self.handle(*args, **options)
|   File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle
|     coverage.save(tested_locations, options)
|   File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run
|     lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False)
| TypeError: __init__() got an unexpected keyword argument 'exit'

This needs further investigation, though independent of this change.
It didn't hit us so far as our repoapi docker image isn't rebuilt
regularly and all previous builds, until this commit - which refreshes
the Dockerfile and we therefore get newer python modules - use older
python modules, including an older pylint version.

Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1

Makefile

@@ -10,6 +10,11 @@ venv_prod: requirements/prod.txt
 	virtualenv --python=python3 $(VAR_DIR)/venv_prod
 	source $(VAR_DIR)/venv_prod/bin/activate && \
 		pip3 install -r ./requirements/prod.txt | tee install.log
+
+venv_dev: requirements/dev.txt
+	virtualenv --python=python3 $(VAR_DIR)/venv_dev
+	source $(VAR_DIR)/venv_dev/bin/activate && \
+		pip3 install -r ./requirements/dev.txt | tee install.log
 ###################################
 
 test:

README.md

@@ -1,33 +1,55 @@
 repoapi
-===========
-
-interface to manage our debian repositories metadata
+=======
 
+interface to manage our Debian repositories metadata.
 
 go away! This is on pre-alpha^4 development stage.
 
+Run docker containers
+---------------------
+
+    $ docker run --rm --hostname repoapi-rabbit --name repoapi-rabbit rabbitmq:3
+    $ docker run --rm -i -t --link repoapi-rabbit:rabbit -v $(pwd):/code:rw docker.mgm.sipwise.com/repoapi-stretch:latest bash
+
+Prepare development environment
+===============================
 
-Devel environment
-=================
-$ make venv_dev
-$ source ./venv_dev/bin/activate
+Inside the repoapi-stretch container run:
+
+    $ export VAR_DIR=/tmp/repoapi
+    $ make venv_dev
+    $ source ${VAR_DIR}/venv_dev/bin/activate
 
 Create DB
 =========
-(repoapi)$ ./manage.py migrate
+
+To ensure `db.sqlite3` exists as needed:
+
+    (repoapi)$ ./manage.py migrate --settings="repoapi.settings.dev"
 
 Create superuser
 ================
-(repoapi)$ ./manage.py createsuperuser
+
+    (repoapi)$ ./manage.py createsuperuser --settings="repoapi.settings.dev"
 
 Run test server
 ================
-(repoapi)$ ./manage.py runserver_plus
+
+If you want to run it on a specific IP, use:
+
+    (repoapi)$ IP=172.17.0.3 # adjust as needed
+    (repoapi)$ ./manage.py runserver_plus $IP:8000 --settings="repoapi.settings.dev"
+
+or just:
+
+    (repoapi)$  make run_dev
 
 Tests
 =====
-(repoapi)$ ./manage.py test
+
+    (repoapi)$ ./manage.py test
 
 Reports
 =======
-(repoapi)$ ./manage.py jenkins
+
+    (repoapi)$ ./manage.py jenkins

repoapi/settings/dev.py

@@ -19,6 +19,9 @@ import os
 # pylint: disable=W0401,W0614,C0413
 from .test import *
 
+# avoid having to hardcode an IP address
+from socket import gethostname, gethostbyname
+
 LOGGING['loggers']['release_dashboard']['level'] = \
     os.getenv('DJANGO_LOG_LEVEL', 'DEBUG')
 
@@ -30,3 +33,8 @@ BROKER_BACKEND = 'amqp'
 CELERY_ALWAYS_EAGER = False
 BROKER_URL = 'amqp://guest:guest@rabbit'
 JBI_BASEDIR = os.path.join(BASE_DIR, 'jbi_files')
+
+# Enable access when not accessing from localhost:
+ALLOWED_HOSTS = [gethostname(), gethostbyname(gethostname()), ]
+# or to manually override:
+# ALLOWED_HOSTS = ['172.17.0.3']

requirements/test.txt

@@ -2,6 +2,6 @@
 django-jenkins==0.19
 flake8==2.6
 pep8
-pylint
+pylint==1.9.3
 coverage
 mock

t/Dockerfile

@@ -5,11 +5,11 @@ FROM docker.mgm.sipwise.com/sipwise-stretch:latest
 # is updated with the current date. It will force refresh of all
 # of the base images and things like `apt-get update` won't be using
 # old cached versions when the Dockerfile is built.
-ENV REFRESHED_AT 2017-07-21
+ENV REFRESHED_AT 2018-10-26
 
 RUN apt-get update
 RUN apt-get install --assume-yes python3 python3-dev \
-	python3-pip git screen
+	python3-pip python3-virtualenv virtualenv sqlite3 git screen
 
 # Get pip to download and install requirements:
 ADD requirements/*.txt /tmp/