Use --break-system-packages option as the new pip3 claims:
error: externally-managed-environment
...
note: If you believe this is a mistake, please contact your
Python installation or OS distribution provider. You can
override this, at the risk of breaking your Python
installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Change-Id: I11b29dfe31a5404779b732923b6703670073a365
When invoking `virtualenv --python=python3 virtualenv` to deploy the
virtual environment, installing the python-debian wheel, and then
re-executing `virtualenv --python=python3 virtualenv` once again, we
might end up without the virtualenv/lib/python*/site-packages/debian
directory, resulting in:
| Traceback (most recent call last):
| File "<string>", line 1, in <module>
| ModuleNotFoundError: No module named 'debian'
This is a bug in setuptools >=51.3.3-1, see
https://bugs.debian.org/1041091 for the details, so currently affecting
all systems with Debian/bookworm.
By excluding installation of setuptools (via --no-setuptools), we would
be able to work around this bootstrapping problem itself, though then
installation of django-jsonify (which we depend on) fails with:
| ModuleNotFoundError: No module named 'setuptools'
Let's take this as a chance to switch from virtualenv to the more
lightweight venv, quoting https://docs.python.org/3/library/venv.html:
| Changed in version 3.5: The use of venv is now recommended for creating virtual environments.
Change-Id: Iad06bf8c49abd96441ab04bb9b7e3917acd1ee59
It will allow richer output on dev
https://www.structlog.org/en/stable/development.html
* call structlog.configure as last step in env settings.
We are changing LOGGING there
Change-Id: I2fc7bed3561062e5e6ead3d62dc009f815856b2a
* don't allow to trigger more than one build for instance of
release-trunk-* or mrX.Y
* autoupdate pre-commit config
Change-Id: I7183b8645155ca017e9796664d2570e88a29c44e
* calling build from API was always unauthorized
* drfapikey only supports Django until 1.10 version
* https://florimondmanca.github.io/djangorestframework-api-key/
supports modern Python and Django versions
- remove useless BUILD_KEY_AUTH preference, it's True always
Change-Id: I5521b07532dba12abea52982d376eb83293f6a38
https: //www.django-rest-framework.org/topics/documenting-your-api/#drf-yasg-yet-another-swagger-generator
* We were getting error on /docs/
> 'AutoSchema' object has no attribute 'get_link'
* provide /docs/ with swagger UI
/redoc/ with ReDoc https://github.com/Redocly/redoc
Change-Id: I455e0c392285760d2450f7d3ec3d51209d20bf2a
- Features deprecated in 2.1:
* {% load staticfiles %} and {% load admin_static %} are deprecated in
favor of {% load static %}, which works the same.
- Features deprecated in 1.10:
* Session verification will be enabled regardless of whether or not
'django.contrib.auth.middleware.SessionAuthenticationMiddleware' is
in MIDDLEWARE_CLASSES.
- django-filters:
* https://django-filter.readthedocs.io/en/stable/guide/rest_framework.html
- django-rest-framework:
https://www.django-rest-framework.org/api-guide/pagination/
"Note that you need to set both the pagination class, and the page size that
should be used. Both DEFAULT_PAGINATION_CLASS and PAGE_SIZE are None by default."
- t/Dockerfile: migrate to bullseye
Change-Id: Ia225eba15ca46446700fa2699e89d7e4601bf1a2
* https://django-structlog.readthedocs.io/en/latest/
django-structlog is a structured logging integration for Django project
using structlog
Logging will then produce additional cohesive metadata on each logs that
makes it easier to track events or incidents.
* https://www.structlog.org/en/stable/
structlog makes logging in Python less painful and more powerful by
adding structure to your log entries.
Change-Id: I087574a8d7e04ddb32d86077b0e5478a9f41e11b
* skip pylint check for now, lots of changes needed to pass
* tmux.sh support using a different tag for repoapi image
this helps to download a tag from a review localy and
hack
Change-Id: I63f034d9d4b73f7e6eb69b0cd3ad6f731281f806
Adds support to export data using filters
to yaml via django admin interface
* add tmux.sh helper and update README.md
* requirements: set max supported versions for django 1.11
https://github.com/praekelt/django-export
Change-Id: Ia9af0610725795c588e0bafd06cd6cfcffd9873f
* for running on dev environment point to README.md on Dockerfile
* add watchdog for dev to decrease IO/CPU workload as documentation suggests
> https://django-extensions.readthedocs.io/en/latest/runserver_plus.html
* coverage.rc file to define where to store data
needed on jenkins environment where only results dir has rw perms
* use my own fork for django-jenkins until project is back alive
Change-Id: If3d9b442759a2841ff3441609c3164987fbbb6c5
While working on the support for triggering $debian_release/master
branches (see previous commit) I noticed a bunch of missing or unclear
instructions as well as incomplete setup/configuration:
* Makefile: add missing `venv_dev` target to populate development
environment
* extend and clarify usage instructions in README.md
* dev environment: avoid having to manually configure ALLOWED_HOSTS,
instead identify hostname and listen on that (useful e.g.
when running the environment inside docker and then accessing
it via the docker IP like http://172.17.0.3:8000/release_panel/)
* Dockerfile: add python3-virtualenv + virtualenv as they are
needed for the venv handling, also install sqlite3 for
handling the sqlite database manually
* pin pylint to version 1.9.3[1]
[1] We need to pin pylint to version 1.9.3, which is the last
working version for us. Newer version of pylint (2.0.0 and up to
2.1.1) fail for us with:
| Executing django_jenkins.tasks.run_pylint...
| Traceback (most recent call last):
| File "./manage.py", line 10, in <module>
| execute_from_command_line(sys.argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
| utility.execute()
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute
| self.fetch_command(subcommand).run_from_argv(self.argv)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
| self.execute(*args, **cmd_options)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
| output = self.handle(*args, **options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle
| coverage.save(tested_locations, options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run
| lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False)
| TypeError: __init__() got an unexpected keyword argument 'exit'
This needs further investigation, though independent of this change.
It didn't hit us so far as our repoapi docker image isn't rebuilt
regularly and all previous builds, until this commit - which refreshes
the Dockerfile and we therefore get newer python modules - use older
python modules, including an older pylint version.
Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
* using https://github.com/manosim/django-rest-framework-api-key
for Permissions. Will generate an Auth KEY for jenkins to be able
to trigger build from there using build REST API
* store the info needed to trigger builds for a set of projects for
a release
- build is a list of projects to be built with branch or tag as orig
and release as destination for a debian distribution
* added release_uuid to JenkinsJobInfo model
- release_uuid is the way to group project builds for a release
- tag works for group of jobs for a project
Change-Id: Iebeaa54308c3e569a167f55d98c184b52248af8a
* provide info of actual images and tags
* refresh docker images/tag via release_dashboard
* keep that info in db
* fix Dockerfile documentation
* cleanup thanks to flake8
Change-Id: I743482da9b4d50f7b1832cad324882f3a49cbcf0
* no need to have files twice
* djangorestframework==3.4 so we keep stable compatibility
- fix small issue with Serializer classes for that version
Change-Id: Ib0f04def39b41adf8cc0a700fe096e405b620cb0
* python-debian installed via pip
We are using virtualenv on production, so no need to
install packages for that
Change-Id: I2db86ffa9e71814b1d6dc9d7675bc9790b2c7ee7
* release-tools-runner sends a note
"hotfix <projectname> <version> triggered"
* TODO: how to be sure that version reaches
public release ??
Change-Id: Ib220f36d642172d8c89f4df35543193be16d37ce
* Dockerfile: add screen and refresh due new requirements
* create settings/test.py just for tests
- celery in memory ( no need of server )
- files in RESULTS
* repoapi/tasks.py for celery tasks
- download console.txt and job.json from jenkins
when a JenkinsBuildInfo is created
* repoapi.ini:
- fire worker from uwsgi
http://uwsgi-docs.readthedocs.io/en/latest/AttachingDaemons.html
Change-Id: Ib23c45194878a6fdbbe547058013e39516ea2d17
Mykola Malkov
Michael Prokop
Victor Seva
dependabot[bot]