As of django-structlog v6.0.0, our django application fails with:
| Traceback (most recent call last):
| File "/var/lib/repoapi/venv_prod/lib/python3.9/site-packages/django/utils/module_loading.py", line 20, in import_string
| return getattr(module, class_name)
| AttributeError: module 'django_structlog.middlewares' has no attribute 'CeleryMiddleware'
Quoting from
https://django-structlog.readthedocs.io/en/latest/upgrade_guide.html#upgrading-to-6-0:
| (If you use celery) Make sure you use DJANGO_STRUCTLOG_CELERY_ENABLED = True
|
| It is only applicable if you use celery integration.
|
| django_structlog.middlewares.CeleryMiddleware has been remove in favor of a django settings.
|
| MIDDLEWARE += [
| "django_structlog.middlewares.RequestMiddleware",
| # "django_structlog.middlewares.CeleryMiddleware", # <- remove this
| ]
|
| DJANGO_STRUCTLOG_CELERY_ENABLED = True # <-- add this
Instead of forcing usage of an older django-structlog version,
let's make sure we have 6.0 or newer and update configuration
accordingly.
Change-Id: I0449c05db682ace22438b8b0ad8d8a4a1dfca71a
This reverts commit 1e2026769e.
This isn't a problem with python-debian, but in setuptools
itself, see https://bugs.debian.org/1041091
Change-Id: I0cef0bc444aa07060b07a1d7ca5a2301c77cd602
* calling build from API was always unauthorized
* drfapikey only supports Django until 1.10 version
* https://florimondmanca.github.io/djangorestframework-api-key/
supports modern Python and Django versions
- remove useless BUILD_KEY_AUTH preference, it's True always
Change-Id: I5521b07532dba12abea52982d376eb83293f6a38
https: //www.django-rest-framework.org/topics/documenting-your-api/#drf-yasg-yet-another-swagger-generator
* We were getting error on /docs/
> 'AutoSchema' object has no attribute 'get_link'
* provide /docs/ with swagger UI
/redoc/ with ReDoc https://github.com/Redocly/redoc
Change-Id: I455e0c392285760d2450f7d3ec3d51209d20bf2a
- Features deprecated in 2.1:
* {% load staticfiles %} and {% load admin_static %} are deprecated in
favor of {% load static %}, which works the same.
- Features deprecated in 1.10:
* Session verification will be enabled regardless of whether or not
'django.contrib.auth.middleware.SessionAuthenticationMiddleware' is
in MIDDLEWARE_CLASSES.
- django-filters:
* https://django-filter.readthedocs.io/en/stable/guide/rest_framework.html
- django-rest-framework:
https://www.django-rest-framework.org/api-guide/pagination/
"Note that you need to set both the pagination class, and the page size that
should be used. Both DEFAULT_PAGINATION_CLASS and PAGE_SIZE are None by default."
- t/Dockerfile: migrate to bullseye
Change-Id: Ia225eba15ca46446700fa2699e89d7e4601bf1a2
* https://django-structlog.readthedocs.io/en/latest/
django-structlog is a structured logging integration for Django project
using structlog
Logging will then produce additional cohesive metadata on each logs that
makes it easier to track events or incidents.
* https://www.structlog.org/en/stable/
structlog makes logging in Python less painful and more powerful by
adding structure to your log entries.
Change-Id: I087574a8d7e04ddb32d86077b0e5478a9f41e11b
Adds support to export data using filters
to yaml via django admin interface
* add tmux.sh helper and update README.md
* requirements: set max supported versions for django 1.11
https://github.com/praekelt/django-export
Change-Id: Ia9af0610725795c588e0bafd06cd6cfcffd9873f
Otherwise building against Debian/buster with python 3.7 fails with:
| File "/build/repoapi-0.3.0+0~20190702124241.289+buster~1.gbp167a8a/venv_prod/lib/python3.7/site-packages/django/contrib/admin/widgets.py", line 151
| '%s=%s' % (k, v) for k, v in params.items(),
| ^
| SyntaxError: Generator expression must be parenthesized
The underlying issue is #32012 in python (see
https://bugs.python.org/issue32012), which disallows ambiguous syntax
f(x for x in [1],), which is fixed only with Django >=1.11.17, see
931c60c521
Quoting from https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django:
| Django version | Python versions
+----------------+-------------------------------------------
| 1.11 | 2.7, 3.4, 3.5, 3.6, 3.7 (added in 1.11.17)
| 2.0 | 3.4, 3.5, 3.6, 3.7
| 2.1, 2.2 | 3.5, 3.6, 3.7
| 3.0 | 3.6, 3.7, 3.8
Change-Id: I4a1db7488792e5374c0f82629dfcfba0a6419be5
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
* using https://github.com/manosim/django-rest-framework-api-key
for Permissions. Will generate an Auth KEY for jenkins to be able
to trigger build from there using build REST API
* store the info needed to trigger builds for a set of projects for
a release
- build is a list of projects to be built with branch or tag as orig
and release as destination for a debian distribution
* added release_uuid to JenkinsJobInfo model
- release_uuid is the way to group project builds for a release
- tag works for group of jobs for a project
Change-Id: Iebeaa54308c3e569a167f55d98c184b52248af8a
* provide info of actual images and tags
* refresh docker images/tag via release_dashboard
* keep that info in db
* fix Dockerfile documentation
* cleanup thanks to flake8
Change-Id: I743482da9b4d50f7b1832cad324882f3a49cbcf0
* no need to have files twice
* djangorestframework==3.4 so we keep stable compatibility
- fix small issue with Serializer classes for that version
Change-Id: Ib0f04def39b41adf8cc0a700fe096e405b620cb0
* python-debian installed via pip
We are using virtualenv on production, so no need to
install packages for that
Change-Id: I2db86ffa9e71814b1d6dc9d7675bc9790b2c7ee7
* Dockerfile: add screen and refresh due new requirements
* create settings/test.py just for tests
- celery in memory ( no need of server )
- files in RESULTS
* repoapi/tasks.py for celery tasks
- download console.txt and job.json from jenkins
when a JenkinsBuildInfo is created
* repoapi.ini:
- fire worker from uwsgi
http://uwsgi-docs.readthedocs.io/en/latest/AttachingDaemons.html
Change-Id: Ib23c45194878a6fdbbe547058013e39516ea2d17
Jose Su
Victor Seva
Michael Prokop
dependabot[bot]
Sergii Kipot