The previous changes help with deleting some of the unused tags (roughly half of
the previous 30~40), but do not help in some cases and it's therefore impossible
to remove the remaining ones.
Currently, this is what happens:
a) repoapi retrieves this url for a project, e.g. for a tag in data-hal-jessie:
https://repoapi.mgm.sipwise.com/docker/tag/3534/ and uses tag.reference
gets the "reference" attribute:
sha256:fa18c06b84b17b19bed1d14b0b9d836bffba1531e68721dc8ad0a65cc7477761
b) manage_docker.sh (which works correctly) uses this complex command:
curl -vsiL -H "${curl_docker_header}" -X GET "${docker_url}/${docker_tag}" | sed -ne 's/^Docker-Content-Digest: \(sha256:[[[:alnum:]]\+\).*$/\1/ip;Tn;q;:n'
and with this input:
curl -vsiL -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -X GET \
'https://docker.mgm.sipwise.com:5000/v2/data-hal-jessie/manifests/I30522e47cadb23b042e48f23b5a509dc9abf6ccb'
gets as reference:
sha256:ebc37e95c8844d37ec174d6208f80d664a5d099802d839cde0a4f74424d317fe
c) from stackoverflow in
https://stackoverflow.com/questions/38795240/delete-image-from-docker-registry-v2
it is suggested to take json.config.digest of the output of the url above (instead
of headers, like manage_docker.sh), and that is:
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4915,
"digest": "sha256:fac5e71f2f690379d5fad98c550cb451a8c5197f529e519adaab7a66cef00019"
},
which is again different from both values above.
So this change adds code to remove in a way that mimics case b), in addition to
the existing one, in another attempt to completely fix the issue.
(It is not very elegant and it would be nice to pinpoint the exact problem
without these trial and error cycles by pushing to production, but it's
difficult when it depends on a running environment and the tools are not well
known to us, e.g. Django.)
Change-Id: I46859e369f039fffa3e191afb92c60df77d7e6e5
Commit 39744330b3 wrongly added code to call
request.delete() with more than one argument:
response = requests.delete(url, headers)
TypeError: delete() takes 1 positional argument but 2 were given
Change-Id: I65b82d9ccef33b295d009b00461f9eeca2da0827
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
* we need to use reference ( docker API it doesn't support tag name )
* fix tag date (value in the first occurrence not last)
Change-Id: I2fa2f7bedaca4cdeeb3a3a2d4a3f4a661d15fad1
* provide info of actual images and tags
* refresh docker images/tag via release_dashboard
* keep that info in db
* fix Dockerfile documentation
* cleanup thanks to flake8
Change-Id: I743482da9b4d50f7b1832cad324882f3a49cbcf0
Victor Seva
Manuel Montecelo
Michael Prokop