We usually use either "ignore" or "branch/master" and those are within
the 15 character limits of BuildTrunkDepForm() and
BuildTrunkReleaseForm().
But when building against "branch/stretch/master" or
"branch/buster/master", we trigger a `form validation error` because
both branch/stretch/master and branch/buster/master have more than 15
characters.
Raise character limit from 15 to 30 characters.
Change-Id: Icc1f8ee46dcb250c2d40da9ca25fef0d6924db1e
Calling sorting after every single project processing causes
huge overload during initial page load because the same DOM
part is removed, sorted and then added back by many callbacks
at the same moment. So let's disable it.
This function sorts the list of projects under each progress bar.
It's not critical to have the list sorted there so let's remove
sorting and see how it works for us.
Change-Id: Iabf4adb3ade3b7835b35818b58855720363c2bf5
Otherwise projects loop in 'update_info' function
returns immediately during the very first call and
no data shown on page for another 30 seconds
Change-Id: Ic94894dcace48b3c1dc0bf01759b5088ca3cea37
With current timer setup the first load happens after 30 sec only and
all this time empty data is shown on panel.
Change-Id: I9200460b74b4d3323316da1406991e1ad768c57a
While working on the support for triggering $debian_release/master
branches (see previous commit) I noticed a bunch of missing or unclear
instructions as well as incomplete setup/configuration:
* Makefile: add missing `venv_dev` target to populate development
environment
* extend and clarify usage instructions in README.md
* dev environment: avoid having to manually configure ALLOWED_HOSTS,
instead identify hostname and listen on that (useful e.g.
when running the environment inside docker and then accessing
it via the docker IP like http://172.17.0.3:8000/release_panel/)
* Dockerfile: add python3-virtualenv + virtualenv as they are
needed for the venv handling, also install sqlite3 for
handling the sqlite database manually
* pin pylint to version 1.9.3[1]
[1] We need to pin pylint to version 1.9.3, which is the last
working version for us. Newer version of pylint (2.0.0 and up to
2.1.1) fail for us with:
| Executing django_jenkins.tasks.run_pylint...
| Traceback (most recent call last):
| File "./manage.py", line 10, in <module>
| execute_from_command_line(sys.argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
| utility.execute()
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute
| self.fetch_command(subcommand).run_from_argv(self.argv)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
| self.execute(*args, **cmd_options)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
| output = self.handle(*args, **options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle
| coverage.save(tested_locations, options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run
| lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False)
| TypeError: __init__() got an unexpected keyword argument 'exit'
This needs further investigation, though independent of this change.
It didn't hit us so far as our repoapi docker image isn't rebuilt
regularly and all previous builds, until this commit - which refreshes
the Dockerfile and we therefore get newer python modules - use older
python modules, including an older pylint version.
Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1
If we want to trigger a trunk build against e.g. Debian/buster we
might need to build certain projects against branch "buster/master",
instead of using the default "master" branch (which works fine for the
current Debian stable release AKA stretch, but might need
non-backwards compatible changes when building for the current Debian
testing release AKA buster ).
I identified two approaches to get there:
1) Statically extend the list of available branches which is available
for selection in the "Version" drop down (which only provide "ignore"
and "branch/master" until now).
This could be done by something like:
| --- a/release_dashboard/utils/__init__.py
| +++ b/release_dashboard/utils/__init__.py
| @@ -14,6 +14,7 @@
| # with this program. If not, see <http://www.gnu.org/licenses/>.
|
| from release_dashboard.models import Project
| +from release_dashboard.forms import rd_settings
|
|
| def get_tags(projectname, regex=None):
| @@ -23,4 +24,8 @@ def get_tags(projectname, regex=None):
|
| def get_branches(projectname, regex=None):
| project, _ = Project.objects.get_or_create(name=projectname)
| - return project.filter_branches(regex)
| + branches = project.filter_branches(regex)
| + for debian_release in rd_settings['debian_supported']:
| + if debian_release != "auto":
| + branches.append(debian_release + "/master")
| + return branches
The disadvantage of this approach is, that we're listing all the
branches all the time, e.g. the very old "squeeze/master" is listed
even though it doesn't exist in any project as such.
2) Identify available branches and extend the regular expression
which identifies the "master" branch by also looking for
"$debian_release/master" branches, iterating over all supported
Debian releases.
Since approach 2) doesn't polute the drop down for branch
selection so much, this is the solution I decided to use here.
JFTR, when testing locally against an unpopulated sqlite database (:=
db.sqlite3 file), then the relevant branches need to exist in the
release_dashboard_project table of the DB. To fake such data use
something like:
| root@16f2201e3229:/code# sqlite3 ./db.sqlite3
| SQLite version 3.16.2 2017-01-06 16:32:41
| Enter ".help" for usage hints.
| sqlite> update release_dashboard_project set json_branches = '[{"ref": "HEAD", "revision": "master"}, {"ref": "refs/heads/master", "revision": "fakedata"},
| {"ref": "refs/heads/buster/master", "revision": "fakedata"}, {"ref": "refs/heads/stretch/master", "revision": "fakeeeeee"}]';
Change-Id: I5f24574e33ac80d1e92210ff5516dce4bf8fef76
The previous changes help with deleting some of the unused tags (roughly half of
the previous 30~40), but do not help in some cases and it's therefore impossible
to remove the remaining ones.
Currently, this is what happens:
a) repoapi retrieves this url for a project, e.g. for a tag in data-hal-jessie:
https://repoapi.mgm.sipwise.com/docker/tag/3534/ and uses tag.reference
gets the "reference" attribute:
sha256:fa18c06b84b17b19bed1d14b0b9d836bffba1531e68721dc8ad0a65cc7477761
b) manage_docker.sh (which works correctly) uses this complex command:
curl -vsiL -H "${curl_docker_header}" -X GET "${docker_url}/${docker_tag}" | sed -ne 's/^Docker-Content-Digest: \(sha256:[[[:alnum:]]\+\).*$/\1/ip;Tn;q;:n'
and with this input:
curl -vsiL -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -X GET \
'https://docker.mgm.sipwise.com:5000/v2/data-hal-jessie/manifests/I30522e47cadb23b042e48f23b5a509dc9abf6ccb'
gets as reference:
sha256:ebc37e95c8844d37ec174d6208f80d664a5d099802d839cde0a4f74424d317fe
c) from stackoverflow in
https://stackoverflow.com/questions/38795240/delete-image-from-docker-registry-v2
it is suggested to take json.config.digest of the output of the url above (instead
of headers, like manage_docker.sh), and that is:
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4915,
"digest": "sha256:fac5e71f2f690379d5fad98c550cb451a8c5197f529e519adaab7a66cef00019"
},
which is again different from both values above.
So this change adds code to remove in a way that mimics case b), in addition to
the existing one, in another attempt to completely fix the issue.
(It is not very elegant and it would be nice to pinpoint the exact problem
without these trial and error cycles by pushing to production, but it's
difficult when it depends on a running environment and the tools are not well
known to us, e.g. Django.)
Change-Id: I46859e369f039fffa3e191afb92c60df77d7e6e5
Commit 39744330b3 wrongly added code to call
request.delete() with more than one argument:
response = requests.delete(url, headers)
TypeError: delete() takes 1 positional argument but 2 were given
Change-Id: I65b82d9ccef33b295d009b00461f9eeca2da0827
To be able to build trunk and its dependencies against
Debian/buster we need to add it to the list of available
Debian releases.
Change-Id: I9bf840759c9cbcc1290a6d03761922b2a7b9fd6e
Otherwise kibana is no available for old release building,
like for mr5.5.3.
This reverts commit 8ddede8b6f.
Change-Id: Ia1721c581695a40fdc4a8261f79fe9c8f15ea69d
- Bump debhelper compatibility version 10.
- Switch from Priority extra to optional.
- New debian/.gitignore file.
- Canonicalize Homepage URL.
- Remove dh-make boilerplate in debian/rules.
Change-Id: Ic731001d847b143ae9d0bbc6ca23202b39003f67
Attempt to fix problem when release panel triggers jenkins jobs
setting branch=none when tag=mrX.X.X.1, when the branch parameter
should have branch=mrX.X.X
A change from string.join() to "".join() is needed for Python 3.
Change-Id: I89f7bb304180880bc7317b61aa675b8d2162f091
Attempt to fix problem when release panel triggers jenkins jobs
setting branch=none when tag=mrX.X.X.1, when the branch parameter
should have branch=mrX.X.X
This fix is basically the same as the one applied a few days ago in
4f4c28d92d for the file
(top)/build/utils.py which seems similar in purpose, I am not sure why
the duplicity, but this one seems to be the file mapped for the url
".../release_panel/build".
Change-Id: I04217f7f0d4a26802e4c3b808ef9732485d27edb
Attempt to fix problem when release panel triggers jenkins jobs
setting branch=none when tag=mrX.X.X.1, when the branch parameter
should have branch=mrX.X.X
Change-Id: Ic510cec2e94400a5c5a3c8bb0935ff19cdf0ce17
After switching repoapi to Python3 we noticed that workfront
issues received a wrong target release set by repoapi:
| Updated Target Release to b'mr5.5.1'
The underlying issue is that meta-release-helper's output
is considered a bytes object, while we need a string object.
Demonstration of what's going on, using just the relevant
(though minimized) repoapi code:
| % cat demo.py
| import subprocess
|
| def executeAndReturnOutput(command, env=None):
| proc = subprocess.Popen(command, stdout=subprocess.PIPE,
| stderr=subprocess.PIPE, env=env)
| stdoutdata, stderrdata = proc.communicate()
| return proc.returncode, stdoutdata, stderrdata
|
| def get_next_release(branch):
| command = [
| "/usr/bin/meta-release-helper",
| "--next-release",
| branch
| ]
| res = executeAndReturnOutput(command)
| if res[0] != 0:
| return None
| val = res[1].rstrip()
| if len(val) > 0:
| return val
| else:
| return None
|
| release = get_next_release("master")
| print(type(release))
|
| % python2 demo.py
| <type 'str'>
| % python3 demo.py
| <class 'bytes'>
NOTE regarding tests: inside our tests (test_utils.py) the call
to meta-release-helper gets mocked/patched, so it's set to a
string object already (which differs from what we get in actual
usage/production). That's why this issue was hidden so far.
NOTE for later debugging: if repoapi source gets changed
don't forget to restart uwsgi service (`systemctl restart uwsgi.service`)
and find its logging output in file `/var/log/uwsgi/app/repoapi.log`
(which is NOT same es `journalctl -u uwsgi.service`!).
Change-Id: I0f797ed3f7ae49c963c2f3c07fd601221e1dcd6c
In Python3 all strings are considered Unicode by default. The
unicode type from Python 2 is called str in Python 3, and str
becomes bytes. In Python3, there aren’t any automatic conversions
between str and bytes though. The json.loads in Python versions
>3 and <3.6 expects a string representation and not bytes. As we
receive bytes we need to decode() it, as this always returns str,
just as needed.
Fixes:
| File "./release_dashboard/views/build.py", line 59, in hotfix_build
| json_data = json.loads(request.body)
| File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
| s.__class__.__name__))
| TypeError: the JSON object must be str, not 'bytes'
See https://docs.djangoproject.com/en/1.11/topics/python3/ and
https://vinta.ws/code/notes-of-porting-to-python-3-for-a-django-project.html
for related documentation.
Change-Id: I3984c5fd1f79b94680e680c8da64dd0ac111cbe9
Lots of changes, time for a new release, since it's a switch
from Python2 to Python3, updating Django and its components
and switch from Debian jessie to stretch let's bump the minor
version number.
Change-Id: Id400901510e34f6c90e8eea538fe0f24acb23dd0
Updates:
* Debian jessie->stretch (to use current stable Debian version)
* Python 2.7->3.5 (since Python v2 won't be supported with the
next Django LTS version any longer and we want to be prepared)
* Improve uwsgi configuration for production usage while at it
(thanks Florian Apolloner for feedback and suggestions)
Necessary changes:
* Debian:
- we need to use uwsgi-plugin-python3 for Python3 support
- replace python-dev with python3-dev for Python3 support
and python with python3 accordingly
- get rid of __pycache__ directories to avoid inclusion in
resulting Debian package
* uwsgi:
- if we hardcode python34 (for jessie) or python35 (for stretch)
then we can't run uwsgi on all supported systems, so instead
try `plugin = python3` and rely on update-alternatives mechanism
of uwsgi-plugin-python3 Debian package to get it right for,
if that shouldn't work as needed then we can look into
/usr/share/python3/runtime.d/uwsgi-plugin-python3.rtupdate
or shipping repoapi.ini via puppet instead
* docker:
- python-distribute is not needed (anymore)
- adjust package names for Python3 usage, support jessie
and stretch at the same time
- note: just replacing 'stretch' with 'jessie' in t/Dockerfile
is enough to get a working repoapi docker image based on jessie
* Python3:
- assertItemsEqual became assertCountEqual
- mocking: `__builtin__.open` became `builtins.open`, see
https://stackoverflow.com/questions/1289894/how-do-i-mock-an-open-used-in-a-with-statement-using-the-mock-framework-in-pyth
- we need six >=1.9.0 for raise_from astraction in py3
Otherwise we're failing on Debian/jessie with:
| AttributeError: 'module' object has no attribute 'raise_from'
See https://github.com/benjaminp/six/blob/master/CHANGES
- assert_has_calls changes, fixes:
| AssertionError: Calls not found.
| Expected: [call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
| Actual: [call('123', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered'),
| call('345', 'hotfix fake.git 3.8.7.4+0~mr3.8.7.4 triggered')]
See https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.Mock.assert_has_calls
- urllib:
- `from urlparse import urlparse` became `from urllib.parse import urlparse`
- `urllib.quote` became `urllib.parse.quote`
- configparser:
- `from ConfigParser import ...` became `from configparser import ...`
* Django:
- 'from views import ...' became 'from .views import ...'
- the "TEMPLATE_DIRS" setting needs to go into "TEMPLATES = [ { 'DIRS': [ ...."
nowadays, see https://docs.djangoproject.com/en/1.11/ref/templates/upgrading/
* celery:
- to use latest celery version 4 django-celery needs to be replaced
by django-celery-beat + django-celery-results
(otherwise we're running into https://github.com/celery/django-celery/issues/497),
see documentation at http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
- adjust systemd unit files to use ExecStart
| ExecStart=/var/lib/repoapi/venv_prod/bin/celery -A repoapi [...]
instead of
| ExecStart=/var/lib/repoapi/venv_prod/bin/python3 ./manage.py celery [...]
and use `--db=/var/lib/repoapi/flower` instead of
`--db=/var/lib/repoapi/flower.db` to avoid ending up with a file
named flower.db.db
Notes for migration (assuming root permissions):
- get rid of existing flower database (not containing relevant data):
mv /var/lib/repoapi/flower.db /var/lib/repoapi/flower.db.old
- re-deploy virtual environment from scratch:
mv /var/lib/repoapi/venv_prod /var/lib/repoapi/venv_prod.old
cd /usr/share/repoapi && make deploy
- service restarts (in case of virtual env changes):
systemctl restart nginx
systemctl restart repoapi-beat.service
systemctl restart repoapi-flower.service
systemctl restart repoapi-worker.service
systemctl restart uwsgi
- usage instructions for celery + django stuff:
/var/lib/repoapi/venv_prod/bin/celery flower --help
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py help --settings="repoapi.settings.prod"
- create superuser for django administration (https://repoapi*.mgm.sipwise.com/admin/):
source /var/lib/repoapi/venv_prod/bin/activate && ./manage.py createsuperuser --settings="repoapi.settings.prod"
Thanks: Victor Seva for assistance with porting
Change-Id: Iacb48bf5dadf4eae97b5fbe944e44370e525f64c
Michael Prokop
Sergii Kipot
Robert Axelsen
Alexander Lutay
Manuel Montecelo
Guillem Jover
Victor Seva