* for running on dev environment point to README.md on Dockerfile
* add watchdog for dev to decrease IO/CPU workload as documentation suggests
> https://django-extensions.readthedocs.io/en/latest/runserver_plus.html
* coverage.rc file to define where to store data
needed on jenkins environment where only results dir has rw perms
* use my own fork for django-jenkins until project is back alive
Change-Id: If3d9b442759a2841ff3441609c3164987fbbb6c5
Set debhelper compat level in Build-Depends instead of debian/compat.
Do not explicitly request the systemd sequence.
Use dh_installsystemd instead of deprecated dh_systemd_*.
Change-Id: Ic8cbf795de90aa8900a899706ce6e0474a8e6eed
Merge two file stanzas for Sipwise copyright, and use the one with the
canonical Sipwise copyright entity.
Change-Id: I68abd326d976d1db99e119c88d834662daf824d8
Fixes:
| No attribute 'code' on urllib.request._HTTPResponse [attribute-error]
as reported by pytype.
The urllib module has been split into parts and renamed in Python 3 to
urllib.request, urllib.parse, and urllib.error, and therefore will fail
with:
| AttributeError: module 'urllib' has no attribute 'request'
This used to work only because other components loaded within repoapi
did the `import urllib.request` for us.
Also the errors were never reported as such, as e.g. 404 errors are
throwing an exception (instead of provided as return code as expected by
our code so far), which we didn't catch so far. And the arguments
provided to logger/ logging were incorrect (number vs string), so fix
this and also slightly improve the logging messages while at it.
JFTR, `import urllib.request` is py3k-only, which is fine for us though,
since repoapi is supporting only py3k nowadays. We should also think
about migrating from urllib towards requests everywhere.
Change-Id: Ibc965daa94216aaad80758dba0f50a74d8658887
Otherwise building against Debian/buster with python 3.7 fails with:
| File "/build/repoapi-0.3.0+0~20190702124241.289+buster~1.gbp167a8a/venv_prod/lib/python3.7/site-packages/django/contrib/admin/widgets.py", line 151
| '%s=%s' % (k, v) for k, v in params.items(),
| ^
| SyntaxError: Generator expression must be parenthesized
The underlying issue is #32012 in python (see
https://bugs.python.org/issue32012), which disallows ambiguous syntax
f(x for x in [1],), which is fixed only with Django >=1.11.17, see
931c60c521
Quoting from https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django:
| Django version | Python versions
+----------------+-------------------------------------------
| 1.11 | 2.7, 3.4, 3.5, 3.6, 3.7 (added in 1.11.17)
| 2.0 | 3.4, 3.5, 3.6, 3.7
| 2.1, 2.2 | 3.5, 3.6, 3.7
| 3.0 | 3.6, 3.7, 3.8
Change-Id: I4a1db7488792e5374c0f82629dfcfba0a6419be5
We usually use either "ignore" or "branch/master" and those are within
the 15 character limits of BuildTrunkDepForm() and
BuildTrunkReleaseForm().
But when building against "branch/stretch/master" or
"branch/buster/master", we trigger a `form validation error` because
both branch/stretch/master and branch/buster/master have more than 15
characters.
Raise character limit from 15 to 30 characters.
Change-Id: Icc1f8ee46dcb250c2d40da9ca25fef0d6924db1e
Calling sorting after every single project processing causes
huge overload during initial page load because the same DOM
part is removed, sorted and then added back by many callbacks
at the same moment. So let's disable it.
This function sorts the list of projects under each progress bar.
It's not critical to have the list sorted there so let's remove
sorting and see how it works for us.
Change-Id: Iabf4adb3ade3b7835b35818b58855720363c2bf5
Otherwise projects loop in 'update_info' function
returns immediately during the very first call and
no data shown on page for another 30 seconds
Change-Id: Ic94894dcace48b3c1dc0bf01759b5088ca3cea37
With current timer setup the first load happens after 30 sec only and
all this time empty data is shown on panel.
Change-Id: I9200460b74b4d3323316da1406991e1ad768c57a
While working on the support for triggering $debian_release/master
branches (see previous commit) I noticed a bunch of missing or unclear
instructions as well as incomplete setup/configuration:
* Makefile: add missing `venv_dev` target to populate development
environment
* extend and clarify usage instructions in README.md
* dev environment: avoid having to manually configure ALLOWED_HOSTS,
instead identify hostname and listen on that (useful e.g.
when running the environment inside docker and then accessing
it via the docker IP like http://172.17.0.3:8000/release_panel/)
* Dockerfile: add python3-virtualenv + virtualenv as they are
needed for the venv handling, also install sqlite3 for
handling the sqlite database manually
* pin pylint to version 1.9.3[1]
[1] We need to pin pylint to version 1.9.3, which is the last
working version for us. Newer version of pylint (2.0.0 and up to
2.1.1) fail for us with:
| Executing django_jenkins.tasks.run_pylint...
| Traceback (most recent call last):
| File "./manage.py", line 10, in <module>
| execute_from_command_line(sys.argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
| utility.execute()
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 355, in execute
| self.fetch_command(subcommand).run_from_argv(self.argv)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 47, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/commands/test.py", line 29, in run_from_argv
| super(Command, self).run_from_argv(argv)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 283, in run_from_argv
| self.execute(*args, **cmd_options)
| File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 330, in execute
| output = self.handle(*args, **options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/management/commands/jenkins.py", line 116, in handle
| coverage.save(tested_locations, options)
| File "/usr/local/lib/python3.5/dist-packages/django_jenkins/tasks/run_pylint.py", line 44, in run
| lint.Run(args, reporter=ParseableTextReporter(output=output), exit=False)
| TypeError: __init__() got an unexpected keyword argument 'exit'
This needs further investigation, though independent of this change.
It didn't hit us so far as our repoapi docker image isn't rebuilt
regularly and all previous builds, until this commit - which refreshes
the Dockerfile and we therefore get newer python modules - use older
python modules, including an older pylint version.
Change-Id: I35c5edb22642851f0b9b20d1b8a1c91fd24e6ac1
If we want to trigger a trunk build against e.g. Debian/buster we
might need to build certain projects against branch "buster/master",
instead of using the default "master" branch (which works fine for the
current Debian stable release AKA stretch, but might need
non-backwards compatible changes when building for the current Debian
testing release AKA buster ).
I identified two approaches to get there:
1) Statically extend the list of available branches which is available
for selection in the "Version" drop down (which only provide "ignore"
and "branch/master" until now).
This could be done by something like:
| --- a/release_dashboard/utils/__init__.py
| +++ b/release_dashboard/utils/__init__.py
| @@ -14,6 +14,7 @@
| # with this program. If not, see <http://www.gnu.org/licenses/>.
|
| from release_dashboard.models import Project
| +from release_dashboard.forms import rd_settings
|
|
| def get_tags(projectname, regex=None):
| @@ -23,4 +24,8 @@ def get_tags(projectname, regex=None):
|
| def get_branches(projectname, regex=None):
| project, _ = Project.objects.get_or_create(name=projectname)
| - return project.filter_branches(regex)
| + branches = project.filter_branches(regex)
| + for debian_release in rd_settings['debian_supported']:
| + if debian_release != "auto":
| + branches.append(debian_release + "/master")
| + return branches
The disadvantage of this approach is, that we're listing all the
branches all the time, e.g. the very old "squeeze/master" is listed
even though it doesn't exist in any project as such.
2) Identify available branches and extend the regular expression
which identifies the "master" branch by also looking for
"$debian_release/master" branches, iterating over all supported
Debian releases.
Since approach 2) doesn't polute the drop down for branch
selection so much, this is the solution I decided to use here.
JFTR, when testing locally against an unpopulated sqlite database (:=
db.sqlite3 file), then the relevant branches need to exist in the
release_dashboard_project table of the DB. To fake such data use
something like:
| root@16f2201e3229:/code# sqlite3 ./db.sqlite3
| SQLite version 3.16.2 2017-01-06 16:32:41
| Enter ".help" for usage hints.
| sqlite> update release_dashboard_project set json_branches = '[{"ref": "HEAD", "revision": "master"}, {"ref": "refs/heads/master", "revision": "fakedata"},
| {"ref": "refs/heads/buster/master", "revision": "fakedata"}, {"ref": "refs/heads/stretch/master", "revision": "fakeeeeee"}]';
Change-Id: I5f24574e33ac80d1e92210ff5516dce4bf8fef76
The previous changes help with deleting some of the unused tags (roughly half of
the previous 30~40), but do not help in some cases and it's therefore impossible
to remove the remaining ones.
Currently, this is what happens:
a) repoapi retrieves this url for a project, e.g. for a tag in data-hal-jessie:
https://repoapi.mgm.sipwise.com/docker/tag/3534/ and uses tag.reference
gets the "reference" attribute:
sha256:fa18c06b84b17b19bed1d14b0b9d836bffba1531e68721dc8ad0a65cc7477761
b) manage_docker.sh (which works correctly) uses this complex command:
curl -vsiL -H "${curl_docker_header}" -X GET "${docker_url}/${docker_tag}" | sed -ne 's/^Docker-Content-Digest: \(sha256:[[[:alnum:]]\+\).*$/\1/ip;Tn;q;:n'
and with this input:
curl -vsiL -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -X GET \
'https://docker.mgm.sipwise.com:5000/v2/data-hal-jessie/manifests/I30522e47cadb23b042e48f23b5a509dc9abf6ccb'
gets as reference:
sha256:ebc37e95c8844d37ec174d6208f80d664a5d099802d839cde0a4f74424d317fe
c) from stackoverflow in
https://stackoverflow.com/questions/38795240/delete-image-from-docker-registry-v2
it is suggested to take json.config.digest of the output of the url above (instead
of headers, like manage_docker.sh), and that is:
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4915,
"digest": "sha256:fac5e71f2f690379d5fad98c550cb451a8c5197f529e519adaab7a66cef00019"
},
which is again different from both values above.
So this change adds code to remove in a way that mimics case b), in addition to
the existing one, in another attempt to completely fix the issue.
(It is not very elegant and it would be nice to pinpoint the exact problem
without these trial and error cycles by pushing to production, but it's
difficult when it depends on a running environment and the tools are not well
known to us, e.g. Django.)
Change-Id: I46859e369f039fffa3e191afb92c60df77d7e6e5
Commit 39744330b3 wrongly added code to call
request.delete() with more than one argument:
response = requests.delete(url, headers)
TypeError: delete() takes 1 positional argument but 2 were given
Change-Id: I65b82d9ccef33b295d009b00461f9eeca2da0827
To be able to build trunk and its dependencies against
Debian/buster we need to add it to the list of available
Debian releases.
Change-Id: I9bf840759c9cbcc1290a6d03761922b2a7b9fd6e
Otherwise kibana is no available for old release building,
like for mr5.5.3.
This reverts commit 8ddede8b6f.
Change-Id: Ia1721c581695a40fdc4a8261f79fe9c8f15ea69d
- Bump debhelper compatibility version 10.
- Switch from Priority extra to optional.
- New debian/.gitignore file.
- Canonicalize Homepage URL.
- Remove dh-make boilerplate in debian/rules.
Change-Id: Ic731001d847b143ae9d0bbc6ca23202b39003f67
Victor Seva
Guillem Jover
Alexander Lutay
dependabot[bot]
Sergii Kipot
Michael Prokop
Robert Axelsen
Manuel Montecelo