- Add pyproject.toml and tox.ini with consistent black and pycodestyle
settings.
- Run black for consistent coding style.
- Manually fix remaining issues:
+ Split overlong strings.
+ Reflow comments.
+ Use «is» instead of «==» for None.
+ Disable checks for explicit empty blank lines in test data.
Change-Id: I6d10b78061974d8f96bdb07eadee939e0e481aa4
The process_pool function already extracts the provided tarball to
initialise an empty git repo. Doing it again when creating the
directories is redundant. Use the appropriate /etc path for config file
output.
Change-Id: I3c598a4b1ddf3a65f7f1759c9b29f1f7d1c098bf
Use a separate temp dir for the state files. Otherwise updates to the
state files pollute the git status of the mock /etc.
Change-Id: I6a784259db63b12c7ffe5666660966ad6c9d60f4
We check whether we have both a sites.yml and a sites/ directory.
If so, and multi-sites support is enabled, then we proceed to load
all the multi-sites configurations, by placing each of them under
«sites.<site-name>.<config>». Adding a «sites.current» link to the
current site, and filling the root tree config with the current site
as well for backwards compatibility.
Change-Id: Ia810f57e8e976eef6a7582bddda46c1701456ed6
This variable has been deprecated for a long time (since mr10.5), and it
is not used anywhere in our entire NGCP code base.
Change-Id: Ic142a347fcef0405f8cf8304728b076af3369e0d
With git v2.37.2-1, as present in current Debian/unstable, our unit
tests might fail - as seen with our Github actions:
| err = ('fatal: detected dubious ownership in repository at '
| "'/tmp/pytest-of-root/pytest-24/test_build_instance_customtt0/ngcpctl-pytest-base/ngcp-config'\n"
| 'To add an exception for this directory, call:\n'
| '\n'
| '\tgit config --global --add safe.directory '
With its underlying working directory looking like:
| root@b5a6b272fb90:/code# ls -la /tmp/pytest-of-root/pytest-24/test_build_instance_customtt0/ngcpctl-pytest-base/ngcp-config
| total 148
| drwxr-xr-x 5 root root 4096 Aug 15 09:12 .
| drwxr-xr-x 3 root root 4096 Aug 15 09:12 ..
| drwxr-xr-x 8 1000 1000 4096 May 18 2021 .git
| -rw-r--r-- 1 1000 1000 305 Jun 11 2020 .gitignore
| -rw-r----- 1 1000 1000 47437 Jun 15 2020 config.yml
| [...]
FTR, this can also be reproduced with our docker setup, when running as
user root, while the underlying ngcpcfg repository is owned by a normal
user:
| root@b5a6b272fb90:/code# ls -la
| total 92
| drwxr-xr-x 16 1000 1000 4096 Aug 15 08:41 .
| drwxr-xr-x 1 root root 4096 Aug 15 08:36 ..
| drwxr-xr-x 8 1000 1000 4096 Aug 15 09:22 .git
| drwxr-xr-x 4 1000 1000 4096 Jun 23 15:37 .github
| -rw-r--r-- 1 1000 1000 125 Jun 23 15:37 .gitignore
| -rw-r--r-- 1 1000 1000 64 Jul 16 2019 .gitreview
| -rw-r--r-- 1 1000 1000 169 Aug 15 08:41 .mailmap
| [...]
Quoting from git's Documentation/RelNotes/2.36.0.txt:
| * With the fixes for CVE-2022-24765 that are common with versions of
| Git 2.30.4, 2.31.3, 2.32.2, 2.33.3, 2.34.3, and 2.35.3, Git has
| been taught not to recognise repositories owned by other users, in
| order to avoid getting affected by their config files and hooks.
| You can list the path to the safe/trusted repositories that may be
| owned by others on a multi-valued configuration variable
| `safe.directory` to override this behaviour, or use '*' to declare
| that you trust anything.
Whereas the following git upstream change checks if a repository is
safe, by verifying the ownership of the worktree (if any), the git
directory, and the gitfile (if any):
| commit 3b0bf2704980b1ed6018622bdf5377ec22289688
| Author: Carlo Marcelo Arenas Belón <carenas@gmail.com>
| Date: Tue May 10 12:35:29 2022 -0700
|
| setup: tighten ownership checks post CVE-2022-24765
This change made it into git v2.30.5, v2.31.4, v2.32.3, v2.33.4,
v2.34.4, v2.35.4, v2.36.2, v2.37.1 + v2.37.2, and we got v2.37.2-1
in Debian/unstable as of 2022-08-12 (see
https://packages.qa.debian.org/g/git/news/20220813T030422Z.html).
This affects us with our mixture of root vs user permissions of the
working directory vs the git directory, so let's make sure the .git
directory has the according permissision as well.
Change-Id: I695fbd5a3b7fb79acc2873f75b8de410c8e3a0aa
The HA_NODE, HA_FILE and HA_CONFIG variables and nomenclature are
deprecated in favor of the NGCP_NODENAME, NODE_FILE and NODE_CONFIG.
Switch the old uses to the new ones. Unify PAIRNAME to NGCP_PAIRNAME.
Leave the old HA_ variables for backwards compatibility
Change-Id: I11f09f1620c2e456f10fc17229b61ec5a35a732b
In more recent versions, Git upstream does an owner check for the
top-level directory (see git upstream commit 8959555ce), also see
https://github.blog/2022-04-12-git-security-vulnerability-announced/
This change is included in git versions >=2.30.3, >=2.31.2, >=2.34.2,
>=2.35.2 + >=2.36.0-rc2, and therefore also affects the Git package
v2.35.2-1 as present in current Debian/unstable (as of 2022-04-16).
Now due to this behavioral change, our unit tests fail with e.g.:
| err = ('fatal: unsafe repository '
| "('/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config' "
| 'is owned by someone else)\n'
| 'To add an exception for this directory, call:\n'
| '\n'
| '\tgit config --global --add safe.directory '
| '/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config\n')
| ex = 128
We're creating many temporary git repositories. Therefore, adding every
single repository via `git config --global --add safe.directory` as
suggested in git's error message isn't really a viable option for us.
Git upstream also recognized this, and as of git rev 0f85c4a30 it's
possible to opt-out of this check via `git config --global --add
safe.directory *`. This change is currently included only in Git
versions 2.30.4, 2.31.3, 2.32.2, 2.33.3, 2.34.3 and 2.35.3, so not
available in Debian/unstable, yet.
But nevertheless, it's not really an ideal option for us, as we don't
want to mess with $HOME/.gitconfig ever, as this might not always be
some random directory inside a testing container, but pointing to an
actual user configuration.
The underlying reason, why this issue showed up in our Github actions is
caused by the fact, that the checkout of the artifacts is running as
user (also see https://github.com/actions/checkout/issues/47):
| uid=1001(runner) gid=121(docker) groups=121(docker),4(adm),101(systemd-journal)
But the docker containers are executed with root permissions in the
following steps. To properly handle this, we set the permissions of the
git repository to $UID/$GID of the executing user.
Even more tricky and worth being aware of, certain git actions might
fail due to permission issues, without telling you directly:
| root@8d1e4156f6d8:/tmp# mkdir testrepo/
| root@8d1e4156f6d8:/tmp# cd testrepo/
| root@8d1e4156f6d8:/tmp/testrepo# git init
| Initialized empty Git repository in /tmp/testrepo/.git/
| root@8d1e4156f6d8:/tmp/testrepo# chown testbuild .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| fatal: --local can only be used inside a git repository
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 128
| root@8d1e4156f6d8:/tmp/testrepo# chown root .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 0
While at it, let's unify our git configuration, by using the following
settings for all our user configuration:
| git config --local user.email pytest@example.com
| git config --local user.name pytest
Change-Id: Icad0ea4c3daf22f17481f23b27fa17750bd623da
- NGCPCTL_BASE
- OUTPUT_DIRECTORY{CONFIG_POOL}
* add empty-git-repo.tar.gz
* fs.py: define check_output() helper
* set DEBUG by default
* ngcpcfgcli, return env and cfg instead of just outdir
* cleanup on tests:
- don't compile the regex, it's useless in a test contex
- remove unused imports and fixtures
- use check_output() from fs
Change-Id: I0eb859b6f7aad4cd3dd8e15f4e211395c5131ce7
gitrepo.from_archive() is suppose to be used at a ``with`` statement
so at the end the temporary directory gitrepo.localpath is removed
> with gitrepo.from_archive(path_tar) as git:
> pass
> # here gitrepo.localpath is gone already
So instead of using from_archive() let's create the temporary directory
and extract the tar file there instead.
Be aware that gitrepo.default has the git repository inside the ``testing``
directory
Change-Id: I7a50baab8c5a444e08aeabc2f1597e3a13d94c84
ngcpcfg service expects that the produced files are inside a git
repository
* ngcpcfgcli:
- set RUN_DIR by default
- set STATE_FILES_DIR by default
- use default-git-repository.tar.gz as base for outdir
- return outdir value too
- print stdout and stderr so on failure we can debug
* t/test_*:
- remove unused imports
- don't create OUTPUT_DIRECTORY unless really necessary
- remove print stdout/stderr
- black indent
Change-Id: I61e475cc9c775c30c3700a8c0e560e7c5b22f0fc
* ngcpcfg: obey SKIP_UPDATE_PERMS
* t/fixtures/programs.py: set SKIP_*_PERMS to true by default
* t/test_ngcpcfg_apply.py:
touching the file was not producing a change in git
Change-Id: Id8317ff09fee2a9a3d4d2015d1cd782c3f9b62e7
Use an absolute path so that the shell can properly resolve the
pathnames, otherwise they do not get resolved correctly at all.
Change-Id: I03c5db4a831ab44f9eb554fe3638c15bf4708774
Switch away from the old daemon/wrapper architecture, which complicates
things and requires more scaffolding code.
We turn the daemon/wrapper and build_config into a single perl process
that will load all YAML files, and then process each input/output pair
on a parallel child, to try to speed up the processing as much as
possible.
For whole rebuilds, it might speed up the generation by at least a
factor of x2, x4 or more, depending on the number of active processors.
Change-Id: I51aa2f90336e34a20983d8733f45b64d9b6fea0b
The yaml files should not be executable. This also makes running the
test suite more painful, as the fixtures in git will get their
permissions changed and will be shown by git as modified.
Change-Id: I05909b5eac316bb46251ccd79a19dc8b437876bc
The commit 92dccb4b10 introduced new group 'ngcp-admin'
and forced ngcpcfg to set proper perms every time we call ngcpcfg.
One of the tests starts failing right after the commit above, but
we didn't notice it until we add error message (the following commit).
Wihtout the fix here ngcpcfg doesn't pass tests:
> test_ngcpcfg_build_network_interfaces.py::test_network_interfaces FAILED
> ...
> stderr:
> b"../scripts//build: line 174: CONFIG_GROUP: unbound variable
> 2019-10-01 15:41:35 b22fa4d1cbb9: Error: Failed to call action 'build' on 'b22fa4d1cbb9' (see logs on 'b22fa4d1cbb9')
Change-Id: Ia514a7e275aef25d01bdaedf71ef7879e84dd8ba
- Remove a spurious trailing ';'.
- Make the hook directory reaction conditional on whether we are running
from within a testsuite, to restore its otherwise fatal treatment.
Change-Id: I3f8a5364d7b2f0ac82bfdc132b94641603225f85
Fixes: commit 505d6566d8
When running the test suite, in case we need to cover the code paths
that are handling the hooks, we should not assume the files are
installed in the system directories.
Change-Id: I1cd09eaf186d28093eb03ad5e9fc096fe7d63a3e
The perl Template::Toolkit is very rich, but its "function" support is a
bit poor. The ways to do it are either via MACRO directives, or by
simulating them with one function per file and then using PROCESS on
these. The problem is that this is very clunky, does not support
nesting, as we'd need different "argument" names for each "function",
and it's quite cumbersome to use, need to assign aguments passed
beforehand, and then assign back a designated return value from another
variable. This is also one of the reasons some of the functions are not
encapsulated, and have been inlined in various loops, because it was not
possible to cleanly PROCESS them from those call sites.
Instead we should use its native support for perl objects and perl
subroutines, which exposes these as proper methods of a designated
variable, and have none of the above mentioned problems. So we'll switch
from constructs such as:
argv.arg-a = variable;
argv.arg-b = 'value';
PROCESS 'path-to-library-dir/function'
result = out
into:
result = ngcp.function(variable, 'value');
In addition this might actually be faster, as it does not require
processing additional files, and it's all just native perl code.
This will be exposed within the NGCP templates as the ngcp object, and
new member functions will start replacing our old and clunky native
Template PROCESS-style library.
Change-Id: Id2f0d181c695a9dd074646881b7d9de3478570af
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
The old testsuite wasn't updated for way too long and since
ngcpcfg receives more and more features we need a decent test
coverage. pytest seems to provide the right level of
abstraction, excellent fixtures and junit-xml reporting as
needed.
Inspired by Vincent Bernat's
https://github.com/vincentbernat/lldpd/tree/master/tests/integration
Thanks Victor Seva <vseva@sipwise.com>, Vincent Bernat <vincent@bernat.im>, Christian Hofstaedtler <christian@hofstaedtler.name> and Lukas Prokop <admin@lukas-prokop.at> for feedback, inspiration and help
Change-Id: Iffed87e8cc540169bed89c00967a03e80859179e
Guillem Jover
Richard Fuchs
Mykola Malkov
Michael Prokop
ngcp-config
Alexander Lutay