As reported at https://jenkins.mgm.sipwise.com/job/ngcpcfg-tap-test/4544/
when building against Debian/trixie
Furthermore fix SC2166 + SC2086 issues within helper/check-for-mysql
as reported by manual shellcheck v0.10.0 runs.
Change-Id: Ic9be0dbc8ef4a501cf3b6e62522574da72f85c99
Usage of IP addresses like 1.2.3.4 + 2.3.4.5 is not recommended, as they
might point to actually used and non-reserved IPs.
Quoting from RFC 5737 AKA "IPv4 Address Blocks Reserved for
Documentation" (see https://datatracker.ietf.org/doc/html/rfc5737):
| The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), and
| 203.0.113.0/24 (TEST-NET-3) are provided for use in documentation.
So use 203.0.113.0/24 in our docs/code/testsuite instead.
Change-Id: Ic786a12c006a1fe11b67b788e01cca377f08771f
We check whether we have both a sites.yml and a sites/ directory.
If so, and multi-sites support is enabled, then we proceed to load
all the multi-sites configurations, by placing each of them under
«sites.<site-name>.<config>». Adding a «sites.current» link to the
current site, and filling the root tree config with the current site
as well for backwards compatibility.
Change-Id: Ia810f57e8e976eef6a7582bddda46c1701456ed6
This variable has been deprecated for a long time (since mr10.5), and it
is not used anywhere in our entire NGCP code base.
Change-Id: Ic142a347fcef0405f8cf8304728b076af3369e0d
This variable has been deprecated for a long time (since mr10.5), and it
is not used anywhere in our entire NGCP code base.
Change-Id: I83b3f00c519c8e152fcb166b4ebeb635246b1915
Previously 'set-value' sets 1 even in case 1.0 was passed.
Check if the value is number and if yes - do not reval() it.
Add a testcase for a float number.
Change-Id: I6b4e014b036fe12ad62206ad3386d0bf0b1225f7
If the element of array is not quoted it causes malformed output:
=======================
ngcpcfg set /tmp/config.yml "geo_cluster.remote_shared_proxy=[172.30.52.170]"
---
geo_cluster:
remote_shared_lb: []
remote_shared_proxy:
- "¬\x1E4ª"
======================
So detect it and print the error.
Add a testcase.
Change-Id: I7119256728c82b123eff5a171aa4e257b67ac5ff
In ngcp-initial-configuration there are a lot of multiple sequential calls
of 'ngcpcfg set' which consume significant amount of time.
So add support for setting multiple options to do it at once.
Change-Id: I8cacdbec78ecefc4681048a0fb085b4cb705d83e
Move perl code which actually modifies yml file to separate helper as
there is no sense to recreate it every call of 'ngcpcfg set'.
Change-Id: I08b10186b1aa127d4a2b81ef36b234dfdd46fc8e
With git v2.37.2-1, as present in current Debian/unstable, our unit
tests might fail - as seen with our Github actions:
| err = ('fatal: detected dubious ownership in repository at '
| "'/tmp/pytest-of-root/pytest-24/test_build_instance_customtt0/ngcpctl-pytest-base/ngcp-config'\n"
| 'To add an exception for this directory, call:\n'
| '\n'
| '\tgit config --global --add safe.directory '
With its underlying working directory looking like:
| root@b5a6b272fb90:/code# ls -la /tmp/pytest-of-root/pytest-24/test_build_instance_customtt0/ngcpctl-pytest-base/ngcp-config
| total 148
| drwxr-xr-x 5 root root 4096 Aug 15 09:12 .
| drwxr-xr-x 3 root root 4096 Aug 15 09:12 ..
| drwxr-xr-x 8 1000 1000 4096 May 18 2021 .git
| -rw-r--r-- 1 1000 1000 305 Jun 11 2020 .gitignore
| -rw-r----- 1 1000 1000 47437 Jun 15 2020 config.yml
| [...]
FTR, this can also be reproduced with our docker setup, when running as
user root, while the underlying ngcpcfg repository is owned by a normal
user:
| root@b5a6b272fb90:/code# ls -la
| total 92
| drwxr-xr-x 16 1000 1000 4096 Aug 15 08:41 .
| drwxr-xr-x 1 root root 4096 Aug 15 08:36 ..
| drwxr-xr-x 8 1000 1000 4096 Aug 15 09:22 .git
| drwxr-xr-x 4 1000 1000 4096 Jun 23 15:37 .github
| -rw-r--r-- 1 1000 1000 125 Jun 23 15:37 .gitignore
| -rw-r--r-- 1 1000 1000 64 Jul 16 2019 .gitreview
| -rw-r--r-- 1 1000 1000 169 Aug 15 08:41 .mailmap
| [...]
Quoting from git's Documentation/RelNotes/2.36.0.txt:
| * With the fixes for CVE-2022-24765 that are common with versions of
| Git 2.30.4, 2.31.3, 2.32.2, 2.33.3, 2.34.3, and 2.35.3, Git has
| been taught not to recognise repositories owned by other users, in
| order to avoid getting affected by their config files and hooks.
| You can list the path to the safe/trusted repositories that may be
| owned by others on a multi-valued configuration variable
| `safe.directory` to override this behaviour, or use '*' to declare
| that you trust anything.
Whereas the following git upstream change checks if a repository is
safe, by verifying the ownership of the worktree (if any), the git
directory, and the gitfile (if any):
| commit 3b0bf2704980b1ed6018622bdf5377ec22289688
| Author: Carlo Marcelo Arenas Belón <carenas@gmail.com>
| Date: Tue May 10 12:35:29 2022 -0700
|
| setup: tighten ownership checks post CVE-2022-24765
This change made it into git v2.30.5, v2.31.4, v2.32.3, v2.33.4,
v2.34.4, v2.35.4, v2.36.2, v2.37.1 + v2.37.2, and we got v2.37.2-1
in Debian/unstable as of 2022-08-12 (see
https://packages.qa.debian.org/g/git/news/20220813T030422Z.html).
This affects us with our mixture of root vs user permissions of the
working directory vs the git directory, so let's make sure the .git
directory has the according permissision as well.
Change-Id: I695fbd5a3b7fb79acc2873f75b8de410c8e3a0aa
* check that host on which instance runs exists
* check that instance names are not dupplicated
* check that instance names in connections are not duplicated
Additionally fix an uniqueness of array members in 'dupe_conn'.
Change-Id: I65fc31107d7e784614974ab9992836885ff50d75
* check doesn't work for 'host' type
* check doesn't test 'type' interface
* introduce test for command
Change-Id: I6a9c37a874aa219f33fde10fb0991f7450906443
The configuration files specific to the ngcpcfg tool should not be
mixed with the site specific configuration, as that's rather confusing,
more so when the actual ngcpcfg configuration contains references to the
location of the ngcp-config dir itself, which makes it self-referential.
Change-Id: I93f59e2ce3441c2c315956f68e22e4b2e9c839ac
We need to specify what to sort on, otherwise we are sorting based on
the references and not the actual keys. Add specific code blocks to sort
on the hash keys we are interested in. This makes the output
reproducible which makes the tests always pass, instead of randomly
failing depending on the ordering used randomly by perl.
Fixes: commit 83003649e3
Change-Id: I3dfc031a6c76db5153b7d05f6058e88d4b101599
Connections between instances were reworked to let easier to read
and configured and to improve the validation.
This commit adapt the script get_all_ips_of_connection_for_instance
to the new format and also the corresponding tests.
Change-Id: I6b9d21912a7db968e6c374e0afe210016ac4e879
This makes it clear this is not an ngcpcfg tool specific file but one
more in line with the cfg-schema and friends. It also gives it a proper
extension to make it clear it is a YAML file.
Change-Id: I6f19945a3ffc0e1406024b60e401688d7d3c68cc
The HA_NODE, HA_FILE and HA_CONFIG variables and nomenclature are
deprecated in favor of the NGCP_NODENAME, NODE_FILE and NODE_CONFIG.
Switch the old uses to the new ones. Unify PAIRNAME to NGCP_PAIRNAME.
Leave the old HA_ variables for backwards compatibility
Change-Id: I11f09f1620c2e456f10fc17229b61ec5a35a732b
In more recent versions, Git upstream does an owner check for the
top-level directory (see git upstream commit 8959555ce), also see
https://github.blog/2022-04-12-git-security-vulnerability-announced/
This change is included in git versions >=2.30.3, >=2.31.2, >=2.34.2,
>=2.35.2 + >=2.36.0-rc2, and therefore also affects the Git package
v2.35.2-1 as present in current Debian/unstable (as of 2022-04-16).
Now due to this behavioral change, our unit tests fail with e.g.:
| err = ('fatal: unsafe repository '
| "('/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config' "
| 'is owned by someone else)\n'
| 'To add an exception for this directory, call:\n'
| '\n'
| '\tgit config --global --add safe.directory '
| '/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config\n')
| ex = 128
We're creating many temporary git repositories. Therefore, adding every
single repository via `git config --global --add safe.directory` as
suggested in git's error message isn't really a viable option for us.
Git upstream also recognized this, and as of git rev 0f85c4a30 it's
possible to opt-out of this check via `git config --global --add
safe.directory *`. This change is currently included only in Git
versions 2.30.4, 2.31.3, 2.32.2, 2.33.3, 2.34.3 and 2.35.3, so not
available in Debian/unstable, yet.
But nevertheless, it's not really an ideal option for us, as we don't
want to mess with $HOME/.gitconfig ever, as this might not always be
some random directory inside a testing container, but pointing to an
actual user configuration.
The underlying reason, why this issue showed up in our Github actions is
caused by the fact, that the checkout of the artifacts is running as
user (also see https://github.com/actions/checkout/issues/47):
| uid=1001(runner) gid=121(docker) groups=121(docker),4(adm),101(systemd-journal)
But the docker containers are executed with root permissions in the
following steps. To properly handle this, we set the permissions of the
git repository to $UID/$GID of the executing user.
Even more tricky and worth being aware of, certain git actions might
fail due to permission issues, without telling you directly:
| root@8d1e4156f6d8:/tmp# mkdir testrepo/
| root@8d1e4156f6d8:/tmp# cd testrepo/
| root@8d1e4156f6d8:/tmp/testrepo# git init
| Initialized empty Git repository in /tmp/testrepo/.git/
| root@8d1e4156f6d8:/tmp/testrepo# chown testbuild .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| fatal: --local can only be used inside a git repository
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 128
| root@8d1e4156f6d8:/tmp/testrepo# chown root .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 0
While at it, let's unify our git configuration, by using the following
settings for all our user configuration:
| git config --local user.email pytest@example.com
| git config --local user.name pytest
Change-Id: Icad0ea4c3daf22f17481f23b27fa17750bd623da
This new action concatenates the various configuration files and outputs
the result as YAML. This is useful because they have a determined
ordering, and include optional host specific files that might not be
obvious how to merge.
This will also be the foundation for verifying all the current host
settings against the cfg-schema.
Change-Id: I72a61193f74caf3b2f7a58a47eefad2ed46c973a
Defining instances connections could be necessary (for example in SEMS)
to get both the IP of the sip_int interface of LB instance and the IP
of the sip_ext interface.
This is not currently possible because we are allowing definitions of
only one interface type per connection.
The new supported structure is:
connections:
lb:
- type: instance
name: A
interfaces:
- name: vlan1720
type: sip_int
- name: vlan1721
type: sip_ext
* pre-commit: exclude dirs from end-of-file-fixer hook
Change-Id: I832645cb916ee7f2febbea6397a439ea06fbe955
Returns an array of IPs from the connection list passed as argument
The ips are retrieved from the definition of other instances or hosts
@param argv.connection the connection list of the instance
@return out the array of IPs.
* ignore vscode files
Change-Id: I1e108f00d771e5af1d479ffa2cbced786d15e25f
* We were missing the scenario when only the build files for an instance
had changes. The service for that instance was not restarted
Change-Id: I85ba86bc57c7fc8b6df1ac2ef1144c2d3952a4da
will provide a new instance_info variable with the all the content
defined at network.yml for the instance
removed intance_name since instance_info.name has the same value
Change-Id: I349d0b430b59280bc383d46d5ba6a61f5d1f9046
The previous error message was really confusing:
> root@spce:~# ngcpcfg get /asdf
> 2021-12-06 18:48:12 sp1: Error: Cannot process template '/tmp/tmp.RBRJg02PEz':
>
> file error - parse error - input file handle line 1: unexpected token (/asdf)
> [% /asdf %]
> 2021-12-06 18:48:12 sp1: Error: Generating /tmp/tmp.NKfB8BtJXv based on /tmp/tmp.RBRJg02PEz: FAILED
It happens due to the way ngcpcfg using TT2 template which
is trying to include the file it the string looks like a PATH.
The new error message is more user friendly
(and generates proper/non-zero exit code):
> root@spce:~# ngcpcfg get /asdf
> 2021-12-07 12:42:50 spce: Error: cannot process request for '/asdf'!
> root@spce:~# echo $?
> 1
> root@spce:~#
Change-Id: Ifaf56dec07adf8600bd9d7cf2e733769d6d82a4f
- NGCPCTL_BASE
- OUTPUT_DIRECTORY{CONFIG_POOL}
* add empty-git-repo.tar.gz
* fs.py: define check_output() helper
* set DEBUG by default
* ngcpcfgcli, return env and cfg instead of just outdir
* cleanup on tests:
- don't compile the regex, it's useless in a test contex
- remove unused imports and fixtures
- use check_output() from fs
Change-Id: I0eb859b6f7aad4cd3dd8e15f4e211395c5131ce7
> # Backward compatibility config for upgrade mr3.4*->mr3.5*
> # it can be removed when the next LTS is released:
Change-Id: Ie93fc45965251dc3384a3b624339d45ad144683f
hook_setup() is called directly by some scripts
* fix test for apply: forcing changes in the repo in order to
have a commit message
Change-Id: I7da6f210826ae817b574bba793e61346ed6cac0d
gitrepo.from_archive() is suppose to be used at a ``with`` statement
so at the end the temporary directory gitrepo.localpath is removed
> with gitrepo.from_archive(path_tar) as git:
> pass
> # here gitrepo.localpath is gone already
So instead of using from_archive() let's create the temporary directory
and extract the tar file there instead.
Be aware that gitrepo.default has the git repository inside the ``testing``
directory
Change-Id: I7a50baab8c5a444e08aeabc2f1597e3a13d94c84
ngcpcfg service expects that the produced files are inside a git
repository
* ngcpcfgcli:
- set RUN_DIR by default
- set STATE_FILES_DIR by default
- use default-git-repository.tar.gz as base for outdir
- return outdir value too
- print stdout and stderr so on failure we can debug
* t/test_*:
- remove unused imports
- don't create OUTPUT_DIRECTORY unless really necessary
- remove print stdout/stderr
- black indent
Change-Id: I61e475cc9c775c30c3700a8c0e560e7c5b22f0fc
* remove support for local result
We need to force using /code volumen as read-only
to detect problems with the test environment
Change-Id: Idce04b1b00caf22562fc4161eb52317459aee03d
* ngcpcfg: obey SKIP_UPDATE_PERMS
* t/fixtures/programs.py: set SKIP_*_PERMS to true by default
* t/test_ngcpcfg_apply.py:
touching the file was not producing a change in git
Change-Id: Id8317ff09fee2a9a3d4d2015d1cd782c3f9b62e7
The junit_family default is going to change, and we get a deprecation
warning about this. Set the value we want explicitly to be future-proof
and to quiesce the warning.
Change-Id: I4c7fda419c15336d4547928da78a50c71c7b1026
Michael Prokop
Guillem Jover
Mykola Malkov
ngcp-config
dzenichev
Marco Capetta
Alexander Lutay