we can define what is the expected origin via
/etc/ngcp-ngcpcfg/ngcpcfg.d/mgmnode.cfg for instance
Change-Id: Ied83502ddc022a1c7bb948f43bee7ccc10efcd88
This off-loads the check to the ngcp-site script, and prints any error
message from it if the script fails.
Only run the check if ngcp-site is present, so that we avoid a
hard-dependency on system-tools.
Change-Id: I640f85383d424980d81a207dee7d586a88422862
This is the symlink pointing to the current site for this site, which
cannot be committed to git, or we'd get constant conflicts when multiple
sites commit their own.
Change-Id: I054afc15bb3a52d0ca2357db5ea4a152cfff035b
The configuration files specific to the ngcpcfg tool should not be
mixed with the site specific configuration, as that's rather confusing,
more so when the actual ngcpcfg configuration contains references to the
location of the ngcp-config dir itself, which makes it self-referential.
Change-Id: I93f59e2ce3441c2c315956f68e22e4b2e9c839ac
We need to specify what to sort on, otherwise we are sorting based on
the references and not the actual keys. Add specific code blocks to sort
on the hash keys we are interested in. This makes the output
reproducible which makes the tests always pass, instead of randomly
failing depending on the ordering used randomly by perl.
Fixes: commit 83003649e3
Change-Id: I3dfc031a6c76db5153b7d05f6058e88d4b101599
Connections between instances were reworked to let easier to read
and configured and to improve the validation.
This commit adapt the script get_all_ips_of_connection_for_instance
to the new format and also the corresponding tests.
Change-Id: I6b9d21912a7db968e6c374e0afe210016ac4e879
The rtpengine project used to be called rtpengine-ng, and before that
mediaproxy-ng, which was an implementation using the same protocol as
the rtpproxy project. It's about time we rename the config.yml section
so that this is less confusing.
Change-Id: I51fc5effc515ccca74b4d2093b9665edb20286f8
This makes it clear this is not an ngcpcfg tool specific file but one
more in line with the cfg-schema and friends. It also gives it a proper
extension to make it clear it is a YAML file.
Change-Id: I6f19945a3ffc0e1406024b60e401688d7d3c68cc
The factory_default files are not created anymore by the installer,
and the systems.cfg file is neither created by the installer nor
needed or used since commit 5e2037cf82.
Change-Id: I9c58021db307f23923b4cfee8a1862561dc09d3d
This special handling was from a time when we did not have any other
generic suffix support to override based on hostname, nodename, role,
etc. It also depends on the now obsolete systems.cfg file which is
redundant and already covered by the information contained in
network.yml.
Our tt2-process tool already handles these suffixes, so there is no
point in special-casing them from within the build script. This way we
simplify our templating and overriding mechanism, and reduce duplication
of information.
Change-Id: Iabf597dbaddf755a7f85504975afc8666a58d272
Starting with mr10.0 heartbeat-2 is no longer supported, starting with
mr10.5 (the next LTS) heartbeat-2 is no longer available.
Use better terminology to avoid confusion.
Change-Id: I585d79f83e12db92df7068f4fbd1ef798ba7e1d0
We want to be able to track down any left-behind tmp files,
so ensure we're creating them with according file names.
Change-Id: I1aa0ccdb8834e8aef919f1f028028c1bb14cbd01
Due to proposal to refactor the 'get_hosts_records_for_db_connections'
some improvements and "shrinking" of that has been applied.
This however doesn't change a result which this script produces.
Change-Id: Ib977a8c05af59e158b66a238a52d4f91f090c162
The HA_NODE, HA_FILE and HA_CONFIG variables and nomenclature are
deprecated in favor of the NGCP_NODENAME, NODE_FILE and NODE_CONFIG.
Switch the old uses to the new ones. Unify PAIRNAME to NGCP_PAIRNAME.
Leave the old HA_ variables for backwards compatibility
Change-Id: I11f09f1620c2e456f10fc17229b61ec5a35a732b
Let's remove the duplicate implementation, and simply off-load to the
command, so that we only need to modify one place.
As a side effect, this removes a bogus fallback value of 'self' for
CE systems, which should have been 'spce'.
Change-Id: I70b48caad3da4123a3399254deda73c381ae67d9
Add support for getting the nodename for any hostname in an NGCP
cluster.
We switch from fetching this information from the filesystem to the one
contained now in network.yml.
We accept up to 9 nodes per "pair", from sp1 to sp9, so for Carrier
setups that means hostnames ending from 'a' to 'i'. On CE the nodename
is called "spce".
If we cannot match the hostname to a known pattern we will error out.
Change-Id: I80c287046d02f60193479febee427615474043e3
Path.is_relative() (as used in t/fixtures/programs.py) is a new feature
as of Python 3.9, see
https://docs.python.org/3.9/whatsnew/changelog.html?highlight=is_relative_to
and https://github.com/python/cpython/issues/81870
When running with an older python3 version, tests fail with:
| env = {}
|
| def prepare_conf(env={}):
| testenv = define_env(env)
| ngcpcfg_path = testenv["NGCPCFG"]
| > if not ngcpcfg_path.is_relative_to(CWD):
| E AttributeError: 'PosixPath' object has no attribute 'is_relative_to'
While python3-pytest depends on python3:any, let's keep the build
dependency slim, by build-depending on the python3-minimal package only.
Problem spotted while trying to debug the git safe.directory issue
in an outdated bullseye based container, providing python3 3.7.5-3.
Change-Id: Ib620a796a780817a81a51d791a766c950b1a541c
In more recent versions, Git upstream does an owner check for the
top-level directory (see git upstream commit 8959555ce), also see
https://github.blog/2022-04-12-git-security-vulnerability-announced/
This change is included in git versions >=2.30.3, >=2.31.2, >=2.34.2,
>=2.35.2 + >=2.36.0-rc2, and therefore also affects the Git package
v2.35.2-1 as present in current Debian/unstable (as of 2022-04-16).
Now due to this behavioral change, our unit tests fail with e.g.:
| err = ('fatal: unsafe repository '
| "('/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config' "
| 'is owned by someone else)\n'
| 'To add an exception for this directory, call:\n'
| '\n'
| '\tgit config --global --add safe.directory '
| '/tmp/pytest-of-root/pytest-0/test_status_build0/ngcpctl-pytest-base/ngcp-config\n')
| ex = 128
We're creating many temporary git repositories. Therefore, adding every
single repository via `git config --global --add safe.directory` as
suggested in git's error message isn't really a viable option for us.
Git upstream also recognized this, and as of git rev 0f85c4a30 it's
possible to opt-out of this check via `git config --global --add
safe.directory *`. This change is currently included only in Git
versions 2.30.4, 2.31.3, 2.32.2, 2.33.3, 2.34.3 and 2.35.3, so not
available in Debian/unstable, yet.
But nevertheless, it's not really an ideal option for us, as we don't
want to mess with $HOME/.gitconfig ever, as this might not always be
some random directory inside a testing container, but pointing to an
actual user configuration.
The underlying reason, why this issue showed up in our Github actions is
caused by the fact, that the checkout of the artifacts is running as
user (also see https://github.com/actions/checkout/issues/47):
| uid=1001(runner) gid=121(docker) groups=121(docker),4(adm),101(systemd-journal)
But the docker containers are executed with root permissions in the
following steps. To properly handle this, we set the permissions of the
git repository to $UID/$GID of the executing user.
Even more tricky and worth being aware of, certain git actions might
fail due to permission issues, without telling you directly:
| root@8d1e4156f6d8:/tmp# mkdir testrepo/
| root@8d1e4156f6d8:/tmp# cd testrepo/
| root@8d1e4156f6d8:/tmp/testrepo# git init
| Initialized empty Git repository in /tmp/testrepo/.git/
| root@8d1e4156f6d8:/tmp/testrepo# chown testbuild .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| fatal: --local can only be used inside a git repository
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 128
| root@8d1e4156f6d8:/tmp/testrepo# chown root .
| root@8d1e4156f6d8:/tmp/testrepo# git config --local user.email pytest@example.com
| root@8d1e4156f6d8:/tmp/testrepo# echo $?
| 0
While at it, let's unify our git configuration, by using the following
settings for all our user configuration:
| git config --local user.email pytest@example.com
| git config --local user.name pytest
Change-Id: Icad0ea4c3daf22f17481f23b27fa17750bd623da
This is a script, which properly prepares hosts records for
connections towards nosql/sql DBs.
It is able to differentiate the Carrier grade from the PRO setup.
It creates the following list of records:
- at least one record towards central node of the 'db' role (db.central);
- at least one record towards central node of the 'db' role (nosql.central) ;
- records for each PRX/LB node pointing to replicated pair and replicated central ;
In the Carrier grade:
- db.central, points to -> shared IP of the central DB (only on Proxy nodes)
- nosql.central, points to -> shared IP of the central Nosql server (only on Proxy nodes)
- LB has records (for all LB nodes) of replicated pair (nosql db)
pointing to its 'ha_int' type IP address
- Proxy has records (for all Proxy nodes) of replicated pair (nosql db, sql db)
and replicated central type (db only), pointing to its 'ha_int' type IP address
In the PRO setup:
- db.central, points to -> own 'ha_int' type IP address
- nosql.central, points to -> own 'ha_int' type IP address
- records (for all nodes) of replicated pair (nosql db, sql db) and
replicated central type (sql db only), pointing to its 'ha_int' type IP address
This gives us a way for the nosql db / sql db be listening on the
internal IP addresses (not on the loopback ifaces) and not use floating IPs
to be reachable constantly by instances.
Meanwhile instances constantly remain using local nosql/sql DB
(of course excluding a case with the Carrier,
where central sql db/nosql db has a remote shared IP).
Change-Id: Ifceb9fccab48cf78856f8aa8752c3a8b769e3f63
Related to previous commit (fa645475), we shouldn't execute
restore-permissions with any actions that aren't modifying system state.
Therefore also skip restore-permissions execution for diff, log + status
actions.
While at it, sort list of actions alphabetically.
Thanks Guillem Jover for raising this.
Change-Id: I803b08eee843d31f12d9810fe33b4327490228da
Execution of `ngcpcfg check` shouldn't modify system state,
and executing the restore-permissions helper script also
takes some notable time (e.g. during ngcp-status execution
it causes ~10% of its runtime).
Change-Id: I8ad861379da3817f0bdf760ea16939c80eae2ce6
We should validate all configuration files by concatenating them, so
that any local addition or override gets schema validated, otherwise
these modifications will pass without notice, until it is probably too
late.
Change-Id: I9c7774eefdeadddb399dd68668779dffcd173696
This new action concatenates the various configuration files and outputs
the result as YAML. This is useful because they have a determined
ordering, and include optional host specific files that might not be
obvious how to merge.
This will also be the foundation for verifying all the current host
settings against the cfg-schema.
Change-Id: I72a61193f74caf3b2f7a58a47eefad2ed46c973a
The YAML::XS module handles its passed or returned data for Load() and
Dump() as raw UTF-8, irrespective of perl's UTF-8 string markers. When
we use these functions we need to use a «:raw» encoding or we will end
up with doubly encoded strings.
We also need to tell the Template Toolkit to use UTF-8 as encoding, and
then setup the file handles to use the appropriate encodings.
Change-Id: I37a6811f0680763d7177c3ad92ddf9b890869e66
We need to add a new script, which finds and grabs
a list needed advertised ip addresses for a particular instance.
Change-Id: Id33c4e2d4de071c5feb34dd9346a33ef4a61381b
Defining instances connections could be necessary (for example in SEMS)
to get both the IP of the sip_int interface of LB instance and the IP
of the sip_ext interface.
This is not currently possible because we are allowing definitions of
only one interface type per connection.
The new supported structure is:
connections:
lb:
- type: instance
name: A
interfaces:
- name: vlan1720
type: sip_int
- name: vlan1721
type: sip_ext
* pre-commit: exclude dirs from end-of-file-fixer hook
Change-Id: I832645cb916ee7f2febbea6397a439ea06fbe955
Otherwise 'ngcpcfg apply' silently aborted on stage 'build'
and error message is not visible due to long/parallel output.
Change-Id: I34b2c14fecad33526d89939e62d85dbafab31535
Returns an array of IPs from the connection list passed as argument
The ips are retrieved from the definition of other instances or hosts
@param argv.connection the connection list of the instance
@return out the array of IPs.
* ignore vscode files
Change-Id: I1e108f00d771e5af1d479ffa2cbced786d15e25f
* We were missing the scenario when only the build files for an instance
had changes. The service for that instance was not restarted
Change-Id: I85ba86bc57c7fc8b6df1ac2ef1144c2d3952a4da
This will be used in the templates for values that are scaled based on
the number of online CPUs on the system, such as process pools, or load
averages.
Change-Id: I3666873f327becc3f7a02be02dba82493b29252e
The prevalent convention in the project is to use «-» instead of «_» to
split words in options. Switch the options to that, and keep backwards
compatibility option alias that emits a deprecation warning while people
transition to the new names. The alias will be removed on the next LTS.
Change-Id: I8bb7f26f8a766f7ea76ba2d9e49b793e5cb8d11e
Instead of having to identify the parent process name (ngcpcfg vs.
ngcp-config) inside subprocesses (like "status"), let's provide the
invocation name through the main ngcpcfg binary via variable
NGCPCFG_NAME.
Thanks to Guillem Jover for the suggestion.
Change-Id: Iafbac535bedb9cc9dc73f3d95861a5bc735cc739
will provide a new instance_info variable with the all the content
defined at network.yml for the instance
removed intance_name since instance_info.name has the same value
Change-Id: I349d0b430b59280bc383d46d5ba6a61f5d1f9046
When invoking `ngcp-config status`, we reported "ngcpcfg":
| [sipwise-lab-trunk] sipwise@sp1:~$ sudo ngcpcfg status
| 2021-12-09 17:30:08 sp1: Checking state of ngcpcfg:
| 2021-12-09 17:30:08 sp1: OK: has been initialised already
| [...]
While what we should report instead is "ngcpcfg-config", like:
| [sipwise-lab-trunk] sipwise@sp1:~$ sudo ngcp-config status
| 2021-12-09 17:31:31 sp1: Checking state of ngcp-config:
| 2021-12-09 17:31:31 sp1: OK: has been initialised already
| [...]
Change-Id: If8fc8076fdf1d214baea3c7388ab208947d66a05
ngcp-config
Guillem Jover
Michael Prokop
Sipwise Jenkins Builder
Marco Capetta
Donat Zenichev
Mykola Malkov
Alexander Lutay