For multi-site support we need to pass all the possible files to load,
and these might be different between sites, so we need to check for
their existence from within tt2-process, instead of from the outside as
that would restrict them to only the current site.
Change-Id: I0965506832aa2f0a1ba67b19511c509bec60922e
This special handling was from a time when we did not have any other
generic suffix support to override based on hostname, nodename, role,
etc. It also depends on the now obsolete systems.cfg file which is
redundant and already covered by the information contained in
network.yml.
Our tt2-process tool already handles these suffixes, so there is no
point in special-casing them from within the build script. This way we
simplify our templating and overriding mechanism, and reduce duplication
of information.
Change-Id: Iabf597dbaddf755a7f85504975afc8666a58d272
We want to be able to track down any left-behind tmp files,
so ensure we're creating them with according file names.
Change-Id: I1aa0ccdb8834e8aef919f1f028028c1bb14cbd01
The HA_NODE, HA_FILE and HA_CONFIG variables and nomenclature are
deprecated in favor of the NGCP_NODENAME, NODE_FILE and NODE_CONFIG.
Switch the old uses to the new ones. Unify PAIRNAME to NGCP_PAIRNAME.
Leave the old HA_ variables for backwards compatibility
Change-Id: I11f09f1620c2e456f10fc17229b61ec5a35a732b
Otherwise 'ngcpcfg apply' silently aborted on stage 'build'
and error message is not visible due to long/parallel output.
Change-Id: I34b2c14fecad33526d89939e62d85dbafab31535
will provide a new instance_info variable with the all the content
defined at network.yml for the instance
removed intance_name since instance_info.name has the same value
Change-Id: I349d0b430b59280bc383d46d5ba6a61f5d1f9046
Switch all code to use a loop to parse options so that these can parse
multiple options (currently some scripts would simply parse one or
another but not several), and to make them easier to extend.
Change-Id: I5e902f2d7d8db78e66e48db1237924fa7d8fdda1
we need to produce different output for each instance defined
from the same source
* helper/instances-info, produces the needed output to feed
helper/tt2-process with the proper arguments to build
instances files. We inject ``instance_name`` with the
proper value to templates and INSTANCE_NAME to hooks
environment
* ${NGCPCTL_MAIN}/ngcpcfg.instances is where we define the
supported instance templates and where the output path is
* helper/tt2-process, add -r --replace <path:path> option
in order to allow generation of different output from
the same source
Change-Id: I067266b5226485ea0d6c3bc3436275c5553a8177
shellcheck v0.8 has new check, known as SC2295 ("Warn about "${x#$y}"
treating $y as a pattern when not quoted").
Fixes:
| In scripts/build line 117:
| build_file="${build_file%%.${host}}"
| ^-----^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.
| In scripts/encrypt line 24:
| x=${file##${NGCPCTL_MAIN}/templates/} # drop leading /etc/ngcp-config
| ^-------------^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.
... as reported by our Github actions at
https://github.com/sipwise/ngcpcfg
Change-Id: I3371388ecf445905cca0bc4681276151aceb43ae
Instead of checking for the user, group and mode configuration knobs in
various scripts (currently «build» and «set»), just make sure the
variables have sane defaults when loading the «ngcpcfg.cfg» file.
This also makes it possible to simplify the chown/chgrp calls into
single chown calls.
Change-Id: I1fcc06a7d311e4ebc4a96cc84ffae333be243c53
To get uniform and unexpected behavior we should use the exact same
precedence used with the tt2 files.
Fixes: commit 354e7578c5
Change-Id: I86281f3bd236687baec58b3a2e29cf80550176da
shellcheck v0.7.1 complains about a bunch of issues:
SC1083: This { is literal. Check expression (missing ;/\n?) or quote it.
SC1083: This } is literal. Check expression (missing ;/\n?) or quote it.
SC1090: Can't follow non-constant source. Use a directive to specify location.
SC2034: foo appears unused. Verify it or export it.
SC2059: Don't use variables in the printf format string. Use printf '..%s..' "$foo".
SC2128: Expanding an array without an index only gives the first element.
SC2148: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.
SC2155: Declare and assign separately to avoid masking return values.
SC2162: read without -r will mangle backslashes.
SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
SC2237: Use [ -z .. ] instead of ! [ -n .. ].
We're still bound to support shellcheck v0.3.4 and sadly this version
fails hard when using the modern "shellcheck source=" directives to
address SC1090 and SC1091. It's failing like:
| not ok 1 source/functions/init:8:28: note: The mentioned parser error was in this then clause. [SC1009]
| not ok 2 source/functions/init:10:3: error: Couldn't parse this shellcheck annotation. [SC1073]
| not ok 3 source/functions/init:10:16: error: Unexpected "s". Fix any mentioned problems and try again. [SC1072]
So until we can get rid of supporting this ancient shellcheck version,
instead of using:
| # shellcheck source=etc/ngcp-config/ngcpcfg.cfg
| # shellcheck source=functions/init
| # shellcheck source=functions/logs
| # shellcheck source=functions/main
we need to place disable=SC1090 and disable=SC1091 accordingly.
Change-Id: I14da87a00102609bffdb60e932045180c6d13d5e
This makes it possible to use more specific config.yml files either for
a pair of nodes, for settings that will always affect both, or for the
HA names, for settings that might affect all A or B nodes.
This matches the support in the template files.
We need to move sourcing the ha_features and carrier_features before
sourcing init, as now the ngcpcfg.cfg file can use these variables if
set.
Change-Id: Ic6b32acceb8f020f48c3afa09d9e84252b3531e7
The current shell code is very inefficient, as it uses files to track
the file lists, and grep to traverse them NxM times per N files.
Change-Id: I8396b4ee232157cea8ef599cba0618be3dd93536
As we now run the processing in parallel, we should not take CPU time
from other more important tasks in the system.
Change-Id: Ifcfdbc6838a377c00025b4ddb30d4ccfbd1dc548
Switch away from the old daemon/wrapper architecture, which complicates
things and requires more scaffolding code.
We turn the daemon/wrapper and build_config into a single perl process
that will load all YAML files, and then process each input/output pair
on a parallel child, to try to speed up the processing as much as
possible.
For whole rebuilds, it might speed up the generation by at least a
factor of x2, x4 or more, depending on the number of active processors.
Change-Id: I51aa2f90336e34a20983d8733f45b64d9b6fea0b
In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
We do abort building on the first broken file due to 'set -e' above.
Having $RC variable here which collects exit code is very confusing.
Change-Id: I415bc77d8becf317c1be6b5ac493be2583479e00
About 50% of previous helper/build_config execution was loading 'functions/main'
and we were performing it again and again for every tt2 files we build
and we have more then hundred files in ngcpcfg.
In fact we need two env variables and logging functions.
Let's extract them into the separate function file and load them only.
Command: time ngcpcfg build
Original results: real 0m27.766s
Current results: real 0m19.976s
Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
according to security framework spec
5.5.4 Allow r/w access /etc/ngcp-config/config.yml for users in security
domain ngcp-admin This file contains operational configuration data of
the NGCP system. Reading and editing it is part of day-to-day
operations. As such it has to be read- and writable by users in security
domain ngcp-admin. It is also readable by users of any other security
domain. This file is also read by many NGCP processes.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chgrp ngcp-admin /etc/ngcp-config/config.yml
Security domain: ngcp-admin User ownership: root Group ownership:
ngcp-admin File permissions: 664
IMPACT: Configuration data can be edited by all users in group
ngcp-admin, the sipwise user, and user root. If for some reason (e.g.
after upgrade) the proposed settings are reverted to default settings,
this does not pose a security risk. However, write access to
configuration data in this file will be denied to named users.
5.5.5 Allow r/w access to /etc/ngcp-config/network.yml for users in
security domain ngcp-admin This file contains information about the
network configuration of the NGCP cluster. This information should not
frequently change. Nevertheless, this file and its content is meant for
the operator and as such is read- and writable by users in security
domain ngcp-admin. It is also readable by users of any other security
domain. This file is also read by many NGCP processes.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chgrp ngcp-admin /etc/ngcp-config/network.yml
Security domain: ngcp-admin User ownership: root Group ownership:
ngcp-admin File permissions: 664
IMPACT: Configuration data can be edited by all users in group
ngcp-admin, the sipwise user, and user root. If for some reason (e.g.
after upgrade) the proposed settings are reverted to default settings,
this does not pose a security risk. However, write access to
configuration data in this file will be denied to named users.
5.5.6 Restrict access to /etc/ngcp-config/constants.yml to users in
security domain root This file contains values set during the
initialization of the NGCP system. It contains passwords used by
different NGCP functions to connect to other secured subsystems (e.g. DB
or lawful intercept). As such the file has a high security impact and is
read- and writeable to users of security domain root only.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chmod 600 /etc/ngcp-config/constants.yml
Security domain: root User ownership: root Group ownership: root File
permissions: 600
IMPACT: Data in this configuration file are usually entered once during
commissioning of the platform. Only users root or sipwise can edit or
read this file. The customer’s named users have no access to this data.
If for some reason (e.g. after upgrade) the proposed settings are
reverted to default settings, this poses a security risk as credentials
used internally may be leaked to unprivileged users.
Change-Id: I49a2994a227b9c296966c805c9370ae3b067de12
This is necessary when the HA_INT addresses are being changed, the communication
is not possible in that scenario.
Change-Id: I658a4f40e4f5b13d6ac224978999789728d8b915
'git status' output has been changed in 2.9 which causes some troubles for us
(we didn't change all the places, we missed ngcpcfg-ha.git repo,
better to use common code in all the places).
Change-Id: Ie664af89ee6d6b31fb3011e763107c41bf4cf00f
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
The new function will simplify initial customtt->patchtt
migration for end users. Some code was merged to be generic.
Also more tests were add here.
Change-Id: I7719f45275018818b2db82f6deee5b7428670a29
The 'ngcpcfg' received support for 'patchtt' files, like
> /etc/ngcp-config/templates/etc/foo/bar.patchtt.tt2
Those 'patchtt' are going to be applied on default 'tt2 template' file:
> /etc/ngcp-config/templates/etc/foo/bar.tt2
and produce 'customtt' on 'ngcpcfg patch':
> /etc/ngcp-config/templates/etc/foo/bar.customtt.tt2
Further 'customtt' will be used to overwrite 'tt2 templates'
on 'ngcpcfg build' or 'ngcpcfg apply'.
NOTE: 'ngcpcfg patch' is executed automatically on every 'ngcpcfg build'.
It should allows to update ngcp-templates easily and support
local modifications without the pain (until the patches can be applied).
Change-Id: Ice4369386313c5d33e4d498346345eade6f3d0d7
This option is essential if we're testing from within the git
repository and aren't standing on active 'master' branch.
Change-Id: I75b62d071116313317c69ad496e4425d6e98ad5a
When installing Debian/stretch we fail at this stage:
| ngcpcfg --no-db-sync commit
On first glance it looks like a mariadb issue, though actually
it's the git status check inside ngcpcfg that's failing. The
reason is that starting with git v2.9.3 there's a different
output in its "status" output:
| commit 2a0e6cdedab306eccbd297c051035c13d0266343
| Author: Lars Vogel <lars.vogel@gmail.com>
| Date: Thu Jun 9 20:19:30 2016 +0200
|
| Use "working tree" instead of "working directory" for git status
|
| Working directory can be easily confused with the current directory.
| In one of my patches I already updated the usage of working directory
| with working tree for the man page but I noticed that git status also
| uses this incorrect term.
|
| Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
| Signed-off-by: Junio C Hamano <gitster@pobox.com>
In Debian/stretch we've 2.11.0-2 and are affected by this change.
Adjust the `git status` checks accordingly to support git in all
its versions.
NOTE: we should consider switching to `git status --porcelain`
(see git commit fc5af3bbb2 for
some history about it) once we've the testsuite in place.
Change-Id: If4223a35083d279e746dce5a80bcb86b52f64025
Make the relationship internal, instead of external. So that anything
that might end up calling the build script will always do the right
thing and we will not forget to perform required actions.
Change-Id: Iafc3bc7230c59750de7ec8bf825005011b48f403
In a previous implementation using timestamps we ran into timing
problems on slower systems. So instead lets record the git commit
ID of the latest ngcpcfg commit and check whether we have a match
between this commit and an according ngcpcfg build.
If the "ngcpcfg build" was executed without an "ngcpcfg commit"
before then we mark it as dirty since we don't have an according
git commit available for testing against. Since that's a common
workflow we don't report it as a real problem though, instead we
inform the user using:
| OK: nothing to build (latest build newer than latest commit)
As we commit AFTER the build in the apply action we can use a
workaround for this situation, so build and commit state match
with each other in that case then.
Change-Id: I48e32db6f42b53fe97a0b88805e7ddbee8576133
We will use first free port instead which will be stored into
/var/run/ngcpcfg.port file while tt2-daemon is up and running.
Change-Id: I1a918f865d87863eef7d34be8abd1032282220b8
When runnning 'ngcpcfg build' (which is also executed when
running 'ngcpcfg apply') we waste quite some time with loading
the same YAML configuration files and merging them again and
again.
In a 'ngcpcfg build' run we don't have to re-load and merge those
configuration files again and again (since they won't modify
during a single run), instead just load+merge them *once* and
re-use the result on all the following execution steps in the
same run.
Original:
| # time ngcpcfg apply >/dev/null
| ngcpcfg apply > /dev/null 149.01s user 9.49s system 96% cpu 2:43.57 total
Reworked optimized runs:
| # time ngcpcfg apply >/dev/null
| ngcpcfg apply > /dev/null 81.25s user 8.62s system 95% cpu 1:34.17 total
For debugging purposes you can execute 'ngcpcfg build' under
NO_REUSE=1 to not use this feature, executing it under 'DEBUG=1'
won't remove the so called reuse file which stores the merged
configuration data.
Thanks: Alexander Lutay for the initial patch and idea
Files in configuration directory EXTRA_CONFIG_DIR (being
/etc/ngcp-config/config.d/ by default) with the suffix yml (so
e.g. a file named pbx.yml can be shipped by an external Debian
package) are additionally taken into consideration as main
configuration files.
Those extra configuration file(s) have higher precedence than
the other yml configuration files (config.yml, network.yml,...),
*except* for constants.yml which has the power to override all
settings, including the ones from files inside EXTRA_CONFIG_DIR.
Closes: sipwise-#3059
Guillem Jover
Mykola Malkov
Michael Prokop
Alexander Lutay
ngcp-config
Manuel Montecelo
Richard Fuchs