Fixing issues reported by new shellcheck v0.9.0:
1) SC2317 (info): Command appears to be unreachable. Check usage (or ignore if invoked indirectly).
https://www.shellcheck.net/wiki/SC2317
(new feature as of shellcheck v0.9.0)
2) SC2086 (info): Double quote to prevent globbing and word splitting.
https://www.shellcheck.net/wiki/SC2086
(behavior change with shellcheck v0.9.0)
Change-Id: I73d50157f72b772c381cf1bff497079133bc6d09
Introduce a list of checks related to connections:
* check that we don't have instance names duplicates
* prove an existence of a preferred for instnace host
* an instance to which instance gets connected exists
* a host to which instance gets connected exists
* interface via which instance gets connected is defined
Change-Id: I24f43b0fb24e308f571a88f30a72a9b6dd04b94d
We add support for YAML schemas, and hook it into the check command to
validate the sites.yml file.
Change-Id: Ic6ed43e36361d395b485230af4ec3cf7e77758bb
we can define what is the expected origin via
/etc/ngcp-ngcpcfg/ngcpcfg.d/mgmnode.cfg for instance
Change-Id: Ied83502ddc022a1c7bb948f43bee7ccc10efcd88
This off-loads the check to the ngcp-site script, and prints any error
message from it if the script fails.
Only run the check if ngcp-site is present, so that we avoid a
hard-dependency on system-tools.
Change-Id: I640f85383d424980d81a207dee7d586a88422862
We want to be able to track down any left-behind tmp files,
so ensure we're creating them with according file names.
Change-Id: I1aa0ccdb8834e8aef919f1f028028c1bb14cbd01
The HA_NODE, HA_FILE and HA_CONFIG variables and nomenclature are
deprecated in favor of the NGCP_NODENAME, NODE_FILE and NODE_CONFIG.
Switch the old uses to the new ones. Unify PAIRNAME to NGCP_PAIRNAME.
Leave the old HA_ variables for backwards compatibility
Change-Id: I11f09f1620c2e456f10fc17229b61ec5a35a732b
We should validate all configuration files by concatenating them, so
that any local addition or override gets schema validated, otherwise
these modifications will pass without notice, until it is probably too
late.
Change-Id: I9c7774eefdeadddb399dd68668779dffcd173696
Previously we had quite complicated schema of origin urls on Pro/Carrier
mgmt/Carrier non-mgmt nodes:
Pro, both sp1 and sp2 - local /mnt/glusterfs/ngcpcfg-share directory.
Carrier mgmt nodes, web01a/web01b - local /mnt/glusterfs/ngcpcfg-share
directory.
Carrier other nodes - web01:/mnt/glusterfs/ngcpcfg-share.
There is no sense in such a complication so it was changed to use
${mgmt}:/mnt/glusterfs/ngcpcfg-share for all installation types and
nodes.
Change-Id: If84f70da51739a89763fb54091faaab9e15ffc9c
To get uniform and unexpected behavior we should use the exact same
precedence used with the tt2 files.
Fixes: commit 354e7578c5
Change-Id: I86281f3bd236687baec58b3a2e29cf80550176da
shellcheck v0.7.1 complains about a bunch of issues:
SC1083: This { is literal. Check expression (missing ;/\n?) or quote it.
SC1083: This } is literal. Check expression (missing ;/\n?) or quote it.
SC1090: Can't follow non-constant source. Use a directive to specify location.
SC2034: foo appears unused. Verify it or export it.
SC2059: Don't use variables in the printf format string. Use printf '..%s..' "$foo".
SC2128: Expanding an array without an index only gives the first element.
SC2148: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.
SC2155: Declare and assign separately to avoid masking return values.
SC2162: read without -r will mangle backslashes.
SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.
SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
SC2237: Use [ -z .. ] instead of ! [ -n .. ].
We're still bound to support shellcheck v0.3.4 and sadly this version
fails hard when using the modern "shellcheck source=" directives to
address SC1090 and SC1091. It's failing like:
| not ok 1 source/functions/init:8:28: note: The mentioned parser error was in this then clause. [SC1009]
| not ok 2 source/functions/init:10:3: error: Couldn't parse this shellcheck annotation. [SC1073]
| not ok 3 source/functions/init:10:16: error: Unexpected "s". Fix any mentioned problems and try again. [SC1072]
So until we can get rid of supporting this ancient shellcheck version,
instead of using:
| # shellcheck source=etc/ngcp-config/ngcpcfg.cfg
| # shellcheck source=functions/init
| # shellcheck source=functions/logs
| # shellcheck source=functions/main
we need to place disable=SC1090 and disable=SC1091 accordingly.
Change-Id: I14da87a00102609bffdb60e932045180c6d13d5e
This makes it possible to use more specific config.yml files either for
a pair of nodes, for settings that will always affect both, or for the
HA names, for settings that might affect all A or B nodes.
This matches the support in the template files.
We need to move sourcing the ha_features and carrier_features before
sourcing init, as now the ngcpcfg.cfg file can use these variables if
set.
Change-Id: Ic6b32acceb8f020f48c3afa09d9e84252b3531e7
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
NGCP uses 'YAML::XS' as only the library to work with YAML files.
Meanwhile pkwalify has a long fallback list:
> @try_order = ('YAML::XS', 'YAML::Syck', 'YAML', 'JSON::XS', 'JSON');
It may hide some issues in validation schema (used by 'ngcpcfg check')
and bypass the testing on Gerrit reviews, but failed to 'ngcpcfg apply'
on the customer's system.
Example:
1) cfg_schema.git had a bug in validation schema committed long time ago
in mr6.1+ , commit b9d69e0b33d316c04d8b70536f1e166fec9a67ac.
The commit passed review and all internal testing:
> https://gerrit.mgm.sipwise.com/#/c/17825/ ->
> https://repoapi.mgm.sipwise.com/job/cfg-schema-ce-validation-docker/849/console.txt
> ...
> + ngcpcfg --validate check
> 2017-12-20 14:34:37: yml configs were validated successfully
The bug was simple, the space was missing in schema:
> "rtcp_logging_facility":{ type: str, required: yes }
The library 'YAML::XS' was not able to load the YAML file.
Meanwhile the library 'YAML' was able to load it and pkwalify happily reported us
"yml configs were validated successfully".
2) Later another developer has noticed the problem with YAML schema here
and fixed it in trunk, commit 4636eadb78e5e91dd6d93378a42a8df2ed84838a.
> - "rtcp_logging_facility":{ type: str, required: yes }
> + "rtcp_logging_facility": { type: str, required: yes }
Unfortunately the fix has been applied in trunk only.
3) Later the third developer committed new code which was valid
for 'YAML::XS' but invalid for 'YAML' library (which is fine, as NGCP uses 'YAML::XS').
Since in trunk pkwalify used 'YAML::XS' the changes passed all the testing:
> commit a79e54fb9fbdf0db61b4dfb9c52e572795e854af
> https://gerrit.mgm.sipwise.com/#/c/28486/ ->
> https://repoapi.mgm.sipwise.com/job/cfg-schema-ce-validation-docker/1987/console.txt
> ...
> 2019-04-05 17:20:08 06828d6f31f5: yml configs were validated successfully
4) Later the third developer backported the commit a79e54fb9fbdf0db
to the previous release mr6.5 (where commit 4636eadb78e was missing).
And nightly upgrade tests has failed on mr6.4->mr6.5 upgrade saying:
> 2019-04-09 04:06:27 sp1: Error: Invalid schema detected for /etc/ngcp-config/config.yml
> Cannot parse </usr/share/ngcp-cfg-schema/validate/config.yml>. Cumulated errors:
> YAML::XS::Load Error: The problem:
> did not find expected key
It happens because pkwalify was not able to load config.yml with any libraries it support.
We have to force pkwalify to use only the library NGCP is using 'YAML::XS'.
This is a commit about it.
Hopefully all the necessary changes have been done by Guillem 2 years ago,
and were uploaded upstream (and even available in default Debian buster):
> https://github.com/eserte/p5-Kwalify/pull/6
Thanks to Guillem Jover for the library fix here and for the help with tracing this issue.
Change-Id: Idbd46b4048a03b5b01a0280a6dcd50406b4222dc
This is necessary when the HA_INT addresses are being changed, the communication
is not possible in that scenario.
Change-Id: I658a4f40e4f5b13d6ac224978999789728d8b915
Otherwise ngcp-installer failed to install PRO/Carrier as
we build the file /etc/ngcp_mgmt_node the very first time on the first MGMT node:
> +08:22:48 cfg_build_templates
> +08:22:48 cfg_build_configs /etc/ngcp_mgmt_node
> +08:22:48 build_configs=($1)
> +08:22:48 declare -a build_configs
> +08:22:48 log_info 'Generating default configuration files /etc/ngcp_mgmt_node'
> +08:22:48 tee -a /tmp/ngcp-installer.log
> +08:22:48 echo 'Generating default configuration files /etc/ngcp_mgmt_node'
> Generating default configuration files /etc/ngcp_mgmt_node
> +08:22:48 ngcpcfg build /etc/ngcp_mgmt_node
> cat: /etc/ngcp_mgmt_node: No such file or directory
> +08:22:48 log_die 'Error running '\''ngcpcfg build'\'''
Checking the file availability is not enough here, the next error is:
> (sp1)root@sp1:/# ngcpcfg build
> /usr/share/ngcp-ngcpcfg/scripts//check: line 83: NGCP_IS_MGMT: unbound variable
> (sp1)root@sp1:/# cat /etc/default/ngcp-roles
> NGCP_TYPE="sppro"
> (sp1)root@sp1:/#
It happens because /etc/default/ngcp-roles is a fake one at this stage.
Handling NGCP_IS_MGMT properly is also not enough:
> (sp1)root@sp1:/# ngcpcfg build
> 2018-07-16 16:49:32: Error: Remote origin of ngcpcfg is '/mnt/glusterfs/ngcpcfg-share', expected: 'sp:/mnt/glusterfs/ngcpcfg-share'.
> 2018-07-16 16:49:32: Error: NOTE: execute `cd /etc/ngcp-config ; git remote set-url origin 'sp:/mnt/glusterfs/ngcpcfg-share'` to adjust setting.
> 2018-07-16 16:49:32: Error: NOTE: perform `ngcpcfg clean --all` to recreate local master branch from remote.
> (sp1)root@sp1:/#
Which happens because the peer is not yet configured at the moment (first node installation).
Introducing new internal option '--no-check-origin' to skip the test into installer.
Change-Id: I0265c65f45972e92ca92320871a7ef29f8904fec
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ie51f1c79859d40ef0877fc0ab75f86ee72e14ea4
This script will validate the network.yml based on a schema constructed
from information only available from the network.yml file itself. This
way we can do the strictest validation, which we could not do before.
Change-Id: I32714e678e901e58d70e4253bcc61a147494c225
This option is essential if we're testing from within the git
repository and aren't standing on active 'master' branch.
Change-Id: I75b62d071116313317c69ad496e4425d6e98ad5a
At the moment we provide no output for the end users, which
confuses them for the checks with and without the validation.
We should clearly report to users what has been checked.
Change-Id: Icc61744637345a1dc51fe8d8dd9bd3fd7f86824d
Mainly nobody validate the schema in trunk right now
as it requires manual changes for ngcpcfg.cfg.
Also having force validation will allow us to add
validation in ngcp-upgrade and Jenkins nightly builds.
Change-Id: Ia48a778ecf7dae30b26bdff0c5dd5b9000c9d089
Verify that the user is operating on branch master,
otherwise the result might be unexpected.
If branch master is checked out:
| root@spce:/etc/ngcp-config# ngcpcfg check
| root@spce:/etc/ngcp-config# ngcpcfg status
| [...]
| 2016-02-23 05:48:32: Checking currently active branch:
| 2016-02-23 05:48:32: OK: branch master active
If a branch other than 'master' is checked out:
| root@spce:/etc/ngcp-config# ngcpcfg check
| 2016-02-23 05:47:42: Error: branch 'mika' in '/etc/ngcp-config' active - please switch to branch 'master' before continuing.
| root@spce:/etc/ngcp-config# echo $?
| 1
| root@spce:/etc/ngcp-config# ngcpcfg status
| [...]
| 2016-02-23 05:47:47: Checking currently active branch:
| 2016-02-23 05:47:47: ACTION_NEEDED: branch 'mika' active - please switch to branch 'master'
| [...]
Change-Id: I5df92075905cafa3b714211581c8cfe749df04ba
This allows us to abort in 'ngcpcfg status' whenever there are
outstanding changes to pull/push from the shared storage. Using
the --no-action-failure option allows the user to continue anyway.
Change-Id: I4062d5bb627bb553b98705bb122575651b035849
Disabled by default as we need a time to check it carefuly.
Also we need to solve the "tilda" problem reported upstream:
https://github.com/eserte/p5-Kwalify/issues/1
Change-Id: Ia2c3d48f0ac6fc6ac9899d44f4291544373806d9
Mykola Malkov
Michael Prokop
Guillem Jover
dzenichev
ngcp-config
Alexander Lutay
Andreas Granig
Manuel Montecelo