'git status' output has been changed in 2.9 which causes some troubles for us
(we didn't change all the places, we missed ngcpcfg-ha.git repo,
better to use common code in all the places).
Change-Id: Ie664af89ee6d6b31fb3011e763107c41bf4cf00f
We have a hard dependency on netcat-openbsd, see:
| commit 26ba0340b6
| Author: Alexander Lutay <alutay@sipwise.com>
| Date: Mon Jun 4 13:22:27 2018 +0200
|
| TT#37401 Fix 50ecc1544: depends on netcat-openbsd since we use 'nc -U' (not available in 'netcat-traditional')
But our docker image still uses netcat-traditional.
Make sure to have netcat-openbsd available, while at
it also adjust Build-Depends accordingly.
Change-Id: I1d3cfd9b4b56047fa51c3ef1d77060122f4d2568
Due to the code in tt2-wrapper, the only errors reported were failures to
connect to the socket to use the back-end performing the parsing. But there was
no attempt to detect errors in the parsing at all.
Since we are only able to use the output stream and not for example exit codes
of the parser at the back-end (libtemplate-perl), we can only rely on finding
some kind of error string, which can change in future versions of
libtemplate-perl but hopefully not very often, and at most it should only change
when we move base to newer Debian releases.
In the case of the original problem reported we could rely on "parse error", but
since there are two such strings ("file error" and "parse error") the first is
more general and also covers cases like "permission denied to read file" or
other errors that might happen. So the implementation of error checking is
basically grepping for "^file error", the string at the beginning of a line, to
limit possible false positives.
With the new checks and an example of an invalid syntax already in the first
line, "ngcpcfg build" detects the error correctly and stops processing, and
shows information about the source of the problem, e.g.:
root@spce:~# ngcpcfg build /etc/test
2018-10-29 11:37:01 spce: yml configs were validated successfuly
2018-10-29 11:37:01 spce: configs were checked successfuly
2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch.
Loading /etc/ngcp-config/config.yml in memory: OK
Loading /etc/ngcp-config/network.yml in memory: OK
Loading /etc/ngcp-config/constants.yml in memory: OK
2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED
2018-10-29 11:37:02 spce: Error: from generated file:
2018-10-29 11:37:02 spce: Error: file error - parse error - input file handle line 1: unexpected end of directive
2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding:
2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2
2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml
2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file>
2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G
2018-10-29 11:37:02 spce: should provide more details.
Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13
When the "mktemp" command is invoked with a template parameter without path
prefixes or without using "--tmpdir", it does not use "/tmp" by default, but
instead it uses the current working directory as the place to create these
temporary files.
This was an oversight based on our side on the unknown behaviour, which was
discovered when working on this task, because we do want to have "/tmp" as the
home for these temporary files.
Change-Id: I1b03dc90cfd8dd9364c8bc3b2bfe0e3c26979e7f
In some conditional branches we make the ip key optional, but never
change netmask to match. Which can end up being required when it
should not, such as on a vlan interface.
Change-Id: I9767434901abb2c0f07c10f8b8800987c2e363c5
The night Carrier trunk installation has failed:
> +02:29:59 (cfg_pro.inc:280): cfg_ngcpcfg_init_mgmt(): cfg_ensure_file_mgmt_node
> ...
> +02:29:59 (cfg_pro.inc:282): cfg_ngcpcfg_init_mgmt(): '[' -x /usr/share/ngcp-ngcpcfg/scripts/init-mgmt ']'
> +02:29:59 (cfg_pro.inc:287): cfg_ngcpcfg_init_mgmt(): DEBUG=yes
> +02:29:59 (cfg_pro.inc:287): cfg_ngcpcfg_init_mgmt(): /usr/share/ngcp-ngcpcfg/scripts/init-mgmt
> /usr/share/ngcp-ngcpcfg/functions//main: line 18: HNAME: unbound variable
The manual execution of script 'init-mgmt' shows no error,
so far if 'DEBUG=yes' is enabled the issue is easily reproducible.
The reason here is debug messages which can be printed on the
very early stage of initialisation and cause the issue here:
> ++ log_debug 'sourced configuration file /etc/ngcp-config/ngcpcfg.cfg'
> ++ '[' -n yes ']'
> ++ logger -t ngcpcfg -- 'Debug: sourced configuration file /etc/ngcp-config/ngcpcfg.cfg'
> ++ console_output 'DEBUG: sourced configuration file /etc/ngcp-config/ngcpcfg.cfg\n'
> ++ '[' -z '+%F %T' ']'
> +++ date '+%F %T'
> ++ local 'timestamp=2018-10-12 10:16:24'
> +++ printf -- '%s: ' '2018-10-12 10:16:24'
> +++ sed 's/./ /g'
> ++ timestamp_replacementchars=' '
> /usr/share/ngcp-ngcpcfg/functions//main: line 18: HNAME: unbound variable
Let's be polite for such kind of log message.
Change-Id: I3647567aba5fad3a103af2e6da35229f6d932bed
It is necessary to improve navigation into long 'ngcpcfg push all'
output on huge customer's systems. Requested by Sipwise Ops team.
Previously:
2018-10-11 10:48:40: Generating /etc/update-motd.d/05-maintenance-mode: OK
2018-10-11 10:48:40: Executing postbuild for /etc/update-motd.d/05-maintenance-mode
2018-10-11 10:48:40: Generating /etc/update-motd.d/10-heartbeat: OK
2018-10-11 10:48:40: Executing postbuild for /etc/update-motd.d/10-heartbeat
2018-10-11 10:48:40: Generating /etc/update-motd.d/20-sysinfo: OK
Now:
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/redis-server.service: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/rsyslog.service.d/override.conf: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/slapd.service.d/override.conf: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/slapd.service: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/snmpd.service.d/override.conf: OK
Change-Id: I3035293b95f64033005cc5f8d031dbbfac1dcc33
Currently `/etc/debian_version` for Debian/buster systems includes:
buster/sid
So until we get the according version number via base-files, also
check for the codename instead. Make it more generic so it works for
any upcoming release that uses ".*/sid", to not run into the issue
again with e.g. the upcoming Debian/release codenamed bullseye (the
release expected after buster).
Thanks: Manuel Montecelo for improving my initial version
Change-Id: I59e6adea98b6c228daa2b5ebe14872c4a35b8920
It was impossible to set value like '10G' previosuly:
>> ngcpcfg set /etc/ngcp-config/config.yml bootenv.netscript.fallbackfssize=10M
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 28, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 30, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 31, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 33, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 34, near "10M"
> (Missing operator before M?)
> syntax error at /tmp/tmp.px3MpkUFZS line 28, near "10M"
> syntax error at /tmp/tmp.px3MpkUFZS line 30, near "10M"
> syntax error at /tmp/tmp.px3MpkUFZS line 39, near "}"
> Execution of /tmp/tmp.px3MpkUFZS aborted due to compilation errors.
The 'case' statement doesn't support regular expresions in Bash,
let's rewrite them into linear check.
Change-Id: I29ced837fca0abdc3cbe1ea18c614c4c1dd6780c
At the end of installer's 'configuration' section we need
to restart all services were touched during the installation
(otherwise we need to restart the node completely).
The option helps us in this case as etckeeper is
not reliable enough during the installation.
P.S. we pass options from 'apply' to 'services' now,
we should remove 'Unsupported option(s) given: $*' check,
otherwise it is failing as 'apply' requires commit message,
which cause error on 'services' level: 'Unsupported option(s) given'
> 2018-09-13 18:08:33: Error: Unsupported option(s) given: my commit message
> 2018-09-13 18:08:34: Did you mean '--dry-run'?
Change-Id: I5dac44cd87dc1fc90f81e4ac3a14af32c00f825e
We need to have some reproducible order of services restart,
otherwise this might lead to different results, depending on the order.
Also unifyer() looks too heavy here since the files can be easily
unified in a moment of writing them into ${TMPFILE}.
Change-Id: I0a683a7d57cd52b74cba0ee9a285f84f77f3dc20
It was initially introduced in the commit:
> commit 7d422b7878
> Author: Victor Seva <vseva@sipwise.com>
> Date: Fri Aug 22 17:03:39 2014 +0200
>
> MT#8763 check for .git before running the services
>
> On a multi directory config, the services depends on git. Check
> first if the git repository exists and warn about it
The subshell here was necessary to isolate 'cd $dir' call.
In the same time the "proper" bash way is to use pushd/popd.
For the history, 'shellcheck disable' was muting the warning:
> This only exits the subshell caused by the (..) group. [SC2106]
Change-Id: I3546fa9615858dbbc4c0f91d74de003e22837d55
NGCP doesn't use NDB anymore and also it looks like non-safe commit
as the commit has been done by user 'ngcp-config <root@4bd5de92c84d>' :-D
Removing it.
This reverts commit 9f4d913f6b.
Change-Id: Ie91442a1f33e9ddcae5589c095d19c9e46cd68cb
To be able to generate an authorized_keys file from templates we need
to fetch the public ssh keys for the root user. This function will make
this task easier.
Change-Id: If8edef0e77a5f3c1167fe8dfd0e92e4d9e468a6a
Previously 'systemd daemon-reload' and 'systemctl preset-all'
were executed only if changes detected in configs.
Otherwise the early exit happened if no .service files were triggered:
> DEBUG: No services file(s) reported - nothing to do.
Move the code into the function systed_daemon_reload_preset and
execute it on the top of the script. Also print info message
informing users about reloaded systemd and newly preset units.
It should provide better visibility here.
Change-Id: I992af9fb274ea93a37b812a51cebcd7af5c54133
Otherwise the code in between the functions can be easily missed
which cause issues as described in the ticket and will be fixed
in the following commit.
Change-Id: I57decfcbcd41691d35d085b13881e1e6b5208f6e
This partially reverts commit bc8ae9e795.
We should not match li_dist with regexes as then we would trap cases
that we should not be covering by the li_dist mapping logic. Just do
exact matches.
This also fixes the hypothetical case with other regexes that are not
the wrongly specified '*'.
Change-Id: I9d9d850969e03fa9dc587f2d47a688bc0359e63c
Corrupted YML schema is the popular way for time spending on debug
mystery behaviour on NGCP. We have validation schema since mr4.5
and it coverts all YML files nowadays. Let's enable it by default,
since in field testing shows good results.
In case of the inconsistent schema, apply is still possible with
ngcpcfg option '--no-validate':
> root@sp1:~# ngcpcfg --no-validate apply 'some changes with inconsistent schema'
> 2018-08-29 13:12:27: Error: Invalid schema detected for /etc/ngcp-config/config.yml
> /etc/ngcp-config/config.yml#0: INVALID
> - [/apps] Expected required key 'malicious_call'
>
> DANGEROUS ZONE: invalid configs detected, continue anyway due to option '--no-validate'
> Checking state of local storage:
> ...
Change-Id: Ifa51c9e0c2fd396696f73760d89eadcbe9763456
Otherwise it matches on 'li/li_dist' cases and the library get_all_ips
returns no IPs if called like:
> argv.role='*';
> argv.type='*';
> PROCESS '/usr/lib/ngcp-ngcpcfg/get_all_ips';
> all_ips = out;
This is a hotfix commit for backporting into mr6.4.1 while the "proper" fix
should be re-considered and committed for mr6.5+
P.S we have 3 tt2 templates where we have * in argv parameters.
P.P.S. The code is from Andreas, tnx for the fix!
Change-Id: I119a687389075d2ebdcd824ed457f768b5fb2123
This will collect all service actions, synthetize them into their
minimal expression and execute them in a single batch per action.
Change-Id: I950d5db32e0ec6327964faac4ce8f15449f90e90
the following additional fields can be used in
admin_export_fields/reseller_export_fields in config.yml:
- FURNISHED_CHARGE_INFO:
the fci data returned by external lnp requests
- HEADER_*:
the value of a sip header. * is the sip header name,
which is case-sensitive.
Change-Id: Ie0c95d341648fc63fff23ea2d3054b70fa2cf9e9
We need to use rename semantics when moving the built template into the
destination, so that we avoid reacing on ETXTBSY for executable files.
But we need to fallback to use copy semantics, because at least Docker
bind mounts /etc/hosts, which means we cannot rename over it.
We'll use perl's File::Copy which gives us the exact semantics we need.
Ref: https://github.com/moby/moby/issues/22281
Change-Id: I6ae6ce2050050c13c7ec9d08b0e6e01fb2801fd6
We should not just check whether these keys are defined, but whether
they contain actual IPs too.
Fixes: commit 80f70d88b2
Change-Id: I122de32629164cc46537ab0800612c8654f1ba14
If we are using DHCP we do not need either an IPv4 nor an IPv6. If we
have an IPv6 we do not need an IPv4. If we have a netmask, a shared
IP, an advertised IP, we should have the matching IP setting.
Change-Id: I13f190086a0540495b0314e58ca301aae6db0453
This adds an additional virtual li_dist role on top of the existing li
role. The difference is that li is the cluster simpli acting in central
mode, while li_dist is the cluster acting in distributed mode.
Change-Id: I930c93380a0e88649c325b7faf3c0fb86d78e5ff
We need to preserve symlinks, as was the case when we were using cat.
This way we'll not get caught in the ETXTBSY race from the kernel, and
we'll have the old semantics.
Bisected-by: Alex Lutay <alutay@sipwise.com>
Fixes: 7480ebe7c5
Change-Id: I0be7473b271cc3807da957bbef1063018d3b42b2
Otherwise ngcp-installer failed to install PRO/Carrier as
we build the file /etc/ngcp_mgmt_node the very first time on the first MGMT node:
> +08:22:48 cfg_build_templates
> +08:22:48 cfg_build_configs /etc/ngcp_mgmt_node
> +08:22:48 build_configs=($1)
> +08:22:48 declare -a build_configs
> +08:22:48 log_info 'Generating default configuration files /etc/ngcp_mgmt_node'
> +08:22:48 tee -a /tmp/ngcp-installer.log
> +08:22:48 echo 'Generating default configuration files /etc/ngcp_mgmt_node'
> Generating default configuration files /etc/ngcp_mgmt_node
> +08:22:48 ngcpcfg build /etc/ngcp_mgmt_node
> cat: /etc/ngcp_mgmt_node: No such file or directory
> +08:22:48 log_die 'Error running '\''ngcpcfg build'\'''
Checking the file availability is not enough here, the next error is:
> (sp1)root@sp1:/# ngcpcfg build
> /usr/share/ngcp-ngcpcfg/scripts//check: line 83: NGCP_IS_MGMT: unbound variable
> (sp1)root@sp1:/# cat /etc/default/ngcp-roles
> NGCP_TYPE="sppro"
> (sp1)root@sp1:/#
It happens because /etc/default/ngcp-roles is a fake one at this stage.
Handling NGCP_IS_MGMT properly is also not enough:
> (sp1)root@sp1:/# ngcpcfg build
> 2018-07-16 16:49:32: Error: Remote origin of ngcpcfg is '/mnt/glusterfs/ngcpcfg-share', expected: 'sp:/mnt/glusterfs/ngcpcfg-share'.
> 2018-07-16 16:49:32: Error: NOTE: execute `cd /etc/ngcp-config ; git remote set-url origin 'sp:/mnt/glusterfs/ngcpcfg-share'` to adjust setting.
> 2018-07-16 16:49:32: Error: NOTE: perform `ngcpcfg clean --all` to recreate local master branch from remote.
> (sp1)root@sp1:/#
Which happens because the peer is not yet configured at the moment (first node installation).
Introducing new internal option '--no-check-origin' to skip the test into installer.
Change-Id: I0265c65f45972e92ca92320871a7ef29f8904fec
We should be exhaustive here, and any unknown pattern should be caught
and reported, as we might miss information about peers and the cluster
layout otherwise.
Change-Id: I56a9a2b5b6c6f776c0c64fc2f8e01f5f76bb8e10
Files that are being executed cannot be modified in place as that
returns EBUSY. But their dentry can be replaced with a rename(2),
which is atomic and does not touch the original inode. This is the
standard procedure to replace running executables in Unix.
So, we just replace the cat(1) with a mv(1), and stop quiescing its
stderr so that we get proper errors reported.
Change-Id: If15ea1cfa749a6140ff4022200c7fc730c76aa3a
Previously it had the following behavior:
Select all records with 'Host LIKE' filtering so if requested host was '%'
this select returned all the record for this user.
Then user was delete with 'DELETE ... Host LIKE' statement so if
requested host was '%' it deleted all the records.
With using 'DROP USER' statement the same behavior can be achieved with
the same selecting but drop user not with requested host but with actual
one from select statement.
Change-Id: I72f1dd1962e139939be700794e0eb025fe1615b2
There are a lot of flushing privileges so in order not to flood the
console move it to debug level.
Change-Id: I98e3247881393d7892799cc23c2a4e5dc865185a
Replace unsafe 'DELETE FROM mysql.user' with recommended 'DROP USER'
statement to avoid problems with DROP/CREATE USER due to missing
'FLUSH PRIVILEGES' like:
Error: Cannot create grant temp user: Operation CREATE USER failed
for 'ngcp-sync-db'@'localhost' at /usr/sbin/ngcp-sync-grants line 322.
Execute 'FLUSH PRIVILEGES' before creating and after dropping of temp user
'ngcp-sync-db'.
Change-Id: I49c29b6c39353d4a47f086851a915af1469ebcdd
This makes it possible to depend on this new package while not having to
pull the huge amount of dependencies.
Change-Id: I2df3d072ecca0751d4d05d30f5b5c1ac0ec4ed25
This file should be installed only for the ngcp-ngcpcfg package, as
that's the one making sure etckeeper is installed, and we only need
one doing the setup, not all the rdepends too.
Change-Id: Ib20111ada44964a3bdfda4c50a84971cf9678eb9
Alexander Lutay
Michael Prokop
Manuel Montecelo
Guillem Jover
Sipwise Jenkins Builder
Rene Krenn
Mykola Malkov