This partially reverts commit bc8ae9e795.
We should not match li_dist with regexes as then we would trap cases
that we should not be covering by the li_dist mapping logic. Just do
exact matches.
This also fixes the hypothetical case with other regexes that are not
the wrongly specified '*'.
Change-Id: I9d9d850969e03fa9dc587f2d47a688bc0359e63c
Corrupted YML schema is the popular way for time spending on debug
mystery behaviour on NGCP. We have validation schema since mr4.5
and it coverts all YML files nowadays. Let's enable it by default,
since in field testing shows good results.
In case of the inconsistent schema, apply is still possible with
ngcpcfg option '--no-validate':
> root@sp1:~# ngcpcfg --no-validate apply 'some changes with inconsistent schema'
> 2018-08-29 13:12:27: Error: Invalid schema detected for /etc/ngcp-config/config.yml
> /etc/ngcp-config/config.yml#0: INVALID
> - [/apps] Expected required key 'malicious_call'
>
> DANGEROUS ZONE: invalid configs detected, continue anyway due to option '--no-validate'
> Checking state of local storage:
> ...
Change-Id: Ifa51c9e0c2fd396696f73760d89eadcbe9763456
Otherwise it matches on 'li/li_dist' cases and the library get_all_ips
returns no IPs if called like:
> argv.role='*';
> argv.type='*';
> PROCESS '/usr/lib/ngcp-ngcpcfg/get_all_ips';
> all_ips = out;
This is a hotfix commit for backporting into mr6.4.1 while the "proper" fix
should be re-considered and committed for mr6.5+
P.S we have 3 tt2 templates where we have * in argv parameters.
P.P.S. The code is from Andreas, tnx for the fix!
Change-Id: I119a687389075d2ebdcd824ed457f768b5fb2123
This will collect all service actions, synthetize them into their
minimal expression and execute them in a single batch per action.
Change-Id: I950d5db32e0ec6327964faac4ce8f15449f90e90
the following additional fields can be used in
admin_export_fields/reseller_export_fields in config.yml:
- FURNISHED_CHARGE_INFO:
the fci data returned by external lnp requests
- HEADER_*:
the value of a sip header. * is the sip header name,
which is case-sensitive.
Change-Id: Ie0c95d341648fc63fff23ea2d3054b70fa2cf9e9
We need to use rename semantics when moving the built template into the
destination, so that we avoid reacing on ETXTBSY for executable files.
But we need to fallback to use copy semantics, because at least Docker
bind mounts /etc/hosts, which means we cannot rename over it.
We'll use perl's File::Copy which gives us the exact semantics we need.
Ref: https://github.com/moby/moby/issues/22281
Change-Id: I6ae6ce2050050c13c7ec9d08b0e6e01fb2801fd6
We should not just check whether these keys are defined, but whether
they contain actual IPs too.
Fixes: commit 80f70d88b2
Change-Id: I122de32629164cc46537ab0800612c8654f1ba14
If we are using DHCP we do not need either an IPv4 nor an IPv6. If we
have an IPv6 we do not need an IPv4. If we have a netmask, a shared
IP, an advertised IP, we should have the matching IP setting.
Change-Id: I13f190086a0540495b0314e58ca301aae6db0453
This adds an additional virtual li_dist role on top of the existing li
role. The difference is that li is the cluster simpli acting in central
mode, while li_dist is the cluster acting in distributed mode.
Change-Id: I930c93380a0e88649c325b7faf3c0fb86d78e5ff
We need to preserve symlinks, as was the case when we were using cat.
This way we'll not get caught in the ETXTBSY race from the kernel, and
we'll have the old semantics.
Bisected-by: Alex Lutay <alutay@sipwise.com>
Fixes: 7480ebe7c5
Change-Id: I0be7473b271cc3807da957bbef1063018d3b42b2
Otherwise ngcp-installer failed to install PRO/Carrier as
we build the file /etc/ngcp_mgmt_node the very first time on the first MGMT node:
> +08:22:48 cfg_build_templates
> +08:22:48 cfg_build_configs /etc/ngcp_mgmt_node
> +08:22:48 build_configs=($1)
> +08:22:48 declare -a build_configs
> +08:22:48 log_info 'Generating default configuration files /etc/ngcp_mgmt_node'
> +08:22:48 tee -a /tmp/ngcp-installer.log
> +08:22:48 echo 'Generating default configuration files /etc/ngcp_mgmt_node'
> Generating default configuration files /etc/ngcp_mgmt_node
> +08:22:48 ngcpcfg build /etc/ngcp_mgmt_node
> cat: /etc/ngcp_mgmt_node: No such file or directory
> +08:22:48 log_die 'Error running '\''ngcpcfg build'\'''
Checking the file availability is not enough here, the next error is:
> (sp1)root@sp1:/# ngcpcfg build
> /usr/share/ngcp-ngcpcfg/scripts//check: line 83: NGCP_IS_MGMT: unbound variable
> (sp1)root@sp1:/# cat /etc/default/ngcp-roles
> NGCP_TYPE="sppro"
> (sp1)root@sp1:/#
It happens because /etc/default/ngcp-roles is a fake one at this stage.
Handling NGCP_IS_MGMT properly is also not enough:
> (sp1)root@sp1:/# ngcpcfg build
> 2018-07-16 16:49:32: Error: Remote origin of ngcpcfg is '/mnt/glusterfs/ngcpcfg-share', expected: 'sp:/mnt/glusterfs/ngcpcfg-share'.
> 2018-07-16 16:49:32: Error: NOTE: execute `cd /etc/ngcp-config ; git remote set-url origin 'sp:/mnt/glusterfs/ngcpcfg-share'` to adjust setting.
> 2018-07-16 16:49:32: Error: NOTE: perform `ngcpcfg clean --all` to recreate local master branch from remote.
> (sp1)root@sp1:/#
Which happens because the peer is not yet configured at the moment (first node installation).
Introducing new internal option '--no-check-origin' to skip the test into installer.
Change-Id: I0265c65f45972e92ca92320871a7ef29f8904fec
We should be exhaustive here, and any unknown pattern should be caught
and reported, as we might miss information about peers and the cluster
layout otherwise.
Change-Id: I56a9a2b5b6c6f776c0c64fc2f8e01f5f76bb8e10
Files that are being executed cannot be modified in place as that
returns EBUSY. But their dentry can be replaced with a rename(2),
which is atomic and does not touch the original inode. This is the
standard procedure to replace running executables in Unix.
So, we just replace the cat(1) with a mv(1), and stop quiescing its
stderr so that we get proper errors reported.
Change-Id: If15ea1cfa749a6140ff4022200c7fc730c76aa3a
Previously it had the following behavior:
Select all records with 'Host LIKE' filtering so if requested host was '%'
this select returned all the record for this user.
Then user was delete with 'DELETE ... Host LIKE' statement so if
requested host was '%' it deleted all the records.
With using 'DROP USER' statement the same behavior can be achieved with
the same selecting but drop user not with requested host but with actual
one from select statement.
Change-Id: I72f1dd1962e139939be700794e0eb025fe1615b2
There are a lot of flushing privileges so in order not to flood the
console move it to debug level.
Change-Id: I98e3247881393d7892799cc23c2a4e5dc865185a
Replace unsafe 'DELETE FROM mysql.user' with recommended 'DROP USER'
statement to avoid problems with DROP/CREATE USER due to missing
'FLUSH PRIVILEGES' like:
Error: Cannot create grant temp user: Operation CREATE USER failed
for 'ngcp-sync-db'@'localhost' at /usr/sbin/ngcp-sync-grants line 322.
Execute 'FLUSH PRIVILEGES' before creating and after dropping of temp user
'ngcp-sync-db'.
Change-Id: I49c29b6c39353d4a47f086851a915af1469ebcdd
This makes it possible to depend on this new package while not having to
pull the huge amount of dependencies.
Change-Id: I2df3d072ecca0751d4d05d30f5b5c1ac0ec4ed25
This file should be installed only for the ngcp-ngcpcfg package, as
that's the one making sure etckeeper is installed, and we only need
one doing the setup, not all the rdepends too.
Change-Id: Ib20111ada44964a3bdfda4c50a84971cf9678eb9
The Template class accepts both a hash reference and a hash to get the
configuration arguments. And the Dancer framework passes those as a
hash.
Change-Id: Ic9ae758bd81a2bb3108900e6f92ad91ad88f3cc9
Move the ngcp object variable into NGCP::Template::Object, and make
NGCP::Template just a customized variant of Template that we can use
instead.
This way we hide all the internal details of how to set up the
environment we need to process NGCP templates.
Change-Id: I690cf1a74551f4751380a506ddcc047b0942ba21
This is another good candidate to replace with a proper ngcp method, as
the function is already implemented as a PERL block, and the most used
"function".
Change-Id: I49dc638a197a0215ea65c0a60c82c6c556aca3fa
This replaces the old has_role Template "function" with a proper method
of the ngcp object, anywhere we need to check for the host role,
including other macros that could not use our previous "function"
before. Which will make it possible to modify the has_role behavior in
a single place, and affect all code paths.
Change-Id: I73526aa77fab3dec8f33361e142164204d7c1481
* ngcp.timezone table is not updated if
new timezone=old timezone
* ngcp.timezone change is not replicated to
preserve stability of the other (active) node
and to address the replication issue during upgrade
Change-Id: Id8f7b291c188792a33093ac3ed706b55d1b0a654
The perl Template::Toolkit is very rich, but its "function" support is a
bit poor. The ways to do it are either via MACRO directives, or by
simulating them with one function per file and then using PROCESS on
these. The problem is that this is very clunky, does not support
nesting, as we'd need different "argument" names for each "function",
and it's quite cumbersome to use, need to assign aguments passed
beforehand, and then assign back a designated return value from another
variable. This is also one of the reasons some of the functions are not
encapsulated, and have been inlined in various loops, because it was not
possible to cleanly PROCESS them from those call sites.
Instead we should use its native support for perl objects and perl
subroutines, which exposes these as proper methods of a designated
variable, and have none of the above mentioned problems. So we'll switch
from constructs such as:
argv.arg-a = variable;
argv.arg-b = 'value';
PROCESS 'path-to-library-dir/function'
result = out
into:
result = ngcp.function(variable, 'value');
In addition this might actually be faster, as it does not require
processing additional files, and it's all just native perl code.
This will be exposed within the NGCP templates as the ngcp object, and
new member functions will start replacing our old and clunky native
Template PROCESS-style library.
Change-Id: Id2f0d181c695a9dd074646881b7d9de3478570af
In this way, if the file fails to be generated (e.g. because of a broken .yml
file) the old file is preserved intact.
It doesn't protect from all failures (in particular, it is ineffective against
wrong content due to programming errors), but it's safer than having an empty
file or truncated as a result of the parsing/generation being stopped due to
incorrect files or external events.
This is added in the context of creating new /etc/hosts and
/etc/network/interfaces files because we found the problem at this point, and
having those files empty completely breaks the system.
Change-Id: I01e8ef9d6971506aa3f8d9da174ce890ea83393c
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
If the output file has been replaced by a symlink then the
permissions of the symlink's target are adjusted. This is clearly
unwanted and unexpected behavior.
This can happen e.g. when masking a service via systemd's systemctl,
demonstration:
| sipwise@sp1:~$ sudo rm /etc/systemd/system/ntp.service
| sipwise@sp1:~$ sudo systemctl mask ntp
| Created symlink /etc/systemd/system/ntp.service → /dev/null.
| sipwise@sp1:~$ ls -la /dev/null
| crw-rw-rw- 1 root root 1, 3 May 12 01:54 /dev/null
| sipwise@sp1:~$ sudo ngcpcfg build /etc/systemd/system/ntp.service
| 2018-05-15 16:06:47: Checking state of local storage:
| 2018-05-15 16:06:47: OK: nothing to pull
| 2018-05-15 16:06:47: configs were checked successfuly
| 2018-05-15 16:06:47: No patchtt files found, nothing to patch.
| Loading /etc/ngcp-config/config.yml in memory: OK
| Loading /etc/ngcp-config/network.yml in memory: OK
| Loading /etc/ngcp-config/constants.yml in memory: OK
| 2018-05-15 16:06:48: Generating /etc/systemd/system/ntp.service: OK
| sipwise@sp1:~$ ls -la /dev/null
| cr--r--r-- 1 root root 1, 3 May 12 01:54 /dev/null
Instead check whether the expected output file is a symlink and
if so don't modify permissions at all, but warn instead.
Change-Id: I1c1bb0941a8a2f599652b7c089efd63ff33ca455
Guillem Jover
Alexander Lutay
Rene Krenn
Sipwise Jenkins Builder
Michael Prokop
Mykola Malkov
Kirill Solomko
Manuel Montecelo
Richard Fuchs