In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
We are calling slow Perl command to move files to workaround Docker problem
and losing 25% of performance on every NGCP installation in production.
Let's call slow workaround when it is necessary only (in Docker).
Command: time ngcpcfg build
Original results: real 0m19.976s
Current results: real 0m15.286s
Change-Id: Iac787c0fcdb4182d71b6305ebbba5ea3193e76c8
About 50% of previous helper/build_config execution was loading 'functions/main'
and we were performing it again and again for every tt2 files we build
and we have more then hundred files in ngcpcfg.
In fact we need two env variables and logging functions.
Let's extract them into the separate function file and load them only.
Command: time ngcpcfg build
Original results: real 0m27.766s
Current results: real 0m19.976s
Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
The dpkg-query command already has an interface to fetch the field
value, so there is no need to use --status, grep and then parse the
output.
Change-Id: I13a2f601bde4ca01bfbd7db35f8a1ff11175fba3
Due to the code in tt2-wrapper, the only errors reported were failures to
connect to the socket to use the back-end performing the parsing. But there was
no attempt to detect errors in the parsing at all.
Since we are only able to use the output stream and not for example exit codes
of the parser at the back-end (libtemplate-perl), we can only rely on finding
some kind of error string, which can change in future versions of
libtemplate-perl but hopefully not very often, and at most it should only change
when we move base to newer Debian releases.
In the case of the original problem reported we could rely on "parse error", but
since there are two such strings ("file error" and "parse error") the first is
more general and also covers cases like "permission denied to read file" or
other errors that might happen. So the implementation of error checking is
basically grepping for "^file error", the string at the beginning of a line, to
limit possible false positives.
With the new checks and an example of an invalid syntax already in the first
line, "ngcpcfg build" detects the error correctly and stops processing, and
shows information about the source of the problem, e.g.:
root@spce:~# ngcpcfg build /etc/test
2018-10-29 11:37:01 spce: yml configs were validated successfuly
2018-10-29 11:37:01 spce: configs were checked successfuly
2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch.
Loading /etc/ngcp-config/config.yml in memory: OK
Loading /etc/ngcp-config/network.yml in memory: OK
Loading /etc/ngcp-config/constants.yml in memory: OK
2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED
2018-10-29 11:37:02 spce: Error: from generated file:
2018-10-29 11:37:02 spce: Error: file error - parse error - input file handle line 1: unexpected end of directive
2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding:
2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2
2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml
2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file>
2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G
2018-10-29 11:37:02 spce: should provide more details.
Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13
When the "mktemp" command is invoked with a template parameter without path
prefixes or without using "--tmpdir", it does not use "/tmp" by default, but
instead it uses the current working directory as the place to create these
temporary files.
This was an oversight based on our side on the unknown behaviour, which was
discovered when working on this task, because we do want to have "/tmp" as the
home for these temporary files.
Change-Id: I1b03dc90cfd8dd9364c8bc3b2bfe0e3c26979e7f
We need to use rename semantics when moving the built template into the
destination, so that we avoid reacing on ETXTBSY for executable files.
But we need to fallback to use copy semantics, because at least Docker
bind mounts /etc/hosts, which means we cannot rename over it.
We'll use perl's File::Copy which gives us the exact semantics we need.
Ref: https://github.com/moby/moby/issues/22281
Change-Id: I6ae6ce2050050c13c7ec9d08b0e6e01fb2801fd6
We need to preserve symlinks, as was the case when we were using cat.
This way we'll not get caught in the ETXTBSY race from the kernel, and
we'll have the old semantics.
Bisected-by: Alex Lutay <alutay@sipwise.com>
Fixes: 7480ebe7c5
Change-Id: I0be7473b271cc3807da957bbef1063018d3b42b2
Files that are being executed cannot be modified in place as that
returns EBUSY. But their dentry can be replaced with a rename(2),
which is atomic and does not touch the original inode. This is the
standard procedure to replace running executables in Unix.
So, we just replace the cat(1) with a mv(1), and stop quiescing its
stderr so that we get proper errors reported.
Change-Id: If15ea1cfa749a6140ff4022200c7fc730c76aa3a
Move the ngcp object variable into NGCP::Template::Object, and make
NGCP::Template just a customized variant of Template that we can use
instead.
This way we hide all the internal details of how to set up the
environment we need to process NGCP templates.
Change-Id: I690cf1a74551f4751380a506ddcc047b0942ba21
* ngcp.timezone table is not updated if
new timezone=old timezone
* ngcp.timezone change is not replicated to
preserve stability of the other (active) node
and to address the replication issue during upgrade
Change-Id: Id8f7b291c188792a33093ac3ed706b55d1b0a654
The perl Template::Toolkit is very rich, but its "function" support is a
bit poor. The ways to do it are either via MACRO directives, or by
simulating them with one function per file and then using PROCESS on
these. The problem is that this is very clunky, does not support
nesting, as we'd need different "argument" names for each "function",
and it's quite cumbersome to use, need to assign aguments passed
beforehand, and then assign back a designated return value from another
variable. This is also one of the reasons some of the functions are not
encapsulated, and have been inlined in various loops, because it was not
possible to cleanly PROCESS them from those call sites.
Instead we should use its native support for perl objects and perl
subroutines, which exposes these as proper methods of a designated
variable, and have none of the above mentioned problems. So we'll switch
from constructs such as:
argv.arg-a = variable;
argv.arg-b = 'value';
PROCESS 'path-to-library-dir/function'
result = out
into:
result = ngcp.function(variable, 'value');
In addition this might actually be faster, as it does not require
processing additional files, and it's all just native perl code.
This will be exposed within the NGCP templates as the ngcp object, and
new member functions will start replacing our old and clunky native
Template PROCESS-style library.
Change-Id: Id2f0d181c695a9dd074646881b7d9de3478570af
In this way, if the file fails to be generated (e.g. because of a broken .yml
file) the old file is preserved intact.
It doesn't protect from all failures (in particular, it is ineffective against
wrong content due to programming errors), but it's safer than having an empty
file or truncated as a result of the parsing/generation being stopped due to
incorrect files or external events.
This is added in the context of creating new /etc/hosts and
/etc/network/interfaces files because we found the problem at this point, and
having those files empty completely breaks the system.
Change-Id: I01e8ef9d6971506aa3f8d9da174ce890ea83393c
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
If the output file has been replaced by a symlink then the
permissions of the symlink's target are adjusted. This is clearly
unwanted and unexpected behavior.
This can happen e.g. when masking a service via systemd's systemctl,
demonstration:
| sipwise@sp1:~$ sudo rm /etc/systemd/system/ntp.service
| sipwise@sp1:~$ sudo systemctl mask ntp
| Created symlink /etc/systemd/system/ntp.service → /dev/null.
| sipwise@sp1:~$ ls -la /dev/null
| crw-rw-rw- 1 root root 1, 3 May 12 01:54 /dev/null
| sipwise@sp1:~$ sudo ngcpcfg build /etc/systemd/system/ntp.service
| 2018-05-15 16:06:47: Checking state of local storage:
| 2018-05-15 16:06:47: OK: nothing to pull
| 2018-05-15 16:06:47: configs were checked successfuly
| 2018-05-15 16:06:47: No patchtt files found, nothing to patch.
| Loading /etc/ngcp-config/config.yml in memory: OK
| Loading /etc/ngcp-config/network.yml in memory: OK
| Loading /etc/ngcp-config/constants.yml in memory: OK
| 2018-05-15 16:06:48: Generating /etc/systemd/system/ntp.service: OK
| sipwise@sp1:~$ ls -la /dev/null
| cr--r--r-- 1 root root 1, 3 May 12 01:54 /dev/null
Instead check whether the expected output file is a symlink and
if so don't modify permissions at all, but warn instead.
Change-Id: I1c1bb0941a8a2f599652b7c089efd63ff33ca455
* sync_smsc_peers() is used to sync the 'smsc_peer' domain
preference
* generic_enum_sync() is extended and accepts boolean args
$usr_pref, $dom_pref, $peer_pref
Change-Id: I376a2ba823bef53b3548b88c38960c893a74efc8
On migration to systemd we have separated mysqld1 and mysqld2 by
separate my.cnf files. We have to use the accordingly here.
Change-Id: I6d291e3b7c9601c09b82103cd32a775ba44911ef
* sync_db_timezones use 'mysql_tzinfo_to_sql'
to load timezone info from /usr/share/zoneinfo into
MariaDB.
* tzdata package version is checked and the timezone data
sync is skipped if the version in ngcp.tzinfo_version
is already up to date.
Change-Id: I92c87fb52fea20df0366c93c2e3568c25833b9bb
* general.timezone value from config.yml
is now populated into ngcp.timezone DB table.column
* it produces an error if general.timezone field is undefined
Change-Id: I975f786a7f8ff42916d92e13f6fe9971d5b1e2f0
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ie51f1c79859d40ef0877fc0ab75f86ee72e14ea4
This module is more compliant and it is faster than the pure perl
implementation. The latter is also deprecated in favor of the former
(see man YAML for more details).
Change-Id: I3fccca4ab57ad7c316b6cf58a81bc4baa1bdabe1
If a template named
/etc/ngcp-config/templates/etc/apt/apt.conf.d/71_no_recommended.tt2 exists
the resulting destination file would be
/etc/apt/apt.conf.d/71_no_recommended. Though if
/etc/apt/apt.conf.d/71_no_recommended exists as directory ngcpcfg should
abort and inform the user about the existing situation with a useful error
message.
Change-Id: I2c6e1e3a4ec485183674c1fe72251631ad9867ac
This let's us control the directory where files should be written to.
Otherwise we'd have to write to the system's /etc/ + /var, which is
clearly unwanted.
Change-Id: I1010afcd4637928868fc81a0adfafebd3bbce874
If we have multiple web0X proxies but need specific
templates on just one of those pairs then we need to
support *.customtt.tt2.$pairname + *.tt2.$pairname
templates.
While at it also support *.customtt.tt2.$nodename +
foo.tt2.$nodename (where nodename is actual hostname
and not sp1 or sp2 as already supported already).
Test script for demonstration (especially for precedence):
mkdir /etc/ngcp-config/templates/etc/mika
cd /etc/ngcp-config/templates/etc/mika
hostname=$(hostname) # e.g. web01b
pairname=${hostname%[ab]} # e.g. web01
nodename=$(cat /etc/ngcp_nodename) # e.g. sp2
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do echo $file > $file ; done
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do
ngcpcfg build /etc/mika/ >/dev/null
if [ "$(cat /etc/mika/foo)" = "$file" ] ; then
echo "Content of file /etc/mika/foo matching $file"
rm "$file" ; rm /etc/mika/foo
else
echo "Problem with precedence of file $file"
return 1
fi
done
Change-Id: If7c222374bb56d7bf243200b37a414e39e050cd4
During the installation we need an ability to fix perms the first time
on PRO sp2 installation. The file /etc/ngcp-config/config.yml is not
available at the moment causing function "main" failed on load:
> /usr/share/ngcp-ngcpcfg/helper/restore-permissions /tmp/tmp.Hrt0sBRYy3
> Error: Configuration file /etc/ngcp-config/config.yml does not exist (unconfigured?) - exiting.
Change-Id: Ieb196690246d7c1ea59e7f4359d72179472c278b
Git doesn't track file permissions (except for the executable
flag). For sensitive data (like the 'ssl' directory and file
'constants.yml' with passwords included) we've to prevent
non-root users from accessing those files.
hooks/pre-commit is inspired and based on the implementation
as present in etckeeper (and luckily we're license compatible)
and takes care of storing the file permissions inside file
/etc/ngcp-config/.ngcpcfg_perms.
The restore-permissions helper script takes care of restoring the
permissions after cloning the ngcpcfg repository via ngcpcfg
itself (being actions decrypt, pull (PRO-only) + initialise
(PRO-only)). It can be executed manually as well via
`usr/share/ngcp-ngcpcfg/helper/restore-permissions /etc/ngcp-config/`
(or wherever the according ngcpcfg repository is placed at).
Regarding the commit integration: git(1) itself doesn't track
file permissions, so we can't detect changes to file permissions
using git itself. Our new pre-commit hook records file
permissions via the .ngcpcfg_perms file. Now by just invoking it
during 'ngcpcfg commit' time we can ensure that even if there
have been any file permission changes in the working directory
the file .ngcpcfg_perms is then up2date and committed.
JFTR: The solution via the git pre-commit hook ensures that no
matter whether you're using 'ngcpcfg commit …' or 'git commit …'
you always get the file permissions handled via .ngcpcfg_perms.
Now if you want to change file permissions in a clean working
directory and commit *without* using 'ngcpcfg commit' but
directly via git itself then you've to use 'git commit
--allow-empty ...' and thanks to the pre-commit hook the file
.ngcpcfg_perms will still be up2date.
Change-Id: I84d608585c626b52112ff649893e232e441c59d8
* fax gateways are no longer needed in kamailio.dispatcher
as asterisk is the only destination for fax2mail now
* error messages are prefixed with Error:
Change-Id: I865fda3ed0aab967c62cea519f3ad42716ed5878
The default Hash::Merge->merge() behaviour is LEFT_PRECEDENT:
http://search.cpan.org/~rehsack/Hash-Merge-0.200/lib/Hash/Merge.pm
Which means:
> The values buried in the left hash will never be lost;
> any values that can be added from the right hash will be attempted.
So, the values from HOST_CONFIG never overwrites default NGCPCTL_CONFIG:
> NGCPCTL_CONFIG="${NGCPCTL_MAIN}/config.yml"
> HOST_CONFIG="${NGCPCTL_MAIN}/config.$(hostname).yml"
We need to use RIGHT_PRECEDENT order to be in sync with 'scripts/build':
> ${NGCPCTL_CONFIG:-} ${HOST_CONFIG:-} ${LOCAL_CONFIG:-} ${NETWORK_CONFIG:-} ${EXTRA_CONFIG_FILES:-} ${CONSTANTS_CONFIG:-}
Change-Id: I597bb082a1791c1f06072d85c4a26fbb8b8320ce
- added quiet option to disable loading output at tt2-daemon
- detect arrays and output values joined by space
Change-Id: I0b7694a366a284ca2b24ea2df55cb285b598e66d
We will use first free port instead which will be stored into
/var/run/ngcpcfg.port file while tt2-daemon is up and running.
Change-Id: I1a918f865d87863eef7d34be8abd1032282220b8
Sometimes it contains very useful information, like:
> /usr/share/ngcp-ngcpcfg/helper//validate-yml /etc/ngcp-config/network.yml
>> YAML::Tiny failed to classify line ' hwaddr:5C:F3:FC:11:67:26 '
>> Can't call method "write_string" without a package or object
>> reference at /usr/share/ngcp-ngcpcfg/helper//validate-yml line 15.
Change-Id: I762b4dc1b0e6b3446f6a431b038fa6ed674e5516
Alexander Lutay
Guillem Jover
Manuel Montecelo
Kirill Solomko
Michael Prokop
Richard Fuchs
Victor Seva