This makes sure we sort each array, instead of sorting the second, and
then letting perl merge the first aray with the result of the second
sort, and then sorting them all as a single array.
Change-Id: I7cd22bf479fee79c104627bf652f0ee674117eec
Fixes: commit 011ce33f32
The current shell code is very inefficient, as it uses files to track
the file lists, and grep to traverse them NxM times per N files.
Change-Id: I8396b4ee232157cea8ef599cba0618be3dd93536
Switch away from the old daemon/wrapper architecture, which complicates
things and requires more scaffolding code.
We turn the daemon/wrapper and build_config into a single perl process
that will load all YAML files, and then process each input/output pair
on a parallel child, to try to speed up the processing as much as
possible.
For whole rebuilds, it might speed up the generation by at least a
factor of x2, x4 or more, depending on the number of active processors.
Change-Id: I51aa2f90336e34a20983d8733f45b64d9b6fea0b
* add support for MariaDB timezones sync sql
delimiter as now in the generated sql stream output
there are sections with '\d' changing the delimiter behaviour.
Change-Id: If226db7018f431ff4248dcf6ff5a614ff96a45c7
The DBI connect call can be specified the PrintError option, which will
make the function print the contents of the $DBI:errstr, but then we do
that again, which results in redudant output, so we disable this option.
The $DBI::errstr variable contains an error string that is *not*
terminated with a newline. The perl die built-in has magic behavior
when the string it gets does not end in a newline character, it will
then emit the source line, file name and a newline where the error
happened, which is in most cases not what we want. We just avoid
this by adding an explicit newline.
Replace a print + exit with a die, and a printf without a terminating
newline which would mess up further output with a warn call with an
explicit newline, which will also make it go to stderr.
And remove the local PrintError overrides which are now unnecessary.
Change-Id: I348b8cc676da7f255c6c823de2a774694dbca826
There is no need to read the text file and search for password inside,
we have properly formatted credentials file /etc/mysql/sipwise_extra.cnf
which is fully supported by MySQL/MariaDB client library and Perl DBI.
Also we have plans to remove /etc/mysql/sipwise.cnf to prevent it usage
(to prevent password leakage through Linux process list).
Change-Id: I62006ae01d9097642a3dae063781a621f5050842
In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
We are calling slow Perl command to move files to workaround Docker problem
and losing 25% of performance on every NGCP installation in production.
Let's call slow workaround when it is necessary only (in Docker).
Command: time ngcpcfg build
Original results: real 0m19.976s
Current results: real 0m15.286s
Change-Id: Iac787c0fcdb4182d71b6305ebbba5ea3193e76c8
About 50% of previous helper/build_config execution was loading 'functions/main'
and we were performing it again and again for every tt2 files we build
and we have more then hundred files in ngcpcfg.
In fact we need two env variables and logging functions.
Let's extract them into the separate function file and load them only.
Command: time ngcpcfg build
Original results: real 0m27.766s
Current results: real 0m19.976s
Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
The dpkg-query command already has an interface to fetch the field
value, so there is no need to use --status, grep and then parse the
output.
Change-Id: I13a2f601bde4ca01bfbd7db35f8a1ff11175fba3
Due to the code in tt2-wrapper, the only errors reported were failures to
connect to the socket to use the back-end performing the parsing. But there was
no attempt to detect errors in the parsing at all.
Since we are only able to use the output stream and not for example exit codes
of the parser at the back-end (libtemplate-perl), we can only rely on finding
some kind of error string, which can change in future versions of
libtemplate-perl but hopefully not very often, and at most it should only change
when we move base to newer Debian releases.
In the case of the original problem reported we could rely on "parse error", but
since there are two such strings ("file error" and "parse error") the first is
more general and also covers cases like "permission denied to read file" or
other errors that might happen. So the implementation of error checking is
basically grepping for "^file error", the string at the beginning of a line, to
limit possible false positives.
With the new checks and an example of an invalid syntax already in the first
line, "ngcpcfg build" detects the error correctly and stops processing, and
shows information about the source of the problem, e.g.:
root@spce:~# ngcpcfg build /etc/test
2018-10-29 11:37:01 spce: yml configs were validated successfuly
2018-10-29 11:37:01 spce: configs were checked successfuly
2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch.
Loading /etc/ngcp-config/config.yml in memory: OK
Loading /etc/ngcp-config/network.yml in memory: OK
Loading /etc/ngcp-config/constants.yml in memory: OK
2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED
2018-10-29 11:37:02 spce: Error: from generated file:
2018-10-29 11:37:02 spce: Error: file error - parse error - input file handle line 1: unexpected end of directive
2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding:
2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2
2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml
2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file>
2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G
2018-10-29 11:37:02 spce: should provide more details.
Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13
When the "mktemp" command is invoked with a template parameter without path
prefixes or without using "--tmpdir", it does not use "/tmp" by default, but
instead it uses the current working directory as the place to create these
temporary files.
This was an oversight based on our side on the unknown behaviour, which was
discovered when working on this task, because we do want to have "/tmp" as the
home for these temporary files.
Change-Id: I1b03dc90cfd8dd9364c8bc3b2bfe0e3c26979e7f
We need to use rename semantics when moving the built template into the
destination, so that we avoid reacing on ETXTBSY for executable files.
But we need to fallback to use copy semantics, because at least Docker
bind mounts /etc/hosts, which means we cannot rename over it.
We'll use perl's File::Copy which gives us the exact semantics we need.
Ref: https://github.com/moby/moby/issues/22281
Change-Id: I6ae6ce2050050c13c7ec9d08b0e6e01fb2801fd6
We need to preserve symlinks, as was the case when we were using cat.
This way we'll not get caught in the ETXTBSY race from the kernel, and
we'll have the old semantics.
Bisected-by: Alex Lutay <alutay@sipwise.com>
Fixes: 7480ebe7c5
Change-Id: I0be7473b271cc3807da957bbef1063018d3b42b2
Files that are being executed cannot be modified in place as that
returns EBUSY. But their dentry can be replaced with a rename(2),
which is atomic and does not touch the original inode. This is the
standard procedure to replace running executables in Unix.
So, we just replace the cat(1) with a mv(1), and stop quiescing its
stderr so that we get proper errors reported.
Change-Id: If15ea1cfa749a6140ff4022200c7fc730c76aa3a
Move the ngcp object variable into NGCP::Template::Object, and make
NGCP::Template just a customized variant of Template that we can use
instead.
This way we hide all the internal details of how to set up the
environment we need to process NGCP templates.
Change-Id: I690cf1a74551f4751380a506ddcc047b0942ba21
* ngcp.timezone table is not updated if
new timezone=old timezone
* ngcp.timezone change is not replicated to
preserve stability of the other (active) node
and to address the replication issue during upgrade
Change-Id: Id8f7b291c188792a33093ac3ed706b55d1b0a654
The perl Template::Toolkit is very rich, but its "function" support is a
bit poor. The ways to do it are either via MACRO directives, or by
simulating them with one function per file and then using PROCESS on
these. The problem is that this is very clunky, does not support
nesting, as we'd need different "argument" names for each "function",
and it's quite cumbersome to use, need to assign aguments passed
beforehand, and then assign back a designated return value from another
variable. This is also one of the reasons some of the functions are not
encapsulated, and have been inlined in various loops, because it was not
possible to cleanly PROCESS them from those call sites.
Instead we should use its native support for perl objects and perl
subroutines, which exposes these as proper methods of a designated
variable, and have none of the above mentioned problems. So we'll switch
from constructs such as:
argv.arg-a = variable;
argv.arg-b = 'value';
PROCESS 'path-to-library-dir/function'
result = out
into:
result = ngcp.function(variable, 'value');
In addition this might actually be faster, as it does not require
processing additional files, and it's all just native perl code.
This will be exposed within the NGCP templates as the ngcp object, and
new member functions will start replacing our old and clunky native
Template PROCESS-style library.
Change-Id: Id2f0d181c695a9dd074646881b7d9de3478570af
In this way, if the file fails to be generated (e.g. because of a broken .yml
file) the old file is preserved intact.
It doesn't protect from all failures (in particular, it is ineffective against
wrong content due to programming errors), but it's safer than having an empty
file or truncated as a result of the parsing/generation being stopped due to
incorrect files or external events.
This is added in the context of creating new /etc/hosts and
/etc/network/interfaces files because we found the problem at this point, and
having those files empty completely breaks the system.
Change-Id: I01e8ef9d6971506aa3f8d9da174ce890ea83393c
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
If the output file has been replaced by a symlink then the
permissions of the symlink's target are adjusted. This is clearly
unwanted and unexpected behavior.
This can happen e.g. when masking a service via systemd's systemctl,
demonstration:
| sipwise@sp1:~$ sudo rm /etc/systemd/system/ntp.service
| sipwise@sp1:~$ sudo systemctl mask ntp
| Created symlink /etc/systemd/system/ntp.service → /dev/null.
| sipwise@sp1:~$ ls -la /dev/null
| crw-rw-rw- 1 root root 1, 3 May 12 01:54 /dev/null
| sipwise@sp1:~$ sudo ngcpcfg build /etc/systemd/system/ntp.service
| 2018-05-15 16:06:47: Checking state of local storage:
| 2018-05-15 16:06:47: OK: nothing to pull
| 2018-05-15 16:06:47: configs were checked successfuly
| 2018-05-15 16:06:47: No patchtt files found, nothing to patch.
| Loading /etc/ngcp-config/config.yml in memory: OK
| Loading /etc/ngcp-config/network.yml in memory: OK
| Loading /etc/ngcp-config/constants.yml in memory: OK
| 2018-05-15 16:06:48: Generating /etc/systemd/system/ntp.service: OK
| sipwise@sp1:~$ ls -la /dev/null
| cr--r--r-- 1 root root 1, 3 May 12 01:54 /dev/null
Instead check whether the expected output file is a symlink and
if so don't modify permissions at all, but warn instead.
Change-Id: I1c1bb0941a8a2f599652b7c089efd63ff33ca455
* sync_smsc_peers() is used to sync the 'smsc_peer' domain
preference
* generic_enum_sync() is extended and accepts boolean args
$usr_pref, $dom_pref, $peer_pref
Change-Id: I376a2ba823bef53b3548b88c38960c893a74efc8
On migration to systemd we have separated mysqld1 and mysqld2 by
separate my.cnf files. We have to use the accordingly here.
Change-Id: I6d291e3b7c9601c09b82103cd32a775ba44911ef
* sync_db_timezones use 'mysql_tzinfo_to_sql'
to load timezone info from /usr/share/zoneinfo into
MariaDB.
* tzdata package version is checked and the timezone data
sync is skipped if the version in ngcp.tzinfo_version
is already up to date.
Change-Id: I92c87fb52fea20df0366c93c2e3568c25833b9bb
* general.timezone value from config.yml
is now populated into ngcp.timezone DB table.column
* it produces an error if general.timezone field is undefined
Change-Id: I975f786a7f8ff42916d92e13f6fe9971d5b1e2f0
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ie51f1c79859d40ef0877fc0ab75f86ee72e14ea4
This module is more compliant and it is faster than the pure perl
implementation. The latter is also deprecated in favor of the former
(see man YAML for more details).
Change-Id: I3fccca4ab57ad7c316b6cf58a81bc4baa1bdabe1
If a template named
/etc/ngcp-config/templates/etc/apt/apt.conf.d/71_no_recommended.tt2 exists
the resulting destination file would be
/etc/apt/apt.conf.d/71_no_recommended. Though if
/etc/apt/apt.conf.d/71_no_recommended exists as directory ngcpcfg should
abort and inform the user about the existing situation with a useful error
message.
Change-Id: I2c6e1e3a4ec485183674c1fe72251631ad9867ac
This let's us control the directory where files should be written to.
Otherwise we'd have to write to the system's /etc/ + /var, which is
clearly unwanted.
Change-Id: I1010afcd4637928868fc81a0adfafebd3bbce874
If we have multiple web0X proxies but need specific
templates on just one of those pairs then we need to
support *.customtt.tt2.$pairname + *.tt2.$pairname
templates.
While at it also support *.customtt.tt2.$nodename +
foo.tt2.$nodename (where nodename is actual hostname
and not sp1 or sp2 as already supported already).
Test script for demonstration (especially for precedence):
mkdir /etc/ngcp-config/templates/etc/mika
cd /etc/ngcp-config/templates/etc/mika
hostname=$(hostname) # e.g. web01b
pairname=${hostname%[ab]} # e.g. web01
nodename=$(cat /etc/ngcp_nodename) # e.g. sp2
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do echo $file > $file ; done
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do
ngcpcfg build /etc/mika/ >/dev/null
if [ "$(cat /etc/mika/foo)" = "$file" ] ; then
echo "Content of file /etc/mika/foo matching $file"
rm "$file" ; rm /etc/mika/foo
else
echo "Problem with precedence of file $file"
return 1
fi
done
Change-Id: If7c222374bb56d7bf243200b37a414e39e050cd4
During the installation we need an ability to fix perms the first time
on PRO sp2 installation. The file /etc/ngcp-config/config.yml is not
available at the moment causing function "main" failed on load:
> /usr/share/ngcp-ngcpcfg/helper/restore-permissions /tmp/tmp.Hrt0sBRYy3
> Error: Configuration file /etc/ngcp-config/config.yml does not exist (unconfigured?) - exiting.
Change-Id: Ieb196690246d7c1ea59e7f4359d72179472c278b
Guillem Jover
Kirill Solomko
Alexander Lutay
Manuel Montecelo
Michael Prokop
Richard Fuchs
Victor Seva