Otherwise we have an error on jenkins after adding
the recent functionality which helps us detect aborted
ngcpcfg execution:
> stderr:
> b"mkdir: cannot create directory '/var/lib/ngcpcfg': Permission denied
> 2019-10-01 15:56:26 jenkins-slave15: Error: Failed to call action 'build' on 'jenkins-slave15' (see logs on 'jenkins-slave15')
Change-Id: I37946855058a86b63169c65816705fd706902ba0
The commit 92dccb4b10 introduced new group 'ngcp-admin'
and forced ngcpcfg to set proper perms every time we call ngcpcfg.
One of the tests starts failing right after the commit above, but
we didn't notice it until we add error message (the following commit).
Wihtout the fix here ngcpcfg doesn't pass tests:
> test_ngcpcfg_build_network_interfaces.py::test_network_interfaces FAILED
> ...
> stderr:
> b"../scripts//build: line 174: CONFIG_GROUP: unbound variable
> 2019-10-01 15:41:35 b22fa4d1cbb9: Error: Failed to call action 'build' on 'b22fa4d1cbb9' (see logs on 'b22fa4d1cbb9')
Change-Id: Ia514a7e275aef25d01bdaedf71ef7879e84dd8ba
In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
It produces security issues when diff is stored in logs (e.g. upgrade logs)
and may contain passwords or another secrets. The special option '--diff'
is introduced for users who require old behaviour.
Change-Id: Ie4fd1ef2ee7fe1a51e87a99e36b5f71f6ecc8084
These lines have to be commented out, the script/helper is not present during
tests:
#PROCESS '/usr/lib/ngcp-ngcpcfg/get_ngcp_version';
#ngcp_version = out;
Change-Id: I9665f205baf1e28ed2a96e8ab01f0f8f80b838ac
- Remove a spurious trailing ';'.
- Make the hook directory reaction conditional on whether we are running
from within a testsuite, to restore its otherwise fatal treatment.
Change-Id: I3f8a5364d7b2f0ac82bfdc132b94641603225f85
Fixes: commit 505d6566d8
When running the test suite, in case we need to cover the code paths
that are handling the hooks, we should not assume the files are
installed in the system directories.
Change-Id: I1cd09eaf186d28093eb03ad5e9fc096fe7d63a3e
Since we manage this as a template, the configuration is getting more and more
complex and it's important to have comprehensive tests that help us to detect
problems when we add options and change templates.
Change-Id: Ieff8449f3d6d1cd74a9b44d49e1642a1bf02efc2
We have a hard dependency on netcat-openbsd, see:
| commit 26ba0340b6
| Author: Alexander Lutay <alutay@sipwise.com>
| Date: Mon Jun 4 13:22:27 2018 +0200
|
| TT#37401 Fix 50ecc1544: depends on netcat-openbsd since we use 'nc -U' (not available in 'netcat-traditional')
But our docker image still uses netcat-traditional.
Make sure to have netcat-openbsd available, while at
it also adjust Build-Depends accordingly.
Change-Id: I1d3cfd9b4b56047fa51c3ef1d77060122f4d2568
Due to the code in tt2-wrapper, the only errors reported were failures to
connect to the socket to use the back-end performing the parsing. But there was
no attempt to detect errors in the parsing at all.
Since we are only able to use the output stream and not for example exit codes
of the parser at the back-end (libtemplate-perl), we can only rely on finding
some kind of error string, which can change in future versions of
libtemplate-perl but hopefully not very often, and at most it should only change
when we move base to newer Debian releases.
In the case of the original problem reported we could rely on "parse error", but
since there are two such strings ("file error" and "parse error") the first is
more general and also covers cases like "permission denied to read file" or
other errors that might happen. So the implementation of error checking is
basically grepping for "^file error", the string at the beginning of a line, to
limit possible false positives.
With the new checks and an example of an invalid syntax already in the first
line, "ngcpcfg build" detects the error correctly and stops processing, and
shows information about the source of the problem, e.g.:
root@spce:~# ngcpcfg build /etc/test
2018-10-29 11:37:01 spce: yml configs were validated successfuly
2018-10-29 11:37:01 spce: configs were checked successfuly
2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch.
Loading /etc/ngcp-config/config.yml in memory: OK
Loading /etc/ngcp-config/network.yml in memory: OK
Loading /etc/ngcp-config/constants.yml in memory: OK
2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED
2018-10-29 11:37:02 spce: Error: from generated file:
2018-10-29 11:37:02 spce: Error: file error - parse error - input file handle line 1: unexpected end of directive
2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding:
2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2
2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml
2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml
2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file>
2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G
2018-10-29 11:37:02 spce: should provide more details.
Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13
The perl Template::Toolkit is very rich, but its "function" support is a
bit poor. The ways to do it are either via MACRO directives, or by
simulating them with one function per file and then using PROCESS on
these. The problem is that this is very clunky, does not support
nesting, as we'd need different "argument" names for each "function",
and it's quite cumbersome to use, need to assign aguments passed
beforehand, and then assign back a designated return value from another
variable. This is also one of the reasons some of the functions are not
encapsulated, and have been inlined in various loops, because it was not
possible to cleanly PROCESS them from those call sites.
Instead we should use its native support for perl objects and perl
subroutines, which exposes these as proper methods of a designated
variable, and have none of the above mentioned problems. So we'll switch
from constructs such as:
argv.arg-a = variable;
argv.arg-b = 'value';
PROCESS 'path-to-library-dir/function'
result = out
into:
result = ngcp.function(variable, 'value');
In addition this might actually be faster, as it does not require
processing additional files, and it's all just native perl code.
This will be exposed within the NGCP templates as the ngcp object, and
new member functions will start replacing our old and clunky native
Template PROCESS-style library.
Change-Id: Id2f0d181c695a9dd074646881b7d9de3478570af
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
The new function will simplify initial customtt->patchtt
migration for end users. Some code was merged to be generic.
Also more tests were add here.
Change-Id: I7719f45275018818b2db82f6deee5b7428670a29
Especially when working on tests for new features it's useful
to run ngcpcfg and pytest interchangeably. Since identifying
all relevant variables is annoying let's provide an example
command line that's known to being useful while working on
the "ngcpcfg patch" tests.
Change-Id: I1ae1499ba2607f553d1dddcb355c7a3758d1a844
By setting variables like CONFIG_POOL and TEMPLATE_POOL_BASE we
can generate templates on-the-fly, without having to provide any
static files as fixtures. Modify t/fixtures/*.cfg accordingly, by
checking if a variable is set already (via any tests) and only if
it's unset set a default.
Change-Id: Ia156447bca368b19e62e94eee81c3949f00aa39a
* ['online', 'inactive'] as default
* introduced in order to be able to filter values at templates
so we have the ability to install new nodes without
impacting the working system
Change-Id: I78739f66f80da310347861e86b682fcf72745080
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ie51f1c79859d40ef0877fc0ab75f86ee72e14ea4
Quoting from https://twitter.com/hackebrot/status/873083574676733953:
| pytest looks up fixtures based on the import path etc.. Importing
| then creates an internal copy, which messes with the cache etc.
Change-Id: I3ee6f5ab859218ce0cfe8681505b670952e47df6
If a template named
/etc/ngcp-config/templates/etc/apt/apt.conf.d/71_no_recommended.tt2 exists
the resulting destination file would be
/etc/apt/apt.conf.d/71_no_recommended. Though if
/etc/apt/apt.conf.d/71_no_recommended exists as directory ngcpcfg should
abort and inform the user about the existing situation with a useful error
message.
Change-Id: I2c6e1e3a4ec485183674c1fe72251631ad9867ac
The old testsuite wasn't updated for way too long and since
ngcpcfg receives more and more features we need a decent test
coverage. pytest seems to provide the right level of
abstraction, excellent fixtures and junit-xml reporting as
needed.
Inspired by Vincent Bernat's
https://github.com/vincentbernat/lldpd/tree/master/tests/integration
Thanks Victor Seva <vseva@sipwise.com>, Vincent Bernat <vincent@bernat.im>, Christian Hofstaedtler <christian@hofstaedtler.name> and Lukas Prokop <admin@lukas-prokop.at> for feedback, inspiration and help
Change-Id: Iffed87e8cc540169bed89c00967a03e80859179e
Guillem Jover
Alexander Lutay
Marco Capetta
Michael Prokop
Manuel Montecelo
Richard Fuchs
Sergii Kipot
Victor Seva