Move the hook selection at the input/output file generation stage, and
refactor the hook execution into a new function.
Change-Id: I010e50d198f946f5e8e5da0e60cf87d5688c45c7
This avoids lots of chmod calls.
We do not need the symlink handling, as we will generate now the files
with the correct permissions, and will respect the symlinks on mv, as
before, and are not doing the explicit chmod anymore.
Change-Id: I9ab75dda79e8b346eb1f41c36b450d6fabb10f11
This is needed for the get command, so that we can exactly control where
to output to, and avoid all the file generation heuristics based on the
input files.
Change-Id: Iace05d1093342722553ec0bbb8ff0e817f77857c
This makes sure we sort each array, instead of sorting the second, and
then letting perl merge the first aray with the result of the second
sort, and then sorting them all as a single array.
Change-Id: I7cd22bf479fee79c104627bf652f0ee674117eec
Fixes: commit 011ce33f32
The current shell code is very inefficient, as it uses files to track
the file lists, and grep to traverse them NxM times per N files.
Change-Id: I8396b4ee232157cea8ef599cba0618be3dd93536
As we now run the processing in parallel, we should not take CPU time
from other more important tasks in the system.
Change-Id: Ifcfdbc6838a377c00025b4ddb30d4ccfbd1dc548
Switch away from the old daemon/wrapper architecture, which complicates
things and requires more scaffolding code.
We turn the daemon/wrapper and build_config into a single perl process
that will load all YAML files, and then process each input/output pair
on a parallel child, to try to speed up the processing as much as
possible.
For whole rebuilds, it might speed up the generation by at least a
factor of x2, x4 or more, depending on the number of active processors.
Change-Id: I51aa2f90336e34a20983d8733f45b64d9b6fea0b
"Utils" TT plugin, to be extended.
provides base64 encoding for now, used by
provisioningtemplates feature.
Change-Id: I2980b57e5dae3d2595ba7017efcdc7e0bd00e469
The alias key is defined as optional in the config.yml schema. This was
producing perl warnings in the templates files:
,---
Argument "" isn't numeric in numeric gt (>) at /usr/lib/ngcp-ngcpcfg/get_cdr_export_fields line 45.
Argument "" isn't numeric in numeric gt (>) at /usr/lib/ngcp-ngcpcfg/get_cdr_export_fields line 45.
Argument "" isn't numeric in numeric gt (>) at /usr/lib/ngcp-ngcpcfg/get_cdr_export_fields line 45.
Argument "" isn't numeric in numeric gt (>) at /usr/lib/ngcp-ngcpcfg/get_cdr_export_fields line 45.
2020-01-14 15:02:30 sp1: Generating /etc/ngcp-cdr-exporter/event-exporter.conf: OK
`---
Change-Id: I46084fbd25e9c88c8cdddb7511c378ac06135632
Apparently the -vv option does not percolate down into the pytest
modules, perhaps due to the import usage. Using the -l, --showlocals
makes it print everything required to debug errors.
Change-Id: I83a2899a142191c32934a8fb152000ff68b80b1e
The yaml files should not be executable. This also makes running the
test suite more painful, as the fixtures in git will get their
permissions changed and will be shown by git as modified.
Change-Id: I05909b5eac316bb46251ccd79a19dc8b437876bc
When the hostname is self or unknown, we do not return any peername,
so should cope with that to avoid doing comparisons that will end up
emitting warnings.
Change-Id: Ib7e70666991a76b99e1f64ca365a3018320728ec
* add support for MariaDB timezones sync sql
delimiter as now in the generated sql stream output
there are sections with '\d' changing the delimiter behaviour.
Change-Id: If226db7018f431ff4248dcf6ff5a614ff96a45c7
Add '--join' option which is used in init-shared command.
We should not guess the required action but it should be defined either
it's necessary to initialize git repo in shared storage or should use it
as source of ngcpcfg files.
Change-Id: I4b0924fc6b35a718e9dfd5019568c0c3a7a37a9c
If the password is not valid, we should retry until the user has
introduced a valid one, or has canceled the action. Otherwise the
script leaves the system in a bad state, where for example ngcp-reset-db
cannot be re-executed because the password is not valid anymore.
Change-Id: Ic793cabb50a15bed56d916454850db046826083c
We use this module instead of IO::Prompter, which is more lightweight
and has less issues, and is already used in NGCP.
Change-Id: I735bceb7df10d37c68ddedbaa4b5f9ce2a4b3e06
The DBI connect call can be specified the PrintError option, which will
make the function print the contents of the $DBI:errstr, but then we do
that again, which results in redudant output, so we disable this option.
The $DBI::errstr variable contains an error string that is *not*
terminated with a newline. The perl die built-in has magic behavior
when the string it gets does not end in a newline character, it will
then emit the source line, file name and a newline where the error
happened, which is in most cases not what we want. We just avoid
this by adding an explicit newline.
Replace a print + exit with a die, and a printf without a terminating
newline which would mess up further output with a warn call with an
explicit newline, which will also make it go to stderr.
And remove the local PrintError overrides which are now unnecessary.
Change-Id: I348b8cc676da7f255c6c823de2a774694dbca826
In the previous commit the dependency was added to to the sources only
as build-depends, we need also here run dependency.
Change-Id: Iebadef656e4768d22791b378daac934aeaae0dac
Sipwise recommends to set password for MariaDB user 'root'.
It brakes ngcp-reset-db which drops database, upload/create the
new one and cannot connect MariaDB using user 'root' to
initialize all DB passwords using ngcp-sync-constants.
ngcp-sync-constants has to use user 'root' to initialize
DB user 'sipwise' from constants.yml.
We have to request user to provide password for DB user 'root' as
it is NOT stored on NGCP platform anywhere.
P.S. also performed 'wrap-and-sort -sat' here.
Change-Id: I15467ef37d6765e972c8c482a2dbe86eececd0ca
It was a widely popular question from users.
As the common sense pushes them to use 'get' when they use 'set/del'.
Let's follow the common logic here.
We still support old 'ngcpcfg values' syntax.
Change-Id: I40a210b7c13d21a4bff4b6988535c39091f53280
After checking with Kirill, it looks like we do not need any longer
"special handling for user sipwise as it is not in constants.yml".
ngcp-installer fully operates with sipwise_extra.cnf which is filled
from constants.yml, tested as PPA, CE/PRO were installed properly.
The following commands also work well:
> root@spce:~# ngcp-sync-constants -v
> --> nodename => sp1
> --> connected to localhost:3306 using '/etc/mysql/sipwise_extra.cnf'
> --> nothing to update
> root@sp1:~#
> root@spce:~# ngcp-sync-constants -v -r
> --> nodename => spce
> --> connected to localhost:3306 using user 'root'
> --> nothing to update
> root@spce:~#
Also reset DB passwords works in both ways:
- using user 'root':
> ngcp-sync-constants -v -i
> ngcp-sync-constants -v -r
- using user sipwise:
> ngcp-sync-constants -v -i
> ngcpcfg build /etc/mysql/sipwise_extra.cnf
> ngcp-sync-constants -v -r
ngcpcfg build there is necessary to update defaults-extra-file with
newly generated password in constants.yml (credentials->mysql->system->p).
Change-Id: Ib9be8723233ec385256d05576ecfbab613a8c1ce
There is no need to read the text file and search for password inside,
we have properly formatted credentials file /etc/mysql/sipwise_extra.cnf
which is fully supported by MySQL/MariaDB client library and Perl DBI.
Also we have plans to remove /etc/mysql/sipwise.cnf to prevent it usage
(to prevent password leakage through Linux process list).
Change-Id: I62006ae01d9097642a3dae063781a621f5050842
lib/get_ngcp_neighbours generated /etc/default/ngcp-roles which contains
two lists "NGCP_HOSTS" and "NGCP_NEIGHBOURS". The first list is "all nodes
in carrier cluster", while the second is "all nodes in carrier cluster
exclusing this node".
On PRO (sp1 node):
> NGCP_HOSTS="sp1 sp2"
> NGCP_NEIGHBOURS="sp2"
On Carrier (web01a node):
> NGCP_HOSTS="db01a db01b lb01a lb01b prx01a prx01b web01a web01b"
> NGCP_NEIGHBOURS="db01a db01b lb01a lb01b prx01a prx01b web01b"
The problem was in get_nodename which is always sp1/sp2 on PRO AND Carrier.
As a result the code worked on PRO but didn't exclude the current node on Carrier.
This is a fix to exclude current node/hostname from neighbours list.
Change-Id: Icef4c3811c7e1b087157f57f9cb723b5a9456943
We need to call action and care about SUMMARY_ONLY without calling ngcpcfg.
main_action will be used in push-parallel to call pull/build/apply actions
directly without calling ngcpcfg. Also 'ngcpcfg push' can be migrated on new
approach '--remote' (recommended by Guillem Jover, BTW, Tnx!) from the current
approach of executing commands through ngcp-ssh on remote host.
Change-Id: I927b8055a040f49d6cfb462a200ad4d4b2d660ff
Sometimes (e.g. on 'push-parallel') we need to call ngcpcfg on
remote host and generating summary only output about the results.
It simplifies complex 'push-parallel' codebase which is critical
for huge Carrier installation.
P.S. we cannot print summary message all the time to console output,
as a lot of code depends on the ngcpcfg output, e.g.:
> root@web01a:~# ngcp-ppa gerrit_alutay_63758_push_parallel_2
> ERROR: There are uncommitted changes in config files:
> 2019-10-02 11:59:38 web01a: Successfully executed 'diff' on 'web01a'
> ERROR: Please commit them or discard
> root@web01a:~#
Instead, let's be backward compatible with printing on console,
but always print this message to log file (as we sometimes need to know
ngcpcfg exit code and it was impossible to find it in log file previously.)
Change-Id: I9aad0912796500cddb23d1f3eaa428b0b39152d0
Otherwise we have an error on jenkins after adding
the recent functionality which helps us detect aborted
ngcpcfg execution:
> stderr:
> b"mkdir: cannot create directory '/var/lib/ngcpcfg': Permission denied
> 2019-10-01 15:56:26 jenkins-slave15: Error: Failed to call action 'build' on 'jenkins-slave15' (see logs on 'jenkins-slave15')
Change-Id: I37946855058a86b63169c65816705fd706902ba0
The commit 92dccb4b10 introduced new group 'ngcp-admin'
and forced ngcpcfg to set proper perms every time we call ngcpcfg.
One of the tests starts failing right after the commit above, but
we didn't notice it until we add error message (the following commit).
Wihtout the fix here ngcpcfg doesn't pass tests:
> test_ngcpcfg_build_network_interfaces.py::test_network_interfaces FAILED
> ...
> stderr:
> b"../scripts//build: line 174: CONFIG_GROUP: unbound variable
> 2019-10-01 15:41:35 b22fa4d1cbb9: Error: Failed to call action 'build' on 'b22fa4d1cbb9' (see logs on 'b22fa4d1cbb9')
Change-Id: Ia514a7e275aef25d01bdaedf71ef7879e84dd8ba
- Remove epub output as asciidoctor requires unpackaged support.
- Remove html and pdf output as it is not known to be used anyway.
Change-Id: I7089e01ef17dbc40df08b796f48bcd8616936145
Long long time ago (around release 2.8) we started filling
the default HA pingnodes because most of our customers didn't care about them.
Lack of pingnodes was causing problems on PRO/Carrier as HA was not
being able to make correct decisions about the cluster state which causes
split brain and both nodes might end up being active.
The idea back then was to use some IPs which MUST always be reachable:
- gateway
- primary DNS server
- secondary DNS server
NGCP is barely usable (or likely unstable) if some of the IPs above is no reachable.
Sure, those IPs can be redefined in config.yml by the customer IF the customer provides more the 3 IPs.
In the past we didn't have GW and DNS IPs in network.yml,
also NGCP did NOT generate /etc/resolv.conf and /etc/network/interfaces from templates.
Nowadays we have all such information AND generate /etc/resolv.conf from network.yml.
However /etc/ha.d/ha.cf was NOT reconfigured against network.yml but still uses workarounds.
The macro used in /etc/ha.d/ha.cf reads /etc/resolv.conf to fetch DNS addresses.
It causes a list of problems on cloud-based installations (and not only there).
Problem description:
When DHCP is enabled, the server receives IP from DHCP and normally DNS servers
are also included in DHCP packet => Linux/networking saves DNS in /etc/resolv.conf on boot.
Later 'ngcpcfg apply' happens and builds all the files in alphabetical order.
Unfortunately the order here is /etc/ha.d/ha.cf ... /etc/resolv.conf.
So, ngcpcfg builds /etc/ha.d/ha.cf first and inserts there DNS IPs from /etc/resolv.conf->DHCP
and not IPs from network.yml. Some seconds later DNS IPs from network.yml will
be written in /etc/resolv.conf (as we have ngcpcfg template for /etc/resolv.conf since ~mr4.5).
As a result we do change /etc/ha.d/ha.cf twice:
- to switch HA from old DNS IPs/pingnodes to pingnodes from DHCP
- to switch HA back from DHCP DNS IPs/pingnodes to pingnodes from network.yml
It causes two unexpected HA restarts and confuses operators a lot.
We should be safer here and use such sensitive data from one place -> network.yml
The GW/DNS IPs search logic here is a copy of logic from template /etc/resolv.conf:
> For your host find the interface in network.yml which has the gateway defined.
> Use GW/DNS IPs for the interface you detected above.
Change-Id: I055e17e20bfd2bc722388c61db12ff5c156e9ecc
In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
Guillem Jover
Sipwise Jenkins Builder
Rene Krenn
Kirill Solomko
Mykola Malkov
Alexander Lutay