* add support for MariaDB timezones sync sql
delimiter as now in the generated sql stream output
there are sections with '\d' changing the delimiter behaviour.
Change-Id: If226db7018f431ff4248dcf6ff5a614ff96a45c7
Add '--join' option which is used in init-shared command.
We should not guess the required action but it should be defined either
it's necessary to initialize git repo in shared storage or should use it
as source of ngcpcfg files.
Change-Id: I4b0924fc6b35a718e9dfd5019568c0c3a7a37a9c
If the password is not valid, we should retry until the user has
introduced a valid one, or has canceled the action. Otherwise the
script leaves the system in a bad state, where for example ngcp-reset-db
cannot be re-executed because the password is not valid anymore.
Change-Id: Ic793cabb50a15bed56d916454850db046826083c
We use this module instead of IO::Prompter, which is more lightweight
and has less issues, and is already used in NGCP.
Change-Id: I735bceb7df10d37c68ddedbaa4b5f9ce2a4b3e06
The DBI connect call can be specified the PrintError option, which will
make the function print the contents of the $DBI:errstr, but then we do
that again, which results in redudant output, so we disable this option.
The $DBI::errstr variable contains an error string that is *not*
terminated with a newline. The perl die built-in has magic behavior
when the string it gets does not end in a newline character, it will
then emit the source line, file name and a newline where the error
happened, which is in most cases not what we want. We just avoid
this by adding an explicit newline.
Replace a print + exit with a die, and a printf without a terminating
newline which would mess up further output with a warn call with an
explicit newline, which will also make it go to stderr.
And remove the local PrintError overrides which are now unnecessary.
Change-Id: I348b8cc676da7f255c6c823de2a774694dbca826
In the previous commit the dependency was added to to the sources only
as build-depends, we need also here run dependency.
Change-Id: Iebadef656e4768d22791b378daac934aeaae0dac
Sipwise recommends to set password for MariaDB user 'root'.
It brakes ngcp-reset-db which drops database, upload/create the
new one and cannot connect MariaDB using user 'root' to
initialize all DB passwords using ngcp-sync-constants.
ngcp-sync-constants has to use user 'root' to initialize
DB user 'sipwise' from constants.yml.
We have to request user to provide password for DB user 'root' as
it is NOT stored on NGCP platform anywhere.
P.S. also performed 'wrap-and-sort -sat' here.
Change-Id: I15467ef37d6765e972c8c482a2dbe86eececd0ca
It was a widely popular question from users.
As the common sense pushes them to use 'get' when they use 'set/del'.
Let's follow the common logic here.
We still support old 'ngcpcfg values' syntax.
Change-Id: I40a210b7c13d21a4bff4b6988535c39091f53280
After checking with Kirill, it looks like we do not need any longer
"special handling for user sipwise as it is not in constants.yml".
ngcp-installer fully operates with sipwise_extra.cnf which is filled
from constants.yml, tested as PPA, CE/PRO were installed properly.
The following commands also work well:
> root@spce:~# ngcp-sync-constants -v
> --> nodename => sp1
> --> connected to localhost:3306 using '/etc/mysql/sipwise_extra.cnf'
> --> nothing to update
> root@sp1:~#
> root@spce:~# ngcp-sync-constants -v -r
> --> nodename => spce
> --> connected to localhost:3306 using user 'root'
> --> nothing to update
> root@spce:~#
Also reset DB passwords works in both ways:
- using user 'root':
> ngcp-sync-constants -v -i
> ngcp-sync-constants -v -r
- using user sipwise:
> ngcp-sync-constants -v -i
> ngcpcfg build /etc/mysql/sipwise_extra.cnf
> ngcp-sync-constants -v -r
ngcpcfg build there is necessary to update defaults-extra-file with
newly generated password in constants.yml (credentials->mysql->system->p).
Change-Id: Ib9be8723233ec385256d05576ecfbab613a8c1ce
There is no need to read the text file and search for password inside,
we have properly formatted credentials file /etc/mysql/sipwise_extra.cnf
which is fully supported by MySQL/MariaDB client library and Perl DBI.
Also we have plans to remove /etc/mysql/sipwise.cnf to prevent it usage
(to prevent password leakage through Linux process list).
Change-Id: I62006ae01d9097642a3dae063781a621f5050842
lib/get_ngcp_neighbours generated /etc/default/ngcp-roles which contains
two lists "NGCP_HOSTS" and "NGCP_NEIGHBOURS". The first list is "all nodes
in carrier cluster", while the second is "all nodes in carrier cluster
exclusing this node".
On PRO (sp1 node):
> NGCP_HOSTS="sp1 sp2"
> NGCP_NEIGHBOURS="sp2"
On Carrier (web01a node):
> NGCP_HOSTS="db01a db01b lb01a lb01b prx01a prx01b web01a web01b"
> NGCP_NEIGHBOURS="db01a db01b lb01a lb01b prx01a prx01b web01b"
The problem was in get_nodename which is always sp1/sp2 on PRO AND Carrier.
As a result the code worked on PRO but didn't exclude the current node on Carrier.
This is a fix to exclude current node/hostname from neighbours list.
Change-Id: Icef4c3811c7e1b087157f57f9cb723b5a9456943
We need to call action and care about SUMMARY_ONLY without calling ngcpcfg.
main_action will be used in push-parallel to call pull/build/apply actions
directly without calling ngcpcfg. Also 'ngcpcfg push' can be migrated on new
approach '--remote' (recommended by Guillem Jover, BTW, Tnx!) from the current
approach of executing commands through ngcp-ssh on remote host.
Change-Id: I927b8055a040f49d6cfb462a200ad4d4b2d660ff
Sometimes (e.g. on 'push-parallel') we need to call ngcpcfg on
remote host and generating summary only output about the results.
It simplifies complex 'push-parallel' codebase which is critical
for huge Carrier installation.
P.S. we cannot print summary message all the time to console output,
as a lot of code depends on the ngcpcfg output, e.g.:
> root@web01a:~# ngcp-ppa gerrit_alutay_63758_push_parallel_2
> ERROR: There are uncommitted changes in config files:
> 2019-10-02 11:59:38 web01a: Successfully executed 'diff' on 'web01a'
> ERROR: Please commit them or discard
> root@web01a:~#
Instead, let's be backward compatible with printing on console,
but always print this message to log file (as we sometimes need to know
ngcpcfg exit code and it was impossible to find it in log file previously.)
Change-Id: I9aad0912796500cddb23d1f3eaa428b0b39152d0
Otherwise we have an error on jenkins after adding
the recent functionality which helps us detect aborted
ngcpcfg execution:
> stderr:
> b"mkdir: cannot create directory '/var/lib/ngcpcfg': Permission denied
> 2019-10-01 15:56:26 jenkins-slave15: Error: Failed to call action 'build' on 'jenkins-slave15' (see logs on 'jenkins-slave15')
Change-Id: I37946855058a86b63169c65816705fd706902ba0
The commit 92dccb4b10 introduced new group 'ngcp-admin'
and forced ngcpcfg to set proper perms every time we call ngcpcfg.
One of the tests starts failing right after the commit above, but
we didn't notice it until we add error message (the following commit).
Wihtout the fix here ngcpcfg doesn't pass tests:
> test_ngcpcfg_build_network_interfaces.py::test_network_interfaces FAILED
> ...
> stderr:
> b"../scripts//build: line 174: CONFIG_GROUP: unbound variable
> 2019-10-01 15:41:35 b22fa4d1cbb9: Error: Failed to call action 'build' on 'b22fa4d1cbb9' (see logs on 'b22fa4d1cbb9')
Change-Id: Ia514a7e275aef25d01bdaedf71ef7879e84dd8ba
- Remove epub output as asciidoctor requires unpackaged support.
- Remove html and pdf output as it is not known to be used anyway.
Change-Id: I7089e01ef17dbc40df08b796f48bcd8616936145
Long long time ago (around release 2.8) we started filling
the default HA pingnodes because most of our customers didn't care about them.
Lack of pingnodes was causing problems on PRO/Carrier as HA was not
being able to make correct decisions about the cluster state which causes
split brain and both nodes might end up being active.
The idea back then was to use some IPs which MUST always be reachable:
- gateway
- primary DNS server
- secondary DNS server
NGCP is barely usable (or likely unstable) if some of the IPs above is no reachable.
Sure, those IPs can be redefined in config.yml by the customer IF the customer provides more the 3 IPs.
In the past we didn't have GW and DNS IPs in network.yml,
also NGCP did NOT generate /etc/resolv.conf and /etc/network/interfaces from templates.
Nowadays we have all such information AND generate /etc/resolv.conf from network.yml.
However /etc/ha.d/ha.cf was NOT reconfigured against network.yml but still uses workarounds.
The macro used in /etc/ha.d/ha.cf reads /etc/resolv.conf to fetch DNS addresses.
It causes a list of problems on cloud-based installations (and not only there).
Problem description:
When DHCP is enabled, the server receives IP from DHCP and normally DNS servers
are also included in DHCP packet => Linux/networking saves DNS in /etc/resolv.conf on boot.
Later 'ngcpcfg apply' happens and builds all the files in alphabetical order.
Unfortunately the order here is /etc/ha.d/ha.cf ... /etc/resolv.conf.
So, ngcpcfg builds /etc/ha.d/ha.cf first and inserts there DNS IPs from /etc/resolv.conf->DHCP
and not IPs from network.yml. Some seconds later DNS IPs from network.yml will
be written in /etc/resolv.conf (as we have ngcpcfg template for /etc/resolv.conf since ~mr4.5).
As a result we do change /etc/ha.d/ha.cf twice:
- to switch HA from old DNS IPs/pingnodes to pingnodes from DHCP
- to switch HA back from DHCP DNS IPs/pingnodes to pingnodes from network.yml
It causes two unexpected HA restarts and confuses operators a lot.
We should be safer here and use such sensitive data from one place -> network.yml
The GW/DNS IPs search logic here is a copy of logic from template /etc/resolv.conf:
> For your host find the interface in network.yml which has the gateway defined.
> Use GW/DNS IPs for the interface you detected above.
Change-Id: I055e17e20bfd2bc722388c61db12ff5c156e9ecc
In the past helper/build_config generated new tmp file for every call
and we called it for hundred+ times. Let's generate it once and pass
to helper/build_config to be reused every time.
Command: time ngcpcfg apply "test"
Old code: real 0m17.194s
New code: real 0m15.477s
Change-Id: Ic1c35ccb5c5f92ddee6328502194a5301123bd9d
We do abort building on the first broken file due to 'set -e' above.
Having $RC variable here which collects exit code is very confusing.
Change-Id: I415bc77d8becf317c1be6b5ac493be2583479e00
We are calling slow Perl command to move files to workaround Docker problem
and losing 25% of performance on every NGCP installation in production.
Let's call slow workaround when it is necessary only (in Docker).
Command: time ngcpcfg build
Original results: real 0m19.976s
Current results: real 0m15.286s
Change-Id: Iac787c0fcdb4182d71b6305ebbba5ea3193e76c8
About 50% of previous helper/build_config execution was loading 'functions/main'
and we were performing it again and again for every tt2 files we build
and we have more then hundred files in ngcpcfg.
In fact we need two env variables and logging functions.
Let's extract them into the separate function file and load them only.
Command: time ngcpcfg build
Original results: real 0m27.766s
Current results: real 0m19.976s
Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
We are performing complex manipulations with list of files to be build
to understand which tt2 file we should use: .tt2/customtt.tt2/tt2.sp1/...
In the same time 90% of our files has only .tt2, so let's check the count here
and use the $file if only one record is available to proceed.
Change-Id: Ib754390563a93a1fbdeb4a4147b066fcb53ed5cc
The previous code had three including loops:
- loop over all folders ${CONFIG_POOL}
- run find to get list of all files
- loop over all files in list
- check if no files requested (build all) then remember file
- loop over all requested files
- check if file was requested then remember file
- process further with all files we remembered above
The new logic is much simpler and faster:
- loop over all folders ${CONFIG_POOL}, remove missing
- run find on all available folders to get list of all files 'filelist_allfiles'
- check IF no files requested (build all)
- THEN just use all files from list 'filelist_allfiles'
- ELSE loop over all requested by user files (normally one/several)
- grep requested file from 'filelist_allfiles'
- process further with all files we remembered above
The two main ideas here:
- just use all files if we build all files, no need to loop one by one
- grep requested files from "all files" instead of searching them in loop
At the end we are 30% faster in case if we build one file only:
> Command: time ngcpcfg build /etc/default/ngcp-roles
> Old code: real 0m1.587s
> New code: real 0m1.060s
Change-Id: I206557b004ff95c0607150a61a9e6e743ab8cd29
for the intermediate-cdr exporter, a similar .conf is
to be generated. intermediate cdr data will be available
in tables with the "int_cdr" prefix.
the macros in get_cdr_export_fields are therefore required
to switch printed table names (ie. "cdr", "cdr_tag_data")
accordingly (ie. "int_cdr", "int_cdr_tag_data").
the existing "type" argv is used as switch param.
Change-Id: I74fc4c91acf285d621b9ecf26a050ccf131139c4
It produces security issues when diff is stored in logs (e.g. upgrade logs)
and may contain passwords or another secrets. The special option '--diff'
is introduced for users who require old behaviour.
Change-Id: Ie4fd1ef2ee7fe1a51e87a99e36b5f71f6ecc8084
This reverts commit 55c5c6df89.
We need to inject scripts that requires ngcpcfg services files to be
executed, which does not work because those are only honored for /etc
pathnames.
Instead we will just create a symlink back to /etc where we will
install all the needed scripts.
Change-Id: I66b61dea93ed320117bbdf019a4c0a4b2582042f
The monit-services OCF Resource Agent for pacemaker needs to be
installed there.
Ideally we would not need this, but as this is the only location
allowed by pacemaker, let's add it, even if we might at some point
end up not using it, as we can just not install any templates in those
directories.
Change-Id: Ib14b68388f018b45bc4d7cf6895ca17d0fc802a3
These files are an intermediate data source, which are being used also
during builds of templates. We cannot make them first stage YAML sources
as they do depend on information coming from the first stage YAML
sources themselves. So we need to special-case them to make sure they
are always built first, so that the other files have up-to-date
information to use when being built.
Change-Id: I6cfd6e80823ddfaf71e2246de5518b9cb6d6a1d2
Kirill Solomko
Mykola Malkov
Guillem Jover
Alexander Lutay
Sipwise Jenkins Builder
Rene Krenn
Richard Fuchs