We need to call action and care about SUMMARY_ONLY without calling ngcpcfg.
main_action will be used in push-parallel to call pull/build/apply actions
directly without calling ngcpcfg. Also 'ngcpcfg push' can be migrated on new
approach '--remote' (recommended by Guillem Jover, BTW, Tnx!) from the current
approach of executing commands through ngcp-ssh on remote host.
Change-Id: I927b8055a040f49d6cfb462a200ad4d4b2d660ff
Sometimes (e.g. on 'push-parallel') we need to call ngcpcfg on
remote host and generating summary only output about the results.
It simplifies complex 'push-parallel' codebase which is critical
for huge Carrier installation.
P.S. we cannot print summary message all the time to console output,
as a lot of code depends on the ngcpcfg output, e.g.:
> root@web01a:~# ngcp-ppa gerrit_alutay_63758_push_parallel_2
> ERROR: There are uncommitted changes in config files:
> 2019-10-02 11:59:38 web01a: Successfully executed 'diff' on 'web01a'
> ERROR: Please commit them or discard
> root@web01a:~#
Instead, let's be backward compatible with printing on console,
but always print this message to log file (as we sometimes need to know
ngcpcfg exit code and it was impossible to find it in log file previously.)
Change-Id: I9aad0912796500cddb23d1f3eaa428b0b39152d0
About 50% of previous helper/build_config execution was loading 'functions/main'
and we were performing it again and again for every tt2 files we build
and we have more then hundred files in ngcpcfg.
In fact we need two env variables and logging functions.
Let's extract them into the separate function file and load them only.
Command: time ngcpcfg build
Original results: real 0m27.766s
Current results: real 0m19.976s
Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f
Otherwise:
> warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125]
Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc
We are performing complex manipulations with list of files to be build
to understand which tt2 file we should use: .tt2/customtt.tt2/tt2.sp1/...
In the same time 90% of our files has only .tt2, so let's check the count here
and use the $file if only one record is available to proceed.
Change-Id: Ib754390563a93a1fbdeb4a4147b066fcb53ed5cc
The previous code had three including loops:
- loop over all folders ${CONFIG_POOL}
- run find to get list of all files
- loop over all files in list
- check if no files requested (build all) then remember file
- loop over all requested files
- check if file was requested then remember file
- process further with all files we remembered above
The new logic is much simpler and faster:
- loop over all folders ${CONFIG_POOL}, remove missing
- run find on all available folders to get list of all files 'filelist_allfiles'
- check IF no files requested (build all)
- THEN just use all files from list 'filelist_allfiles'
- ELSE loop over all requested by user files (normally one/several)
- grep requested file from 'filelist_allfiles'
- process further with all files we remembered above
The two main ideas here:
- just use all files if we build all files, no need to loop one by one
- grep requested files from "all files" instead of searching them in loop
At the end we are 30% faster in case if we build one file only:
> Command: time ngcpcfg build /etc/default/ngcp-roles
> Old code: real 0m1.587s
> New code: real 0m1.060s
Change-Id: I206557b004ff95c0607150a61a9e6e743ab8cd29
These files are an intermediate data source, which are being used also
during builds of templates. We cannot make them first stage YAML sources
as they do depend on information coming from the first stage YAML
sources themselves. So we need to special-case them to make sure they
are always built first, so that the other files have up-to-date
information to use when being built.
Change-Id: I6cfd6e80823ddfaf71e2246de5518b9cb6d6a1d2
Otherwise ngcpcfg fails in trunk with error:
> +03:29:10 (cfg.inc:378): cfg_common_configuration(): ngcpcfg set /etc/ngcp-config/config.yml bootenv.netscript.fallbackfssize=10M
> Save size of 'code' partitions to /etc/ngcp-config/config.yml
> /usr/share/ngcp-ngcpcfg/functions//main: line 68: NGCP_TESTSUITE: unbound variable
Change-Id: I32961cda14550278662acd40ed5863c19412de3c
- Remove a spurious trailing ';'.
- Make the hook directory reaction conditional on whether we are running
from within a testsuite, to restore its otherwise fatal treatment.
Change-Id: I3f8a5364d7b2f0ac82bfdc132b94641603225f85
Fixes: commit 505d6566d8
Unfortunately git doesn't track POSIX file permissions,
which cause perms drop on some git commands like 'git stash/pull/reset', etc.
We must be sure 'restore-permissions' always executed at the end of 'ngcpcfg'.
Also we must be sure we update .ngcpcfg_perms whenever we start 'ngcpcfg',
otherwise we might loose not-yet committed changes in .ngcpcfg_perms.
Example during the upgrade mr7.0->mr7.1:
1) JFYI, ngcpcfg mr7.1+ copies perms from source tt2 file to the destination config
2) on mr7.0->mr7.1, ngcp-upgrade installs new package ngcp-templates-pro package
with new perms on tt2 files (which are not committed by apt as /etc/ngcp-config
is not covered by 'etckeeper', it is by design).
3) later ngcp-upgrade executes 'ngcpcfg build' to build some config.
4) on PRO/Carrier 'ngcpcfg build' executes 'ngcpcfg pull' first to
ensure there is no outstanding changes on shared git repo
(has been done on Ops request in MT#15803).
5) 'ngcpcfg pull' does 'git stash/pop' before pulling which effectively
reset new perms which came from package in step 1 (as git doesn't track
perms on files, by design).
Summary/fix:
whenever we stash something we should ensure perms are stored as well,
which means we need to generate new .ngcpcfg_perms and store it in stash
together with all the rest local changes.
Change-Id: I5292d4433a39ff2751874d63083f093f19b9cfe2
When running the test suite, in case we need to cover the code paths
that are handling the hooks, we should not assume the files are
installed in the system directories.
Change-Id: I1cd09eaf186d28093eb03ad5e9fc096fe7d63a3e
'git status' output has been changed in 2.9 which causes some troubles for us
(we didn't change all the places, we missed ngcpcfg-ha.git repo,
better to use common code in all the places).
Change-Id: Ie664af89ee6d6b31fb3011e763107c41bf4cf00f
The night Carrier trunk installation has failed:
> +02:29:59 (cfg_pro.inc:280): cfg_ngcpcfg_init_mgmt(): cfg_ensure_file_mgmt_node
> ...
> +02:29:59 (cfg_pro.inc:282): cfg_ngcpcfg_init_mgmt(): '[' -x /usr/share/ngcp-ngcpcfg/scripts/init-mgmt ']'
> +02:29:59 (cfg_pro.inc:287): cfg_ngcpcfg_init_mgmt(): DEBUG=yes
> +02:29:59 (cfg_pro.inc:287): cfg_ngcpcfg_init_mgmt(): /usr/share/ngcp-ngcpcfg/scripts/init-mgmt
> /usr/share/ngcp-ngcpcfg/functions//main: line 18: HNAME: unbound variable
The manual execution of script 'init-mgmt' shows no error,
so far if 'DEBUG=yes' is enabled the issue is easily reproducible.
The reason here is debug messages which can be printed on the
very early stage of initialisation and cause the issue here:
> ++ log_debug 'sourced configuration file /etc/ngcp-config/ngcpcfg.cfg'
> ++ '[' -n yes ']'
> ++ logger -t ngcpcfg -- 'Debug: sourced configuration file /etc/ngcp-config/ngcpcfg.cfg'
> ++ console_output 'DEBUG: sourced configuration file /etc/ngcp-config/ngcpcfg.cfg\n'
> ++ '[' -z '+%F %T' ']'
> +++ date '+%F %T'
> ++ local 'timestamp=2018-10-12 10:16:24'
> +++ printf -- '%s: ' '2018-10-12 10:16:24'
> +++ sed 's/./ /g'
> ++ timestamp_replacementchars=' '
> /usr/share/ngcp-ngcpcfg/functions//main: line 18: HNAME: unbound variable
Let's be polite for such kind of log message.
Change-Id: I3647567aba5fad3a103af2e6da35229f6d932bed
It is necessary to improve navigation into long 'ngcpcfg push all'
output on huge customer's systems. Requested by Sipwise Ops team.
Previously:
2018-10-11 10:48:40: Generating /etc/update-motd.d/05-maintenance-mode: OK
2018-10-11 10:48:40: Executing postbuild for /etc/update-motd.d/05-maintenance-mode
2018-10-11 10:48:40: Generating /etc/update-motd.d/10-heartbeat: OK
2018-10-11 10:48:40: Executing postbuild for /etc/update-motd.d/10-heartbeat
2018-10-11 10:48:40: Generating /etc/update-motd.d/20-sysinfo: OK
Now:
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/redis-server.service: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/rsyslog.service.d/override.conf: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/slapd.service.d/override.conf: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/slapd.service: OK
2018-10-11 11:55:28 web01a: Generating /etc/systemd/system/snmpd.service.d/override.conf: OK
Change-Id: I3035293b95f64033005cc5f8d031dbbfac1dcc33
The 'ngcpcfg' received support for 'patchtt' files, like
> /etc/ngcp-config/templates/etc/foo/bar.patchtt.tt2
Those 'patchtt' are going to be applied on default 'tt2 template' file:
> /etc/ngcp-config/templates/etc/foo/bar.tt2
and produce 'customtt' on 'ngcpcfg patch':
> /etc/ngcp-config/templates/etc/foo/bar.customtt.tt2
Further 'customtt' will be used to overwrite 'tt2 templates'
on 'ngcpcfg build' or 'ngcpcfg apply'.
NOTE: 'ngcpcfg patch' is executed automatically on every 'ngcpcfg build'.
It should allows to update ngcp-templates easily and support
local modifications without the pain (until the patches can be applied).
Change-Id: Ice4369386313c5d33e4d498346345eade6f3d0d7
The old testsuite wasn't updated for way too long and since
ngcpcfg receives more and more features we need a decent test
coverage. pytest seems to provide the right level of
abstraction, excellent fixtures and junit-xml reporting as
needed.
Inspired by Vincent Bernat's
https://github.com/vincentbernat/lldpd/tree/master/tests/integration
Thanks Victor Seva <vseva@sipwise.com>, Vincent Bernat <vincent@bernat.im>, Christian Hofstaedtler <christian@hofstaedtler.name> and Lukas Prokop <admin@lukas-prokop.at> for feedback, inspiration and help
Change-Id: Iffed87e8cc540169bed89c00967a03e80859179e
If we have multiple web0X proxies but need specific
templates on just one of those pairs then we need to
support *.customtt.tt2.$pairname + *.tt2.$pairname
templates.
While at it also support *.customtt.tt2.$nodename +
foo.tt2.$nodename (where nodename is actual hostname
and not sp1 or sp2 as already supported already).
Test script for demonstration (especially for precedence):
mkdir /etc/ngcp-config/templates/etc/mika
cd /etc/ngcp-config/templates/etc/mika
hostname=$(hostname) # e.g. web01b
pairname=${hostname%[ab]} # e.g. web01
nodename=$(cat /etc/ngcp_nodename) # e.g. sp2
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do echo $file > $file ; done
for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do
ngcpcfg build /etc/mika/ >/dev/null
if [ "$(cat /etc/mika/foo)" = "$file" ] ; then
echo "Content of file /etc/mika/foo matching $file"
rm "$file" ; rm /etc/mika/foo
else
echo "Problem with precedence of file $file"
return 1
fi
done
Change-Id: If7c222374bb56d7bf243200b37a414e39e050cd4
Git doesn't track file permissions (except for the executable
flag). For sensitive data (like the 'ssl' directory and file
'constants.yml' with passwords included) we've to prevent
non-root users from accessing those files.
hooks/pre-commit is inspired and based on the implementation
as present in etckeeper (and luckily we're license compatible)
and takes care of storing the file permissions inside file
/etc/ngcp-config/.ngcpcfg_perms.
The restore-permissions helper script takes care of restoring the
permissions after cloning the ngcpcfg repository via ngcpcfg
itself (being actions decrypt, pull (PRO-only) + initialise
(PRO-only)). It can be executed manually as well via
`usr/share/ngcp-ngcpcfg/helper/restore-permissions /etc/ngcp-config/`
(or wherever the according ngcpcfg repository is placed at).
Regarding the commit integration: git(1) itself doesn't track
file permissions, so we can't detect changes to file permissions
using git itself. Our new pre-commit hook records file
permissions via the .ngcpcfg_perms file. Now by just invoking it
during 'ngcpcfg commit' time we can ensure that even if there
have been any file permission changes in the working directory
the file .ngcpcfg_perms is then up2date and committed.
JFTR: The solution via the git pre-commit hook ensures that no
matter whether you're using 'ngcpcfg commit …' or 'git commit …'
you always get the file permissions handled via .ngcpcfg_perms.
Now if you want to change file permissions in a clean working
directory and commit *without* using 'ngcpcfg commit' but
directly via git itself then you've to use 'git commit
--allow-empty ...' and thanks to the pre-commit hook the file
.ngcpcfg_perms will still be up2date.
Change-Id: I84d608585c626b52112ff649893e232e441c59d8
Verify that the user is operating on branch master,
otherwise the result might be unexpected.
If branch master is checked out:
| root@spce:/etc/ngcp-config# ngcpcfg check
| root@spce:/etc/ngcp-config# ngcpcfg status
| [...]
| 2016-02-23 05:48:32: Checking currently active branch:
| 2016-02-23 05:48:32: OK: branch master active
If a branch other than 'master' is checked out:
| root@spce:/etc/ngcp-config# ngcpcfg check
| 2016-02-23 05:47:42: Error: branch 'mika' in '/etc/ngcp-config' active - please switch to branch 'master' before continuing.
| root@spce:/etc/ngcp-config# echo $?
| 1
| root@spce:/etc/ngcp-config# ngcpcfg status
| [...]
| 2016-02-23 05:47:47: Checking currently active branch:
| 2016-02-23 05:47:47: ACTION_NEEDED: branch 'mika' active - please switch to branch 'master'
| [...]
Change-Id: I5df92075905cafa3b714211581c8cfe749df04ba
In a previous implementation using timestamps we ran into timing
problems on slower systems. So instead lets record the git commit
ID of the latest ngcpcfg commit and check whether we have a match
between this commit and an according ngcpcfg build.
If the "ngcpcfg build" was executed without an "ngcpcfg commit"
before then we mark it as dirty since we don't have an according
git commit available for testing against. Since that's a common
workflow we don't report it as a real problem though, instead we
inform the user using:
| OK: nothing to build (latest build newer than latest commit)
As we commit AFTER the build in the apply action we can use a
workaround for this situation, so build and commit state match
with each other in that case then.
Change-Id: I48e32db6f42b53fe97a0b88805e7ddbee8576133
This reverts commit db373d3927.
Conflicts:
etc/ngcp-config/ngcpcfg.cfg
Until we've a working solution for this let's revert
the broken change so we avoid this being a release
stopper.
Change-Id: If03d9b3913de23c698b430d583dc7babfcc4ff04
If the latest git commit has a newer timestamp than
our latest "build" action then a "build" (or apply)
run is required, inform the user about it.
Change-Id: I83e2ff47ba54da733d368d78b6616d13d31a66d4
Previously we guessed the status of current branch by parsing
'git status' output. Unfortunately we have to care about a
corner cases in this way. In the same time we can easily
consider current branch status comparing commits.
Introducing the new function to provide current branch status:
up-2-date, ahead, behind or diverged with origin.
Change-Id: I4d09c8867069ccafc97e5d31638e8bc94caa58cd
Files in configuration directory EXTRA_CONFIG_DIR (being
/etc/ngcp-config/config.d/ by default) with the suffix yml (so
e.g. a file named pbx.yml can be shipped by an external Debian
package) are additionally taken into consideration as main
configuration files.
Those extra configuration file(s) have higher precedence than
the other yml configuration files (config.yml, network.yml,...),
*except* for constants.yml which has the power to override all
settings, including the ones from files inside EXTRA_CONFIG_DIR.
Closes: sipwise-#3059
Alexander Lutay
Guillem Jover
Michael Prokop
Victor Seva
Kirill Solomko