These files are an intermediate data source, which are being used also
during builds of templates. We cannot make them first stage YAML sources
as they do depend on information coming from the first stage YAML
sources themselves. So we need to special-case them to make sure they
are always built first, so that the other files have up-to-date
information to use when being built.
Change-Id: I6cfd6e80823ddfaf71e2246de5518b9cb6d6a1d2
according to security framework spec
5.5.4 Allow r/w access /etc/ngcp-config/config.yml for users in security
domain ngcp-admin This file contains operational configuration data of
the NGCP system. Reading and editing it is part of day-to-day
operations. As such it has to be read- and writable by users in security
domain ngcp-admin. It is also readable by users of any other security
domain. This file is also read by many NGCP processes.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chgrp ngcp-admin /etc/ngcp-config/config.yml
Security domain: ngcp-admin User ownership: root Group ownership:
ngcp-admin File permissions: 664
IMPACT: Configuration data can be edited by all users in group
ngcp-admin, the sipwise user, and user root. If for some reason (e.g.
after upgrade) the proposed settings are reverted to default settings,
this does not pose a security risk. However, write access to
configuration data in this file will be denied to named users.
5.5.5 Allow r/w access to /etc/ngcp-config/network.yml for users in
security domain ngcp-admin This file contains information about the
network configuration of the NGCP cluster. This information should not
frequently change. Nevertheless, this file and its content is meant for
the operator and as such is read- and writable by users in security
domain ngcp-admin. It is also readable by users of any other security
domain. This file is also read by many NGCP processes.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chgrp ngcp-admin /etc/ngcp-config/network.yml
Security domain: ngcp-admin User ownership: root Group ownership:
ngcp-admin File permissions: 664
IMPACT: Configuration data can be edited by all users in group
ngcp-admin, the sipwise user, and user root. If for some reason (e.g.
after upgrade) the proposed settings are reverted to default settings,
this does not pose a security risk. However, write access to
configuration data in this file will be denied to named users.
5.5.6 Restrict access to /etc/ngcp-config/constants.yml to users in
security domain root This file contains values set during the
initialization of the NGCP system. It contains passwords used by
different NGCP functions to connect to other secured subsystems (e.g. DB
or lawful intercept). As such the file has a high security impact and is
read- and writeable to users of security domain root only.
The current default setup of the NGCP does not implement permissions as
required by tightened user security. Required permission settings have
to be applied manually issuing command:
chmod 600 /etc/ngcp-config/constants.yml
Security domain: root User ownership: root Group ownership: root File
permissions: 600
IMPACT: Data in this configuration file are usually entered once during
commissioning of the platform. Only users root or sipwise can edit or
read this file. The customer’s named users have no access to this data.
If for some reason (e.g. after upgrade) the proposed settings are
reverted to default settings, this poses a security risk as credentials
used internally may be leaked to unprivileged users.
Change-Id: I49a2994a227b9c296966c805c9370ae3b067de12
The dpkg-query command already has an interface to fetch the field
value, so there is no need to use --status, grep and then parse the
output.
Change-Id: I13a2f601bde4ca01bfbd7db35f8a1ff11175fba3
An alias for a vlan interface will not have a raw device, stop requiring
it. And an alias for a bond interface might not have miimon, mode or
slaves attributes.
Change-Id: I4b0f547f1c1d02ee826ad97db1a1f44ab91c0e63
The point of an interface alias is to have a different IP, so let's make
that required for any alias, but not for vlan interfaces themselves,
because they might be used for bridging for example.
Change-Id: I7b0fdc0d5cb327c774d6e98ccd7a61c4e6c19b8b
These services files just setup some sysctl settings and should always
be safe to be executed first. In this case we do require them to be
executed first, because many of our services are binding into the shared
IP, which will not be present on the standby node, and would otherwise
fail.
Change-Id: I0df095403461268cdff8517ee5c369355ca9228c
We already know what we have processed and what remains from the
complete list and the new partial list. Use that instead of hard-coding
the directories, which is error-prone as it requires keeping both lists
in sync.
Change-Id: Ia16ac6999beaf89f4c2fb5b74216d0bf17de8414
NGCP uses 'YAML::XS' as only the library to work with YAML files.
Meanwhile pkwalify has a long fallback list:
> @try_order = ('YAML::XS', 'YAML::Syck', 'YAML', 'JSON::XS', 'JSON');
It may hide some issues in validation schema (used by 'ngcpcfg check')
and bypass the testing on Gerrit reviews, but failed to 'ngcpcfg apply'
on the customer's system.
Example:
1) cfg_schema.git had a bug in validation schema committed long time ago
in mr6.1+ , commit b9d69e0b33d316c04d8b70536f1e166fec9a67ac.
The commit passed review and all internal testing:
> https://gerrit.mgm.sipwise.com/#/c/17825/ ->
> https://repoapi.mgm.sipwise.com/job/cfg-schema-ce-validation-docker/849/console.txt
> ...
> + ngcpcfg --validate check
> 2017-12-20 14:34:37: yml configs were validated successfully
The bug was simple, the space was missing in schema:
> "rtcp_logging_facility":{ type: str, required: yes }
The library 'YAML::XS' was not able to load the YAML file.
Meanwhile the library 'YAML' was able to load it and pkwalify happily reported us
"yml configs were validated successfully".
2) Later another developer has noticed the problem with YAML schema here
and fixed it in trunk, commit 4636eadb78e5e91dd6d93378a42a8df2ed84838a.
> - "rtcp_logging_facility":{ type: str, required: yes }
> + "rtcp_logging_facility": { type: str, required: yes }
Unfortunately the fix has been applied in trunk only.
3) Later the third developer committed new code which was valid
for 'YAML::XS' but invalid for 'YAML' library (which is fine, as NGCP uses 'YAML::XS').
Since in trunk pkwalify used 'YAML::XS' the changes passed all the testing:
> commit a79e54fb9fbdf0db61b4dfb9c52e572795e854af
> https://gerrit.mgm.sipwise.com/#/c/28486/ ->
> https://repoapi.mgm.sipwise.com/job/cfg-schema-ce-validation-docker/1987/console.txt
> ...
> 2019-04-05 17:20:08 06828d6f31f5: yml configs were validated successfully
4) Later the third developer backported the commit a79e54fb9fbdf0db
to the previous release mr6.5 (where commit 4636eadb78e was missing).
And nightly upgrade tests has failed on mr6.4->mr6.5 upgrade saying:
> 2019-04-09 04:06:27 sp1: Error: Invalid schema detected for /etc/ngcp-config/config.yml
> Cannot parse </usr/share/ngcp-cfg-schema/validate/config.yml>. Cumulated errors:
> YAML::XS::Load Error: The problem:
> did not find expected key
It happens because pkwalify was not able to load config.yml with any libraries it support.
We have to force pkwalify to use only the library NGCP is using 'YAML::XS'.
This is a commit about it.
Hopefully all the necessary changes have been done by Guillem 2 years ago,
and were uploaded upstream (and even available in default Debian buster):
> https://github.com/eserte/p5-Kwalify/pull/6
Thanks to Guillem Jover for the library fix here and for the help with tracing this issue.
Change-Id: Idbd46b4048a03b5b01a0280a6dcd50406b4222dc
This is necessary when the HA_INT addresses are being changed, the communication
is not possible in that scenario.
Change-Id: I658a4f40e4f5b13d6ac224978999789728d8b915
Change introduced in commit bc1095e596 as part of
the task:
TT#37257 Add option 'ngcpcfg apply --force-all-services' to force all services restart
Change-Id: Iabfd6c78091db98a74a41a0d050f7d090f9443a7
We will gather the actions from within policy-rc.d and then execute all
of them in a single batch.
This will make sure any required restart or reload is performed even if
there has been no template file changes.
Change-Id: I6458936860a86ddb10cd738a8f2770c109d9ed37
The options offload-lro and -tso were required by customers to set up properties
for individual network interfaces, and this tool implements similar options as
helper to set-up network.yml and ultimately /etc/network/interfaces.
Change-Id: Ib6f1c2b6be2c88da10cc8576f83a9f1c72d78d58
Create a new function to set a key conditionally on its value being
defined. And rename the set_yml_config and its arguments to make it
clearer what it is doing.
Change-Id: Idefd89314b5ff76ed0f386135b3791a3309db34c
We should print the deferred service actions that will be performed
before executing them. To add traceability.
Change-Id: I912af91a6b609d20a312182b18f4d959a159de65
These lines have to be commented out, the script/helper is not present during
tests:
#PROCESS '/usr/lib/ngcp-ngcpcfg/get_ngcp_version';
#ngcp_version = out;
Change-Id: I9665f205baf1e28ed2a96e8ab01f0f8f80b838ac
The previous logic had several issues that need to be addressed, when
doing an early exit:
- The temporary file was not getting removed.
- The services sync-state step was not being executed.
This makes the code more future-proof, and fixes the above problems,
by moving the sync-state into its own function, removing the early
exit, and making the service explicit state changes conditional on
the previous early exit conditional.
Change-Id: Ie420cbb94ea6d16a83c7ae3f546cd3eb38435dd8
These commands do not perfom dangerous git calls,
at the same time we slowed them down a lot which
causes very slow nightly testing.
Previously:
> root@web01a:/tmp# time ngcpcfg values eventexport.transfer.password
> ...
> real 0m5.505s
With the new changes:
> root@web01a:/tmp# time ngcpcfg values eventexport.transfer.password
> ...
> real 0m0.127s
Change-Id: I937b4522eb8dbe4f91ada57c978ed4183e7e23fd
Current problem: CE system doesn't pass ngcpcfg
validation if interface name is 'ens0', installation
is aborted on the final 'ngcpcfg apply'.
There were a lot of code changes here and the original logic
if fallback validation has been lost.
The situation here is very unpleasant as all Jenkins tests
reported no issues for us, as interface name is 'eth0' there.
We have enabled validation be default in mr6.5, in the same
time the validation schema we have is not yet perfect, but
we must focus on 'trust' to validation functionality we
introduced otherwise it will not be used by users.
There is not point to have fallback in validation at this stage.
As it can be triggered for random physical and logical interfaces
in the system. Keep in mind mr6.5 generates /etc/network/interfaces
from network.yml, so all interfaces must be described in network.yml.
Change-Id: I1d90c64941bfb7eb8afad51c64c19f72b4ec65d9
ngcpcfg initialise setup git hooks, we should not try to
check/use hooks before they are configured.
Change-Id: Ie74c08fe53f8c8464ee057094637deb03be1d31c
We need to initialize ngcpcfg without HA repo,
to solve chicken-eggs problem about git hooks handking.
Also it will be useful to build glusterfs configs
from tt2 templates and yml configuration.
Then start glusterfs and store HA repo on glusterfs.
In this case ngcp-installer will be able to call:
> ngcpcfg initialise --without_shared
> ngcpcfg set ...
> ngcpcfg build ...
> ngcpcfg init-shared
> ngcpcfg apply "Applying everything together"
In the past it was not so critical problem until we started
taking care about the files permissions which are coming from
ngcp-template-* packages and transferred to POSIX perms via ngcpcfg.
One of the problems there is 'git clone' (which is a part of
HA setup initialisation). It resets file perms if they are not
yet stored into file '.ngcpcfg_perms'.
Change-Id: I3cdc93f309f0b899e626779afb1e71f64b2cb953
Otherwise ngcpcfg fails in trunk with error:
> +03:29:10 (cfg.inc:378): cfg_common_configuration(): ngcpcfg set /etc/ngcp-config/config.yml bootenv.netscript.fallbackfssize=10M
> Save size of 'code' partitions to /etc/ngcp-config/config.yml
> /usr/share/ngcp-ngcpcfg/functions//main: line 68: NGCP_TESTSUITE: unbound variable
Change-Id: I32961cda14550278662acd40ed5863c19412de3c
The li match was triggering for all wildcards and when li mode was not
enabled in some way then we were getting negative results.
The proper way to solve the original problem is to relegate the matching
of the li modes to the end, so that we can properly match on wildcards
and not interfere with other roles.
Change-Id: I7cb559a8dc10e8a832c2e2bcd78f1effc1cc101a
Fixes: commit 236b56d990
- Remove a spurious trailing ';'.
- Make the hook directory reaction conditional on whether we are running
from within a testsuite, to restore its otherwise fatal treatment.
Change-Id: I3f8a5364d7b2f0ac82bfdc132b94641603225f85
Fixes: commit 505d6566d8
Unfortunately git doesn't track POSIX file permissions,
which cause perms drop on some git commands like 'git stash/pull/reset', etc.
We must be sure 'restore-permissions' always executed at the end of 'ngcpcfg'.
Also we must be sure we update .ngcpcfg_perms whenever we start 'ngcpcfg',
otherwise we might loose not-yet committed changes in .ngcpcfg_perms.
Example during the upgrade mr7.0->mr7.1:
1) JFYI, ngcpcfg mr7.1+ copies perms from source tt2 file to the destination config
2) on mr7.0->mr7.1, ngcp-upgrade installs new package ngcp-templates-pro package
with new perms on tt2 files (which are not committed by apt as /etc/ngcp-config
is not covered by 'etckeeper', it is by design).
3) later ngcp-upgrade executes 'ngcpcfg build' to build some config.
4) on PRO/Carrier 'ngcpcfg build' executes 'ngcpcfg pull' first to
ensure there is no outstanding changes on shared git repo
(has been done on Ops request in MT#15803).
5) 'ngcpcfg pull' does 'git stash/pop' before pulling which effectively
reset new perms which came from package in step 1 (as git doesn't track
perms on files, by design).
Summary/fix:
whenever we stash something we should ensure perms are stored as well,
which means we need to generate new .ngcpcfg_perms and store it in stash
together with all the rest local changes.
Change-Id: I5292d4433a39ff2751874d63083f093f19b9cfe2
When running the test suite, in case we need to cover the code paths
that are handling the hooks, we should not assume the files are
installed in the system directories.
Change-Id: I1cd09eaf186d28093eb03ad5e9fc096fe7d63a3e
Since we manage this as a template, the configuration is getting more and more
complex and it's important to have comprehensive tests that help us to detect
problems when we add options and change templates.
Change-Id: Ieff8449f3d6d1cd74a9b44d49e1642a1bf02efc2
The configuration is getting more complex and it's important to have
comprehensive tests that help us to detect problems with the new options.
Change-Id: I5d0004ce5a23b46d48e85294d6fb3bf14073b253
There was a fix for the similar issue in Mantis ticket #7577,
see commit fc5af3bbb2 for more details.
Unfortunately one corner case was missing there:
"git status" doesn't show untracked files in untracked folders by default.
It requires "-u" option with value "all":
>> man git status
> ...
> -u[<mode>], --untracked-files[=<mode>]
> Show untracked files.
>
> The mode parameter is used to specify the handling of untracked files.
> It is optional: it defaults to all, and if specified,
> it must be stuck to the option (e.g. -uno, but not -u no).
>
> The possible options are:
> · no - Show no untracked files.
> · normal - Shows untracked files and directories.
> · all - Also shows individual files in untracked directories.
The ngcpcfg need to see complete path and filename to untracked file
to properly detect .services files and execute them (if configs were changed):
> root@web01a:/etc# git status
> On branch master
> nothing to commit, working tree clean
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> root@web01a:/etc# touch test1
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> test1
> root@web01a:/etc# mkdir test2
> root@web01a:/etc# touch test2/test2_file
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> test1
> test2/
> root@web01a:/etc# git status -uall --porcelain | sed 's/^...//'
> test1
> test2/test2_file
> root@web01a:/etc#
Change-Id: I18286fd76c511061159f90e8aa9841a6e98c7199
Guillem Jover
Alexander Lutay
Richard Fuchs
Sipwise Jenkins Builder
Michael Prokop
Rene Krenn
Andreas Granig
Manuel Montecelo
Marco Capetta
Andrew Pogrebennyk
Mykola Malkov