We should print the deferred service actions that will be performed
before executing them. To add traceability.
Change-Id: I912af91a6b609d20a312182b18f4d959a159de65
The previous logic had several issues that need to be addressed, when
doing an early exit:
- The temporary file was not getting removed.
- The services sync-state step was not being executed.
This makes the code more future-proof, and fixes the above problems,
by moving the sync-state into its own function, removing the early
exit, and making the service explicit state changes conditional on
the previous early exit conditional.
Change-Id: Ie420cbb94ea6d16a83c7ae3f546cd3eb38435dd8
We need to initialize ngcpcfg without HA repo,
to solve chicken-eggs problem about git hooks handking.
Also it will be useful to build glusterfs configs
from tt2 templates and yml configuration.
Then start glusterfs and store HA repo on glusterfs.
In this case ngcp-installer will be able to call:
> ngcpcfg initialise --without_shared
> ngcpcfg set ...
> ngcpcfg build ...
> ngcpcfg init-shared
> ngcpcfg apply "Applying everything together"
In the past it was not so critical problem until we started
taking care about the files permissions which are coming from
ngcp-template-* packages and transferred to POSIX perms via ngcpcfg.
One of the problems there is 'git clone' (which is a part of
HA setup initialisation). It resets file perms if they are not
yet stored into file '.ngcpcfg_perms'.
Change-Id: I3cdc93f309f0b899e626779afb1e71f64b2cb953
When running the test suite, in case we need to cover the code paths
that are handling the hooks, we should not assume the files are
installed in the system directories.
Change-Id: I1cd09eaf186d28093eb03ad5e9fc096fe7d63a3e
There was a fix for the similar issue in Mantis ticket #7577,
see commit fc5af3bbb2 for more details.
Unfortunately one corner case was missing there:
"git status" doesn't show untracked files in untracked folders by default.
It requires "-u" option with value "all":
>> man git status
> ...
> -u[<mode>], --untracked-files[=<mode>]
> Show untracked files.
>
> The mode parameter is used to specify the handling of untracked files.
> It is optional: it defaults to all, and if specified,
> it must be stuck to the option (e.g. -uno, but not -u no).
>
> The possible options are:
> · no - Show no untracked files.
> · normal - Shows untracked files and directories.
> · all - Also shows individual files in untracked directories.
The ngcpcfg need to see complete path and filename to untracked file
to properly detect .services files and execute them (if configs were changed):
> root@web01a:/etc# git status
> On branch master
> nothing to commit, working tree clean
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> root@web01a:/etc# touch test1
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> test1
> root@web01a:/etc# mkdir test2
> root@web01a:/etc# touch test2/test2_file
> root@web01a:/etc# git status --porcelain | sed 's/^...//'
> test1
> test2/
> root@web01a:/etc# git status -uall --porcelain | sed 's/^...//'
> test1
> test2/test2_file
> root@web01a:/etc#
Change-Id: I18286fd76c511061159f90e8aa9841a6e98c7199
Although the NGCPCFG_RUNNING environment variable might be useful for
some of the subprocesses, it is definitely not useful for anything
that gets started via systemd. In that case we have to transmit this
information via the filesystem.
In the future we should probably extend this with some proper locking,
but for our current problem with ngcp-eaddres we'll still want to
completely avoid running if such file exists, not just prevent
multiple concurrent executions.
Change-Id: I0546620d8ef4263177c5d04f7d2546345c42fbb6
This will make sure any service that should not be running is stopped,
and any that should is started.
Change-Id: I7ab181b66ed3554c8f93f2d9f46dc08ee087a87d
'git status' output has been changed in 2.9 which causes some troubles for us
(we didn't change all the places, we missed ngcpcfg-ha.git repo,
better to use common code in all the places).
Change-Id: Ie664af89ee6d6b31fb3011e763107c41bf4cf00f
It was impossible to set value like '10G' previosuly:
>> ngcpcfg set /etc/ngcp-config/config.yml bootenv.netscript.fallbackfssize=10M
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 28, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 30, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 31, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 33, near "10M"
> (Missing operator before M?)
> Bareword found where operator expected at /tmp/tmp.px3MpkUFZS line 34, near "10M"
> (Missing operator before M?)
> syntax error at /tmp/tmp.px3MpkUFZS line 28, near "10M"
> syntax error at /tmp/tmp.px3MpkUFZS line 30, near "10M"
> syntax error at /tmp/tmp.px3MpkUFZS line 39, near "}"
> Execution of /tmp/tmp.px3MpkUFZS aborted due to compilation errors.
The 'case' statement doesn't support regular expresions in Bash,
let's rewrite them into linear check.
Change-Id: I29ced837fca0abdc3cbe1ea18c614c4c1dd6780c
At the end of installer's 'configuration' section we need
to restart all services were touched during the installation
(otherwise we need to restart the node completely).
The option helps us in this case as etckeeper is
not reliable enough during the installation.
P.S. we pass options from 'apply' to 'services' now,
we should remove 'Unsupported option(s) given: $*' check,
otherwise it is failing as 'apply' requires commit message,
which cause error on 'services' level: 'Unsupported option(s) given'
> 2018-09-13 18:08:33: Error: Unsupported option(s) given: my commit message
> 2018-09-13 18:08:34: Did you mean '--dry-run'?
Change-Id: I5dac44cd87dc1fc90f81e4ac3a14af32c00f825e
We need to have some reproducible order of services restart,
otherwise this might lead to different results, depending on the order.
Also unifyer() looks too heavy here since the files can be easily
unified in a moment of writing them into ${TMPFILE}.
Change-Id: I0a683a7d57cd52b74cba0ee9a285f84f77f3dc20
It was initially introduced in the commit:
> commit 7d422b7878
> Author: Victor Seva <vseva@sipwise.com>
> Date: Fri Aug 22 17:03:39 2014 +0200
>
> MT#8763 check for .git before running the services
>
> On a multi directory config, the services depends on git. Check
> first if the git repository exists and warn about it
The subshell here was necessary to isolate 'cd $dir' call.
In the same time the "proper" bash way is to use pushd/popd.
For the history, 'shellcheck disable' was muting the warning:
> This only exits the subshell caused by the (..) group. [SC2106]
Change-Id: I3546fa9615858dbbc4c0f91d74de003e22837d55
Previously 'systemd daemon-reload' and 'systemctl preset-all'
were executed only if changes detected in configs.
Otherwise the early exit happened if no .service files were triggered:
> DEBUG: No services file(s) reported - nothing to do.
Move the code into the function systed_daemon_reload_preset and
execute it on the top of the script. Also print info message
informing users about reloaded systemd and newly preset units.
It should provide better visibility here.
Change-Id: I992af9fb274ea93a37b812a51cebcd7af5c54133
Otherwise the code in between the functions can be easily missed
which cause issues as described in the ticket and will be fixed
in the following commit.
Change-Id: I57decfcbcd41691d35d085b13881e1e6b5208f6e
This will collect all service actions, synthetize them into their
minimal expression and execute them in a single batch per action.
Change-Id: I950d5db32e0ec6327964faac4ce8f15449f90e90
Otherwise ngcp-installer failed to install PRO/Carrier as
we build the file /etc/ngcp_mgmt_node the very first time on the first MGMT node:
> +08:22:48 cfg_build_templates
> +08:22:48 cfg_build_configs /etc/ngcp_mgmt_node
> +08:22:48 build_configs=($1)
> +08:22:48 declare -a build_configs
> +08:22:48 log_info 'Generating default configuration files /etc/ngcp_mgmt_node'
> +08:22:48 tee -a /tmp/ngcp-installer.log
> +08:22:48 echo 'Generating default configuration files /etc/ngcp_mgmt_node'
> Generating default configuration files /etc/ngcp_mgmt_node
> +08:22:48 ngcpcfg build /etc/ngcp_mgmt_node
> cat: /etc/ngcp_mgmt_node: No such file or directory
> +08:22:48 log_die 'Error running '\''ngcpcfg build'\'''
Checking the file availability is not enough here, the next error is:
> (sp1)root@sp1:/# ngcpcfg build
> /usr/share/ngcp-ngcpcfg/scripts//check: line 83: NGCP_IS_MGMT: unbound variable
> (sp1)root@sp1:/# cat /etc/default/ngcp-roles
> NGCP_TYPE="sppro"
> (sp1)root@sp1:/#
It happens because /etc/default/ngcp-roles is a fake one at this stage.
Handling NGCP_IS_MGMT properly is also not enough:
> (sp1)root@sp1:/# ngcpcfg build
> 2018-07-16 16:49:32: Error: Remote origin of ngcpcfg is '/mnt/glusterfs/ngcpcfg-share', expected: 'sp:/mnt/glusterfs/ngcpcfg-share'.
> 2018-07-16 16:49:32: Error: NOTE: execute `cd /etc/ngcp-config ; git remote set-url origin 'sp:/mnt/glusterfs/ngcpcfg-share'` to adjust setting.
> 2018-07-16 16:49:32: Error: NOTE: perform `ngcpcfg clean --all` to recreate local master branch from remote.
> (sp1)root@sp1:/#
Which happens because the peer is not yet configured at the moment (first node installation).
Introducing new internal option '--no-check-origin' to skip the test into installer.
Change-Id: I0265c65f45972e92ca92320871a7ef29f8904fec
If the file /etc/hosts gets handled by templates, as ongoing changes
attempt to do, there's a problem because /etc/hosts gets truncated to
create a new file when it's needed to generate itself, to resolve the
"localhost" string as hostname, so the generation fails and /etc/hosts
gets empty at that point, affecting the system until restored.
The simplest way to break that cycle is to switch to Unix sockets.
Change-Id: I1e33ead6a134625346b9cf1beb09a7bdbfdfc8d2
Remove any stale .wants/ relationships as systemd does not always seem
to be able to cleanup those by itself. This makes sure we always start
from a clean slate.
Change-Id: I1b241d4530c45658b7245cdbca33e34dde5beaf6
We should be as safe as possible here (at least at the beginning)
to prevent 'patch' command patching wrong code parts.
Change-Id: Ide012169bec1f329309baae32fe186fd77cb9592
The new function will simplify initial customtt->patchtt
migration for end users. Some code was merged to be generic.
Also more tests were add here.
Change-Id: I7719f45275018818b2db82f6deee5b7428670a29
The 'ngcpcfg' received support for 'patchtt' files, like
> /etc/ngcp-config/templates/etc/foo/bar.patchtt.tt2
Those 'patchtt' are going to be applied on default 'tt2 template' file:
> /etc/ngcp-config/templates/etc/foo/bar.tt2
and produce 'customtt' on 'ngcpcfg patch':
> /etc/ngcp-config/templates/etc/foo/bar.customtt.tt2
Further 'customtt' will be used to overwrite 'tt2 templates'
on 'ngcpcfg build' or 'ngcpcfg apply'.
NOTE: 'ngcpcfg patch' is executed automatically on every 'ngcpcfg build'.
It should allows to update ngcp-templates easily and support
local modifications without the pain (until the patches can be applied).
Change-Id: Ice4369386313c5d33e4d498346345eade6f3d0d7
Under systemd we need to issue "daemon-reload" after applying
templates, to avoid spurious warnings when adding/changing/removing
systemd unit files.
Change-Id: Ifd474bec25b94ee9cb1c247b1390103a29c7b7b3
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ie51f1c79859d40ef0877fc0ab75f86ee72e14ea4
This script will validate the network.yml based on a schema constructed
from information only available from the network.yml file itself. This
way we can do the strictest validation, which we could not do before.
Change-Id: I32714e678e901e58d70e4253bcc61a147494c225
This option is essential if we're testing from within the git
repository and aren't standing on active 'master' branch.
Change-Id: I75b62d071116313317c69ad496e4425d6e98ad5a
At the moment we provide no output for the end users, which
confuses them for the checks with and without the validation.
We should clearly report to users what has been checked.
Change-Id: Icc61744637345a1dc51fe8d8dd9bd3fd7f86824d
It is hard to clean ngcpcfg framework for users with
limited git knowledge, lets introduce action 'clean'.
It should allows users easily reset to 'previous safe state'
in the case 'if something went wrong'.
Also remove old and unreliable error handling hint from manuals,
as we have switched to fast-forward rebase long time ago.
Change-Id: I961e681d55cac15ba8d772b9345c668218313bf4
When installing Debian/stretch we fail at this stage:
| ngcpcfg --no-db-sync commit
On first glance it looks like a mariadb issue, though actually
it's the git status check inside ngcpcfg that's failing. The
reason is that starting with git v2.9.3 there's a different
output in its "status" output:
| commit 2a0e6cdedab306eccbd297c051035c13d0266343
| Author: Lars Vogel <lars.vogel@gmail.com>
| Date: Thu Jun 9 20:19:30 2016 +0200
|
| Use "working tree" instead of "working directory" for git status
|
| Working directory can be easily confused with the current directory.
| In one of my patches I already updated the usage of working directory
| with working tree for the man page but I noticed that git status also
| uses this incorrect term.
|
| Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
| Signed-off-by: Junio C Hamano <gitster@pobox.com>
In Debian/stretch we've 2.11.0-2 and are affected by this change.
Adjust the `git status` checks accordingly to support git in all
its versions.
NOTE: we should consider switching to `git status --porcelain`
(see git commit fc5af3bbb2 for
some history about it) once we've the testsuite in place.
Change-Id: If4223a35083d279e746dce5a80bcb86b52f64025
/var/log/ngcp/ngcpcfg.log might not contain error messages from
ngcpcfg's sync-db script. This might be confusing when comparing
output from the command line with the logged actions from
ngcpcfg.log. As sync-db doesn't log all error messages to stderr
anyway and we ignore exit code of the sync-db script as well (on
purpose) let's just redirect stderr and stdout to the same file
descriptor and then provide output to the console as well as
syslog's ngcpcfg tag.
Output before this change:
| root@sp1:~# ngcpcfg commit "before"
| 2016-10-25 22:12:20: Executing sync-db:
| DBI connect('database=provisioning;host=localhost;port=3306','sipwise',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/ngcp-ngcpcfg/helper//sync-db line 70.
| Error: Could not connect to database 'provisioning' at 'localhost:3306' as 'sipwise': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
| 2016-10-25 22:12:20: OK: nothing to commit.
| 2016-10-25 22:12:20: Synchronizing MySQL grants/credentials
| Can't connect to MySQL database 'mysql': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/sbin/ngcp-sync-grants line 66.
| root@sp1:~# tail -f /var/log/ngcp/ngcpcfg.log
| [...]
| Oct 25 22:12:20 sp1 (notice) ngcpcfg: Executing sync-db:
| Oct 25 22:12:20 sp1 (notice) ngcpcfg: OK: nothing to commit.
| Oct 25 22:12:20 sp1 (notice) ngcpcfg: Checking state of /etc files
| Oct 25 22:12:20 sp1 (notice) ngcpcfg: OK: nothing to commit.
| Oct 25 22:12:20 sp1 (notice) ngcpcfg: Synchronizing MySQL grants/credentials
Output after this change gets applied:
| root@sp1:~# ngcpcfg commit "after"
| 2016-10-25 22:12:59: Executing sync-db:
| DBI connect('database=provisioning;host=localhost;port=3306','sipwise',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/ngcp-ngcpcfg/helper//sync-db line 70.
| Error: Could not connect to database 'provisioning' at 'localhost:3306' as 'sipwise': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
| 2016-10-25 22:12:59: OK: nothing to commit.
| 2016-10-25 22:12:59: Synchronizing MySQL grants/credentials
| Can't connect to MySQL database 'mysql': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/sbin/ngcp-sync-grants line 66.
| root@sp1:~# tail -f /var/log/ngcp/ngcpcfg.log
| [...]
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: Executing sync-db:
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: DBI connect('database=provisioning;host=localhost;port=3306','sipwise',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/ngcp-ngcpcfg/helper//sync-db line 70.
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: Error: Could not connect to database 'provisioning' at 'localhost:3306' as 'sipwise': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: OK: nothing to commit.
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: Checking state of /etc files
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: OK: nothing to commit.
| Oct 25 22:12:59 sp1 (notice) ngcpcfg: Synchronizing MySQL grants/credentials
Change-Id: Iabb0c0ab24e212b79b0dfa85947cb274a161cc00
We need to use some folder on glusterfs to ensure it is reachable.
/mnt/glusterfs/mgmt-share is good enough here as /mnt/glusterfs/shared_config
is no longer available.
Change-Id: I9f604db9604af41405cfd7b088abb332a463fe4d
Git doesn't track file permissions (except for the executable
flag). For sensitive data (like the 'ssl' directory and file
'constants.yml' with passwords included) we've to prevent
non-root users from accessing those files.
hooks/pre-commit is inspired and based on the implementation
as present in etckeeper (and luckily we're license compatible)
and takes care of storing the file permissions inside file
/etc/ngcp-config/.ngcpcfg_perms.
The restore-permissions helper script takes care of restoring the
permissions after cloning the ngcpcfg repository via ngcpcfg
itself (being actions decrypt, pull (PRO-only) + initialise
(PRO-only)). It can be executed manually as well via
`usr/share/ngcp-ngcpcfg/helper/restore-permissions /etc/ngcp-config/`
(or wherever the according ngcpcfg repository is placed at).
Regarding the commit integration: git(1) itself doesn't track
file permissions, so we can't detect changes to file permissions
using git itself. Our new pre-commit hook records file
permissions via the .ngcpcfg_perms file. Now by just invoking it
during 'ngcpcfg commit' time we can ensure that even if there
have been any file permission changes in the working directory
the file .ngcpcfg_perms is then up2date and committed.
JFTR: The solution via the git pre-commit hook ensures that no
matter whether you're using 'ngcpcfg commit …' or 'git commit …'
you always get the file permissions handled via .ngcpcfg_perms.
Now if you want to change file permissions in a clean working
directory and commit *without* using 'ngcpcfg commit' but
directly via git itself then you've to use 'git commit
--allow-empty ...' and thanks to the pre-commit hook the file
.ngcpcfg_perms will still be up2date.
Change-Id: I84d608585c626b52112ff649893e232e441c59d8
Guillem Jover
Andrew Pogrebennyk
Mykola Malkov
Alexander Lutay
Michael Prokop
Richard Fuchs
Manuel Montecelo