For some reason some of the existing directories are tested while others
are not.
Add them, as some of them are created in the installer, so they will be
missing in upgrades installing through second root partition, and we
want to catch those cases before we start upgrading real systems.
Change-Id: If74db0fa91a16202d0c1297acc3d20abc3c09962
There were an simplifications in templates.git
commit 0141070a5077b8e5a07c2246d3fa2410584b13e2
which allows to use both v1 and v2 in parallel.
The proper URL for customer interface is now simple:
https://x.x.x.x (on default port and no extra URL location).
Change-Id: I5b9be3d25a874a970809bb94faa28788cd59e2ec
Fixes commit 837cde059a
The current output after the last change is "1", which then is evaluated
as different from "true" (see below), so set "true"/"false" explicitly.
Service: victoria-metrics.service: enabled: doesn't match, expect: [1] found: [true]
Change-Id: If1fec840e70e26a352516ce2ceee983d0f0d74fe
mr9.5/bulsseye+ has to provide access on /etc/mysql/debian.cnf
for mysql2 on port 3308 (Carrier proxy node).
Change-Id: Ic08d210477a5fc7b422ed284699ae3e4bfea1353
In templates.git, the template-conditions for ngcp-service.services.yml are:
victoria-metrics:
group:
- ngcp-upgrade-ignore
- monitoring
systemd: victoria-metrics.service
monit: victoria-metrics
enable: [% monitoring.backend == 'prometheus' && monitoring.prometheus_server == 'victoria-metrics' ? 'yes' : 'no' %]
so the service is enabled in all nodes, not only management, and thus
system-tests complains:
https://jenkins.mgm.sipwise.com/job/system-tester/399308/tapTestReport/
194 - - Service: victoria-metrics.service: enabled: doesn't match, expect: [false] found: [true]
195 - - Service: victoria-metrics.service: running: doesn't match, expect: [false] found: [true]
So setting the same conditions for these tests as for its enablement in
the services config file.
Change-Id: I1be7ca2fc21463685ba0119746ad5bbe9a0c14a0
Quoting from "man bash" about `-E` (AKA errtrace):
| If set, any trap on ERR is inherited by shell functions, command
| substitutions, and commands executed in a subshell environment.
| The ERR trap is normally not inherited in such cases.
To demonstrate the problem see this short shell script:
| % cat foo
| set -eu -o pipefail
|
| bailout() {
| echo "Bailing out because of error" >&2
| exit 1
| }
| trap bailout 1 2 3 6 9 14 15 ERR
|
| foo() {
| echo "Executing magic"
| magic
| }
|
| foo
| echo end
If "magic" can't be executed, then this fails as follows:
| % bash ./foo
| Executing magic
| ./foo: line 11: magic: command not found
But it doesn't invoke the bailout function via trap.
When using `set -eE` (AKA errexit + errtrace), instead of only
`set -e` (errexit), then it behaves as expected though:
| % bash ./foo
| Executing magic
| ./foo: line 11: magic: command not found
| Bailing out because of error
Change-Id: I3feca2af2276cd64cd17f8768eb9bf130e170f92
With newer systemd versions the is-enabled command returns «alias»
and a success exit code for service names that are aliases instead
of «disabled» and an error exit code, as was previously the case
with older versions, which breaks the expected goss checks.
Change-Id: I94d56cf5c9cd32d8b7579a4ce12136295876a472
This reverts commit 4e164bf39f.
The change of enabling "::" as listening interface for sshd is only
applied in new installations, not on upgrades (neither as upgrade step
nor as cfg-schema script), so this test only works on newly installed
systems.
For systems installed before mr9.3 and upgraded, all systems fail at
this point of the tests. Thus, all upgrade tests upgrading to mr9.3 are
failing at the moment because of this (and will keep failing for future
versions as well).
So unless and until there are plans to extend this change to systems
installed before mr9.3 by upgrading this configuration, we have to
revert this test.
Change-Id: Ic5e87651fab3daad9462c7135be81f37ef346950
Check for ngcp-io-scheduler.service instead of io-scheduler.service,
as more recent versions of systemd (as present in Debian/bullseye)
resolve aliases and no longer report them as enabled but as alias.
State on Debian/buster:
| sipwise@spce:~$ systemctl list-unit-files '*scheduler*'
| UNIT FILE STATE
| io-scheduler.service enabled
| ngcp-io-scheduler.service enabled
|
| 2 unit files listed.
State on Debian/bullseye:
| root@spce:~# systemctl list-unit-files '*scheduler*'
| UNIT FILE STATE VENDOR PRESET
| io-scheduler.service alias -
| ngcp-io-scheduler.service enabled enabled
|
| 2 unit files listed.
Change-Id: I57b1b68dce32290055aa216084287aa314fc0de5
It's no longer running as mysqld but as mariadbd process.
The mysql.service is present (as in: reachable) but not a
separate systemd unit any longer:
| root@spce:~# systemctl status mysql.service | head -1
| ● mariadb.service - MariaDB 10.5.8 database server
| root@spce:~# systemctl show -p Names --value mariadb.service
| mariadb.service mysqld.service mysql.service
Fixes:
| not ok 505 - Process: mysqld: running: doesn't match, expect: [true] found: [false]
| not ok 205 - Service: mysql.service: enabled: doesn't match, expect: [true] found: [false]
Change-Id: I0740461e0dd77a3d5d8c285fd66018192e3b6308
Some of our scripts and templates rely on sshd listening on the
localhost addresses (e.g. the port test in monitrc). Normally this is
achieved either by having an ssh_ role assigned to the `lo` interface,
or by listening the addresses explicitly in the `listen_addresses` list.
Add an explicit test for this in case it is missing.
Change-Id: Iee409551e8237fbdec56fe33d1bb63d56d0af802
As of systemd v243 the kernel.core_pattern for systemd-coredump (once
again[1]) no longer includes the hostname, see systemd.git commit:
| commit 47cf786c0a13fccd777c334bed4b1e7b02f18d42
| Author: Franck Bui <fbui@suse.com>
| Date: Fri Jun 21 13:12:41 2019 +0200
|
| coredump: rely on /proc exclusively to get the name of the crashing process
[1] This is similar to a revert of our commit a78509496d,
which mentioned:
| commit f45b8015513d38ee5f7cc361db9c5b88c9aae704
| Author: Jakub Filak <jakub@thefilaks.net>
| Date: Thu Feb 15 12:12:46 2018 +0100
|
| coredump: accept hostname on command line (#8033)
We can't just revert the change though, as we need to distinguish between
Debian 11/bullseye and older releases, while commit a78509496d was
for Debian 10/buster and older releases.
Change-Id: I844bc6f23b1acd2ce583bc59f67ea70956863653
Add support for selecting which prometheus scraper and database server
to use, so we are only going to be running with either prometheus or
victoria-metrics, as they will use the same port.
Change-Id: I24dad2c825cf74946ef807fcc0056880375b25ab
InfluxDB is going to be running either when it is the current monitoring
backend, or when it is not but its data has not yet been migrated. We
want to check both cases so make sure the service is there and contains
data.
Change-Id: I45c71f41b6e2106b1ecf3a2665f669c920e59074
The new installations are going to be using Prometheus, and old ones
will be fine with the new influxdb alternatives. This simplifies the
installation process, as then we do not need to install the prometheus
packages and remove the influxdb packages as independent steps.
Change-Id: Id47b0713fa47a698f1d2ffc0505d0763483ff01b
Merge 3 tests into one which is basically equivalent and simpler. In
this way we reduce noise, since it shortens the file and often the
reported errors (when it fails it will often fail several of the
constraints at once), and errors are more direct and clear, reporting
this:
File: /var/log/debug: mode: doesn't match, expect: ["0640"] found: ["0644"]
instead of:
Command: find /var/log/ -type f -name debug ! -group adm: stdout: patterns not found: [!/./]
which tells you the current permissions deemed wrong or owner/group
used, so it's a bit more info to investigate.
Presumably this was done in a separate way because of thinking that the
"exists-or" didn't work or because sometimes files were created empty
without proper permissions, but this presumably solved now.
Change-Id: I77252743f0a204dffb838d4f7841e05689036c01
We need to truncate the process name to 15 characters, which is the
Linux kernel COMM process field, otherwise the process check will
not match.
Change-Id: Iaad2ae99876970cc779b6b915d9eccc23fcdf46d
The check of /var/log/messages was not strict enough, and it failed:
22 - - Command: find /var/log/ -type f -name messages ! -group adm:
stdout: patterns not found: [!/./] 0 ms
34 - - Command: find /var/log/ -type f -name messages ! -perm 640:
stdout: patterns not found: [!/./] 0 ms
51 - - Command: find /var/log/ -type f -name messages ! -user root:
stdout: patterns not found: [!/./] 0 ms
due to matches of /var/log/asterisk/messages which were not intended, and
doesn't follow the same ownership and permissions rules:
root@spce:~# ls -l /var/log/asterisk/messages
-rw-rw---- 1 asterisk asterisk 0 Dec 10 2019 /var/log/asterisk/messages
So convert /var/log/messages back to be a basic check of type "file", as it was
initially, because now it's guaranteed to be created (by systemd-tmpfiles).
Change-Id: Ibd392a0adef78d09e7b232ef3ccae7f9a7e83f56
Add monitoring of udp:5080 (b2b.bindport) port.
The ports tcp:5088, tcp:8098, udp:5088 are no longer listen by sems-b2b.
Change-Id: I896d7235926db0bdcb45311f38fdf667e7ff0873
These files are created by rsyslog with default conf file. Usually it's
done after the deployment.sh's reboot.
But when NGCP is installed via installer package on existing debian
system there is no reboot between installation of rsyslog (as part of
NGCP) and ngcp-initial-configuration which modifies /etc/rsyslog.conf.
So in this case these files are not created.
So check the ownership and permission of these files only if they exist.
Change-Id: I33be13a4e78baee3926de8bd0135d655b89d5bb6
Since mr9.0 sems-b2b is enabled by default and ngcp-sems is disabled.
Change list of open ports accordingly.
Change-Id: I762f61cce282c84276de0741e4e8c6dbfadf63c4
According to nsservices.yml service ngcp-voisniff should be run on
'proxy' and 'li' roles not 'lb'.
So remove it to prevent false alarm.
Change-Id: Iaf180d8e69ae9f962fa2c750bd3afc8088e12230
Now nginx uses rsyslog for logging but all logs processed by rsyslog
have root:adm ownership so change the test accordingly.
Change-Id: I902f6a588ea60c5a0412a1c0a59fc74a2e36faa8
Since the recent change in templates (commit
b95a6ef4ecb9bc3abe44e54506c029569d5a393c) to let systemd-coredump be
able to write to the /var/lib/systemd/coredump directory, it has started
to change the permissions for that pathname 0755 in some circumstances.
This being a symlink, the permissions do not really matter, so we accept
the 0755 permissions matching the target directory (/ngcp-data/coredumps/)
and the 0777 permissions for a default symlink.
Change-Id: Ifc084cb56541e43bd9fa0245464606c1ca992c97
The ownership of nginx logs is www-data:adm. Also check empty files to
catch wrong permissions as soon as possible.
Change-Id: Icb9e1e1c9590bbef7021e3826a19419f7c21f9ce
We don't actually care whether the log files exists or not, we really
only care about their permissions and ownerships.
Change-Id: I094e9b262b4c4c775023beba79acd3787ec616ac
Remove log files that are not guaranteed to exist. Special case api.log
as it's owned by group _ngcp-li while also being optional.
Change-Id: Ie45a6039bbf120cc3b76ad63b9fd35310966ab48
The name of the variables SKIP_DNS_CHECK_TEST and
SKIP_STRICT_HOSTNAME_TEST so if they are 'true' the check should be
skipped. But in code it has direct branching, if the variable true -
run the test.
So add the negation to checks of these variables.
Change-Id: Iaf1e252dbb30cd2dc624db71742fa2cee8edb888
Manuel Montecelo
Alexander Lutay
Guillem Jover
Sipwise Jenkins Builder
Michael Prokop
Mykola Malkov
Victor Seva
Sergii Kipot
Richard Fuchs