Add support for selecting which prometheus scraper and database server
to use, so we are only going to be running with either prometheus or
victoria-metrics, as they will use the same port.
Change-Id: I24dad2c825cf74946ef807fcc0056880375b25ab
InfluxDB is going to be running either when it is the current monitoring
backend, or when it is not but its data has not yet been migrated. We
want to check both cases so make sure the service is there and contains
data.
Change-Id: I45c71f41b6e2106b1ecf3a2665f669c920e59074
The new installations are going to be using Prometheus, and old ones
will be fine with the new influxdb alternatives. This simplifies the
installation process, as then we do not need to install the prometheus
packages and remove the influxdb packages as independent steps.
Change-Id: Id47b0713fa47a698f1d2ffc0505d0763483ff01b
Merge 3 tests into one which is basically equivalent and simpler. In
this way we reduce noise, since it shortens the file and often the
reported errors (when it fails it will often fail several of the
constraints at once), and errors are more direct and clear, reporting
this:
File: /var/log/debug: mode: doesn't match, expect: ["0640"] found: ["0644"]
instead of:
Command: find /var/log/ -type f -name debug ! -group adm: stdout: patterns not found: [!/./]
which tells you the current permissions deemed wrong or owner/group
used, so it's a bit more info to investigate.
Presumably this was done in a separate way because of thinking that the
"exists-or" didn't work or because sometimes files were created empty
without proper permissions, but this presumably solved now.
Change-Id: I77252743f0a204dffb838d4f7841e05689036c01
We need to truncate the process name to 15 characters, which is the
Linux kernel COMM process field, otherwise the process check will
not match.
Change-Id: Iaad2ae99876970cc779b6b915d9eccc23fcdf46d
The check of /var/log/messages was not strict enough, and it failed:
22 - - Command: find /var/log/ -type f -name messages ! -group adm:
stdout: patterns not found: [!/./] 0 ms
34 - - Command: find /var/log/ -type f -name messages ! -perm 640:
stdout: patterns not found: [!/./] 0 ms
51 - - Command: find /var/log/ -type f -name messages ! -user root:
stdout: patterns not found: [!/./] 0 ms
due to matches of /var/log/asterisk/messages which were not intended, and
doesn't follow the same ownership and permissions rules:
root@spce:~# ls -l /var/log/asterisk/messages
-rw-rw---- 1 asterisk asterisk 0 Dec 10 2019 /var/log/asterisk/messages
So convert /var/log/messages back to be a basic check of type "file", as it was
initially, because now it's guaranteed to be created (by systemd-tmpfiles).
Change-Id: Ibd392a0adef78d09e7b232ef3ccae7f9a7e83f56
Add monitoring of udp:5080 (b2b.bindport) port.
The ports tcp:5088, tcp:8098, udp:5088 are no longer listen by sems-b2b.
Change-Id: I896d7235926db0bdcb45311f38fdf667e7ff0873
These files are created by rsyslog with default conf file. Usually it's
done after the deployment.sh's reboot.
But when NGCP is installed via installer package on existing debian
system there is no reboot between installation of rsyslog (as part of
NGCP) and ngcp-initial-configuration which modifies /etc/rsyslog.conf.
So in this case these files are not created.
So check the ownership and permission of these files only if they exist.
Change-Id: I33be13a4e78baee3926de8bd0135d655b89d5bb6
Since mr9.0 sems-b2b is enabled by default and ngcp-sems is disabled.
Change list of open ports accordingly.
Change-Id: I762f61cce282c84276de0741e4e8c6dbfadf63c4
According to nsservices.yml service ngcp-voisniff should be run on
'proxy' and 'li' roles not 'lb'.
So remove it to prevent false alarm.
Change-Id: Iaf180d8e69ae9f962fa2c750bd3afc8088e12230
Now nginx uses rsyslog for logging but all logs processed by rsyslog
have root:adm ownership so change the test accordingly.
Change-Id: I902f6a588ea60c5a0412a1c0a59fc74a2e36faa8
Since the recent change in templates (commit
b95a6ef4ecb9bc3abe44e54506c029569d5a393c) to let systemd-coredump be
able to write to the /var/lib/systemd/coredump directory, it has started
to change the permissions for that pathname 0755 in some circumstances.
This being a symlink, the permissions do not really matter, so we accept
the 0755 permissions matching the target directory (/ngcp-data/coredumps/)
and the 0777 permissions for a default symlink.
Change-Id: Ifc084cb56541e43bd9fa0245464606c1ca992c97
The ownership of nginx logs is www-data:adm. Also check empty files to
catch wrong permissions as soon as possible.
Change-Id: Icb9e1e1c9590bbef7021e3826a19419f7c21f9ce
We don't actually care whether the log files exists or not, we really
only care about their permissions and ownerships.
Change-Id: I094e9b262b4c4c775023beba79acd3787ec616ac
Remove log files that are not guaranteed to exist. Special case api.log
as it's owned by group _ngcp-li while also being optional.
Change-Id: Ie45a6039bbf120cc3b76ad63b9fd35310966ab48
The name of the variables SKIP_DNS_CHECK_TEST and
SKIP_STRICT_HOSTNAME_TEST so if they are 'true' the check should be
skipped. But in code it has direct branching, if the variable true -
run the test.
So add the negation to checks of these variables.
Change-Id: Iaf1e252dbb30cd2dc624db71742fa2cee8edb888
Also check that there are no expired keys
The key was used to sign repos for old unsupported releases.
Change-Id: Ia5dd91c4d8b2e54f4fc18626ef3968c95265bdf6
Sometimes there is divergence in schemes which are inevitable or
harmless.
So add exceptions' list which is easy to maintain. The format is the
following:
=================================
<element-type>/<schema-name>/<element-name>/<element-attribute>
Where:
element-type - tables, columns, views, etc
schema-name - name of the schema
element-name - name of the element (table, index, etc)
=================================
F.e.:
views/ldap/ldap_entries/view_definition
For columns columns/<schema-name>/<table-name>_<column-name>
columns/billing/table1_column1/is_nullable
Add 1st exception from TT#74728. The reason of this diff is that view
was created without schema name on sp1 but the replication statement was
written with schema name.
Change-Id: I29dd109cefb560fe89d92a430cd6522e4513f7d5
Use compare_dbs as a wrapper to check all the used schemes.
Add different formats of output to compare_dbs.pl - tap and human
readable one.
Process all the schemes in one run of compare_dbs.pl and do it in one db
connection.
If the tap formatter is used - output the result of all schemes in a
single file.
Change-Id: I2696680aa30b56658f130bd1cea116099c086753
Nowadays goss yml files are generated by ngcpcfg. But their content
depends on whether the node is active or not, which can change
dynamically during the lifetime of the generated files from the
templates. Which will make the goss checks fail when the node state
changes.
Switch the node state into a run-time variable, and use go template
support to parametrize this into the goss checks.
For the Template Toolkit values that get instantiated within the go
template conditionals, we need to make sure we always set a boolean
value, as the Template Toolkit will return undef on false values,
which would be considered a non-existining argument. Even though the
has_role method will always return either 0 or 1, we still force
a boolean value to make the code future-proof in case more logic is
added on the same check, which would then turn it into a Template
Toolkit boolean evaluation and possibly return nothing on false.
Change-Id: I19c3ef48f09c8d2e79613ef118adc362399a6e78
The semantics of the environment variables are different, as they are
always considered strings. The variables file makes it possible to
specify types, given that it's a yaml file. So the SKIP variables
will now be proper booleans.
Change-Id: Ibbc02c047fd743b8bff9a9da52cffd76db5d7524
The system test package is installed on a system so it's easier to
manage code and run the test as it's branched and no need to copy
scripts to system host.
Change-Id: I9ad3ff7fb16dd843d81a4c9ecfbbe1c0d42fe1c6
We get dangling symlink errors on a fresh installation as
/etc/ngcp-system-tests has not been generated yet, but we do not
currently have any users of the old /etc/ngcp-tests, so we can just
remove it.
Change-Id: I6c3369b6c4c948135ffcef608a1d358729b06e82
We are not matching to new prosody version 0.11.2-1 here as it can
be changed any time by Debian (as we migrated to upstream prosody version).
Also goss doesn't have support for regexps for 'versions' field,
so we cannot check something like:
> prosody:
> installed: true
> versions:
> - /0.11.2-.*/
After a long trying with goss 'matches/match-regexp/semver-constraint'
I decided to stop on the current version, where we inside old version is not available.
Change-Id: I626be7a23fd2b31d468f4cd9411530b24eb7216b
Let us introduce a system-wide symlink from /var/backup to /ngcp-data/backup, to
fix recurrent problems like the missing /var/backup/cdr or
/var/backup/ngcp-upgrade backups taking too much space.
This should have happened when changing the partition schema to use /ngcp-data/
and have a root partition with minimal amount of space, because there were
several paths migrated to /ngcp-data like /var/backup, and we failed to migrate
all those paths in sync with the change to the partitioning and also failed to
have a back-up solution like this symlink.
Some (hopefully most, or all!) of the problems have been sorted out by now, but
they've been biting us years after the change, like this TT#72908, so maybe it
can still save some pain in the future, and there's little cost to it.
Change-Id: Ief4ba33bf57bb5f02cc54b000b64540667ebba48
Guillem Jover
Sipwise Jenkins Builder
Manuel Montecelo
Mykola Malkov
Victor Seva
Sergii Kipot
Richard Fuchs
Alexander Lutay