* --db-scripts-dir option enables override of the
default database scripts directory location
* docstring a bit more compact formatting
* apply_revisions(): improve non-existing directory
detection and error output
Change-Id: Ic1d74244244df6406fb9a9258b7c36b971497513
* it's not a responsiblity of the script to start/stop
any external services and therefore, mariadb start
is removed
Change-Id: I220ebab7764a7016b15e6f08a8ac5ae01fe08523
* new command line option --db-defaults-conf-file
* new command line option --db-connect-method
* connect_db(): improve socket or config selection
* add mysql connect method as mysql command line options when
run_cmd('mysql' ...) is invoked, so that the mysql console connection
matches the pymysql connection
Change-Id: Ice8ff1e79c133d634f3de5bce204c7889f69b42d
* add new command line option --db-socket that overrides
the default socket file location
* MYSQL_SOCKET env is now considered as the socket file location
replacement, only if there is no --db-socket option set
* config.skip_sync_db is renamed into config.skip_ro_db_sync to
be consistent with the command line option
Change-Id: I4f113cdc8ad430f17ce022cdbbceef44e769857d
* ngcp-update-db-schema is rewritten in python. that simplifies
the code, as well as enables flexibility and more complex features
support, as well as performance improvements
* full strict mypy compliant
* flake8 compliant
* google docstring complied
* add .down scripts support
* add docstring full documentation
* add mypy, isort, flake8 checks during build phase
* add command line arguments that support --to-revision, to stop
.up/.down scripts execution when it reaches a certain revision
* better support for ngcp-sync-db
* preserve AUTOMATED_INSTALL_MODE and SKIP_SYNC_DB env vars support
for backward compatibility, in the future it is recommented to
switch to the existing (--automated, --skip-ro-sync-db) options
* experimental batch mode support where instead of applying changes
one by one, they are accumulated into a file along with db_schema
revision tracking changes and applied at once
Change-Id: I4632393d52d1a60c945b4d87a779f1a073ed9a5f
* provisioning.voip_fax_data replication to 3308
is now ignored because it is not in use on proxies
Change-Id: I8d7af9d3fa44f684bb7557cd66133946a802f5c7
For preparation of dumps (sql and json) it's useful to use separate
mysql instance which is run and handled manually.
So add support of such instance.
Add variable MYSQL_SOCKET which shows that we want to connect to custom
mysql instance via defined socket.
In such a case don't try to start a service mysql.
Change-Id: I3f40a3857125fc72158aabf277f892c7cd541bf6
* provisioning.autoprov_firmwares_data is ignored
during master-slave 3308 replication sync, as this
table can be potentially large and not used on the
proxy 3308 instance
Change-Id: I19f7feeb2f12f10abc6d6110fe691583b0d560b4
The bullsyeye-based MariaDB 10.5.9 has invalid root password set by default.
It is not possible to connect DB instance from network, the only option is
to connect DB instance through the socket.
The ngcp-installer calls ngcp-sync-db to replicate db01:3306->prx01:3308
on proxy node for the Carrier case. We have to open networking root access
temporary. ngcp-sync-db will replicate 'root' password from db01 the next moment.
Also update '1>/dev/null 2>/dev/null' to Sipwise style '&>/dev/null'.
Change-Id: I12ff71efdd8283531f03ceaf87345f3cc4e71419
After migration to bullseye, ngcp-update-db-schema is broken
inside Docker containers (Jenkins PPA tests):
> https://jenkins.mgm.sipwise.com/job/db-schema-test-db-up-docker/629/console
> ...
> + ngcp-update-db-schema -f
> mysql: unrecognized service
> Error: can't connect to local MariaDB port 3306 using '/etc/mysql/sipwise_extra.cnf'
> ...
Historically, the script ngcp-update-db-schema starts DB (if
it is not running) using the command "service mysql start".
The wrapper 'service' has to use SysV init script as
systemd unit cannot be used inside Docker.
The package mariadb provided backward compatibility symlink
in buster, but it has been removed in bullseye:
> root@11bb38c12e16:/# ls -la /etc/init.d/mysql
> ls: cannot access '/etc/init.d/mysql': No such file or directory
> root@11bb38c12e16:/# ls -la /etc/init.d/mariadb
> -rwxr-xr-x 1 root root 5966 Dec 18 20:25 /etc/init.d/mariadb
> root@11bb38c12e16:/#
The fix here is to use new/proper service name 'mariadb'.
Change-Id: Icf0d663f4a0d5256e1fc403d4fc0f3dd484ebbab
* determine the master db node name to use
for ngcp-sync-db master-slave replication restoration
if the CENTRAL_DBHOST is set by default to db01 only.
if the current nodename ends with "a" then db01a is used,
otehrwise db01b is used as the master host. that is
to make sure that during the upgrade the correct "side"
is used to replicate the data from and avoid lossy
statements that break replication.
Change-Id: I1a51bd0d975390d9269ef5208ce8275f9e055a1f
* --init-replication is required now for the central-db
because it uses multiple sources and they need to be
readjusted after the data sync
Change-Id: I10c12d479dbe7d73e6e489d3aa9ab37b7216af54
We are in the process of removing the obsolete init scripts, so trying
to use them directly will make these operations not work at all.
Change-Id: Ifff400388a947294ca661b3b8bad793b755ee830
* host and port by ngcp-sync-db is now used from
/etc/default/ngcp-db to ensure the correct R/O instance db
Change-Id: Ib7d9dde15cc4e5fa4ca95bb2d5b6ea508e12f89a
* ngcp database contains a 'timezone' table that
is used in replicated billing database
and therefore, needs to be also replicated
Change-Id: Iaaa23ff8cece68f4e9944a690bfc47c243d64f9f
We suppress some warning here to mark the repo as clean
while further cleanup can be done to remove some labels
'shellcheck disable'.
The first step here is to prevent new errors appearing.
Change-Id: I23ca9dd6fdf80bab6b9ec465dd4a53983e34a52a
* extra ngcp-sync-db options in AUTOMATED_INSTALL_MODE
* ngcp-sync-db is invoked only once now (after all revisions)
Change-Id: I6db3c29f186e1a61df1ccce17b0a646efaa1b055
Running ngcp-check_active for each single script causes quite
some overhead, it should be enough to just check for result of
ngcp-check_active *once* and then re-use the result.
Scripts that need to be executed on all hosts (the ones with
filename *_not_replicated.up) but have been executed on sp1
already are listed as "already executed" on sp2 if the
ngcp-check-rev-applied call doesn't use the node specific check.
So generate a list of *_not_replicated.up files and run
node specific checks while generating the list of missing
revision scripts.
Without this change the following ordering is present:
3.0: http://paste.mgm.sipwise.com/show/513/
vs
trunk: http://paste.mgm.sipwise.com/show/514/
So /usr/share/ngcp-db-schema/db_scripts/base/0070_not_replicated.up
is executed between 0060_* and 0080_* on ngcp 3.0 but after 6746.up
(the last script without the _not_replicated.up suffix) and
together with all the other *_not_replicated scripts before the
next bunch of scripts (being /usr/share/ngcp-db-schema/db_scripts/diff/).
This adds a short penalty on execution time because we re-run
ngcp-check-rev-applied for the *_not_replicated scripts but it's
the only way to get it right if we want to execute all db-schema
scripts in sequential order sorted by their ID revision number.
Without this change the following ordering is present:
3.0: http://paste.mgm.sipwise.com/show/513/
vs
trunk: http://paste.mgm.sipwise.com/show/514/
So /usr/share/ngcp-db-schema/db_scripts/base/0070_not_replicated.up
is executed between 0060_* and 0080_* on ngcp 3.0 but after 6746.up
(the last script without the _not_replicated.up suffix) and
together with all the other *_not_replicated scripts before the
next bunch of scripts (being /usr/share/ngcp-db-schema/db_scripts/diff/).
This adds a short penalty on execution time because we re-run
ngcp-check-rev-applied for the *_not_replicated scripts but it's
the only way to get it right if we want to execute all db-schema
scripts in sequential order sorted by their ID revision number.
This dramatically speeds up the ngcp-update-db-schema process if
everything has been applied already or if just a few single
scripts have to be executed.
Execution times for old mode vs new (batch) mode with all present
scripts executed already:
| root@spce:~# time { /usr/sbin/ngcp-update-db-schema.orig >/dev/null ; }
|
| real 0m53.281s
| user 0m45.439s
| sys 0m5.100s
vs.
| root@spce:~# time /usr/sbin/ngcp-update-db-schema
| [ ok ] Starting MySQL database server: mysqld already running.
| Synchronizing passwords with MySQL
|
| real 0m4.096s
| user 0m1.812s
| sys 0m0.472s
The main behaviour of ngcp-check-rev-applied hasn't been changed,
it's backwards compatible (so also cfg-schema which uses
ngcp-check-rev-applied continues to work as it is), now it's just
possible to provide multiple arguments to the "--revision" option
and its output is parsed by ngcp-update-db-schema then.
So we had the 9999->10000 switch in our svn now.
Scripts >=10000 need to be executed after <=9999.
This code has never happend. Nothing to see here, move along… ٩(͡๏̯͡๏)۶
From: Michael Prokop <mprokop@sipwise.com>
The way it is now won't provide any benefit, so let's just drop it.
Let's see how we can resolve this issue in the installer...
Thanks for review + comments, Andrew
From: Michael Prokop <mprokop@sipwise.com>
Kirill Solomko
Volodymyr Fedorov
Mykola Malkov
Alexander Lutay
Guillem Jover
Victor Seva
Andreas Granig
Michael Prokop
Michael Prokop