* add_row() now uses list[any] instead of tuple[any]
to be compatible with PrettyTable coming with debian/trixie
Change-Id: I98cf5db91c6f63e13577f29ce437046898c5ba1c
This is rollback case if during the upgrade there is no
not_replicated sql scripts.
In 1st run of 'ngcp-update-db-schema --downgrade-release' deletes all
the records will be deleted on sp1. So the next run on sp2 finds no
records to delete and it's ok.
So don't fail on this case.
Change-Id: Ifcdd6cb369eaa0da0167c50023e5502e06aa3a8a
* freeswitch table is now synced to 3308 by ngcp-update-db-schema
because it's fully replicated in my.mysqld2.conf
Change-Id: Iadbd99e11df054e95e37bb56aa7576c387761e84
* new command line boolean option --ngcp-upgrade that enables
ngcp upgrade related logic (particularily sync_ro_db())
if not specified - NGCP_UPGRADE env var is checked to
auto enable it
* sync_ro_db(): rework master node detection:
- preferred master node selection is now only performed when
ngcp_upgrade is enabled
- fix and improve node side detection by either using regexp
or checking if the name is 'sp1' or 'sp2'
this logic is important because during the upgrade the sync must
be done from the db node of the same side, as otherwise old database
version and might be synced.
- mysql database is no longer synchronised to 3308 (it's already
removed from replication)
Change-Id: I151c725058627ec409b65a9e889a81a08f3a9129
* --defaults-extra-file arg is now provided first in the list
of args for the mysql command, otherwise it fails with:
/usr/bin/mysql: unknown variable
'defaults-extra-file=/etc/mysql/sipwise_extra.cnf
* get_db_schema() goes to shutdown(1) if db_schema is empty
and mode is 'down' or if --downgrade-release provided with a
non-existing release version
Change-Id: Ia7621f93fed2a4e7054af11930698348a57df92d
* 'proxy' role is not correctly checked not only that it does
not exists in the config.node_roles object but if it exists
it is checked that it's True/False, to correctly skip sync_ro_db()
if it's False
Change-Id: I03061362c508e474782cf98e81020b067b4ff583
* 15800 revision contains additiona of `ngcp.release` column and
this column is crucial during downgrades and cannot be removed,
therefore, this revision is automatically skipped during downgrades
as non-removable.
Change-Id: Ic16119cf08cd239db089e5bb202ec5154bfb1713
* use join instead of tuple to build
WHERE revision IN config._post_set_release_for_revisions
because in case of one element tuple represents it as
(15708,) which is invalid sql syntax.
Change-Id: I48f408556ace6dbed303a10fa3c4ba4759e5eaf4
* add support for db_schema release column.
* a new command line option '--set-release'
* when .up scripts are applied - the db_schema release column
is set from either --set-release option or /etc/ngcp_version
as the fallback.
* the release column is updated for all applied scripts after
all of them are applied, that is to include revisions that
were applied by the script before the 'release' column.
applied_at timestamp is preserved to keep the original time
when the revision was applied.
* new command line options --get-release-info and --as-json:
when --get-release-info is specified - a table with
"release,min_revision,max_revision,count" is returned and
no scripts processing takes place. With --as-json the data
is returned as a JSON string instead (for possible parsing
automation purposes).
* new command line option --downgrade-release:
- mutually exclusve with --mode
- automatically enables --mode=down
- removes all revisions that match the specified release version
* fix --mode=down --batch-mode, add missing ';' between delete
and commit sql statements.
Change-Id: I529ab523d7886a94848f6d9bbcf4a0348d0842ef
* get_db_schema() now issues a commit after selects to cleanup
any pending ngcp.db_schema row locks as otherwise applying
an .up script that modifies ngcp.db_schema (usually applied outside
of the connection handle by invoking 'mysql' command) causes
wait locks.
Change-Id: Ie0557a1a7648baeba2ec6ded3665c51667877101
* with --mode=down and --to-revision, the revisions are now downgraded
to, and excluding --to-revision (before they included --to-revision,
which is incorrect because downgrades to
package version X.Y, usually do that to the specified package version
Change-Id: I6f250e47f130fc050cbed65d7945a58cde3341a1
* node_state variable is now correctly checked
in the 'if' expression
* improve node state check output
Change-Id: Ie5da62cccf11e9c21cf43918670bf82c3cc2feb4
* --force-ro-db-sync runs ngcp-sync-db on the Carrier
proxy nodes regardless if at least one not_replicated script
was applied. This option is mutually exclusive with --skip-ro-db-sync
* rename config.db_local_socket into config.db_ro_socket
* when ro_sync_db() is invoked, privileges for the sipwise user
are updated (was only in automated mode). This is simplify
the usage as password for the sipwise user can potentially be
changed in the constants.yml (or different master database is
imported).
Change-Id: Ie21ff5a663a504082535834ec309c78857b6e911
* on CE platforms there is only one node assumed and therefore,
ngcp-update-db-schema should run there as expected
Change-Id: Iea9a75d34d77fa282585b365e50d221ca6c60528
* r/o database synchronisation, if run in the automated mode
sets grants for the sipwise user credentials taken from
/etc/mysql/sipwise_extra.conf and does it via the socket
/run/mysql/mysqld2.sock, that is because ngcp-sync-db
does not work via socket and there can be a condition when
no connections are allowed yet via tcp to the database.
* minor docstring typo fix
Change-Id: I536bca36320fdb6aa7ee8832ef50870c0dd5e9a1
* set session wait_timeout to 600 seconds
* in batch mode also the following session vars are set:
- net_read_timeout = 600
- net_write_timeout = 600
Change-Id: Ie774f7b5776d07e5c424d0391a5a5ddf4af2d98d
* config._run_cmd_mysql_options now includes -u db_host
-P db_port options to match the pymysql connection behaviour
Change-Id: I81955f4382c485acb17e123cfa42e27601171d05
* --db-scripts-dir option enables override of the
default database scripts directory location
* docstring a bit more compact formatting
* apply_revisions(): improve non-existing directory
detection and error output
Change-Id: Ic1d74244244df6406fb9a9258b7c36b971497513
* it's not a responsiblity of the script to start/stop
any external services and therefore, mariadb start
is removed
Change-Id: I220ebab7764a7016b15e6f08a8ac5ae01fe08523
* new command line option --db-defaults-conf-file
* new command line option --db-connect-method
* connect_db(): improve socket or config selection
* add mysql connect method as mysql command line options when
run_cmd('mysql' ...) is invoked, so that the mysql console connection
matches the pymysql connection
Change-Id: Ice8ff1e79c133d634f3de5bce204c7889f69b42d
* add new command line option --db-socket that overrides
the default socket file location
* MYSQL_SOCKET env is now considered as the socket file location
replacement, only if there is no --db-socket option set
* config.skip_sync_db is renamed into config.skip_ro_db_sync to
be consistent with the command line option
Change-Id: I4f113cdc8ad430f17ce022cdbbceef44e769857d
* ngcp-update-db-schema is rewritten in python. that simplifies
the code, as well as enables flexibility and more complex features
support, as well as performance improvements
* full strict mypy compliant
* flake8 compliant
* google docstring complied
* add .down scripts support
* add docstring full documentation
* add mypy, isort, flake8 checks during build phase
* add command line arguments that support --to-revision, to stop
.up/.down scripts execution when it reaches a certain revision
* better support for ngcp-sync-db
* preserve AUTOMATED_INSTALL_MODE and SKIP_SYNC_DB env vars support
for backward compatibility, in the future it is recommented to
switch to the existing (--automated, --skip-ro-sync-db) options
* experimental batch mode support where instead of applying changes
one by one, they are accumulated into a file along with db_schema
revision tracking changes and applied at once
Change-Id: I4632393d52d1a60c945b4d87a779f1a073ed9a5f
* provisioning.voip_fax_data replication to 3308
is now ignored because it is not in use on proxies
Change-Id: I8d7af9d3fa44f684bb7557cd66133946a802f5c7
For preparation of dumps (sql and json) it's useful to use separate
mysql instance which is run and handled manually.
So add support of such instance.
Add variable MYSQL_SOCKET which shows that we want to connect to custom
mysql instance via defined socket.
In such a case don't try to start a service mysql.
Change-Id: I3f40a3857125fc72158aabf277f892c7cd541bf6
* provisioning.autoprov_firmwares_data is ignored
during master-slave 3308 replication sync, as this
table can be potentially large and not used on the
proxy 3308 instance
Change-Id: I19f7feeb2f12f10abc6d6110fe691583b0d560b4
The bullsyeye-based MariaDB 10.5.9 has invalid root password set by default.
It is not possible to connect DB instance from network, the only option is
to connect DB instance through the socket.
The ngcp-installer calls ngcp-sync-db to replicate db01:3306->prx01:3308
on proxy node for the Carrier case. We have to open networking root access
temporary. ngcp-sync-db will replicate 'root' password from db01 the next moment.
Also update '1>/dev/null 2>/dev/null' to Sipwise style '&>/dev/null'.
Change-Id: I12ff71efdd8283531f03ceaf87345f3cc4e71419
After migration to bullseye, ngcp-update-db-schema is broken
inside Docker containers (Jenkins PPA tests):
> https://jenkins.mgm.sipwise.com/job/db-schema-test-db-up-docker/629/console
> ...
> + ngcp-update-db-schema -f
> mysql: unrecognized service
> Error: can't connect to local MariaDB port 3306 using '/etc/mysql/sipwise_extra.cnf'
> ...
Historically, the script ngcp-update-db-schema starts DB (if
it is not running) using the command "service mysql start".
The wrapper 'service' has to use SysV init script as
systemd unit cannot be used inside Docker.
The package mariadb provided backward compatibility symlink
in buster, but it has been removed in bullseye:
> root@11bb38c12e16:/# ls -la /etc/init.d/mysql
> ls: cannot access '/etc/init.d/mysql': No such file or directory
> root@11bb38c12e16:/# ls -la /etc/init.d/mariadb
> -rwxr-xr-x 1 root root 5966 Dec 18 20:25 /etc/init.d/mariadb
> root@11bb38c12e16:/#
The fix here is to use new/proper service name 'mariadb'.
Change-Id: Icf0d663f4a0d5256e1fc403d4fc0f3dd484ebbab
* determine the master db node name to use
for ngcp-sync-db master-slave replication restoration
if the CENTRAL_DBHOST is set by default to db01 only.
if the current nodename ends with "a" then db01a is used,
otehrwise db01b is used as the master host. that is
to make sure that during the upgrade the correct "side"
is used to replicate the data from and avoid lossy
statements that break replication.
Change-Id: I1a51bd0d975390d9269ef5208ce8275f9e055a1f
* --init-replication is required now for the central-db
because it uses multiple sources and they need to be
readjusted after the data sync
Change-Id: I10c12d479dbe7d73e6e489d3aa9ab37b7216af54
We are in the process of removing the obsolete init scripts, so trying
to use them directly will make these operations not work at all.
Change-Id: Ifff400388a947294ca661b3b8bad793b755ee830
* host and port by ngcp-sync-db is now used from
/etc/default/ngcp-db to ensure the correct R/O instance db
Change-Id: Ib7d9dde15cc4e5fa4ca95bb2d5b6ea508e12f89a
* ngcp database contains a 'timezone' table that
is used in replicated billing database
and therefore, needs to be also replicated
Change-Id: Iaaa23ff8cece68f4e9944a690bfc47c243d64f9f
We suppress some warning here to mark the repo as clean
while further cleanup can be done to remove some labels
'shellcheck disable'.
The first step here is to prevent new errors appearing.
Change-Id: I23ca9dd6fdf80bab6b9ec465dd4a53983e34a52a
* extra ngcp-sync-db options in AUTOMATED_INSTALL_MODE
* ngcp-sync-db is invoked only once now (after all revisions)
Change-Id: I6db3c29f186e1a61df1ccce17b0a646efaa1b055
Kirill Solomko
Mykola Malkov
Marco Capetta
Volodymyr Fedorov
Alexander Lutay
Guillem Jover