ngcpcfg/docs/ngcpcfg.txt

ngcpcfg(8)
==========

Name
----
ngcpcfg - central and template based Configuration Management

Synopsis
--------
ngcpcfg [<global option(s)>] <action> [ <action specific option(s)> ]

Introduction
------------

ngcpcfg is a Configuration Management System, developed for the Sipwise Next
Generation Platform. It provides a central mechanism for handling configuration
changes, updates and synchronisation between servers through a main
configuration which is simple and easy to read and modify.

tl;dr? - ngcpcfg for the impatient
----------------------------------

The main system configuration is done in the file _/etc/ngcp-config/config.yml_.
After modifying the file execute 'ngcpcfg apply "<summary of changes>"' to build
the according configuration files.

Taxonomy
--------

*central yml files*:: *.yml files inside _/etc/ngcp-config/_

*High Availability setups*:: ngcpcfg running in a cluster setup (e.g.
sip:providerPRO or sip:carrier), depends on Debian package ngcp-ngcpcfg-ha.

*local repository*:: the directory _/etc/ngcp-config/_, being a Git repository

*remote systems*:: the other nodes inside High Availability setups, as defined
in _/etc/ngcp-config/network.yml_

*shared repository*:: Git repository shared amongst nodes inside a High
Availability setup, as defined by the GLUSTERFS setting in
/etc/ngcp-ngcpcfg/ngcpcfg.d/shared_storage.cfg (requires ngcp-ngcpcfg-ha) and
being _/mnt/glusterfs_ by default

*templates*:: template toolkit files with .tt2 suffix, found in
_/etc/ngcp-config/templates/etc/_

[[configfiles]]
Configuration files
-------------------

Main configuration files
~~~~~~~~~~~~~~~~~~~~~~~~

* _/etc/ngcp-config/config.yml_: central configuration file, to be configured with
$EDITOR, webfrontend,...

* _/etc/ngcp-config/config.$NGCP_HOSTNAME.yml_: host specific configuration file,
depending on the ngcp_hostname (:= $NGCP_HOSTNAME) of the system.

* _/etc/ngcp-config/config.$NGCP_PAIRNAME.yml_: pair specific configuration file,
depending on the pairname (:= $NGCP_PAIRNAME) of the system (for example _sp_ on
CE/PRO or _web01_ on CARRIER).

* _/etc/ngcp-config/config.$NGCP_NODENAME.yml_: HA node specific configuration
file, depending on the HA node name of the system. $NGCP_NODENAME is determined
using the __ngcp-nodename__ program (being _spce_ for CE systems, and _spN_
where N is the index of the node in the pair).

* _/etc/ngcp-config/config.local.yml_: local configuration, not being host
specific.

* _/etc/ngcp-config/constants.yml_: configuration file that has precedence over
any other .yml file _/etc/ngcp-config/_, defining important constant settings.
This file is *not* supposed to be modified by the user (without having a very
good reason).

* _/etc/ngcp-config/config.d/_: optional configuration directory with additional
yml configuration files. Files inside this directory aren't supposed to be
added or modified by the user. The purpose of this configuration directory is
that software packages can enable/disable certain features through configuration
options on demand.

* _/etc/ngcp-ngcpcfg/ngcpcfg.cfg_: main configuration file for ngcpcfg itself,
provides global variables used inside ngcpcfg and its helper scripts. This file
is *not* supposed to be modified by the user (without having a very good
reason).

* _/etc/ngcp-ngcpcfg/ngcpcfg.d/_: configuration directory for ngcpcfg itself.
Files with suffix '.cfg' inside this directory are sourced after
/etc/ngcp-ngcpcfg/ngcpcfg.cfg has been read. Files inside this directory are
*not* supposed to be modified by the user (without having a very good reason).

[IMPORTANT]
Configuration file priority: constants.yml takes precedence over
all files in _/etc/ngcp-config/config.d/_, over config.local.yml,
over config.$NGCP_HOSTNAME.yml, over config.$NGCP_PAIRNAME.yml,
over config.$NGCP_NODENAME.yml, over config.yml.

High Availability specific configuration files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* _/etc/ngcp_nodename_: defines the node name in High Availability setups,
being _spce_ for CE systems, and _spN_ where N is the index of the node in
the pair.

Supported template files
~~~~~~~~~~~~~~~~~~~~~~~~

Example for generating a configuration file named _/etc/foobar/baz_ (from lower
precedence to higher precedence):

* _/etc/ngcp-config/templates/etc/foobar/baz.tt2_: main and default template
file, used by template-handler for generating /etc/foobar/baz. Configuration
file is usually provided by a Debian package.

* _/etc/ngcp-config/templates/etc/foobar/baz.tt2.inst-$INSTANCE_NAME_: ngcp_instance specific
template file. $INSTANCE_NAME in lowercase specified at instances part on network.yml.

* _/etc/ngcp-config/templates/etc/foobar/baz.tt2.$NGCP_NODENAME_: node specific
template file. $NGCP_NODENAME is determined using the __ngcp-nodename__ program
(being _spce_ for CE systems, and _spN_ where N is the index of the node in the
pair).
Whereas _*customtt.tt2_ files are used on all nodes in a High Availability
setup the, _*.tt2.$NGCP_NODENAME*_ file is specific for the single node only
of each pair.
A common use case is master vs. slave configuration of a service.
The configuration file is usually provided by a Debian package.
Note: This feature is available in High Availability setups only.

* _/etc/ngcp-config/templates/etc/foobar/baz.tt2.$NGCP_PAIRNAME_: pair specific
template file. Given a PRO pair in a CARRIER environment where two nodes are
named _web01a_ and _web01b_ then $NGCP_PAIRNAME corresponds to _web01_. This is
useful if there are multiple pairs available but only a specific pair of them
should include some specific configuration.
Note: This feature is available in High Availability setups only.

* _/etc/ngcp-config/templates/etc/foobar/baz.tt2.$NGCP_HOSTNAME_: ngcp_hostname specific
template file. While $NGCP_NODENAME is usually bound to _sp1_ for the first node and
_sp2_ via __ngcp-nodename__, it's possible to use host specific template file by
referring to its ngcp_hostname ($NGCP_HOSTNAME is determined via ngcp-hostname).
Note: This feature is available in High Availability setups only.

* _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2_: system specific
template file, but configuration usually isn't provided by a Debian package and
can be modified independently from any Debian package mechanism.

* _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2.inst-$INSTANCE_NAME_:
configuration file similar to
_/etc/ngcp-config/templates/etc/foobar/baz.tt2.inst-$INSTANCE_NAME_ but it's guaranteed
that the file won't be part of any Debian package mechanism. $INSTANCE_NAME in
lowercase specified at instances part on network.yml.

* _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2.$NGCP_NODENAME_: node
specific template file.
Regarding $NGCP_NODENAME the same as for _baz.tt2.$NGCP_NODE_NAME_
applies (see the previous bullet), but the configuration file usually isn't
provided by a Debian package but can be modified independently from any
Debian package mechanism.
Note: This feature is available in High Availability setups only.

* _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2.$NGCP_PAIRNAME_:
configuration file similar to
_/etc/ngcp-config/templates/etc/foobar/baz.tt2.$NGCP_PAIRNAME_ but it's guaranteed
that the file won't be part of any Debian package mechanism.  Note: This feature is
available in High Availability setups only.

* _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2.$NGCP_HOSTNAME_:
configuration file similar to
_/etc/ngcp-config/templates/etc/foobar/baz.tt2.$NGCP_HOSTNAME_ but it's guaranteed
that the file won't be part of any Debian package mechanism.  Note: This feature is
available in High Availability setups only.

[IMPORTANT]
Configuration file precedence (highest to lowest):
*.customtt.tt2.inst-$INSTANCE_NAME
*.customtt.tt2.$NGCP_HOSTNAME
*.customtt.tt2.$NGCP_PAIRNAME
*.customtt.tt2.$NGCP_NODENAME
*.customtt.tt2
*.tt2.inst-$INSTANCE_NAME
*.tt2.$NGCP_HOSTNAME
*.tt2.$NGCP_PAIRNAME
*.tt2.$NGCP_NODENAME
*.tt2

Customisation for default template files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can overwrite the default template file
(e.g. _/etc/ngcp-config/templates/etc/foobar/baz.tt2_)
using a _customtt_ file, like _/etc/ngcp-config/templates/etc/foobar/baz.customtt.tt2_.
Or even host a specific _customtt_ file (see above in "Supported template files").
This approach is NOT recommended as _customtt_ will become outdated very soon,
hence new template files can be released by upstream any time.

The better way is to handle modifications using _patchtt_ files
(e.g. _/etc/ngcp-config/templates/etc/foobar/baz.patchtt.tt2_).
In this case, on every "ngcpcfg patch", _patchtt_ file will be applied on top
of the tt2 file and the result will be saved into the _customtt_ file, which
in the future will be used in a common way. "ngcpcfg patch" is the first step
on "ngcpcfg build" that guarantees the latest upstream templates with the
availability of the necessary local changes on every configuration apply.

[IMPORTANT]
The patch to be applied to the corresponding tt2 template file is selected in
the following order (highest to lowest):
*.patchtt.tt2.$NGCP_HOSTNAME
*.patchtt.tt2.$NGCP_PAIRNAME
*.patchtt.tt2.$NGCP_NODENAME
*.patchtt.tt2

[IMPORTANT]
If a suitable _patchtt_ file is found for a template, then the **ngcpcfg patch**
command will overwrite the corresponding _customtt_ file, if any exists. You can
find the old version of the customtt in ngcpcfg the git repository (if necessary).

Support action related files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Example for handling a configuration file named _/etc/foobar/baz_:

* _/etc/ngcp-config/templates/etc/foobar/baz.services_: service file, defining
actions that have to be executed when file /etc/foobar/baz was modified.

* _/etc/ngcp-config/templates/etc/foobar/ngcpcfg.services_: service file,
defining actions that have to be executed whenever *any* file inside /etc/foobar
was modified.

* _/etc/ngcp-config/templates/etc/foobar/baz.postbuild_: script which is
executed *after* a configuration file (/etc/foobar/baz) has been generated. The
environment variable "$output_file" containing the file name of the generated
configuration file (/etc/foobar/baz) is available within the postbuild script. A
common usage case for postbuild files is adjusting file permissions.

* _/etc/ngcp-config/templates/etc/foobar/baz.prebuild_: script which is
executed *before* a configuration file (/etc/foobar/baz) is generated. The
environment variable "$output_file" containing the file name of the generated
configuration file (/etc/foobar/baz) is available within the postbuild script.

Syntax and layout of configuration files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* _/etc/ngcp-ngcpcfg/ngcpcfg.cfg_: plain shell syntax style "key=value" entries

* _central yml files_ (*.yml): YAML configuration syntax (see http://yaml.org/)

* _template files_ (*.tt2): whatever the according software (Kamailio,
MySQL,...) needs. Any variables that should be replaced by the configuration
management system (based on the main configuration files *.yml inside
/etc/ngcp-config/) have to be written in the YAML syntax (see http://yaml.org/)

* _service files_ (*.services) and _build files_ ({post,pre}build): file will
be executed under "bash $FILE", so you can use commands e.g. like
"ngcp-service foobar restart" and any further shell scripting syntax inside
the services files

NGCP instances
~~~~~~~~~~~~~~

ngcp-templates package provides the *TEMPLATE_INSTANCES* file defining what are
the supported templates for instances. Meaning that ngcpcfg will generate
one additional output per instance following the specifications provide by the
file.

ngcpcfg will generate that output with the template variable **instance_name** defined
and the environment variable **INSTANCE_NAME** set for ngcpcfg.service scripts

Global Options
--------------

  **--debug** <action> [<further_options>]::

Run actions (see next section) in verbose mode, useful for debugging.
Debug output will be sent to syslog (available as /var/log/ngcp/ngcpcfg.log on
NGCP systems) as well as stderr.

  **--help**::

Display usage information and exit.

  **--jobs[=<n>]** <action> [<further_options>]::

The _build_ and _apply_ actions process templates in parallel by default.
This option can be used to control the number of jobs used to process
templates, '0' for an unlimited amount, and '1' for serial execution to
for example get ordered output.

  **--no-action-failure** <action> [<further_options>]::

The _check_ and _apply_ actions check for any possibly outstanding pull
actions on the shared storage. If any outstanding actions are identified then
the script aborts to avoid running into tricky configuration merge situations.
If this option is enabled then ngcpcfg instead doesn't abort on outstanding
pull actions. This option should be used with care and only if you know
what you're doing.

Note: This option is available in High Availability setups only.

  **--no-db-sync** <action> [<further_options>]::

Skip DB synchronisation part of 'commit' action. Normally it
should NOT be skipped as it is mandatory part of ngcpcfg routine.
Necessary during the new installation of distributed platform setup when
database is not yet installed/available.

  **--no-check-origin** <action> [<further_options>]::

Skip Git branch origin check (part of 'check' action). Normally it
should NOT be skipped as it is mandatory selfcheck part of ngcpcfg.
Necessary during the new installation of distributed platform setup when
some initial files have to be generated from templates before ngcpcfg final setup.

  **--no-validate** <action> [<further_options>]::

Ignore schema validation results for YAML files (syntax check is still performed).

  **--summary-only** <action> [<further_options>]::

Hide all the output except the last summary line.
Useful for integration of ngcpcfg in other scripts/tools.

  **--validate** <action> [<further_options>]::

Force schema validation for YAML files even if validation is disabled in config.

  **--version**::

Display ngcpcfg version and exit.

Actions and action specific options
-----------------------------------

  **apply** [--dry-run] [--force-all-services] [--ignore-branch-check] [--ignore-shared-storage-check] [--modified-only] [<commit_message>]::

Executes the _check_, _build_, _services_ and _commit_ commands in a batch
(assuming each command worked as expected). This option serves as a shortcut for
the most commonly executed commands. If there are any outstanding changes that
need to be committed, then the commit message needs to be provided. This is
meant so the configuration change history (accessible e.g. via 'ngcpcfg log')
provides useful information.
Options for _services_, _build_ and _check_ can be used here too.

  **cat** [<config-type>...]::

Concatenates the YAML configuration files, including any local override files
and prints them to standard output.
If no _config-type_ has been specified, then all configuration files found
are used.
Otherwise one or more of *config*, *network* and *constants* can be specified
in any order (they will get ordered properly internally) to specify what
subset of the YAML configuration files to load and output.

  **check** [--ignore-branch-check] [--ignore-shared-storage-check] [<config_files_or_directories>|<pattern>]::

Check syntax of YAML files and validate schema before performing any further actions.
The *--ignore-branch-check* option doesn't fail the check if the
current active branch doesn't match 'master'.
The option *--ignore-shared-storage-check* can be used to skip check of shared
storage, for example in situations like when the HA_INT addresses are being
changed, because the communication is not possible in that scenario.
In High Availability setups possibly outstanding pull actions are
checked as well.

  **build** [--ignore-branch-check] [--ignore-shared-storage-check] [--modified-only] [<files_or_directories>|<pattern>]::

Generate configuration files, based on values defined the central yml files and
based on the templates in the configuration tree (/etc/ngcp-config/templates by
default). Central yml files will be validated for integrity before building using
_check_ action. The *--modified-only* option checks for _modified_ and _uncommitted_
configuration files (central yml files and templates) and builds the relevant
files only. The *--ignore-branch-check* option doesn't fail the build if the
current active branch doesn't match 'master'.
The option *--ignore-shared-storage-check* can be used to pass down to the
"check" subcommand in order to skip check of shared storage, for example in
situations like when the HA_INT addresses are being changed, because the
communication is not possible in that scenario.
If a central yml file is modified it builds all configuration files.
If changes are in template files only just the according template files are
considered for rebuild. If a file (e.g. _/etc/rsyslog.conf_) or directory (e.g.
_/etc/mysql/_) is provided as argument to the _build_ option only the specified
file / files inside the directory will be generated. If the argument doesn't
start with the /etc prefix the argument is considered as pattern, matching all
files/directories which include the specified pattern. The pattern can include
shell globbing patterns - so argument 'mo..t' will match the files
/etc/ngcp-monitoring-tools/collective-check.conf, /etc/monit/monitrc,
/etc/ha.d/resource.d/monit-services and /etc/default/monit iff they are present.
You can combine _<files_or_directories>_ and _<pattern>_ and use multiple
arguments.

  **clean** [--all] [--branches] [--force] [--help] [--reset-master] [--stashes] [--tracked-files] [--untracked-files]::

Clean ngcpcfg from not-yet committed/applied changes, reset git state and remove
any possibly existing git stashes. It also proposes to delete unnecessary git
branches if available.
Note: the action (possibly) deletes some files inside /etc/ngcp-config,
it should be used with care!.
Option _--all_ is an alias for _--branches_ + _--reset-master_ + _--stashes_ +
_--tracked-files_ + _--untracked-files_. It should restore local
/etc/ngcp-config in a clean state.
Note: The option _--reset-master_ is involved in High Availability setups only.
Option _--branches_ proposes to clean all the unnecessary local git branches.
Active branch will be switched to 'master'.
Option _--force_ answers 'yes' automatically on all the cleanup questions.
Option _--help_ shows list of options for the action 'clean'.
Option _--reset-master_ proposes to back up local branch 'master' and restore
branch 'master' from origin. It will effectively clean all locally committed and
not-yet pushed changes.
Note: Already pushed changes won't be removed while they can be reverted using
'git revert <commit>'.
Note: This option is available in High Availability setups only.
Option _--stashed_ proposes to clean all the changes in git stash.
Option _--tracked-files_ resets cached git files and restore all tracked files
to HEAD of branch master.
Option _--untracked-files_ deletes all untracked files available in
/etc/ngcp-config/, including files ignored by /etc/ngcp-config/.gitignore.

  **commit** [<commit_message>]::

Commit all modified files in _/etc/ngcp-config_ and record changes in _/etc_ by
executing the etckeeper(8) command through
'/usr/share/ngcp-ngcpcfg/scripts/etckeeper'. To check whether there are any
pending changes to be committed execute 'ngcpcfg status'.

  **decrypt**::

Decrypt /etc/ngcp-config.tgz.pgp
(or /etc/ngcp-config-crypted.tgz.gpg for backwards compatibility)
and restore configuration files,
doing the reverse operation of the _encrypt_ option.
Note: This feature is only available if the ngcp-ngcpcfg-locker package is
installed.

  **del** [<options>] <file> <option>::

Delete given <option> in <file>. Option '--diff' shows applied changes.

  **diff** [<options_for_git_diff>]::

Show changes between ngcpcfg's Git repository and the working tree inside
_/etc/ngcp-config_. You can specify options for the underlying 'git diff'
command, e.g. 'ngcpcfg diff HEAD^' will show all changes that have been recorded
with the last _commit_ operation (see manpage git-rev-parse(1) for details how
to specify revisions). If the tool doesn't report anything it means that there
are neither any uncommitted changes nor any new or removed files (files which are
not yet (un)registered to the repository).

  **edit**::

Helper to edit the ngcpcfg's YML configuration files in a faster way
(it prevents unnecessary <TAB> pressing).

  **encrypt**::

Encrypt /etc/ngcp-config and all resulting configuration files with a user
defined password and save the result as /etc/ngcp-config.tgz.pgp.
Note: This feature is only available if the ngcp-ngcpcfg-locker package is
installed.

  **get** <key>::

Prints the value of <key> using ngcp config files as source.

  **help**::

Display usage information and exit.

  **initialise [--without-shared]**::

If ngcpcfg was installed but isn't configured yet the 'initialise' option sets
up ngcpcfg accordingly. Follow the instructions from the <<setup_instructions,
Setup instructions>> section. Option '--without-shared' allows to skip shared
repository setup (which is necessary in High Availability setups only).

  **init-shared [--join]**::

The ngcpcfg configures shared repository to synchronise configurations between
all nodes in installation. It is necessary if ngcpcfg was initialised using
option '--without-shared'. If the option '--join' is specified the ngcpcfg
configuration is taken from already configured shared repository. This option
should be used only for configuration of the node which is joining the existing
cluster. Note: This feature is available in High Availability setups only.

  **log** [-p]::

Prints the log of local changes. Expand all changes if '-p' option is specified.

  **patch** [--help | --from-customtt [<customtt file(s)>] | <patchtt file(s)>]::

Generate customtt files using default templates and patchtt files.
Using patchtt files automates template updating and simplifies customtt management.
You can perform initial migration from customtt to patchtt using option '--from-customtt',
ngcpcfg will process all/requested customtt files and generate patchtt files as a
difference between template and current customtt files. You should normally
execute '--from-customtt' only once during migration on patchtt.

  **pull**::

Retrieve modifications from shared repository.
Note: This option is available in High Availability setups only.

  **push** [--nobuild] [--noapply] [--shared-only] [<host(s)>]::

Push modifications to shared repository and remote systems. After changes have been
pushed to the node the _apply_ operation will be executed on each remote system
to rebuild the configuration files (unless the '--noapply' operation has been
specified, then only the _build_ operation will be executed). To skip building
any configuration files at all the '--nobuild' option can be specified (implying
to also skip the _apply_ operation).  If the '--shared-only' option is set then
the any pending commits will be pushed only to the shared repository only, any
foreign hosts will be skipped then.  If ngcp_hostname(s) or IP address(es) is given
as argument then the changes will be pushed to the shared repository and to the
given hosts only. If no host has been specified then the hosts for this pair
specified in _/etc/ngcp-config/network.yml_ are used.
The magic word 'all' can be used as a 'host' to push the changes on all the
nodes from the network.yml except the local one.
Note: This option is available in High Availability setups only.

  **push-parallel** [<options>] [all|sp1|sp2|<host(s)>]::

Like the action 'push' above, action 'push-parallel' pushes modifications to the
shared repository and remote systems _in parallel_. After the changes have been
pushed to the nodes the _apply_ operation will be executed on the remote systems
to rebuild the configuration files and restart services if necessary.

The operations 'build' and 'apply' can be skipped on remote side using options
'--nobuild' and/or '--noapply'  (similar to action 'push'). The action 'push' can be
skipped on local side using option '--skip-shared-storage-push'.

If no hosts have been specified then the 'push' will happen only on all HA inactive nodes
(unless the option '--apply-on-all-nodes' is specified).
Note: an HA switchover might happen if you are pushing changes which trigger
a HA service restart. In this case you should NOT push to the peer nodes
simultaneously (the option '--apply-on-all-nodes' is dangerous in this case,
as this can cause a HA service restart on both sp1 and sp2 nodes
simultaneously).

The 'host' can be defined as 'sp1' or 'sp2' which is a ngcpcfg filter where to push/apply.
The actions will be performed if the HA node ngcp_hostname is 'sp1' or 'sp2' ignoring the HA active state.
It is safe to push to all sp1 nodes at once, as the peer (sp2 node) will be activated
in case the current node does a HA service restart.

If the special host 'all' is used the push will be performed in parallel in two steps:
 - the first step pushes and applies changes to all sp1 nodes (ignoring the HA state),
 - the second step pushes and applies changes to all sp2 nodes (ignoring the HA state).
Note: the second step will NOT be executed if any issue is detected on the first step.

If one or multiple ngcp_hostnames or IP addresses are given as arguments then the
changes will be only pushed to the shared repository and the actions performed
only on the given hosts.

All 'ngcpcfg' output is muted on STDOUT but available in the log file
'/var/log/ngcp/ngcpcfg.log' on each node.

Note: This option is available in High Availability setups only.

  **services** [--dry-run] [--force-all-services]::

Execute any defined service actions for modified configuration files. If the
_--dry-run_ option is present the services won't be executed but you'll be
noticed which service files would be executed if being invoked without the
_--dry-run__ option.
_--force-all-services_ option causes all services to be restarted, including
services without changes in config files.

  **set** [<options>] <file> <option>=<value> [<option>=<value>]::

Set given <option> to <value> in <file>. Using this command you can add/change
the value(s) for every YML file used by ngcpcfg. Use _del_ option to delete the value/option.
Add option '--diff' to display applied changes.
Here you can find the most popular cases for YML configs modification:

  # change rtpengine max port number:
  ngcpcfg set /etc/ngcp-config/config.yml "rtpengine.maxport=44123"
  # disable asterisk service:
  ngcpcfg set /etc/ngcp-config/config.yml "asterisk.enable='no'"
  # add value with spaces
  ngcpcfg set /etc/ngcp-config/config.yml "sshd.authorized_keys_file='%h/.ssh/authorized_keys %h/.ssh/sipwise_vagrant_key'"
  # add the element to the array/hash:
  ngcpcfg set /etc/ngcp-config/config.yml "sshd.listen_addresses.APPEND='203.0.113.42'"
  # replace the first element in the array/hash:
  ngcpcfg set /etc/ngcp-config/config.yml "sshd.listen_addresses.0='203.0.113.42'"
  # add the list of hashes to iproute2 section (also display the applied changes):
  ngcpcfg set --diff  /etc/ngcp-config/config.yml iproute2.rt_tables.APPEND='{"name","my_route","id",205}'
  #  iproute2:
  # -  rt_tables: []
  # +  rt_tables:
  # +  - id: 205
  # +    name: my_route
  # change number of b2b threads:
  ngcpcfg set /etc/ngcp-config/config.yml "b2b.media_processor_threads='10'" "b2b.session_processor_threads='10'"

  **show** [<change_id>]::

Display the change details for specified <change_id>. Show the latest change if
no <change_id> is specified.

  **status** [--local-only]::

Display the current state of the configuration system, like modified
configuration files and pending actions.
Note: The _--local-only_ option is available in High Availability setups
only and disables checking any remote systems (useful when the remote
system isn't available).

  **values** <key>::

Obsolete synonym for 'get'.

Usage examples
--------------

The main workflow *without* High Availability setup is:

  ngcpcfg status                      # check for pending operations
  $EDITOR /etc/ngcp-config/config.yml # adjust/extend configuration
  ngcpcfg apply "summary of changes"  # build configs, run services + commit changes

If you do not want to execute the _apply_ shortcut command but instead run the
single actions, then you can execute:

  ngcpcfg status                      # check for pending operations
  $EDITOR /etc/ngcp-config/config.yml # adjust/extend configuration
  ngcpcfg build                       # generate/update configuration files
  ngcpcfg services                    # restart services for modified configs
  ngcpcfg commit "summary of changes" # register changes

The main workflow *with* High Availability setup is:

  ngcpcfg status                      # check for pending operations
  ngcpcfg pull                        # retrieve possibly pending updates
  $EDITOR /etc/ngcp-config/config.yml # adjust/extend configuration
  ngcpcfg build                       # generate/update configuration files
  ngcpcfg services                    # restart services for modified configs
  ngcpcfg commit "summary of changes" # register changes
  ngcpcfg push                        # upload changes to shared storage + remote systems
  ngcpcfg status                      # check for possibly outstanding issues

[[faq]]
Frequently asked questions
--------------------------

What does ngcpcfg mean?
~~~~~~~~~~~~~~~~~~~~~~~

'ngcp' is the http://www.sipwise.com/[Sipwise Next Generation Communication
Platform] and 'cfg' stands for the ConFiGuration management system of the
ngcp system.

Does ngcpcfg require a shared storage?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Only if you are using ngcpcfg in High Availability setups and using the 'push'
functionality (which is highly recommended in a HA Setup).

How does the configuration management system work?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ngcpcfg takes the central yml files (/etc/ngcp-config/\*.yml) as input for
configuration, searches for supported templates files (*.tt2) in the template
pool (_/etc/ngcp-config/templates/etc/_) and builds the resulting configuration
files in _/etc_.

Does ngcpcfg also track file permissions?
-----------------------------------------

When using ngcpcfg in a HA setup (PRO/CARRIER only) you want to ensure that
changing file permissions inside the ngcpcfg repository on one node propagates
to the other node(s) as well.  ngcpcfg is using git underneath, though git
itself doesn't track file permissions (except for the execution bit).

To be able to track file permissions ngcpcfg includes a pre-commit hook which
records file permissions via the _.ngcpcfg_perms_ file inside the ngcpcfg
repository (being _/etc/ngcp-config/.ngcpcfg_perms_ by default).  Whenever you
run 'ngcpcfg commit' this pre-commit hook looks at the existing file permissions
and records them through the _.ngcpcfg_perms_ file. After pushing the changes to
the shared storage and when pulling then from the git repository on the foreign
node(s) via 'ngcpcfg pull' the file permissions get restored automatically (via
_/usr/share/ngcp-ngcpcfg/helper/restore-permissions_ which can be used also
manually if needed).

The solution via the git pre-commit hook ensures that no matter whether you're
using 'ngcpcfg commit …' or 'git commit …' directly (for whatever reason) you
always get the file permissions handled via .ngcpcfg_perms.

If you want to change file permissions in a *clean* working directory and commit
*without* using 'ngcpcfg commit' but directly via git itself then you've to use
'git commit --allow-empty ...'. Thanks to the pre-commit hook the file
_.ngcpcfg_perms_ will get updated accordingly.

[[errorhandling]]
Error handling - what does this error message mean?
---------------------------------------------------

Error message:

  Error: Failed hosts:  ngcp_hostname (down)

Background: The system "ngcp_hostname" is not reachable via ICMP, which means it's
not reachable for the ngcpcfg system.

How to fix: Make sure the system is up and running and reachable via ssh from
the system where you are running the "pull" action.

Error message:

  error: Your local changes to 'config.yml' would be overwritten by merge. Aborting.

Background: the file 'config.yml' (in this example) on the system where you are
pushing to contains local modifications which are uncommitted yet. To avoid data
damage the system can't resolve this situation automatically.

How to fix:

1) Throw away the changes on the remote system (where you are pushing to,
important: your changes on the system where you are executing the commands will
be gone!):

  cd /etc/ngcp-config ; git checkout -- config.yml

To get rid of *any* modified files (included added/removed files) the following
commands will do the job (use with care and only execute it if you're aware of
the consequences):

  cd /etc/ngcp-config
  git checkout -- .
  git clean -xdf

2) Resolve the merge conflict:

  cd /etc/ngcp-config
  git pull
  $EDITOR config.yml # resolve conflict
  ngcpcfg commit 'merge conflict on system sp2'
  ngcpcfg pull # just to make sure
  ngcpcfg push # push changes back to the other systems

[[setup_instructions]]
Setup instructions
------------------

Requirements for High Availability setups:

* glusterfs is supposed to be set up working and deployed at /mnt/glusterfs
  (as defined via /etc/ngcp-ngcpcfg/ngcpcfg.cfg)

* hosts are expected to be set up for automatic ssh login (ssh-keygen && ssh-copy-id)

* node names are expected to be sp1 and sp2

Configuration on sp1 first (expected to be master node in initial setup),
sequentially followed by configuration on sp2 (and any further existing
system):

 ngcpcfg initialise
 ngcpcfg build

Please note that you do not have to run this steps if you are using the Sipwise
Next Generation Platform since the installation steps are fully automated.

Involved frameworks
-------------------

* git: Distributed Version Control system
* tt2: Template Toolkit
* make: Utility for Directing compilation
* yaml: Generic data serialization language

Limitations
-----------

ngcpcfg was designed specifically for the Sipwise Next Generation Platform,
though with being as generic as possible in mind. The system is supposed to be
useful for configuration management on other systems/platforms and it's possible
to adapt it for different needs through a variety of configuration parameters,
though keep in mind that ngcpcfg was implemented for some very specific use
cases.

Author
------
Michael Prokop <mprokop@sipwise.com>

/////////////////////////////
// vim: ft=asciidoc tw=80 ai
/////////////////////////////