21 Commits (1141490a64cd240093e83dae7013aeeebcdcaef1)

Author	SHA1	Message	Date
Alexander Lutay	1141490a64	TT#65907 Speedup 'ngcpcfg build' on 25% for non-docker installations ... We are calling slow Perl command to move files to workaround Docker problem and losing 25% of performance on every NGCP installation in production. Let's call slow workaround when it is necessary only (in Docker). Command: time ngcpcfg build Original results: real 0m19.976s Current results: real 0m15.286s Change-Id: Iac787c0fcdb4182d71b6305ebbba5ea3193e76c8	7 years ago
Alexander Lutay	8209a5a136	TT#65907 Speedup 'ngcpcfg build' on 20% percent (when we build all files) ... About 50% of previous helper/build_config execution was loading 'functions/main' and we were performing it again and again for every tt2 files we build and we have more then hundred files in ngcpcfg. In fact we need two env variables and logging functions. Let's extract them into the separate function file and load them only. Command: time ngcpcfg build Original results: real 0m27.766s Current results: real 0m19.976s Change-Id: I6896ac437b1548888db7702de92b0ec79ed9d03f	7 years ago
Alexander Lutay	80ead123ab	TT#65907 Convert EXTRA_CONFIG_FILES from string to array to make shellcheck happy here ... Otherwise: > warning: Brace expansions and globs are literal in assignments. Quote it or use an array. [SC2125] Change-Id: I5a209cc1b50cea37e13aef40dfbd86c6178b53dc	7 years ago
Manuel Montecelo	a529f8e0b0	TT#46601 Detect and report errors parsing tt2 templates ... Due to the code in tt2-wrapper, the only errors reported were failures to connect to the socket to use the back-end performing the parsing. But there was no attempt to detect errors in the parsing at all. Since we are only able to use the output stream and not for example exit codes of the parser at the back-end (libtemplate-perl), we can only rely on finding some kind of error string, which can change in future versions of libtemplate-perl but hopefully not very often, and at most it should only change when we move base to newer Debian releases. In the case of the original problem reported we could rely on "parse error", but since there are two such strings ("file error" and "parse error") the first is more general and also covers cases like "permission denied to read file" or other errors that might happen. So the implementation of error checking is basically grepping for "^file error", the string at the beginning of a line, to limit possible false positives. With the new checks and an example of an invalid syntax already in the first line, "ngcpcfg build" detects the error correctly and stops processing, and shows information about the source of the problem, e.g.: root@spce:~# ngcpcfg build /etc/test 2018-10-29 11:37:01 spce: yml configs were validated successfuly 2018-10-29 11:37:01 spce: configs were checked successfuly 2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch. Loading /etc/ngcp-config/config.yml in memory: OK Loading /etc/ngcp-config/network.yml in memory: OK Loading /etc/ngcp-config/constants.yml in memory: OK 2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED 2018-10-29 11:37:02 spce: Error: from generated file: 2018-10-29 11:37:02 spce: Error: file error - parse error - input file handle line 1: unexpected end of directive 2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding: 2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2 2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml 2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml 2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml 2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file> 2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G 2018-10-29 11:37:02 spce: should provide more details. Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13	8 years ago
Manuel Montecelo	20fb1d9d87	TT#46601 Generate temporary files under /tmp ... When the "mktemp" command is invoked with a template parameter without path prefixes or without using "--tmpdir", it does not use "/tmp" by default, but instead it uses the current working directory as the place to create these temporary files. This was an oversight based on our side on the unknown behaviour, which was discovered when working on this task, because we do want to have "/tmp" as the home for these temporary files. Change-Id: I1b03dc90cfd8dd9364c8bc3b2bfe0e3c26979e7f	8 years ago
Guillem Jover	6e6c308ba5	TT#41050 On error when moving the built template fallback to use copy semantics ... We need to use rename semantics when moving the built template into the destination, so that we avoid reacing on ETXTBSY for executable files. But we need to fallback to use copy semantics, because at least Docker bind mounts /etc/hosts, which means we cannot rename over it. We'll use perl's File::Copy which gives us the exact semantics we need. Ref: https://github.com/moby/moby/issues/22281 Change-Id: I6ae6ce2050050c13c7ec9d08b0e6e01fb2801fd6	8 years ago
Guillem Jover	8fc290600a	TT#40300 Use «readlink -f» for the destination file of the mv ... We need to preserve symlinks, as was the case when we were using cat. This way we'll not get caught in the ETXTBSY race from the kernel, and we'll have the old semantics. Bisected-by: Alex Lutay <alutay@sipwise.com> Fixes: 7480ebe7c5 Change-Id: I0be7473b271cc3807da957bbef1063018d3b42b2	8 years ago
Guillem Jover	7480ebe7c5	TT#28113 Rename instead of overwriting the target config file ... Files that are being executed cannot be modified in place as that returns EBUSY. But their dentry can be replaced with a rename(2), which is atomic and does not touch the original inode. This is the standard procedure to replace running executables in Unix. So, we just replace the cat(1) with a mv(1), and stop quiescing its stderr so that we get proper errors reported. Change-Id: If15ea1cfa749a6140ff4022200c7fc730c76aa3a	8 years ago
Manuel Montecelo	13bc4ca34b	TT#37401 Add safety measures, create tmp files before moving to output file ... In this way, if the file fails to be generated (e.g. because of a broken .yml file) the old file is preserved intact. It doesn't protect from all failures (in particular, it is ineffective against wrong content due to programming errors), but it's safer than having an empty file or truncated as a result of the parsing/generation being stopped due to incorrect files or external events. This is added in the context of creating new /etc/hosts and /etc/network/interfaces files because we found the problem at this point, and having those files empty completely breaks the system. Change-Id: I01e8ef9d6971506aa3f8d9da174ce890ea83393c	8 years ago
Michael Prokop	57701e3ed0	TT#36827 Do not set permissions on output files of type symlink ... If the output file has been replaced by a symlink then the permissions of the symlink's target are adjusted. This is clearly unwanted and unexpected behavior. This can happen e.g. when masking a service via systemd's systemctl, demonstration: | sipwise@sp1:~$ sudo rm /etc/systemd/system/ntp.service | sipwise@sp1:~$ sudo systemctl mask ntp | Created symlink /etc/systemd/system/ntp.service → /dev/null. | sipwise@sp1:~$ ls -la /dev/null | crw-rw-rw- 1 root root 1, 3 May 12 01:54 /dev/null | sipwise@sp1:~$ sudo ngcpcfg build /etc/systemd/system/ntp.service | 2018-05-15 16:06:47: Checking state of local storage: | 2018-05-15 16:06:47: OK: nothing to pull | 2018-05-15 16:06:47: configs were checked successfuly | 2018-05-15 16:06:47: No patchtt files found, nothing to patch. | Loading /etc/ngcp-config/config.yml in memory: OK | Loading /etc/ngcp-config/network.yml in memory: OK | Loading /etc/ngcp-config/constants.yml in memory: OK | 2018-05-15 16:06:48: Generating /etc/systemd/system/ntp.service: OK | sipwise@sp1:~$ ls -la /dev/null | cr--r--r-- 1 root root 1, 3 May 12 01:54 /dev/null Instead check whether the expected output file is a symlink and if so don't modify permissions at all, but warn instead. Change-Id: I1c1bb0941a8a2f599652b7c089efd63ff33ca455	8 years ago
Michael Prokop	0441810530	TT#17401 Fail if destination file matches an existing directory ... If a template named /etc/ngcp-config/templates/etc/apt/apt.conf.d/71_no_recommended.tt2 exists the resulting destination file would be /etc/apt/apt.conf.d/71_no_recommended. Though if /etc/apt/apt.conf.d/71_no_recommended exists as directory ngcpcfg should abort and inform the user about the existing situation with a useful error message. Change-Id: I2c6e1e3a4ec485183674c1fe72251631ad9867ac	9 years ago
Michael Prokop	55fa597f54	MT#16391 Support custom output directories ... This let's us control the directory where files should be written to. Otherwise we'd have to write to the system's /etc/ + /var, which is clearly unwanted. Change-Id: I1010afcd4637928868fc81a0adfafebd3bbce874	9 years ago
Alexander Lutay	150d2a5e2e	TT#15824 Update code based on comments from previous code review ... Change-Id: Ic5adbb93eb7fc908c013f257321e82a67d61fbf1	9 years ago
Alexander Lutay	6e554b5717	TT#15824 Fix shellcheck warnings ... Change-Id: I1e35b67b06ef7e628a34d113a328b5aea2758801	9 years ago
Michael Prokop	e00d7e36bc	MT#22241 Support hostname and pairname specific templates (PRO-/CARRIER-only) ... If we have multiple web0X proxies but need specific templates on just one of those pairs then we need to support *.customtt.tt2.$pairname + *.tt2.$pairname templates. While at it also support *.customtt.tt2.$nodename + foo.tt2.$nodename (where nodename is actual hostname and not sp1 or sp2 as already supported already). Test script for demonstration (especially for precedence): mkdir /etc/ngcp-config/templates/etc/mika cd /etc/ngcp-config/templates/etc/mika hostname=$(hostname) # e.g. web01b pairname=${hostname%[ab]} # e.g. web01 nodename=$(cat /etc/ngcp_nodename) # e.g. sp2 for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do echo $file > $file ; done for file in foo.customtt.tt2.$hostname foo.customtt.tt2.$pairname foo.customtt.tt2.$nodename foo.customtt.tt2 foo.tt2.$hostname foo.tt2.$pairname foo.tt2.$nodename foo.tt2 ; do ngcpcfg build /etc/mika/ >/dev/null if [ "$(cat /etc/mika/foo)" = "$file" ] ; then echo "Content of file /etc/mika/foo matching $file" rm "$file" ; rm /etc/mika/foo else echo "Problem with precedence of file $file" return 1 fi done Change-Id: If7c222374bb56d7bf243200b37a414e39e050cd4	10 years ago
Alexander Lutay	e3eda0cb3b	MT#5027 Speed up 'build' runs by using Perl daemon	13 years ago
Alexander Lutay	9206aa868a	Revert "MT#4141 Speed up 'build' runs by loading YAML files only once" ... This reverts commit 335d53c95c.	13 years ago
Michael Prokop	335d53c95c	MT#4141 Speed up 'build' runs by loading YAML files only once ... When runnning 'ngcpcfg build' (which is also executed when running 'ngcpcfg apply') we waste quite some time with loading the same YAML configuration files and merging them again and again. In a 'ngcpcfg build' run we don't have to re-load and merge those configuration files again and again (since they won't modify during a single run), instead just load+merge them *once* and re-use the result on all the following execution steps in the same run. Original: | # time ngcpcfg apply >/dev/null | ngcpcfg apply > /dev/null 149.01s user 9.49s system 96% cpu 2:43.57 total Reworked optimized runs: | # time ngcpcfg apply >/dev/null | ngcpcfg apply > /dev/null 81.25s user 8.62s system 95% cpu 1:34.17 total For debugging purposes you can execute 'ngcpcfg build' under NO_REUSE=1 to not use this feature, executing it under 'DEBUG=1' won't remove the so called reuse file which stores the merged configuration data. Thanks: Alexander Lutay for the initial patch and idea	13 years ago
Michael Prokop	928cd68cf2	Simplify default configuration code	13 years ago
Michael Prokop	c3e1cb68d4	support additional configuration yml files via configuration directory ... Files in configuration directory EXTRA_CONFIG_DIR (being /etc/ngcp-config/config.d/ by default) with the suffix yml (so e.g. a file named pbx.yml can be shipped by an external Debian package) are additionally taken into consideration as main configuration files. Those extra configuration file(s) have higher precedence than the other yml configuration files (config.yml, network.yml,...), *except* for constants.yml which has the power to override all settings, including the ones from files inside EXTRA_CONFIG_DIR. Closes: sipwise-#3059	13 years ago
Michael Prokop	3436c08361	Initial checkin	13 years ago