TT#46601 Detect and report errors parsing tt2 templates

Due to the code in tt2-wrapper, the only errors reported were failures to
connect to the socket to use the back-end performing the parsing.  But there was
no attempt to detect errors in the parsing at all.

Since we are only able to use the output stream and not for example exit codes
of the parser at the back-end (libtemplate-perl), we can only rely on finding
some kind of error string, which can change in future versions of
libtemplate-perl but hopefully not very often, and at most it should only change
when we move base to newer Debian releases.

In the case of the original problem reported we could rely on "parse error", but
since there are two such strings ("file error" and "parse error") the first is
more general and also covers cases like "permission denied to read file" or
other errors that might happen.  So the implementation of error checking is
basically grepping for "^file error", the string at the beginning of a line, to
limit possible false positives.

With the new checks and an example of an invalid syntax already in the first
line, "ngcpcfg build" detects the error correctly and stops processing, and
shows information about the source of the problem, e.g.:

  root@spce:~# ngcpcfg build /etc/test
  2018-10-29 11:37:01 spce: yml configs were validated successfuly
  2018-10-29 11:37:01 spce: configs were checked successfuly
  2018-10-29 11:37:01 spce: No patchtt files found, nothing to patch.
  Loading /etc/ngcp-config/config.yml in memory: OK
  Loading /etc/ngcp-config/network.yml in memory: OK
  Loading /etc/ngcp-config/constants.yml in memory: OK
  2018-10-29 11:37:02 spce: Error: Generating /etc/test based on /etc/ngcp-config/templates//etc/test.customtt.tt2: FAILED
  2018-10-29 11:37:02 spce: Error: from generated file:
  2018-10-29 11:37:02 spce: Error:   file error - parse error - input file handle line 1: unexpected end of directive
  2018-10-29 11:37:02 spce: NOTE: Check those files for valid syntax and encoding:
  2018-10-29 11:37:02 spce: /etc/ngcp-config/templates//etc/test.customtt.tt2
  2018-10-29 11:37:02 spce: /etc/ngcp-config/config.yml
  2018-10-29 11:37:02 spce: /etc/ngcp-config/network.yml
  2018-10-29 11:37:02 spce: /etc/ngcp-config/constants.yml
  2018-10-29 11:37:02 spce: Running /usr/share/ngcp-ngcpcfg/helper/tt2-wrapper <file>
  2018-10-29 11:37:02 spce: or inspecting temporary /tmp/ngcpcfg.test.PwGvShIm9G
  2018-10-29 11:37:02 spce: should provide more details.

Change-Id: Ic305bdab20a6ce15eca13f19586a2572a90b4e13

helper/build_config

@@ -102,6 +102,7 @@ log_debug "      and: move ${tt_tmp_output_file} ${output_file}"
 # We need to use «readlink -f» so that we do not destroy any symlink pointing
 # to the real file, which we were previously preserving while using «cat».
 if "$TT_WRAPPER" "${input_file}" > "${tt_tmp_output_file}" 2>/dev/null &&
+   ! grep -q -E '^file error' "${tt_tmp_output_file}" 2>/dev/null &&
    move "${tt_tmp_output_file}" "$(readlink -f "${output_file}")" ; then
   log_info "Generating ${output_file}: OK"
   RC=0
@@ -109,6 +110,11 @@ else
   log_error "Generating ${output_file} based on ${input_file}: FAILED"
   RC=1
 
+  if [[ -r "${tt_tmp_output_file}" ]] && grep -q -E '^file error' "${tt_tmp_output_file}" ; then
+    log_error "from generated file:"
+    log_error "  $(grep -E '^file error' "${tt_tmp_output_file}")"
+  fi
+
   log_info "NOTE: Check those files for valid syntax and encoding:"
   for f in "${input_file}" ${host_conf:-} ${local_conf:-} "$NGCPCTL_CONFIG" "${NETWORK_CONFIG:-}" ${EXTRA_CONFIG_FILES:-} "$CONSTANTS_CONFIG" ; do
     [ -r "$f" ] && log_info "$f"

t/fixtures/output/bad-syntax.txt

@@ -0,0 +1,2 @@
+file error - parse error - input file handle line 1: unexpected end of directive
+  [% IF %]

t/fixtures/repos/templates/etc/bad-syntax.txt.tt2

@@ -0,0 +1 @@
+[% IF %]

t/test_ngcpcfg_build_syntax.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env py.test-3
+
+import filecmp
+import os
+import pytest
+import re
+import tempfile
+import sys
+
+
+@pytest.mark.tt_46601
+def test_bad_syntax(ngcpcfgcli, tmpdir):
+    tmpdir = tempfile.mkdtemp(prefix='ngcp-', suffix='-pytest-output')
+    out = ngcpcfgcli("build", "--ignore-branch-check",
+                     "/etc/bad-syntax.txt",
+                     env={
+                         'NGCP_BASE_TT2': os.getcwd(),
+                         'NGCP_SOCKETFILE': '/tmp/ngcpcfg.socket',
+                         'OUTPUT_DIRECTORY': tmpdir,
+                         'NGCPCFG': 'fixtures/ngcpcfg_carrier.cfg',
+                     })
+
+    # debug, only printed in logs in case of error
+    print("stdout:")
+    print(out.stdout.replace("\\n", "\n"))
+    print("stderr:")
+    print(out.stderr.replace("\\n", "\n"))
+
+    regex1 = re.compile(r"NOTE: Check those files for valid syntax and "
+                        "encoding.*etc/bad-syntax\.txt\.tt2.*or inspecting "
+                        "temporary /tmp/ngcpcfg\.bad-syntax\.txt\.")
+    assert re.search(regex1, out.stdout)
+
+    regex2 = re.compile(r"Error: Generating /tmp/ngcp-.*-pytest-output.*"
+                        "/etc/bad-syntax.txt based on .*"
+                        "/etc/bad-syntax\.txt\.tt2: FAILED")
+    assert re.search(regex2, out.stderr)
+
+    regex3 = re.compile(r"Error:   file error - parse error - input file "
+                        "handle line 1: unexpected end of directive")
+    assert re.search(regex3, out.stderr)
+
+    output_file = os.path.join(tmpdir, "etc/bad-syntax.txt")
+    test_file = "fixtures/output/bad-syntax.txt"
+    output_temp_file = re.sub(r".* or inspecting temporary ", "", out.stdout)
+    output_temp_file = re.sub(r"\\.*", "", output_temp_file)
+    # debug
+    print("Output temp file: '%s'" % (output_temp_file))
+
+    assert not os.path.exists(output_file)
+    assert os.path.exists(output_temp_file)
+    assert os.path.exists(test_file)
+
+    # debug
+    if not filecmp.cmp(output_temp_file, test_file):
+        print("output_temp_file:")
+        print(open(output_temp_file).read())
+        print("test_file:")
+        print(open(test_file).read())
+    assert filecmp.cmp(output_temp_file, test_file)