16 Commits (85ba31c39330e2bc1761ed0e044831bec180b51b)

Author	SHA1	Message	Date
Nico Schedel	731a56ce95	TT#56376 selenium: add "experimental" launch option ... testrunner: - add option to launch a experimental test run, which will include all tests including those which havent been added to the normal test plan yet. - this can be triggered by typing "experimental" or "exp" instead of a filename. runner.t: - create experimental test plan if experimental arg is used. Change-Id: I921664addbc0fa5c06c5d1a619baec82590eb8c0	6 years ago
Nico Schedel	11cbf1007d	TT#56376 selenium: dont execute every .t file in directory by default ... testrunner: - fix so that *.t doesnt select every file in directory, even ones who are not added to the default test plan yet. Change-Id: I6ed9057fed87244b7968cdc05f60825941675579	6 years ago
Nico Schedel	f9ced2f830	TT#56376 selenium: Multithread tests ... Functions.pm: - add create_driver(). It will check which port can be used and returns a webdriver element. testrunner: - start 4 instances of geckodriver instead of 1 - start runner.t by default. it is still possible to run other files if needed. runner.t will first run a test login to see if the server is here. If login doesnt work, it will abort the test. runner.t will then exectute the tests (max. 4 in parallel). Test files have been modified accordingly to either work with runner.t or to execute independently. Change-Id: I628ea82ad786a6ebd713747fe07dda45a538a01c	6 years ago
Alexander Lutay	7dbe4a13e2	TT#56376 Show selenium/rest-api tests execution time on Jenkins ... XML files generated by prove will contain the time we spend in every test case, it allows us profile our tests and fix the most slop parts. Change-Id: I965af2d95f57d8975613da0f46ce8abf1be4c114	6 years ago
Alexander Lutay	f0e1f74fc6	TT#56376 Restore exit code propogation in t/selenium/testrunner ... Jenkins uses wrapper jenkins_docker_run which will mask exit code anyway and this is what we want currently: we need always zero exit code here on Jenkins. Jenkins will detect errors in produced artifacts (XML files) if any as a result the Jenkins job will be "yellow" instead of "red". Change-Id: I249fcde9103ee25e6798fcb835ee4a408f8f0f57	6 years ago
Nico Schedel	376e3e80c1	TT#56376 selenium: various t/selenium/testrunner improvements ... testrunner: - removed -m from local Perl launch options. this would cause diag messages to only print when the script is finished. Without it we can see in realtime where scripts get stuck. still used for Jenkins - can now specify which test cases should be executed in launch options. - code cleanups jenkins_docker_run: - remove hardcoded old parameters as Jenkins job path all the necessary parameters already in the proper order, we just need to proxy them into the shell script 'testrunner' to run it under the user 'selenium'. Change-Id: Icaced3c3ce65cd23f3037517296e49afce4a9414	6 years ago
Alexander Lutay	653291b07a	TT#46791 Teach Jenkins job ngcp-panel-test-selenium-docker run one test only ... It should speedup the testing when we debug one specific test only. Change-Id: I7a2235bd3225836f3dcbdda6af2eaa6b65e71cff	6 years ago
Alexander Lutay	c2ae23fff9	TT#46791 Improve 'prove' output for Selenium and REST API tests ... It improves readability slightly. Change-Id: Iaecd72eded4f495a38e2009eef89445f6517b688	6 years ago
Alexander Lutay	571c09fdc8	TT#46791 Fix exit code passing in pipe for selenium testrunner ... The old code didn't work as expected, as RC receives 'tee' exit code (which is zero in most of the cases) instead of 'prove' exit code. The new code gets 'prove' exit code and it changes nothing at the end, as we just print it into debug message. Anyway Jenkins operates with JUnit output to detect issues. It is better to have 'correct' code here to prevent misunderstanding in the future. Change-Id: I09f399813179590a67c25509f310c91f7e297f3a	7 years ago
Gerhard Jungwirth	b47cf5f70f	TT#17853 TT#21767 make new selenium library visible to jenkins ... adapt PERL5LIB in a way, so that jenkins is also able to find it (it uses /home/selenium/jenkins_docker_run) Change-Id: I2d5ab35c77abd2dda9f676768da8cd6a36c4ddfb	8 years ago
Alexander Lutay	cccc6a9a41	TT#17032 prove STDERR output is not a part of Jenkins's result XML ... Which confuses developers which are debugging failed tests, as details like below are not a part of XML Jenkins displays on a WEB: > # Failed test 'create test customer w/o profile package' > # at t/api-rest/api-balanceintervals.t line 1578. > # got: '500' > # expected: '201' > Use of uninitialized value in concatenation (.) or string at t/api-rest/api-balanceintervals.t line 1580. > Thread 4 terminated abnormally: malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<!DOCTYPE HTML>\n<!-...") at /usr/share/perl5/JSON.pm line 171. 'man prove' says: > -m, --merge Merge test scripts' STDERR with their STDOUT. Change-Id: I945d4d8867fc9c061f96d067be14125af7cc5a46	8 years ago
Michael Prokop	2b021efb56	TT#17853 Fix selenium related toolchain, replacing selenium with geckodriver ... An unintended docker image rebuild of ngcp-panel-tests-selenium-jessie resulted in failures with our selenium tests. A series of problems started: 1) Updating the Selenium version resulted in an error message by Selenium, requesting the geckodriver binary (https://github.com/mozilla/geckodriver - being a proxy for using W3C WebDriver-compatible clients to interact with Gecko-based browsers). So Selenium versions >=3 depend on geckodriver. 2) Recent versions of Selenium also depend on Java >=8, available only via backports for jessie. Since we avoid usage of backports as much as possible and we'd have to upgrade once anyway we decided to base our fresh docker image on stretch. 3) We have to backport Selenium::Remote::Driver (Debian package libselenium-remote-driver-perl) for stretch (since it's not available in Debian, even though there's an ITP: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839569). It has a bug which prevents its usage for us: https://github.com/gempesaw/Selenium-Remote-Driver/issues/319 -> We fixed the problem via our own backport and submitted a PR towards upstream: https://github.com/gempesaw/Selenium-Remote-Driver/pull/321 4) Firefox from Debian/stretch (firefox-esr 52.2.0esr-1~deb9u1) fails with geckodriver e.g. in click events, the current Firefox version 54 works fine though. So instead install only the relevant runtime packages from Debian and use the Firefox version v54 from upstream. Resulting changes: * switch from Debian/jessie to Debian/stretch * no longer use selenium at all but only geckodriver instead (recent versions of selenium depend on geckodriver anyway) * do all package/file changes in /home/selenium/ (to clearly see what's from us/docker and what are actual temporary files) * testrunner related changes: - invoke vncserver with `-localhost no` option to accept VNC connections not only from localhost only - adjust vncserver process name for Xvnc4->Xtigervnc switch - start geckodriver instead of selenium * package related changes: - backported libtest-pretty-perl (v0.32) - backported libselenium-remote-driver-perl (v1.20) - add curl to package list since it's handy for debugging test failures, like e.g. `curl -X POST localhost:4444/session --data {}` - add procps to have pkill binary available inside testrunner - openjdk/java is no longer needed since we don't depend on selenium anymore - vnc4server is now a transitional package for tigervnc-standalone-server, so directly depend on it Change-Id: I607ffd4fce80f9e84b76c3c7a19824819ac4503c	8 years ago
Alexander Lutay	6d35b85a6b	MT#22099 Docker >=1.11 no longer has /.dockerinit, adjust docker run check accordingly ... File /.dockerinit no longer exists. With Docker versions 1.11 and 1.12 there's /.dockerenv available, though it's not yet sure whether this will be present in the future too, so check for either /.dockerenv or a proper cgroup setup inside the docker container. File /proc/1/cgroup with Docker version 1.12.1 on Debian/jessie looks like: | root@4c3a1c6d773c:/# cat /proc/1/cgroup | 8:freezer:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 7:perf_event:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 6:net_cls,net_prio:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 5:cpuset:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 4:blkio:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 3:cpu,cpuacct:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 2:devices:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 | 1:name=systemd:/docker/4c3a1c6d773c1a5675f24ff58540ff1a290489e7325e1e66ec2c3392aa3a1145 See https://github.com/docker/docker/issues/18355 and https://github.com/docker/docker/pull/19490 for details. Change-Id: I79fff0eead1a11f80b461791b2285a65d706262a	9 years ago
Michael Prokop	a236b39bef	MT#15071 Jenkins<->docker integration support ... The jenkins_docker_run script is invoked from Jenkins when running the docker image. The testrunner script now takes two additional arguments to support sharing test results via another docker volume and to control output format ("pretty" as default but junit for usage on Jenkins). While at it fix some quoting issue identified by shellcheck. Change-Id: Ided716e0e0ca23751b0ee2961f3e301acb80128d	10 years ago
Gerhard Jungwirth	7f2076e8ab	MT#15071 some docker conveniences ... Change-Id: I901a6012ad00e5f318ac6eda5c107e5c4d1241b7	10 years ago
Michael Prokop	b17bc4da82	MT#14585 Docker setup for selenium tests ... This provides a setup to run ngcp-panel tests against selenium-server using docker. It's based on Debian/jessie and assumes are reachable NGCP system for running the tests against (e.g. via `vagrant up ce-trunk`). For usage instructions refer to the Dockerfile. NOTE: replacing 'jessie' with 'wheezy' in the Dockerfile provides a setup based on Debian/wheezy. Change-Id: I79ca4d44c3a7c454f99148889d4b0d075a79e82b	10 years ago