From 87305d49ef3d10136da8d68079a6bb690d5af842 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 4 May 2018 14:21:02 +0200 Subject: [PATCH] TT#36177 Terminate ssh connection during reboot via timeout This is supposed to fix the hanging SSH connection when triggering the reboot: | jenkins@jenkins-slave7:~$ ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i [...] sudo reboot -f | [...] | Rebooting. | Timeout, server ec2-[...].eu-west-1.compute.amazonaws.com not responding. Without ServerAliveInterval + ServerAliveCountMax options the session might get stuck at "Rebooting", timing out only after 2 hours at worst (which is the TCP keepalive default timeout, see /proc/sys/net/ipv4/tcp_keepalive_time). Also drop the force option of reboot, which is causing troubles on sysv systems. The ssh process isn't terminated cleanly then, but ssh is running into the TCP keepalive timeout then. It seems also our systemd backport with the async reboot/halt/poweroff patch works fine, while with the "-f" option it's failing similar to the sysv situation. Change-Id: I3c2c36234d6282f96073ef558fe287d0ac3fa192 --- ec2-create-ce | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ec2-create-ce b/ec2-create-ce index 9f46853..ef341f4 100755 --- a/ec2-create-ce +++ b/ec2-create-ce @@ -443,7 +443,7 @@ ssh -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -i "${KEY_FI if $_opt_skip_reboot ; then echo "*** Skipping system reboot of instance ID ${INSTANCE_ID} as requested via --skip-reboot ***" else - ssh -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -i "${KEY_FILE}" "admin@$HOSTNAME" "sudo reboot -f" + ssh -o "ServerAliveInterval=10" -o "ServerAliveCountMax=1" -o "StrictHostKeyChecking=no" -o "UserKnownHostsFile=/dev/null" -i "${KEY_FILE}" "admin@$HOSTNAME" "sudo reboot" retry=120 # up to 10 minutes STATUS="" while [ "$STATUS" != "running" ] && [ $retry -ne 0 ] ; do