puppet: Use existing autossh tunnels as OpenSSH "master" sockets.

A number of autossh connections are already left open for
port-forwarding Munin ports; autossh starts the connections and
ensures that they are automatically restarted if they are severed.

However, this represents a missed opportunity.  Nagios's monitoring
uses a large number of SSH connections to the remote hosts to run
commands on them; each of these connections requires doing a complete
SSH handshake and authentication, which can have non-trivial network
latency, particularly for hosts which may be located far away, in a
network topology sense (up to 1s for a no-op command!).

Use OpenSSH's ability to multiplex multiple connections over a single
socket, to reuse the already-established connection.  We leave an
explicit `ControlMaster no` in the general configuration, and not
`auto`, as we do not wish any of the short-lived Nagios connections to
get promoted to being a control socket if the autossh is not running
for some reason.

We enable protocol-level keepalives, to give a better chance of the
socket being kept open.

puppet/zulip_ops/files/nagios_ssh_config

@@ -0,0 +1,6 @@
+Host *
+  ControlMaster no
+  ControlPath /tmp/ssh-%C
+
+  ServerAliveInterval 30
+  ServerAliveCountMax 3

puppet/zulip_ops/manifests/profile/nagios.pp

@@ -140,6 +140,14 @@ class zulip_ops::profile::nagios {
     require => File['/var/lib/nagios'],
   }
 
+  file { '/var/lib/nagios/.ssh/config':
+    ensure => file,
+    mode   => '0644',
+    owner  => 'nagios',
+    group  => 'nagios',
+    source => 'puppet:///modules/zulip_ops/nagios_ssh_config',
+  }
+
   # Disable apparmor for msmtp so it can read the above config file
   file { '/etc/apparmor.d/disable/usr.bin.msmtp':
     ensure => link,

puppet/zulip_ops/templates/supervisor/conf.d/munin_tunnels.conf.erb

@@ -5,7 +5,7 @@ i = 0
 @hosts.each do |host|
 -%>
 [program:munin-tunnel-<%= host %>]
-command=autossh -N -M <%= 20000 + 2 * i %> -L <%= 5000 + i %>:localhost:4949 nagios@<%= host %><% unless host.include?(".") %>.<%= @default_host_domain %><% end %>
+command=autossh -N -M <%= 20000 + 2 * i %> -L <%= 5000 + i %>:localhost:4949 -o ControlMaster=yes nagios@<%= host %><% unless host.include?(".") %>.<%= @default_host_domain %><% end %>
 priority=200                   ; the relative start priority (default 999)
 autostart=true                 ; start at supervisord start (default: true)
 autorestart=true               ; whether/when to restart (default: unexpected)