Hardening Cockpit with systemd (socket activation)³

Background

A major future goal for Cockpit is support for client-side TLS authentication, primarily with smart cards. I created a Proof of Concept and a demo long ago, but before this can be called production-ready, we first need to harden Cockpit’s web server cockpit-ws to be much more tamper-proof than it is today.

This heavily uses systemd’s socket activation. I believe we are now using this in quite a unique and interesting way that helped us to achieve our goal rather elegantly and robustly.

Level 1: Good old times - The lonely unit

Cockpit’s web server had been a single process for a long time. It handled all HTTP and HTTPS connections, i. e. it multiplexed an arbitrary number of parallel user sessions. Of course the session processes themselves each run in their own logind cgroup and such (like any other gdm/ssh/VT Linux session), but they are all connected and controlled by the browser’s JavaScript and thus everything gets routed through the web server – that essentially translates the browser’s TCP streams to the session’s stdin/out JSON protocol. Plus, the initial login authentication (passwords, kerberos, auth tokens, etc.) of course also has to go via the web server.

That’s a lot of responsibility to bear – any possible vulnerability in HTTP processing in Cockpit itself or any of its libraries (glib, gio, GnuTLS, glibc, etc.) could lead to owning all current user sessions or being able to sniff password inputs in the worst case.

From a systemd service point of view, this wasn’t yet that interesting. There was a simple socket unit that activated the web service as soon as the first TCP connection came in, and the web server timed out after 90 seconds of inactivity to conserve resources – classic socket activation like mom used to make it. :-)

Of course the web server runs as an unprivileged system user cockpit-ws. The authentication happens in a separate and very small suid root helper cockpit-session. But due to all the session multiplexing from above and the sheer hopelessness of auditing so much code that only helped so far – if you manage to arbitrarily manipulate http streams, you can eventually 0wn the box.

Level 2: Up the ante - Split the service

In a smart card world where your authentication completely hinges on trusting that the user has presented the private key to a public TLS certificate, the above is simply not good enough. If an attacker ever manages to run arbitrary code in cockpit-ws, they could just present any user’s public certificate to the subsequent authentication stack (PAM and sssd in this case), and would get a session as that user.

The new design was to split out TLS termination into a separate program cockpit-tls which would act as a reverse proxy for cockpit-ws. This only does the TLS handshake and decryption, but almost no interpretation of the HTTP stream: The only thing it looks at is if the first byte after the connection is 22, which indicates TLS; if not, it’s HTTP. The interpretation is then delegated (proxied) to a cockpit-ws instance for the given client TLS certificate, plus another instance for “TLS without client certificate”, plus yet another instance for “plain http”.

The only thing that cockpit-tls does is to shovel raw data between two sockets, which reduces the attack surface dramatically. Also, it uses plain C with glibc and GnuTLS only, so it’s small enough to audit for humans or Coverity. And splitting cockpit-ws across session trust boundaries means that if an attacker can run arbitrary code in it, they can’t influence other sessions any more or fake spoof the authentication.

This was introduced in multiple steps, to be able to land it in finite time. The first version just directly forked cockpit-ws as the same user; it did the structural ground work, but didn’t yet have any actual security benefits. These then arrived when we changed it to use systemd socket activation instead: Now cockpit-tls and the cockpit-ws instances run as different system users, and http and https sessions are isolated from each other:

+---------+  http://machine:9090                           +------------------------+
|Browser A|+----------------------+                    +-->|cockpit-ws http instance|
+---------+                       |                    |   +------------------------+
                                  |    +------------+  |
                                  +--->| cockpit-tls|--+   plain HTTP over Unix socket
                                  |    +------------+  |
+---------+  https://machine:9090 |                    |   +-------------------------+
|Browser B|+----------------------+                    +-->|cockpit-ws https instance|
+---------+                                                +-------------------------+

To implement this, we introduced a second layer of socket activation. The first layer is still by and large the same cockpit.socket as always, but cockpit.service now starts the socket activation for the http and https instances 1:

Requires=cockpit-wsinstance-http.socket cockpit-wsinstance-https.socket
After=cockpit-wsinstance-http.socket cockpit-wsinstance-https.socket

The involved units: http socket and service, and the https socket and service.

Now we have an unprivileged system user process spawning another unprivileged process running as a different system user on demand, in a race free, simple, robust, and secure manner (no suid helpers involved here!). cockpit-tls can just connect to the instances’ Unix sockets and systemd does the rest for you.

Level 3: Full Induction - Per-certificate instances

But that still doesn’t meet our goal – we want to shield TLS sessions with different client certificates from each other to safely do smart card authentication. We need a way to not just have one https instance, but arbitrarily many:

+---------+  http://machine:9090                           +------------------------+
|Browser A|+----------------------+                    +-->|cockpit-ws http instance|
+---------+                       |                    |   +------------------------+
                                  |                    |
                                  |                    +   plain HTTP over Unix socket
                                  |                    |
+---------+  https://machine:9090 |    +------------+  |   +------------------------------------+
|Browser B|+----------------------+--->| cockpit-tls|--+-->|cockpit-ws https instance for cert B|
+---------+    client cert B      |    +------------+  +   +------------------------------------+
                                  |                    |
                                  |                    |
+---------+  https://machine:9090 |                    |   +------------------------------------+
|Browser C|+----------------------+                    +-->|cockpit-ws https instance for cert C|
+---------+    client cert C                               +------------------------------------+

So we need .. an instance factory. Fortunately, systemd has that built in via template units and a socket’s Accept= option!

I have a pull request to implement this, and it was surprisingly straightforward: The previous single https instance now becomes a template unit with c-w-https@.socket and c-w-https@.service, which is just a file rename and adding two %i. Thus each instance now listens to its own /run/cockpit/wsinstance/https@%i.sock Unix socket.

Then there is the new c-w-https-factory.socket which has Accept=yes. Each connection to its Unix socket runs a new temporary instance of c-w-https-factory@.service which is launches a new https instance with the same %i name, tells the caller its socket path, and exits again2:

[Unit]
Description=Cockpit Web Service https instance factory
Requires=cockpit-wsinstance-https@%i.socket
After=cockpit-wsinstance-https@%i.socket

[Service]
ExecStart=/bin/sh -ec 'echo -n https@%i.sock >&3'`

So you can e. g. do nc -U /run/cockpit/wsinstance/https-factory.sock, and it will respond with https@0.sock (the name in /run/cockpit/wsinstance/) and launch the corresponding cockpit-ws socket. Each time you connect to the factory you get a new cockpit-ws instance.

cockpit-tls itself does not need to do much – it mostly just needs to keep a mapping of client certificate → ws Unix socket path, and call the factory for a previously unseen certificate. There are no races, no need to ever delete the instances, and no complicated housekeeping. All instances are now running isolated from each other and from the TLS termination and certificate checking. Voilà!

Conclusion

Take a step back and ponder for a bit what that means: Initially there are zero running processes, so no resources wasted as long as nothing talks to Cockpit. Then, on a connection to port 9090,

  • systemd activates cockpit.service (level 1),
  • which starts sockets for the http and https factories (level 2),
  • whose service is then started as soon as cockpit-tls determines the right one it needs to talk to,
  • which for a TLS connection is another https instance socket unit (level 3),
  • which finally starts the per-instance cockpit-ws service.

socket activation cubed – All of that has no races, is isolated, secure, and easy to work with in plain C.

If you have comments or question, please respond to my tweet.

  1. For simplicity, this blog post skips the https-redirect instance; it does not add any new aspect here. [return]
  2. The real implementation calls a C program, not echo, because that works better with SELinux. But it essentially does the same. [return]
Home