But first, let us discuss …

The POSIX Resource Limited sandbox

POSIX documents setrlimit(2). Disabling the ability to open new files, sockets, etc, or create new processes is actually pretty powerful.

Thanks to the privsep dhcpcd now has to support both Capsicum and Pledge, this turned out to be pretty easy to implement. The only issue with this is the poll(2) interface which dhcpcd makes great use of. Implementations found on Linux, OpenBSD and Solaris return EINVAL when the nfds argument is greater than RLIMIT_NOFILE where-as the other OS’s dhcpcd supports don’t. This means that on Linux, OpenBSD and Solaris an attacker could close an exiting file descriptor and try to open a new one. On OpenBSD this is not that much of a big deal because pledge should stop that from happening. On Linux and Solaris, they can’t open a file thanks to the chrooted empty directory but they can create a new one. But thanks to RLIMIT_FSIZE they can’t actually write to it. At most they could create a network socket and send arbitary data over it. For Linux, we could look into using seccomp to stop this.

For implementations such as NetBSD and FreeBSD, setting RLIMIT_NOFILE to zero with poll(2) still working means they cannot create any few file descriptor. This means that if an attacker breaches a resource limited process it can only work with the resources it has because it cannot fork another process, nor open any files, sockets, etc. It’s also running as an unpriviledged user locked in an empty directory, so there is nothing to see or do there. All the resources it currently has are:

• PF_INET, PF_LINK, etc sockets that can only query for data
• network proxy process (receives only, does not send)
• BPF processes (send and receive, the read and write filters are also locked)
• privileged actioneer (filters ioctls, validates paths and outbound traffic)

So the only way to do anything outside of what dhcpcd normally does is to use a facility that does not need to create a file or socket, or fork a process. The only remaining avenue of attack is to break the privileged actioneer process- that cannot be protected by any sandboxing as it needs to do a lot of stuff. Both OpenBSD’s and FreeBSD’s dhclient have such a privileged process as well.

The priviledged actioneer process itself doesn’t do a great deal. For example to add a route on BSD you create a RTM_ADD message. The master process will do this and instead of writing to the PF_ROUTE socket itself it will pass the message to the privilged actioneer process which in turn writes to it’s PF_ROUTE socket. Every ioctl, path accessed or network bound packet is validated by the privileged actioneer. Even though it’s generic, it’s also locked down.

So what extra to Capsicum and Pledge bring to the table?

Both bring system call filtering. For example, sysctl(2) does not need to create a new resource. Information available to the ordinal user such as uname may not be desirable to leak to these sandboxed processes. But then the question to ask is what can they do with it?

Pledge overcomes the RLIMIT_NOFILE limitation for poll(2) on OpenBSD.

Capsicum goes a bit futher by limiting rights of each resource you have in terms of what you can do with them.

What they both bring to the table though is making sandboxing easier. Here, Pledge is the outright winner. As I pointed out in my initial blog post, Pledge is really easy. But make it too easy and it’s not as secure as it could be. Capsicum is harder and the resource limited sandbox is harder still.

The take-away point from this is while Capsicum and Pledge are nice, they don’t beat a good design.

Capsicum is hard to develop for

It’s either on or off. You can limit each FD with capabilites mode off, but I’m not sure what that gains as it’s mainly there to allow the FD to be used in the restricted world so we can treat it as either on or off really.

Pledge is granular, you can turn areas on or off. For example, you can allow full file access while restricting various ioctls. Ideally, the only thing you want to pledge is stdio, which is kinda needed for the Privilege Separation IPC to work. However, we end up pledging dns inet and route as well.

So why is Capsicum harder? Well, the short answer is that it’s not. The long answer is that it’s perceived to be harder because you have to do all the work up front to get your code working whereas Pledge allows you do it it piece meal.

Pledge is as secure as Capsicum

As I said in my prior post, I’m not a security expert, but here’s my findings:

You can make Pledge as secure as Capsicum, but you don’t have to.

As a user on OpenBSD I can see lots of processes marked with p to indicate they have pledged something. What that something is though is a black box. You can use ps-o pledge-p $pid to show the promises a process has pledged. The more promises you see the more that process is allowed to do. Short list good, long list bad.

As a user on FreeBSD I can see lots of processes marked with ‘C’ to indicate they are in Capsicum Capabilites Mode. Because this is just on or off, I know exactly what this process can and cannot do.

Armed with this knowledge I would trust a Capsicum enabled application more than a Pledge enabled application. But to fully trust either you still need to audit the code.

They both have limitations

With a Pledged process you cannot change a BPF filter with the bpf pledge. This is unfortunate, a possible solution could be to spawn a process AND filter for every address you want to monitor for DaD or ARP pinging. That could be a lot of resources wasted as BPF is neither infinite nor cheap.

With a Capsicum Capability Mode enabled process you cannot call sendto(2) that is not connected. Unlike the above, there is no easy solution. I can’t spawn off another process because unlike normal communication, because I need to bind to the bootpc and dhcpv6-client ports so I can dictate which source port is used to send the message. The binding would then fail because the network proxy process is already using the port. A possible solution is to call RTM_GET to work out which host I send to on the local network, craft the packet by hand and send it out using a BPF socket.

I strongly feel that both should be fixed upstream:

• the bpf Pledge should allow changing the BPF filter.
• Capsicum should allow sendto unconnected if bound to a priviledged port or another capability created to grant to the socket.

As it stands, dhcpcd does not Pledge for BPF processes and does not run the generic network proxy process in Capsicum Capabilites mode. However, both processes are using the unprivileged user and chrooted to an empty directory. Neither do any processing of data coming in, they just ferry it back to the master process via IPC which runs under the same restrictions but is also sandboxed with Pledge or Capsicum so I’m confident that it’s good enough.

The good news is that basic Capsicum support has been enabled in this commit by ensuring all the file descriptors of the network facing processes are limited in their capability. The bad news is that capability mode is only enabled for the BPF processes- the end goal is that the master process is at least placed in capability mode.

So what’s the problem? Well, the first problem is that Capsicum is really hard to use, a lot of this is not helped by the documentation. It took a long time to work out how it works and how to use it. The second problem is that capability mode is really hard core- it stops a lot of stuff working dead in it’s tracks by killing access not capability limited in the global namespace.

From dhcpcds perspective the big issue is that gethostname(3) and getifaddrs(3) won’t work. This is the biggest show stopper, hard to IPC.

EDIT: gethostname(3) does work in Capsicum capabilities mode.

Let’s just take a moment to talk about getifaddrs(3)- it’s a really powerful interface to work out what interfaces are available to work with. In some OS it’s the only mechanism to get the hardware address assigned to an interface. Also, it’s possible for the route(4) (or netlink(7)) socket to overflow so it’s required to re-learn the system state which means it cannot be cached. Not that FreeBSD supports reporting route(4) overflow, but that’s another story.

Due to the getifaddrs(3) show stopper, I decided to look into OpenBSDs pledge(2). While the documentation is still very technical, it’s easier to read and understand compared to Capsicum. It’s also really really really easy to implement in comparison as you only pledge features rather than each file descriptor. Because of this, it’s also very granular. The initial commit to enable pledge is mainly allowing the current ioctl IPC framework to pass data back again.

But can we do better? Can we operate with the same Pledges that OpenBSDs dhclient(8) uses? The answer is of course YES! By caching some initial data that does not change such as utsname and machine uuid AND moving all file IO to privsep, this commit allows almost perfect Pledge support. We could go better if privsep gains support for the SIOCGIFGROUP ioctl as this would allow the inet pledge to be dropped. It’s not quite perfect because a Pledged process cannot change a BPF filter. We would need to space a BPF process per address which is then limited by the number of BPF instances you are allowed. I’m not sure this a good thing, and ironically BPF processes are the one thing we have in Capsicums capability mode. This is also a massive improvement over the initial dhcpcd-9 release because it means we no longer need any files in the chroot directory and /var/empty can be used again. This alone should make current and new FreeBSD users of dhcpcd quite happy. It will also go some way to enabling capabilities mode for the master dhcpcd process in the future.

Now, I can’t say which technology is more secure- that’s really not my field of expertise. I can’t even say if either are really needed or actually make security worse because Privilege Separation IPC adds more code, complexity and as such could make it more prone to bugs some of which could be a security attack vector in itself.

But what I can say is that Pledge is much easier to develop for because it’s really granular unlike Capsicum which is either on or off or per fd limited.

I’ll continue working on Capsicum support in dhcpcd just to see if I can enable Capabilites Mode in the master process. When that’s finished I will do a more in-depth comparison of these two technologies.