But first, let us discuss …

The POSIX Resource Limited sandbox

POSIX documents setrlimit(2). Disabling the ability to open new files, sockets, etc, or create new processes is actually pretty powerful.

Thanks to the privsep dhcpcd now has to support both Capsicum and Pledge, this turned out to be pretty easy to implement. The only issue with this is the poll(2) interface which dhcpcd makes great use of. Implementations found on Linux, OpenBSD and Solaris return EINVAL when the nfds argument is greater than RLIMIT_NOFILE where-as the other OS’s dhcpcd supports don’t. This means that on Linux, OpenBSD and Solaris an attacker could close an exiting file descriptor and try to open a new one. On OpenBSD this is not that much of a big deal because pledge should stop that from happening. On Linux and Solaris, they can’t open a file thanks to the chrooted empty directory but they can create a new one. But thanks to RLIMIT_FSIZE they can’t actually write to it. At most they could create a network socket and send arbitary data over it. For Linux, we could look into using seccomp to stop this.

For implementations such as NetBSD and FreeBSD, setting RLIMIT_NOFILE to zero with poll(2) still working means they cannot create any few file descriptor. This means that if an attacker breaches a resource limited process it can only work with the resources it has because it cannot fork another process, nor open any files, sockets, etc. It’s also running as an unpriviledged user locked in an empty directory, so there is nothing to see or do there. All the resources it currently has are:

• PF_INET, PF_LINK, etc sockets that can only query for data
• network proxy process (receives only, does not send)
• BPF processes (send and receive, the read and write filters are also locked)
• privileged actioneer (filters ioctls, validates paths and outbound traffic)

So the only way to do anything outside of what dhcpcd normally does is to use a facility that does not need to create a file or socket, or fork a process. The only remaining avenue of attack is to break the privileged actioneer process- that cannot be protected by any sandboxing as it needs to do a lot of stuff. Both OpenBSD’s and FreeBSD’s dhclient have such a privileged process as well.

The priviledged actioneer process itself doesn’t do a great deal. For example to add a route on BSD you create a RTM_ADD message. The master process will do this and instead of writing to the PF_ROUTE socket itself it will pass the message to the privilged actioneer process which in turn writes to it’s PF_ROUTE socket. Every ioctl, path accessed or network bound packet is validated by the privileged actioneer. Even though it’s generic, it’s also locked down.

So what extra to Capsicum and Pledge bring to the table?

Both bring system call filtering. For example, sysctl(2) does not need to create a new resource. Information available to the ordinal user such as uname may not be desirable to leak to these sandboxed processes. But then the question to ask is what can they do with it?

Pledge overcomes the RLIMIT_NOFILE limitation for poll(2) on OpenBSD.

Capsicum goes a bit futher by limiting rights of each resource you have in terms of what you can do with them.

What they both bring to the table though is making sandboxing easier. Here, Pledge is the outright winner. As I pointed out in my initial blog post, Pledge is really easy. But make it too easy and it’s not as secure as it could be. Capsicum is harder and the resource limited sandbox is harder still.

The take-away point from this is while Capsicum and Pledge are nice, they don’t beat a good design.

Capsicum is hard to develop for

It’s either on or off. You can limit each FD with capabilites mode off, but I’m not sure what that gains as it’s mainly there to allow the FD to be used in the restricted world so we can treat it as either on or off really.

Pledge is granular, you can turn areas on or off. For example, you can allow full file access while restricting various ioctls. Ideally, the only thing you want to pledge is stdio, which is kinda needed for the Privilege Separation IPC to work. However, we end up pledging dns inet and route as well.

So why is Capsicum harder? Well, the short answer is that it’s not. The long answer is that it’s perceived to be harder because you have to do all the work up front to get your code working whereas Pledge allows you do it it piece meal.

Pledge is as secure as Capsicum

As I said in my prior post, I’m not a security expert, but here’s my findings:

You can make Pledge as secure as Capsicum, but you don’t have to.

As a user on OpenBSD I can see lots of processes marked with p to indicate they have pledged something. What that something is though is a black box. You can use ps-o pledge-p $pid to show the promises a process has pledged. The more promises you see the more that process is allowed to do. Short list good, long list bad.

As a user on FreeBSD I can see lots of processes marked with ‘C’ to indicate they are in Capsicum Capabilites Mode. Because this is just on or off, I know exactly what this process can and cannot do.

Armed with this knowledge I would trust a Capsicum enabled application more than a Pledge enabled application. But to fully trust either you still need to audit the code.

They both have limitations

With a Pledged process you cannot change a BPF filter with the bpf pledge. This is unfortunate, a possible solution could be to spawn a process AND filter for every address you want to monitor for DaD or ARP pinging. That could be a lot of resources wasted as BPF is neither infinite nor cheap.

With a Capsicum Capability Mode enabled process you cannot call sendto(2) that is not connected. Unlike the above, there is no easy solution. I can’t spawn off another process because unlike normal communication, because I need to bind to the bootpc and dhcpv6-client ports so I can dictate which source port is used to send the message. The binding would then fail because the network proxy process is already using the port. A possible solution is to call RTM_GET to work out which host I send to on the local network, craft the packet by hand and send it out using a BPF socket.

I strongly feel that both should be fixed upstream:

• the bpf Pledge should allow changing the BPF filter.
• Capsicum should allow sendto unconnected if bound to a priviledged port or another capability created to grant to the socket.

As it stands, dhcpcd does not Pledge for BPF processes and does not run the generic network proxy process in Capsicum Capabilites mode. However, both processes are using the unprivileged user and chrooted to an empty directory. Neither do any processing of data coming in, they just ferry it back to the master process via IPC which runs under the same restrictions but is also sandboxed with Pledge or Capsicum so I’m confident that it’s good enough.

The good news is that basic Capsicum support has been enabled in this commit by ensuring all the file descriptors of the network facing processes are limited in their capability. The bad news is that capability mode is only enabled for the BPF processes- the end goal is that the master process is at least placed in capability mode.

So what’s the problem? Well, the first problem is that Capsicum is really hard to use, a lot of this is not helped by the documentation. It took a long time to work out how it works and how to use it. The second problem is that capability mode is really hard core- it stops a lot of stuff working dead in it’s tracks by killing access not capability limited in the global namespace.

From dhcpcds perspective the big issue is that gethostname(3) and getifaddrs(3) won’t work. This is the biggest show stopper, hard to IPC.

EDIT: gethostname(3) does work in Capsicum capabilities mode.

Let’s just take a moment to talk about getifaddrs(3)- it’s a really powerful interface to work out what interfaces are available to work with. In some OS it’s the only mechanism to get the hardware address assigned to an interface. Also, it’s possible for the route(4) (or netlink(7)) socket to overflow so it’s required to re-learn the system state which means it cannot be cached. Not that FreeBSD supports reporting route(4) overflow, but that’s another story.

Due to the getifaddrs(3) show stopper, I decided to look into OpenBSDs pledge(2). While the documentation is still very technical, it’s easier to read and understand compared to Capsicum. It’s also really really really easy to implement in comparison as you only pledge features rather than each file descriptor. Because of this, it’s also very granular. The initial commit to enable pledge is mainly allowing the current ioctl IPC framework to pass data back again.

But can we do better? Can we operate with the same Pledges that OpenBSDs dhclient(8) uses? The answer is of course YES! By caching some initial data that does not change such as utsname and machine uuid AND moving all file IO to privsep, this commit allows almost perfect Pledge support. We could go better if privsep gains support for the SIOCGIFGROUP ioctl as this would allow the inet pledge to be dropped. It’s not quite perfect because a Pledged process cannot change a BPF filter. We would need to space a BPF process per address which is then limited by the number of BPF instances you are allowed. I’m not sure this a good thing, and ironically BPF processes are the one thing we have in Capsicums capability mode. This is also a massive improvement over the initial dhcpcd-9 release because it means we no longer need any files in the chroot directory and /var/empty can be used again. This alone should make current and new FreeBSD users of dhcpcd quite happy. It will also go some way to enabling capabilities mode for the master dhcpcd process in the future.

Now, I can’t say which technology is more secure- that’s really not my field of expertise. I can’t even say if either are really needed or actually make security worse because Privilege Separation IPC adds more code, complexity and as such could make it more prone to bugs some of which could be a security attack vector in itself.

But what I can say is that Pledge is much easier to develop for because it’s really granular unlike Capsicum which is either on or off or per fd limited.

I’ll continue working on Capsicum support in dhcpcd just to see if I can enable Capabilites Mode in the master process. When that’s finished I will do a more in-depth comparison of these two technologies.

I just wanted to thank you for such a great piece of software (dhcpcd)…

I often forget the people that make all this cool stuff that make linux. bsd and all the other unixes work… so rather sending a bug report, I’d just though i’d send you a hello! and thanks for giving my machines ip addresses!!!!!

I was diagnosing a race condition between gnome network manager, wireless, wpa_supplicant and dhcpcd…

but that for further research… (your software is doing exactly what is should do). :-).

Regards,

Lee

THIS is one of the main reasons I enjoy this hobby of mine …. random thoughts of happiness that come in from total strangers that appreciate that I made something for free that in turn made them happy :)

At a bare minimum, the hardware address of the interface is sent- this is required to work.

So, how to solve this dilema of wanting total anonymity? The answer is to randomise the hardware address. This will happen when the carrier is down OR dhcpcd starts with the interface down. Then, dhcpcd will use this random hardware address to set a DUID LL which will be used inplace of any saved DUID and set the IAID to all zeros. This combo is used by DHCP and DHCPv6 to identify a lease. As this is randomised each time the carrier comes up you get a different IP address!

Try not to use this on an unstable link as it could drain the DHCP server of addresses :(

But we can’t stop there! dhcpcd also sends some identifying options as well! For example, this is sent in the vendor class identifier:
dhcpcd-8.99.0:NetBSD-9.99.17:amd64:x86_64

It does not identify you or the device in anyway, but it does say what software is being used on which hardware. This could be used by DHCP servers to hand out a specific image to download and boot from TFTP for network boot clients.

Now, there are a gazzillion and one DHCP options out there- we don’t know what you’ve configured. So dhcpcd will mask all of them when anonymous mode is activated, unless they are essential for enabling dhcpcd to work correctly on the network. But wait! What if you really want to leak something? Like say your on a corporate network that uses DHCP security and still want to remain anonymous? Well you can! Any request or option after the anonymous option in dhcpcd.conf is turned on. So the placing of the anonymous directive is important, unlike other dhcpcd options. So far this is the only implementation of RFC 7844 which does this :)

This is NOT enabled by default because most people want stable addresses AND a flappy link could drain addresses as disussed earlier.

So far, my structures in dhcpcd are long lived- or rather the scope is design to live outside of where it was created. As such they are created on the heap and are at the mercy of malloc. Generally I use calloc so that the whole area is inited to zero as uninitialised memory is bad.

So I decided to start out and see if I can just create the structures I need on the stack. Turns out I could! Yay! Now, how to you initialise a structure on the stack to all zeros? First let us consider this structure:

struct ps_addr {
	sa_family_t psa_family;
	union {
		struct in_addr psau_in_addr;
		struct in6_addr psau_in6_addr;
	} psa_u;
#define	psa_in_addr		psa_u.psau_in_addr
#define	psa_in6_addr	psa_u.psau_in6_addr
};

The first way is memset:

struct ps_addr psa;

memset(&psa, 0, sizeof(psa));
psa.psa_family = AF_INET;

But what if you could avoid memset? Luckily the C standard allows setting any member and will zero all other members. So we can do this:

struct ps_addr psa = { .psa_family = AF_INET };

Wow!!! So simple. This reduces binary size a fair bit. But then I turned on the Memory Sanitiser and boom, it crashed hard. Why?

The answer is simple- padding. Eric S Raymond gives a very good writeup about the problem. Basically, the standard will initialise any unintialised members to zero- but padding added for alignent isn’t a member! So we need to ensure that our structure requires zero padding.

Here is the new struct:

struct ps_addr {
	sa_family_t psa_family;
	uint8_t psa_pad[4- sizeof(sa_family_t)];
	union {
		struct in_addr psau_in_addr;
		struct in6_addr psau_in6_addr;
	} psa_u;
#define	psa_in_addr		psa_u.psau_in_addr
#define	psa_in6_addr	psa_u.psau_in6_addr
};

And it allows the former structure initialisation to work and memory sanitisers are happy- so happy days :) Now, if anyone can tell me what I can use instead of the magic number 4 in the above I’d be even happier!

Now, to be clear, you have been able to do this since forever using fopen, writing to the file and then allocating your area based on the final file size. Still, it requires some memory management still but more importantly it writes to a file. Writing to a file is slow and reduces the life span of the disk you’re writing to. It’s only been fairly recently that tmpfs was a thing, but even today not all OS’s have /tmp mounted as tmpfs. Requiring this isn’t exactly ideal for a generic program to do- the setup and install should be easy. Because of all these reasons, most programs worked the string length needed and either allocated an area or string and then finally write the string. However, while saving the disk, it’s also a lot more error prone because you need to work out the length of everything and that’s not always trivial, especially for things like a DHCP client which is always building strings based on the information given by the DHCP server.

Here’s an example of open_memstream in action:

/*
 * Example program which manages an area of environment strings
 * to send to child programs.
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static const char *foo = "foo";
static const char *bar = "bar";

int main(void)
{
        char *argv[] = { "/usr/bin/env", NULL };
        char *buf, *ep, *p, **env, **envp;
        size_t buflen, nenv, i;
        FILE *fp = open_memstream(&buf, &buflen);

        fprintf(fp, "FOO=%s", foo);
        fputc('\0', fp);
        fprintf(fp, "BAR=%s", bar);
        fputc('\0', fp);

        /* We could keep fp around as our area and just rewind it. */
        fclose(fp);

        /* execve relies on a trailing NULL */
        nenv = 1;
        for (p = buf, ep = p + buflen; p < ep; p++) {
                if (*p == '\0')
                        nenv++;
        }

        /* reallocarray(3) should be standard really */
        envp = env = malloc(nenv * sizeof(char *));
        *envp++ = buf;
        for (p = buf, ep--; p < ep; p++) {
                if (*p == '\0')
                        *envp++ = p + 1;
        }
        *envp = NULL;

        execve(argv[0], argv, env);
}

As you can see, we only manage the environment array handed to to execve. open_memstream is managing our string area and fprintf is working out the length each string needs to be for us. This vastly reduces the complexity and increases the security and reliabilty of creating large environment strings, which most DHCP clients do. We could also write a helper function to write the string AND the trailing NULL terminator for the string to be more efficient. You’ll get to see this in dhcpcd-8 which should be released later this year.

• Very low extra memory needed for it’s implementation
• Fast at insertion and removal operation- both are O(1)

However, it’s just a list. And like any list, to find something we need to pick a starting point and then search in a single direction and match each item until we find it. This is described as O(N) and it’s performance roughly corrolates to the Number of objects in the list. If you need to insert at an arbitary point (for example to order the list) you need to search it, so this is O(N) also.

But is this slow?

Typically for dhcpcd, the answer is no because on a desktop, tablet, phone or SOHO router you only have a handful of objects in all the lists. However, someone ran dhcpcd on a switch with millions of routes and dhcpcd was taking minutes of CPU time for routing table modifications. So in this corner case … yes! dhcpcd is slow! A suggestion to improve matters was given in that report- use a Red-Black Tree instead of a Linked List. A RB tree guarantees O(log N) performance for searching. A patch was even submitted based on OpenBSD’s tree implementation! This reduced the dhcpcd runtime from minutes to seconds in his situation.

Now you can’t just swap in a RB Tree for a Linked List. Each object within an RB Tree has to have a unique key. Also, you need to have comparison function to define an order to the objects within the tree. Only then can you start to implement it. To find an object within the tree, you just need to know it’s key and off you go. I settled on using NetBSD’s rbtree(3) implementation instead of the initial suggestion to use the generic BSD tree(3). This mainly to reduce the binary size of dhcpcd because it’s in libc on NetBSD and on other systems it will creates a smaller binary due to the complexity of the tree(3) macros. Testing also showed it was slightly faster and also easier to actually develop with.

So should you replace all LinkedLists with RB Trees? No! That would be silly :) After all, sometimes you don’t actually need to search or order a list- a lot of the time, lists are used for push/pop operations and thus gain nothing from RB Tree.

You’ll get to see this in dhcpcd-8 which should be released later this year.

When I discovered FreeBSD and then NetBSD making dhcpcd work with the same features provided troublesome. For the IPv4 case, we needed to patch the kernel so that IFA_ROUTE remained sane. For the IPv6 case it was a lot more complicated as parts of the IPv6 stack rely on the kernel processing Router Advertisements instead of a 3rd party tool. Working this out was a lot more time consuming and complicated, but we got there with minimal fuss!

So as of now, dhcpcd can fully replace the NetBSD and Linux in kernel Router Advertisement handling code.

But why was I right after all these years? Because quite a few knowledgeable NetBSD folk assured me that the kernel was fine and it was likely dhcpcd at fault. As it turned out, it really was the kernels fault.

Anyhoo, since GW2 is a lot more casual than the others the idea is that I have more time for working on open source stuff, like say dhcpcd. Well, I finally found some time over the past few months and put DHCPv6 INFORM support into dhcpcd. This is only triggered when dhcpcd recieves an IPv6 RA with the O flag set. Luckily, I put IPv6 RS support into dhcpcd a few months ago but didn’t feel the urge to blog about it. Well, I do now!

Before the end of the year, I aim to have DHCPv6 IA support triggered via IPv6 RA with the M flag set as well. Do I have any other goals? Well, all my other projects are pretty much complete. One of the reasons why I starting gaming so much is that I’m pretty happy with the current state of affairs and anything else is just a bonus. That being said, there are a lot more DHCPv6 things to do besides IA support such as better separation between IPv4 and IPv6, moving to TAILQ lists intsead of our hand rolled and other misc stuff such as improving dhcpcd-dbus and dhcpcd-gtk.

This led to an interesting debtate about my internet connection. In a nutshell my Drayetk Vigor 100 is broken, but apparently by design. The problem is this- for IPv4 goes through my NAT whose public IP has an MTU of 1492 so Path MTU Discovery works fine. However IPv6 does not need NAT and the PPPoE interface does not have a public IPv6 and nor do the internal clients use it even it there was one. So the path MTU is 1500, which is too big for the PPPoE to handle.

The correct solution is to obviously terminate the IPv6 on a PPPoA connection, but only the Cisco 877 router does this and that’s outside of my price range, even on eBay. One solution is to clamp the MSS on the router to 1430 so that all clients work. But this is a hack. The best solution with my hardware is to force the MTU to 1492 for all nodes inside my network, so they share the same MTU as the PPPoE so that Path MTU Discovery works.

You can see this working for yourself on IPv6 now, as this site is IPv6 enabled and sits behind the PPPoE link. You’ll need to query IPv4 servers for the address though as I don’t yet have a glue record for IPv6, but as I’m changing registrar that should change in a few weeks :)

Because of this, I’ve made the dhcpcd default to request and use the MTU value if offered by the DHCP server. You’ll be seeing this in dhcpcd-5.0.5 (now out). dhcpcd-5.0.6 will feature restoring the MTU correctly between leases.