The new treatment plan is a pill based one which targets the cancer itself to stop it from growing. This, however, is not a cure and the life expectancy is not great. The median is one year and depending on which study you read 20-40% survive 5 years after which they seem to stop tracking. At the clinic I go to, one lady reportedly is on 7 years and still ticking.

So there’s potentially life in this old boy yet!

Not having a spare 12V 1A adapter with the right barrel made testing this pretty darn hard and desperate times convinced me to buy the USG (which is still working like a champ, no regrets here!). So I waited until everyone was in bed (not hard due to my medication as I don’t sleep much) and used the USG power supply. Still didn’t boot, but performing a hardware reset did!

Unsure what the hardware reset does, maybe update some internal flash a little, unsure if it touches the USB stick or relies on EdgeOS being installed rather than say NetBSD, but it’s done the trick! I bought a cheapo 12V 1A AC adapter with the right barrel size and plugged it in today and it still boots.

This means I can keep the USG powering my internet at blazingly fast speeds and the ERLite for NetBSD development as it doesn’t support any of the network hardware offload the ERLite has, so throughput suffers as a result.

If anyone from Ubiquiti reads this, please do something about your power adapters as it seems this has happened to quite a few customers after reading past community posts on your forums!

Thanks to the wonderful folks over at the NetBSD port-mips mailing list for their valuable assistance.

This is a 3 port router. Each port handles 1 Gigabits of traffic. It’s perfect for connecting your ISP with your LAN with a spare port left over. This is a router for technical people, it’s not for others as the setup is tricky. The Web UI is very dated but functional. It’s good enough to get things going. For more exotic things like say a VPN you need to use the command line and use the custom configuration tool.

Internally the CPU is a dual MIPS64 Cavium Octeon with 512Mb RAM and more importantly a removable USB drive which houses the OS. Which means we can put any OS on we want :) I had a lot of fun working on NetBSD with this, making things tick for MIPS64 and the exotic 64-bit kernel with 32-bit userland compat gunk.

But as a performance router, anything other than the stock OS is a bad choice as you then forgo Hardware Offloading which is a massive hit. I get 200mbs down with my ISP, but with NetBSD I was only getting about 80mbs. You have been warned.

So anyway, it died. I was sad. No shop sold a new ERLite either! I replaced it with a Unify Security Gateway which ironically enough is the same hardware as the ERLite, just in a different casing.

The USG is the one with the bright blue square at the top. And yes, my cable management is lacking!

The big difference between the ERLite and the USG is the software. The ERLite runs EdgeOS you need to configure it on the router directly. The USG needs to be setup by a Unify Controller which is a Java application that can be hosted anywhere. I of course host mine on my NetBSD server! You can also use a smartphone application as a controller or even better - talk to your existing one so you don’t leave the sofa. Now, you only need the controller for setup, you don’t need to leave it running all the time unless you’re interested in collecting metrics of your network over time.

I already used the controller to manage my UAP-AC-Pro for quite a while so I’m quite used to it. The Web UI is always being improved and it’s the most reliable Wireless Access Point I’ve ever had. Once you use this, nothing else will simply do! For reliable home use anyway. I can play football with the kids on the common land next to my house and still enjoy good wireless!

Anyway, the USG integrates with this. First off the bat, don’t use the beta interface. Unlike the UAP, the USG setup is poor here as the firewall setup is in two places which is odd. Also, there is an issue setting up IPv6 with a tunnel related to the firewall. Using the classic setup with the v6 controller you can do pretty much all this aside from the tunnel.

As on the ERLite, my IPv6 tunnel needs to be setup on the commandline. This time around, I used a provisioning script to setup the tunnel AND the needed parts in the firewall for it just incase I use the beta screens again.

Coupling the USG with the Unify 8-Port Managed Gigabit Switch and my UAP-AC-Pro wireless point I get nice metrics on the Unify Controller about the state of my network. And I do like a good network :)

But what about the performance? Well, here’s the interesting part. Even though it’s the same hardware I consistently hit my ISP limits a lot more than with the ERLite. I would like to think it’s due the the software it runs as it’s different, but it’s also likely to be some configuration issue on my part. Or it could have been a sign that the ERLite was already failing somehow. Either way, I’m very very happy with this!

Now, this is not a cheap setup. However, because I self host, I cannot use my ISP’s router. So this is my solution, mainly due to being burned with useless access poinst over the years and I just went with Ubiquiti from there. The company itself basically sells near enterprise level devices at SOHO prices making itself very attractive to people like me.

At the time, Abbey was a front-line worker at a care home and a supermarket. Luckily her job allowed her to help home school the kids during the lockdown allowing me to work and then look after the kids while she worked nights and weekends.

For this, we were incredibly lucky compared to others.

Then just as summer started, Abbey found a lump on my left shoulder close to my neck. Really quite small, smaller than the tip of a snooker cue. Only detectable by pressing quite hard - I was getting a much needed massage at the time. We looked at it again 2 weeks later and lump still there so off to the doctors! Doctor said absoloutely nothing to worry about, but just to be sure they will stick a camera down my throat.

2 months later (yay Covid-19) I finally got to see the throat specialist. He put a camera up my nose and down my throat. Yuck!!! He couldn’t see anything wrong. However, the lump was bigger. As the camera turned up nothing, a biopsy was required.

Now the NHS started to take a bit more seriously and the appointemts came quicker. Over the coming months I had numerous tests all coming back blank, the doctors were confused as to the growth. I then had a more invasive biopsy that finally returned the cause - Melanoma. This came as quite a shock to everyone, including the doctors, as there was no obvious primary site as it starts on top of the skin and apparently lumps from it don’t travel too far. So I was examined from head to toe and no primary site found. They even stuck the hateful camera down my throat again via the nose as the throat specialist wanted another look. Again, nothing.

But this was not a shock to me because I had a golf ball sized lump of Melanoma removed from my back almost 20 years ago. At the time it was a simple “We’ll cut that out right here” in the doctors surgery and I went to work afterwards. Then I got a letter saying I needed an operation as it was maligant. The operation was deemed a resounded success and after numerous checkups I was given the all clear.

But of course, like the best evil villans they always come back for the sequel. This time around though it’s a lot more serious - due to the now sizeable growth I was sporting and it’s was now pressing against my neck an operation was impossible. Too much vital stuff to cut through as they need to extract healthy tissue as well to prove they got it all.

Now medical science has come a long way and I embarked on a course of Dual Immunotherapy. This was only approved for use on the NHS in November 2018 so I’m very glad the cancer waited for this to be available.

Unlike the prior surgery, this is a lot more dangerous as it takes the brakes off your immune system and it can attack healthy parts too. Day zero, my left knee buckled and left me hardly able to walk or get up stairs. Doctors still doubtful if caused by immunotherapy as it was hours after the first session, normally it’s like over a month. I’ve lost a lot of weight, I barely ate, the pain in my knee was agonising and left me with little sleep. I was weak, tired and grumpy. And the lump on my neck got bigger and bigger.

Here is a picture of me and my lump, November 15th. Now clearly visible and the pressure againt my neck is uncomfortable at all times.

At the end of November, a blood test prior to my 3rd treatment showed that my liver had gone critical. If not for Covid-19, I would have likely been hospitalised, but instead we agreed I come come in every day for a blood test and based on the results, given a dose of Prednisolone, a steroid intravenously and then sent home. Did I mention I had little sleep due to my knee? Now I get even less. At the worst, I got about 2 hours sleep a night at best it was 4. Sadly this also meant that I had to stop the immunotherapy as it won’t work with steroids.

The week before Christmas my liver showed signs of recovery and I was allowed to stay at home taking steroids pills (lower dose I think) and do a blood test every other day due to the holiday season so they can remotely monitor my recovery. But the lump grew again and was pressing really quite hard. I started to get a lot of pain and the shoulder became impossible to move. I was getting pretty scared now. Barely any sleep, plenty of pain and discomfort.

But Christmas itself was a joyous afair. Even with Covid-19, my side of the family was able to make the yearly trip which I was very thankful for. I kept on saying to myself Roy, this is the last time you’ll see them. Of course they all knew what I was going through and they could see I was really struggling just moving around.

After Christmas, the pain was too much. You can see from the below picture just how much the lump has grown in just over a month.

I had a meeting with my specialist over the phone and based on what I said, she wanted to put me on the targeted tablet approach. This is not immunotherapy, it’s much less invasive to the system and can be taken on steroids. It also quickly reduces the size of the growth, giving comfort. But it also has a low life expectancy - the median is 11.5 months. Not even a year. Still, more than the 6 months with no treatment.

I did not accept this. I have young kids to watch grow up and a loving wife to grow old with. Life and time are the two most precious commodoties we will ever have. Grudgingly, the specialist agreed that we would do more tests to see how my body was otherwise coping. The most important test was the PET scan which would show if the cancer had spread. Even if the tests come back fine, the doctors did not want to offer me a false hope of restarting the immunotherapy, because they made it clear right there that the drugs do not always work. But we knew this already.

So this is how I saw in the New Year. With the growing pain and growing lump, increasing confusion, short term memory issues and feeling very light headed, I was already expecting the worst, but I still wanted the best. I resolved that if I am given the choice I would beg to continue the immunotherapy and gamble my life but I had also accepted that if my body can’t take it then I’ll take that last year.

Today, 5 days later, I got the results of the my tests. Not only had the cancer not spread, my immune system was killing it!!! So, the cancer was being killed from the inside but it was growing on the outside. My liver is almost back to normal levels as well and in 10 weeks I will be steroid free so if I need more immunotherapy then I can continue with a single drug rather than two. It’s kind of a suppliment to the dual dose of different drugs which you have 4 times. I got 2 doses, so half way.

So my hope is that I have enough of the dual inside me to kill this off and if not, I can live long enough to become free of the steroids to continue the immunotherapy treatment.

From being convinced I was facing terminal cancer in under a year, I find myself in the very lucky position of facing the possibility of much longer as the treatment is working, albeit wrecking other parts of me in the process. But that’s a small price to pay…

Do I dare to dream about the possibility of reaching No Evidence of Disease?

Who Dares Wins.

If you made it to here, thanks for reading. I was in two minds about writing this as I removed myself from many online groups including gaming and IRC technical channels some time ago just giving the grounds of ill health. I’ve always been and still will be reachable via email as I can respond when I am able to. This is the first time I’ve talked about my condition publicly. I have no immediate plans to return to the online groups I have left, but I also felt I needed to write something after todays news that maybe one day in the future, I might be back.

Upstream has already accepted my patch to allow a build on NetBSD which is nice.

However, there is still a serious issue - GoAccess burns a LOT of CPU. With older versions it wasn’t too onerous, but now it uses 4 threads and CPU was going through the roof on my poor web server. Luckily it turned out it was a simple patch.

This is a good example of why we always want to check for errors. POSIX usleep documents that if a value of 1000000 or greater is given then EINVAL is returned immediately. The default value GoAccess was using for following the tail of the web log file was also defined as 1000000, which means on a POSIX compliant OS like say NetBSD, the usleep call did exactly nothing. Hence burning the CPU time. With this patch, CPU time is now barely measureable :)

But we can still make it GoFaster :D

While the CPU was being pegged I noticed it was stuck in the select state. Well, select(2) is not the greatest API ever made. Luckiy, we now have poll(2) which is a massive improvement and as it’s a POSIX standard, highly portable as well. After a few hours of work, I turned out a a fairly complex patch and it works very well! It’s been running on my web server all night and the websocket just ticks over. The web page I’m looking at constantly updated itself.

Huge win for my low powered server, now consuming less power and a great start to the New Year!

Why change?

Well, don’t get me wrong, vuepress is a great product for what it is. I generally like the Vue.js framework over the others. If you recall, I changed from Grav to vuepress back at the start of last year

However, the vuepress blog plugin is not really that easy to setup, configure and get going. It’s also not easy to maintain when the tree of plugins breaks. That’s the thing with complex stuff - one small part breaks and it all falls down.

And this is the reason - a plugin the plugin used broke. It didn’t upgrade. My fault for using NetBSD I know. I did sort of manage to get it working, but the site on my mobile phone showed the desktop view which made me mad. Tried to upgrade it, no dice. A this point, I figured I should look around and see if there’s anything easier and voila. Here we are.

What makes this better?

Well, unlike vuepress, Hugo is in pkgsrc which makes it a snap to install on NetBSD. Yes, there is sometimes a large lag between updates in pkgsrc to Hugo which is why I ruled it out when I changed to vuepress a few years ago as the pkgsrc version was too old for the themes I wanted. Technically this was because the pkgsrc golang infrastructure was quite new and not up to it I think.

It’s also a lot faster to build the site on my server, by several orders of magnitude. Speed is always important.

Are you done yet?

I’m an engineer! It’s never done, it can always be improved! There are some things I don’t like about Geekblog and Geekdoc. Both seem to get frequent changes, so hopefully they can be resolved soon. The Projects and About Me links are temporary as they don’t belong in the tag cloud.

I also need to edit my old blog posts and fix the links where I can. If you see a broken link, please let me know!

Sadly, I seem to have lost a lot of pictures from Grav. I thought I had them all backed up, but I’ve since re-used the disk. Also, a lot date to before I used a smart phone to take pictures which are now auto magically backed up to the cloud. A real shame, good memories lost.

But first, let us discuss …

The POSIX Resource Limited sandbox

POSIX documents setrlimit(2). Disabling the ability to open new files, sockets, etc, or create new processes is actually pretty powerful.

Thanks to the privsep dhcpcd now has to support both Capsicum and Pledge, this turned out to be pretty easy to implement. The only issue with this is the poll(2) interface which dhcpcd makes great use of. Implementations found on Linux, OpenBSD and Solaris return EINVAL when the nfds argument is greater than RLIMIT_NOFILE where-as the other OS’s dhcpcd supports don’t. This means that on Linux, OpenBSD and Solaris an attacker could close an exiting file descriptor and try to open a new one. On OpenBSD this is not that much of a big deal because pledge should stop that from happening. On Linux and Solaris, they can’t open a file thanks to the chrooted empty directory but they can create a new one. But thanks to RLIMIT_FSIZE they can’t actually write to it. At most they could create a network socket and send arbitary data over it. For Linux, we could look into using seccomp to stop this.

For implementations such as NetBSD and FreeBSD, setting RLIMIT_NOFILE to zero with poll(2) still working means they cannot create any few file descriptor. This means that if an attacker breaches a resource limited process it can only work with the resources it has because it cannot fork another process, nor open any files, sockets, etc. It’s also running as an unpriviledged user locked in an empty directory, so there is nothing to see or do there. All the resources it currently has are:

• PF_INET, PF_LINK, etc sockets that can only query for data
• network proxy process (receives only, does not send)
• BPF processes (send and receive, the read and write filters are also locked)
• privileged actioneer (filters ioctls, validates paths and outbound traffic)

So the only way to do anything outside of what dhcpcd normally does is to use a facility that does not need to create a file or socket, or fork a process. The only remaining avenue of attack is to break the privileged actioneer process- that cannot be protected by any sandboxing as it needs to do a lot of stuff. Both OpenBSD’s and FreeBSD’s dhclient have such a privileged process as well.

The priviledged actioneer process itself doesn’t do a great deal. For example to add a route on BSD you create a RTM_ADD message. The master process will do this and instead of writing to the PF_ROUTE socket itself it will pass the message to the privilged actioneer process which in turn writes to it’s PF_ROUTE socket. Every ioctl, path accessed or network bound packet is validated by the privileged actioneer. Even though it’s generic, it’s also locked down.

So what extra to Capsicum and Pledge bring to the table?

Both bring system call filtering. For example, sysctl(2) does not need to create a new resource. Information available to the ordinal user such as uname may not be desirable to leak to these sandboxed processes. But then the question to ask is what can they do with it?

Pledge overcomes the RLIMIT_NOFILE limitation for poll(2) on OpenBSD.

Capsicum goes a bit futher by limiting rights of each resource you have in terms of what you can do with them.

What they both bring to the table though is making sandboxing easier. Here, Pledge is the outright winner. As I pointed out in my initial blog post, Pledge is really easy. But make it too easy and it’s not as secure as it could be. Capsicum is harder and the resource limited sandbox is harder still.

The take-away point from this is while Capsicum and Pledge are nice, they don’t beat a good design.

Capsicum is hard to develop for

It’s either on or off. You can limit each FD with capabilites mode off, but I’m not sure what that gains as it’s mainly there to allow the FD to be used in the restricted world so we can treat it as either on or off really.

Pledge is granular, you can turn areas on or off. For example, you can allow full file access while restricting various ioctls. Ideally, the only thing you want to pledge is stdio, which is kinda needed for the Privilege Separation IPC to work. However, we end up pledging dns inet and route as well.

So why is Capsicum harder? Well, the short answer is that it’s not. The long answer is that it’s perceived to be harder because you have to do all the work up front to get your code working whereas Pledge allows you do it it piece meal.

Pledge is as secure as Capsicum

As I said in my prior post, I’m not a security expert, but here’s my findings:

You can make Pledge as secure as Capsicum, but you don’t have to.

As a user on OpenBSD I can see lots of processes marked with p to indicate they have pledged something. What that something is though is a black box. You can use ps-o pledge-p $pid to show the promises a process has pledged. The more promises you see the more that process is allowed to do. Short list good, long list bad.

As a user on FreeBSD I can see lots of processes marked with ‘C’ to indicate they are in Capsicum Capabilites Mode. Because this is just on or off, I know exactly what this process can and cannot do.

Armed with this knowledge I would trust a Capsicum enabled application more than a Pledge enabled application. But to fully trust either you still need to audit the code.

They both have limitations

With a Pledged process you cannot change a BPF filter with the bpf pledge. This is unfortunate, a possible solution could be to spawn a process AND filter for every address you want to monitor for DaD or ARP pinging. That could be a lot of resources wasted as BPF is neither infinite nor cheap.

With a Capsicum Capability Mode enabled process you cannot call sendto(2) that is not connected. Unlike the above, there is no easy solution. I can’t spawn off another process because unlike normal communication, because I need to bind to the bootpc and dhcpv6-client ports so I can dictate which source port is used to send the message. The binding would then fail because the network proxy process is already using the port. A possible solution is to call RTM_GET to work out which host I send to on the local network, craft the packet by hand and send it out using a BPF socket.

I strongly feel that both should be fixed upstream:

• the bpf Pledge should allow changing the BPF filter.
• Capsicum should allow sendto unconnected if bound to a priviledged port or another capability created to grant to the socket.

As it stands, dhcpcd does not Pledge for BPF processes and does not run the generic network proxy process in Capsicum Capabilites mode. However, both processes are using the unprivileged user and chrooted to an empty directory. Neither do any processing of data coming in, they just ferry it back to the master process via IPC which runs under the same restrictions but is also sandboxed with Pledge or Capsicum so I’m confident that it’s good enough.

The good news is that basic Capsicum support has been enabled in this commit by ensuring all the file descriptors of the network facing processes are limited in their capability. The bad news is that capability mode is only enabled for the BPF processes- the end goal is that the master process is at least placed in capability mode.

So what’s the problem? Well, the first problem is that Capsicum is really hard to use, a lot of this is not helped by the documentation. It took a long time to work out how it works and how to use it. The second problem is that capability mode is really hard core- it stops a lot of stuff working dead in it’s tracks by killing access not capability limited in the global namespace.

From dhcpcds perspective the big issue is that gethostname(3) and getifaddrs(3) won’t work. This is the biggest show stopper, hard to IPC.

EDIT: gethostname(3) does work in Capsicum capabilities mode.

Let’s just take a moment to talk about getifaddrs(3)- it’s a really powerful interface to work out what interfaces are available to work with. In some OS it’s the only mechanism to get the hardware address assigned to an interface. Also, it’s possible for the route(4) (or netlink(7)) socket to overflow so it’s required to re-learn the system state which means it cannot be cached. Not that FreeBSD supports reporting route(4) overflow, but that’s another story.

Due to the getifaddrs(3) show stopper, I decided to look into OpenBSDs pledge(2). While the documentation is still very technical, it’s easier to read and understand compared to Capsicum. It’s also really really really easy to implement in comparison as you only pledge features rather than each file descriptor. Because of this, it’s also very granular. The initial commit to enable pledge is mainly allowing the current ioctl IPC framework to pass data back again.

But can we do better? Can we operate with the same Pledges that OpenBSDs dhclient(8) uses? The answer is of course YES! By caching some initial data that does not change such as utsname and machine uuid AND moving all file IO to privsep, this commit allows almost perfect Pledge support. We could go better if privsep gains support for the SIOCGIFGROUP ioctl as this would allow the inet pledge to be dropped. It’s not quite perfect because a Pledged process cannot change a BPF filter. We would need to space a BPF process per address which is then limited by the number of BPF instances you are allowed. I’m not sure this a good thing, and ironically BPF processes are the one thing we have in Capsicums capability mode. This is also a massive improvement over the initial dhcpcd-9 release because it means we no longer need any files in the chroot directory and /var/empty can be used again. This alone should make current and new FreeBSD users of dhcpcd quite happy. It will also go some way to enabling capabilities mode for the master dhcpcd process in the future.

Now, I can’t say which technology is more secure- that’s really not my field of expertise. I can’t even say if either are really needed or actually make security worse because Privilege Separation IPC adds more code, complexity and as such could make it more prone to bugs some of which could be a security attack vector in itself.

But what I can say is that Pledge is much easier to develop for because it’s really granular unlike Capsicum which is either on or off or per fd limited.

I’ll continue working on Capsicum support in dhcpcd just to see if I can enable Capabilites Mode in the master process. When that’s finished I will do a more in-depth comparison of these two technologies.

I just wanted to thank you for such a great piece of software (dhcpcd)…

I often forget the people that make all this cool stuff that make linux. bsd and all the other unixes work… so rather sending a bug report, I’d just though i’d send you a hello! and thanks for giving my machines ip addresses!!!!!

I was diagnosing a race condition between gnome network manager, wireless, wpa_supplicant and dhcpcd…

but that for further research… (your software is doing exactly what is should do). :-).

Regards,

Lee

THIS is one of the main reasons I enjoy this hobby of mine …. random thoughts of happiness that come in from total strangers that appreciate that I made something for free that in turn made them happy :)

I’ve recently purchased an AMD 2600x which I’ve fitted out with 32GB RAM. This will be my new developer machine (not to self, NOT to use for gaming), but I need the 512GB SSD from the old server for this one. As I have a few spare 120GB SSDs, I decided to downgrade!

In the process I decided to try out ZFS as it was recently improved quite a bit in NetBSD. Turned out there were still a few bugs to iron out, but I worked out a way to have Root On ZFS. With lz4 compression enabled, the space savings for my usage are huge! I’m really pleased with ZFS, it seems a lot better than using LVM + FFS + WAPBL.

Taking advantage of a new disk, all the software is very uptodate as well, and required some curses changes so I could try a few other things out…. but more on that later!

Since I was the main protagonist in bringing terminfo to NetBSD, I had a reasonable grasp on how our curses worked and I had looked at this bug before, but left scratching my head over it.

Since I looked at it, Valery Ushakov added a simple test case to demonstrate the issue.

#include <curses.h>
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
	const char *errmsg = NULL;
	int err;

#define CHECKERR(_msg)					\
	do {						\
		if (err != OK) {			\
			errmsg = _msg " failed";	\
			goto out;			\
		}                                       \
	} while (0)


	WINDOW *scrw = initscr();
	if (scrw == NULL)
		errx(EXIT_FAILURE, "initscr failed\n");

	if (!has_colors()) {
		errmsg = "no colors";
		goto out;
	}

	err = start_color();
	CHECKERR("start_color");

	err = init_pair(2, COLOR_WHITE, COLOR_RED);
	CHECKERR("init_pair");

	int maxy, maxx;
	getmaxyx(stdscr, maxy, maxx);

	WINDOW *lborder = newwin(maxy- 1, 1, 0, 0);
	if (lborder == NULL) {
		errmsg = "newwin failed";
		goto out;
	}

	wbkgdset(lborder, COLOR_PAIR(2));

	werase(lborder);

	mvwaddstr(lborder, 0, 0, "A");
	mvwaddstr(lborder, maxy- 2, 0, "Z");

	wnoutrefresh(stdscr);
	wnoutrefresh(lborder);
	doupdate();

	sleep(5);
out:
	endwin();

	if (err != OK) {
		if (errmsg == NULL)
			errmsg = "unknown error";
		fprintf(stderr, "%s\n", errmsg);
		return EXIT_FAILURE;
	}

	return EXIT_SUCCESS;
}

He then noted that using wbkgd instead of wbkgdset makes it work.

Now, the difference between the two is that wbkgd forces new attributes on the whole window background, whereas wbkgdset sets new attributes for new actions on the window. Keen observers will notice that the following function called is werase, which does what it says on the tin and erases the window but also resetting the attributes of the contents. So, I looked at this function and noticed the attribute checking code was faulty and a simple fix was enough to solve it!

Valery then pointed out that wclrtoeol probably needed fixing to and as it turned out, wclrtobot likewise. As it transpired, all 3 had different logic in working out when to erase something! We went through a few iterations of patches to harmonise the code and have settled on a macro that all 3 now share, so hopefully no more erasing issues in our curses.

As a follow on task, I should go through the NetBSD bug backlog and see if this fixes any other reported issues.

Infact, such a terminfo description was imported where colour pairs for screen-256color went up to 65536 which exposed a bug in the existing implementation where it was set to zero. Because the number might mean something more than a range, we need to be able to store it accurately.

This requires a version bump because whilst the API hasn’t changed thanks to C int promotion, the ABI has. Also the underlying database structure has changed as well- we now store the numeric parameter inside a uint32_t field rather than a uint16_t one. Whilst this change can still read the old style database, the old one cannot read the new one and thus we now maintain the database as terminfo2.cdb, leaving the old library and database alone so old programs still work fine.

libcurses, libfrom, libmenu and libpanel have also been bumped to accomoate this change.

Internally, Grav uses markdown files to store the user pages using frontmatter to store meta-data such as tags, publication date, etc. It seems that a log of blog and cms systems use this of late as it entirely removes the need for a database and isn’t that hard to pickup.

What went wrong?

Well, every time you login to the Grav admin page it checks for updates and casually lets you know there’s an upgrade via a popup. It also checks all installed modules as well. It makes upgrading painless ….. except when it goes wrong. Basically Grav complained about the gateway timing out via a popup.

Oh dear!

Refresh the page and boom, presented with a Please run bin/grav install banner on a white screen.

Okay…. this might have been a harmless timeout. So I login via ssh, goto the needed directory and run the installer and I see many errors like so: tls_process_server_certificate:certificate verify failed.

This is because, like all systems today, Grav uses many 3rd party things and downloads them from other sites. Why this suddenly started happening recently I don’t know- I’ve not updated my server at all.

To be clear, this isn’t Grav’s fault or error- the failure is somewhere in PHP, OpenSSL or the website in questions HTTPS configuration.

What can be done about it?

It’s possible to fix it for sure, but maybe this is the wrong approach. The vast majority of my site does not accept any data and Static Site Generators have come a long way since I last looked at them. The idea being that a static site has zero runtime dependencies so it’s always working. The downside being it can still go wrong as can any other application but you, the end user, does not see this. The other downside is that they build a lot of files and this takes time.

All of them seem to use Markdown and Frontmatter like Grav as their data storage so in theory it should be fairly simple to install some and try them out.

I settled on VuePress mainly because I use Vue.js on a side project I’m working on. VuePress is mainly for documentation though (ie, perfect for my open source projects) but as it turns out isn’t that great for blogs! Luckily there is a blog plugin, but the the downside is that you need to set it up as a different site. Thanks to too much cleverness in VuePress, my blog link looks like an external site which I’m not overly keen on. Also, the blog insits on using the date in URLs which I’m not too sure about either.

But it’s a start, it’s up and working and renders really well on mobile devices. Let’s see how long this lasts :)

At a bare minimum, the hardware address of the interface is sent- this is required to work.

So, how to solve this dilema of wanting total anonymity? The answer is to randomise the hardware address. This will happen when the carrier is down OR dhcpcd starts with the interface down. Then, dhcpcd will use this random hardware address to set a DUID LL which will be used inplace of any saved DUID and set the IAID to all zeros. This combo is used by DHCP and DHCPv6 to identify a lease. As this is randomised each time the carrier comes up you get a different IP address!

Try not to use this on an unstable link as it could drain the DHCP server of addresses :(

But we can’t stop there! dhcpcd also sends some identifying options as well! For example, this is sent in the vendor class identifier:
dhcpcd-8.99.0:NetBSD-9.99.17:amd64:x86_64

It does not identify you or the device in anyway, but it does say what software is being used on which hardware. This could be used by DHCP servers to hand out a specific image to download and boot from TFTP for network boot clients.

Now, there are a gazzillion and one DHCP options out there- we don’t know what you’ve configured. So dhcpcd will mask all of them when anonymous mode is activated, unless they are essential for enabling dhcpcd to work correctly on the network. But wait! What if you really want to leak something? Like say your on a corporate network that uses DHCP security and still want to remain anonymous? Well you can! Any request or option after the anonymous option in dhcpcd.conf is turned on. So the placing of the anonymous directive is important, unlike other dhcpcd options. So far this is the only implementation of RFC 7844 which does this :)

This is NOT enabled by default because most people want stable addresses AND a flappy link could drain addresses as disussed earlier.

So far, my structures in dhcpcd are long lived- or rather the scope is design to live outside of where it was created. As such they are created on the heap and are at the mercy of malloc. Generally I use calloc so that the whole area is inited to zero as uninitialised memory is bad.

So I decided to start out and see if I can just create the structures I need on the stack. Turns out I could! Yay! Now, how to you initialise a structure on the stack to all zeros? First let us consider this structure:

struct ps_addr {
	sa_family_t psa_family;
	union {
		struct in_addr psau_in_addr;
		struct in6_addr psau_in6_addr;
	} psa_u;
#define	psa_in_addr		psa_u.psau_in_addr
#define	psa_in6_addr	psa_u.psau_in6_addr
};

The first way is memset:

struct ps_addr psa;

memset(&psa, 0, sizeof(psa));
psa.psa_family = AF_INET;

But what if you could avoid memset? Luckily the C standard allows setting any member and will zero all other members. So we can do this:

struct ps_addr psa = { .psa_family = AF_INET };

Wow!!! So simple. This reduces binary size a fair bit. But then I turned on the Memory Sanitiser and boom, it crashed hard. Why?

The answer is simple- padding. Eric S Raymond gives a very good writeup about the problem. Basically, the standard will initialise any unintialised members to zero- but padding added for alignent isn’t a member! So we need to ensure that our structure requires zero padding.

Here is the new struct:

struct ps_addr {
	sa_family_t psa_family;
	uint8_t psa_pad[4- sizeof(sa_family_t)];
	union {
		struct in_addr psau_in_addr;
		struct in6_addr psau_in6_addr;
	} psa_u;
#define	psa_in_addr		psa_u.psau_in_addr
#define	psa_in6_addr	psa_u.psau_in6_addr
};

And it allows the former structure initialisation to work and memory sanitisers are happy- so happy days :) Now, if anyone can tell me what I can use instead of the magic number 4 in the above I’d be even happier!

All of the really needed kernel improvements are now in and dhcpcd doesn’t log any more compile warnings, but there is more work to be done such as RFC 5227 support, restarting DaD on link state up and denying the use of an address until validated. I’m quite enjoying working on DragonFlyBSD … their SMP approach is very interesting and in many ways much easier to work with than NetBSDs fine grained locking approach.

And then out of the blue, a discussion crops up on the FreeBSD mailing list about putting dhcpcd into the FreeBSD base system! This has led into me working on Priviledge Seperation which seems to be the only show stopper for FreeBSD acceptance. I have a reasonable idea on how this should work and hopefully this will be enough.

Now, to be clear, you have been able to do this since forever using fopen, writing to the file and then allocating your area based on the final file size. Still, it requires some memory management still but more importantly it writes to a file. Writing to a file is slow and reduces the life span of the disk you’re writing to. It’s only been fairly recently that tmpfs was a thing, but even today not all OS’s have /tmp mounted as tmpfs. Requiring this isn’t exactly ideal for a generic program to do- the setup and install should be easy. Because of all these reasons, most programs worked the string length needed and either allocated an area or string and then finally write the string. However, while saving the disk, it’s also a lot more error prone because you need to work out the length of everything and that’s not always trivial, especially for things like a DHCP client which is always building strings based on the information given by the DHCP server.

Here’s an example of open_memstream in action:

/*
 * Example program which manages an area of environment strings
 * to send to child programs.
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static const char *foo = "foo";
static const char *bar = "bar";

int main(void)
{
        char *argv[] = { "/usr/bin/env", NULL };
        char *buf, *ep, *p, **env, **envp;
        size_t buflen, nenv, i;
        FILE *fp = open_memstream(&buf, &buflen);

        fprintf(fp, "FOO=%s", foo);
        fputc('\0', fp);
        fprintf(fp, "BAR=%s", bar);
        fputc('\0', fp);

        /* We could keep fp around as our area and just rewind it. */
        fclose(fp);

        /* execve relies on a trailing NULL */
        nenv = 1;
        for (p = buf, ep = p + buflen; p < ep; p++) {
                if (*p == '\0')
                        nenv++;
        }

        /* reallocarray(3) should be standard really */
        envp = env = malloc(nenv * sizeof(char *));
        *envp++ = buf;
        for (p = buf, ep--; p < ep; p++) {
                if (*p == '\0')
                        *envp++ = p + 1;
        }
        *envp = NULL;

        execve(argv[0], argv, env);
}

As you can see, we only manage the environment array handed to to execve. open_memstream is managing our string area and fprintf is working out the length each string needs to be for us. This vastly reduces the complexity and increases the security and reliabilty of creating large environment strings, which most DHCP clients do. We could also write a helper function to write the string AND the trailing NULL terminator for the string to be more efficient. You’ll get to see this in dhcpcd-8 which should be released later this year.

• Very low extra memory needed for it’s implementation
• Fast at insertion and removal operation- both are O(1)

However, it’s just a list. And like any list, to find something we need to pick a starting point and then search in a single direction and match each item until we find it. This is described as O(N) and it’s performance roughly corrolates to the Number of objects in the list. If you need to insert at an arbitary point (for example to order the list) you need to search it, so this is O(N) also.

But is this slow?

Typically for dhcpcd, the answer is no because on a desktop, tablet, phone or SOHO router you only have a handful of objects in all the lists. However, someone ran dhcpcd on a switch with millions of routes and dhcpcd was taking minutes of CPU time for routing table modifications. So in this corner case … yes! dhcpcd is slow! A suggestion to improve matters was given in that report- use a Red-Black Tree instead of a Linked List. A RB tree guarantees O(log N) performance for searching. A patch was even submitted based on OpenBSD’s tree implementation! This reduced the dhcpcd runtime from minutes to seconds in his situation.

Now you can’t just swap in a RB Tree for a Linked List. Each object within an RB Tree has to have a unique key. Also, you need to have comparison function to define an order to the objects within the tree. Only then can you start to implement it. To find an object within the tree, you just need to know it’s key and off you go. I settled on using NetBSD’s rbtree(3) implementation instead of the initial suggestion to use the generic BSD tree(3). This mainly to reduce the binary size of dhcpcd because it’s in libc on NetBSD and on other systems it will creates a smaller binary due to the complexity of the tree(3) macros. Testing also showed it was slightly faster and also easier to actually develop with.

So should you replace all LinkedLists with RB Trees? No! That would be silly :) After all, sometimes you don’t actually need to search or order a list- a lot of the time, lists are used for push/pop operations and thus gain nothing from RB Tree.

You’ll get to see this in dhcpcd-8 which should be released later this year.

I’ve reported various bugs upstream, sometimes supplied patches. dhcpcd these days works pretty darned well on all BSD platforms out of the box, but there are still some caveats where dhcpcd has to work differently to do the same job. This article focuses on the big three, NetBSD, OpenBSD and FreeBSD.

The Good

Phabricator is written in PHP which means I don’t have to install Yet Another Framework. I use quite a few things that depend on PHP on this site already, such as Grav and RoundCube. So of course, it allows me to self host. Or you can rent a Phabricator VPS @ Phacility.

The sign up process (to my Phabricator instance, not somewhere else) is very straight-forward, allowing email/password with ReCaptha or use a OAuth2 provider such as Google. So this is very socially acceptable and should be secure from spambots.

The core work is based around the ease of code auditing and review of patches. There is even a pastebin so users can upload config files and logs for analysis. Doing all this in a mailing list over the years results in things being here, there and everywhere …. and then expiring. Having it all centralised means nothing is lost. But more importantly, it’s much easier to look at and work with, so this is a massive quality of life improvement.

Tickets (or tasks in Phabricator) very user friendly, showing a collapsable history with full links to related objects such as commits, reviews, logs, etc. Infact the linking is extremely easy, one can reference some more of the popular objects by using a single letter follows by the id. Such as T1. Tickets can be related to one or more Projects and in turn Projects can display Tasks on a KanBan Board.

Phabricator can host your code in your SCM of choice for you and defaults to not allowing destructive changesets by default which saves me from messing around with custom hooks. This allows the same feature as Fossil’s immutable history on the server- you can do what you like to your own clone still.

It’s fast! No, it’s not as fast as Fossil, but it’s still more than fast enough especially when you consider the extra toys you get- syntax highlighting, desktop notifications (on supported browsers, which is most recent ones), user icons, in-depth tooltips. It’s certainly faster than other solutions I’ve looked at recently and bar Fossil, probably the fastest.

You get a chat room (does require a NodeJS server on the host for automatic updates though it seems) and a wiki. I still use IRC on FreeNode, but the advantage here is that this is web based and persistent so you don’t loose anything if you get disconnected. Still, unsure how useful either be as I don’t recall users editing any publically editable wiki pages I’ve had over the years- are my man pages really that good? Heh.

The Bad

Phabricator is written in PHP. Now I did say that was a good thing earlier, but it’s a double edged sword. PHP does have a bad reputation for both security and language structure. I would argue that this is no different from how C is today. This is also bad, because my site ran on PHP-7.0 and that was soooo much faster than earlier versions it was silly. But Phabricator didn’t support PHP-7 until PHP-7.1 in early Feb this year. Something to think about for long term support, but this equally applies to other languages, especially the Python-2 vs Python-3 issue as my box has two Python versions due mainly to certbot needing Python-2.7

Phabricator requires MySQL (I installed MariaDB, the fork from MySql). I was very happy with PostgreSQL but my box does not have the resources to run both. Pretty much all other software I use allows the choice of DB, so this actually took me by surprise. And just like the PHP reaction others have, I was concerned by using MySQL, but as I’m not really into being a DBA I’m quite happy with MySQL so far.

The linking is really bad for DHCP, because we always talk about T1 and T2 as timers. This is important, because my main product is of course dhcpcd. In Phabricator T1 and T2 are shorthand to link to Task 1 and Task 2. You can fix this by stopping Phabricator from linking via a matchig regex, but I quite like the ease of use and solved the problem via changing the AUTO_INCREMENT value in some tables from 1 to 101. This reduces the potential collision with other things, such as Z1 and allows the same workflow. Upstream rejected my change and even went as far as to remove me posting my fix if anyone else has the same issue claiming this would make support hard. As it turns out, something with my change isn’t quite right- either Phabricator or MySQL resets the AUTO_INCREMENT value. I don’t know which one, or what action I did or if it’s a general Garbage Collection going on. This could be why they didn’t like the change, but heh ho most of the important tables now have values in at 101 and higher so it shouldn’t be a problem anymore.

The Continuous Integration support is limiting, but it is there. Apparently you can at least call out to Jenkins or BuildBot.

Because Phabricator is based on and developed in a DevOps fashion, there is practically no support for managed releases or milestones. This isn’t a problem for me as such, but I would like a feature to track important things that went into a release better.

The Ugly

Phabricator is NOT ugly. It’s quite visually appealing. However, it is quite possibly the most complex installation I’ve ever done as it uses many databases and as many configuration options as sysctl on a good BSD. This wasn’t helped by running on NetBSD-current and a gcc built PHP with Phabricator just didn’t work and I spent a long time working out why. My fix was to build everthing with clang which required a lot of personal effort from me at the time due to the recent UEFI booting support breaking the build and a the new clang-4 compiler not working with the NetBSD build knobs I was using. On the plus side, the Phabricator docmentation is good and about 95% of the issues I had were easily searchable on StackOverflow or the (mostly) friendly Phabricator community helped me out in their chat channel- which oddly enough is also a Phabricator application.

Phabricator workflow with more than one dev, or the best way of submitting patches, is to use the Arcanist tool. They admit it’s not great and things should be manageable directly through the SCM. We’ll see how that progresses. In the meantime, posting patches to the Differential application is quite easy and allows easy patch review.

I had to stop using Fossil because Fossil is more than just a SCM, it strives to be a complete one stop solution. Obviously that won’t work for the desire to use Phabricator for all the good reasons, so I needed to pick a SCM to use. Luckily Phabricator quite a few- GIT, Mercurial and SVN.

But what about the source code control?

It’s importance cannot be understated- the code is everything, the history of the code is everything. This has been known since the dawn of time. At this point though, the SCM just becomes a tool in the box, just like sed.

Eh what?

Every SCM solution out there has pretty much the same set of basic features you need- atomic checkins (ok, CVS lacks this), changesets, branching, tagging. That’s all you pretty much need at a basic level- the rest of the features are predominently driven by workflow.

Tools exist to export data from one to the other, and tools are being created to allow a more transparent bridge again making the choice of SCM even less important than it was before. The only real issue is the importance of meta data that has no place-holder in the other SCM you want to use. A good example of this would git the Author vs Commiter git attribute on the commit.

Then, you need to understand that the SCM is only for developers. End users don’t care a hoot about it. What they do care about is an easy to use system which handles the lifetime of their issue where dicussion, patches, logs, reviews and audits can happen. Hopefully they can even get a fixed build at the end. This is basically part of Application Lifecycle Management.

This is a massive change from the way we’ve worked before and it excites me! The bad news is that I am spending some of my hobby time on this because it’s fun! But I should be getting back onto dhcpcd / NetBSD related stuff soon once the initial prototypes are in place and the new tech feeling wears off.

So, I can ping the router but nothing else as I don’t have a public IPv6 IP address. No address via RA, no reply from my DHCPv6 solicitations- which is odd as the router says I can get a managed address and other information. Maybe they have yet to turn that part on? Please, turn it on soon Virgin!

So I shopped around and found Rspamd. For my needs it sounded wonderful- no more need for MySQL (it’s a hard dependency of OpenDMARC) as I much prefer PostgreSQL. SPF, DKIM and DMARC all integrated. Written in C and LUA which is a massive improvement over Perl and Python. Also sports a shiny Web UI to monitor the server and do basic config. Speaking of config, it’s still not entirely easy, but it’s much easier than configuring the stack I used to have! I did have to patch the build so that it works with OpenSSL-1.1 which is now in pkgsrc. All in all, I anticpated a nice memory reduction once I had it all configured. So far it’s using about 200Mb less memory, but it’s early days. How much better or worse than SA it is at actual spam filtering remains to be seem, but I have high hopes.

While here, I also replaced procmail with PigeonHole. I didn’t really need to do this, but I thought “As I’m here…..”. Actually the end result is much nicer as I now only have one Spam folder instead of another two Spam folders for training ham and spam. I just need to hook this final part into how I manage spam on my mlmmj email lists.

Many improvements have been made with regards to compability with NetBSD and the others, but there are the following caveats:

• NetBSD will only resize stdscr, curscr and virtscr windows (like PDCurses)
• NetBSD clears the window when resized
• NetBSD will re-size and re-position the ripped off lines (like ncurses)
• NetBSD will not change LINES when ripoffline is called and may offset stdscr to it’s relative position on the screen. This means that portable application should check getbegyx and getmaxyx for the size and location of stdscr.

The expectation is that the client will re-draw all windows, including ripped off windows on a KEY_RESIZE event. It’s also expected that the client will re-size any other windows as it has a clearer idea of where things should be on the screen to it’s proportion rather than curses guesstimating.

Now, the very colourful Roguelike game Ascent into the Depths of Beyond now works on NetBSD with a minor patch

Thanks for openresolv. If only the whole world used it... 

Rome wasn’t built in a day. But it’s getting there- openresolv can be found in NetBSD and FreeBSD base systems. It’s available in most other OS’s package respositories to at least depend upon.

Thanks

Nice error. Googling for it doesn’t reveal much on how to fix it. The good news is that I only use MySQL for Phabricator and PostreSQL for everything else. The bad news is that my Phabricator instance is no longer working. The worse news is that I get the same error when trying to use backups, so there must be something else in play here.

Ideas on how to resolve this are welcome!

Trying to connect to the printer from a Windows machine (I can see the printer find in the server share) gives an error that it cannot connect, but nothing appears in the Samba or CUPS logs.

This is 2017, surely we have better diagnostics to solve these issues!

But this came with some issues:

• There is a window where the IP address doesn’t exist, and the kernel may wipe out the subnet route at that point also.
• DHCP renews didn’t come through to the right interface.
• Some kernels didn’t like the address moving interfaces.

Still, to the best of my knowledge, no other product has this feature and for the most part, it did work well allowing almost seamless switching of wired-> wireless and back again with both using the same IP address. But that wasn’t good enough - I was challenged to do better!

So I took up the bat and changed the behaviour to this:

• Each applicable interface will have the shared ip address.
• Whenever the address is added, the most preferred address will be ARP announced.

And lo - IT WORKS!!! The changeover when plugging/removing the wired interface is 100% seamless for me. ssh, ping, etc get zero interuption. Of course, YMMV ;)
But there are some costs:

• Thanks to ARP, only the primary interface will receive DHCP unicast messages for other interfaces.
  As such we need to re-direct them to the correct interface by examining xid and chaddr.
  This means we have to relax the BPF filters to allow more through.
• Kernels supporting RFC5227 will double ARP announce the address.
• NetBSD-8 kernels needed some love to get it to work and there’s still an issue with it not working when an address is deleted from the interface.

Only the last bullet is really important, which is mainly why the changeset hasn’t hit the master branch yet. But that should be fixed soon. The other points can be fixed as and when.

Summary of changes since dhcpcd-6.11.5:

• source file locations reworked: dhcpcd source is in src dhcpcd hooks are in hooks compat is in compat
• README split into README.md and BUILDING.md
• internal routing is now protocol agnostic
• avoid using __packed and use compile time asserts instead
• addresses some alignment issues
• disable some ARP code on kernels which support RFC5227
• BSD IPv6 kernel settings are now updated to reflect dhcpcd config
• custom logger has been removed, syslog handles everything as such, the --logfile option has been removed as well. If you need better/earlier logging, get a better syslogger!
• distinfo and signed distinfo files are now available alongside release taraballs from this point onwards
• default DBDIR has changed from /var/db to /var/db/dhcpcd
• /etc/dhcpcd.duid moves to DBDIR/duid
• /etc/dhcpcd.secret moves to DBDIR/secret
• lease file names have dhcpcd removed from them as they are now inside a directory of the same name
• fixed issues with reject routes not working on some platforms
• improved nl80211 support on Linux for working out the SSID
• no longer request NTP by default in dhcpcd.conf
• fix detecting IPv6 DAD on OpenBSD
• remove custom Solaris DLPI filtering in favour of BPF (note there seems to be a kernel issue where the DHCP fd receives ARP’s as well, the only side effect is a noisy syslog)
• BPF filtering vastly improved so dhcpcd only wake up on ARP or DHCP packets destined for it
• support for MUD URL (draft-ietf-opsawg-mud-05)
• if the kernel isn’t doing DAD, don’t insist on waiting for it to actually do it
• fix a potential crash where the DHCP or ARP states could be freed before the packet processing loop naturally breaks
• removed gateway and nogateway options (these can be controlled by the nooption directive which works for more than just gateways)
• removed ipv6ra_own and ipv6ra_own_default options (these can be controled by the ipv6rs/noipv6rs directive)
• fix a memory leak on systems where posix_spawnattr_init allocates memory by calling posix_spawnattr_destroy afterwards
• fix a crash receiving SIGUSR1

I’ve not done everything I’ve wanted to, but I feel that many issues have now been addressed and on the whole dhcpcd is in a very good state right now.

Let me know of any issues you find!

• BSD license.
• One binary, easy installation, very low maintainence cost.
• Integrated CGI web front end.
• Integrated Wiki, Tickets- which are also distributed.
• Sane command line UI.
• Stores everything in a SQLite database.
• Repository is not joined to the checkout, supports different checkout from the same cloned repository.
• I have had a Fossil commit bit - my change allows a near perfect Fossil <> GIT bridge.

And naturally, after many years of use, there are some negative points:

• The ticketing system is very basic and has no email support- you’re expected to use each tickets RSS feed, but this is not clear.
• It’s not extendable.
• It’s possible for an admin in the upstream repo to wipe out parts or the whole of your cloned repo.
• It’s not social.

That’s actually a very small list of negative points. It shows that Fossil is a great product, with a great team behind it. Let’s address these these negatives in more detail though.

The ticketing system is is poor

Yes, the tickets are distributed, but that’s the only good point. The UI to progress the ticket needs a lot of work and is not intuitive to use. Tickets don’t support markdown. It’s not clear to the end user that the only feedback they get is a RSS feed. My initial attempt to fix this was about 3 years ago but was met with silence. I could try and improve this by creating a fossil branch just to add RSS icons to the ticket UI.

Fossil is not extendable

This isn’t actually that bad, what it does have works well enough (aside from the ticketing). And to be fair, there is a 3rd party library to extend fossil but it doesn’t seem to be used by anything I can find. However, based on recent experiences at my day job (where I don’t use Fossil), code reviews are turning out to be quite critical and the tools we were using suck. Well, Fossil doesn’t have any code review feature nor any easy way of hooking it into an automated build system for continuous integration So we’re left reviewing changes via pastebin where links expire or diffs via email. Now diffs via email have been standard on many open source projects, and still are in many. Most of the time I can read them fine, but sometimes they are hard to review in an email. Sometimes I end up using a code review tool, loading the dhcpcd source code, copy and paste the diff into it, reviewing and then copying my review comments back into the email. This is hardly ideal and quite time consuming.

Fossil history can be wiped out

Fossil has an ability to delete anything from your cloned repo - it’s called shunning. While this is an awesome feature for corporations (I work for one, I understand the problems and wish my day job had this feature), but equally I believe it’s entirely un-suitable for open source use. This is my PC, it contains my contributions to a project which could wipe out my copy of said project and published contributions. History, gone. Now, it’s entirely likely that this will never happen, I like to believe in the good in people, but the possibilty remains someone could push the button. OK, there’s a bit more to it than that- the default fossil setting is to auto-sync the shun list. However, the code is disabled for auto-sync (ie sync on commit) but is enabled for the manual pull/push commands. While that removes the item from your checkout (if it’s there), it won’t actually remove it from the repository itself until the repository is rebuilt, which is sometimes forced on you when upgrading fossil. Yes, this is probably a knee-jerk reaction to a none-issue, but it still grates. This is also a reason why I love to self host and would never consider having GitHub or similar being the one sole place where I publish my work. I have always, and always will do, self host.

Fossil is not social

By it’s very nature, you can’t contribue to the club unless you’re in the club- at least not using just Fossil. It’s designed (from my perspective anyway) to be a distributed CVS/SVN + wiki + tickets. By this, I mean there is one master repository everyone clones from and pushes to. This makes it impossible to have my own branch outside of the main repo and publish it to others (equivalent of GIT Fork and Pull). It can also be argued that this is a good thing because it encourages people to work together and just like the prior point, this is a good feature for corporate setups. But equally sometimes someone needs to maintain a patchset unsuitable for upstream for valid reasons. This is rare, but it has happened. And I hate losing users for any reason. Could they have a branch they maintain in my repo? Quite possibly, but Fossil’s security isn’t that granular AFAIK and I would dislike someone messing around in the other branches. Maybe that’s anti-social of me, but equally no-one has ever asked for commit access to my repos either.

EDIT: Dr Richard Hipp pointed out privately that this is The Cathedral and the Bazaar. Fossil is the pre-eminent solution for The Cathedral, while others are more suited to the Bazaar.

In summary

Taking the above into account, I can no longer justify the use of Fossil in my Open Source projects. For other projects, Fossil is still an awesome tool if that’s all you need.

So, to get myself on GitHub (as a mirror only), there needs to be a bridge between Fossil and Git. Fossil documentation implies this is quite easy. Sadly, this isn’t the case as the <=Fossil-1.37 releases (note there is no guaranntee that future versions of follow will not have these flaws- my branch may not be comitted to trunk) have the following flaws:

• Branch and Tag name mangling (dhcpcd-6 becomes dhcpcd_6)
• Silent master branch renaming into trunk on inport, but not on export
• No tag comments (Fossil lacks the feature) which means syncing tags back and forth results in tag conflict due to signature change

I submitted some initial patches the the Fossil mailing list and I now have a Fossil commit bit! You can find my branch here to fix the Fossil Git bridge.

But that’s not the end of the story. A bridge has two ends. With my initial setup, the Git end was bare bones repository which I pushed to GitHub. This is no longer the case- I now need a staging repository to pull both ends. And this requires a script because Git needs a little more hand-holding to completely track a remote. The below script is tailored for my needs, yours may differ. It also reflects the above initial design and the subsequent change- as such it it may need editing if you need to create a git clone from fossil. This comes with no support, just as an idea of how you might implement such a bridge.

#!/bin/sh

fossildir=/var/fossil
# Cannot be a bare directory for git as we cannot write to the host directly.
# So we have a staging directory instead.
# This requires a bit of hand-holding to track all the branches.
gitdir=/var/git-staging

marksdir=/var/scm-marks

# Respect default naming at either end
fossil_export_opts="--rename-trunk master"
fossil_import_opts="--rename-master trunk"

# Only used when creating a git bare bones repo from Fossil.
export_fossil_to_git_new()
{

        rm-f "$fossilmarks" "$gitmarks"
        git init
        fossil export--git \
--export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
--export-marks="$gitmarks"
}

export_fossil_to_git()
{

        fossil export--git \
--import-marks "$fossilmarks"--export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
--import-marks="$gitmarks"--export-marks="$gitmarks"
}

export_git_to_fossil()
{

        git fast-export--all \
--import-marks="$gitmarks"--export-marks="$gitmarks" | \
                fossil import--git--incremental \
--import-marks "$fossilmarks"--export-marks "$fossilmarks" \
                $fossil_import_opts "$fossildir/$fossilrepo"
}

pull_git()
{
        local remote

        git fetch--all
        # Track all remote branches
        git branch-r | grep-v '\->' | while read remote; do
                if [-z "$(git branch--list "${remote#origin/}")" ]; then
                        git branch--track "${remote#origin/}" "$remote"
                fi
        done
        git branch--list | sed-e 's/^\* //' | while read branch; do
                git checkout "$branch"
                git merge--ff-only
        done
}

push_git()
{

        git push--all
        git push--tags
        # Reset the current branch checkout.
        # If we don't, the next run will complain about unstashed changes.
        # This maybe a bug in git, but maybe not because the live checkout
        # *is* behind at this point as we just fast-imported.
        git reset--hard
}

echo "Syncing git and fossil."
for repo in "$fossildir"/*.fossil; do
        fossilrepo=${repo#${fossildir}/*}
        repo=${fossilrepo%.fossil}
        gitrepo="$repo"
        fossilmarks="$marksdir/$repo.fossil.marks"
        gitmarks="$marksdir/$repo.git.marks"

        # We just sync old fossil repos to new phab clones
        if [-d "$gitdir/$gitrepo" ]; then
                cd "$gitdir/$gitrepo"
                pull_git # staging only
                export_git_to_fossil
                export_fossil_to_git
                push_git # staging only
# Enable the below if pusing to a bare git repo from fossil
#       else
#               export_fossil_to_git_new
        fi
done

Direct download to script

• gcc toolchain is broken for some packages which affected running any PHP build
• clang toolchain was broken for my config (USE_SSP= yes and CPUFLAGS+=-march=core2)
• clang compiles as a whole were broken due to a recent efiboot import

In hind-sight, I could have had the box up and running a lot sooner if I used NetBSD-7 guests (or maybe just a NetBSD-7 build box), but no, I just had to get-current running. It offers more than-7 and prior exerience told me that tracking-7 was very problematic … but that could have been due to my settings and wanting to compile everything with clang. I’ve currently got 3-current images there now … the Xen DOM0, the actual server itself as Xen DomU (without any compiler or tools) and another Xen DomU which just builds stuff for other guests to use.

Anyway, the box itself is now up and running and all relevant services have been moved to it. During this move, I decided to modernise things a little and setup HTTP/2. If you’re reading this then it’s working :) As such, I’ve re-directed all basic HTTP traffic to HTTPS and it does seem to load a lot faster. One side effect of this is that I’ve stoped using my own self signed certificates and I’m now using the nice Let’s Encrypt service. pkgsrc users should use the py-certbot package and don’t bother searching for any letsencrypt package as it’s been renamed. I found this quite confusing as there was no reference to the rename in pkgsrc I could find and wasted a little time on it.

But now it’s live I can finally look into bringing online some other development utilities to play around with as the server has a lot more room to grow :)

Currently we lack Soft Label Key functions as specified by POSIX. What are these? Glad you asked! Soft Label Keys ripoff a line from the bottom of the screen and allow for up to 8 keys to be displayed there. You can define labels for them, colours, etc.

“BIG WHOOP!" I hear you say. “My app already does this, why should I care?"

Because SLK also works with the terminfo database to work out if the terminal natively supports labels (via plab_norm, label_on, etc)! This means a line doesn’t have to be ripped off from the screen so no real-estate is lost. Sadly, no terminal actually seems to support these markings. But if one is ever made, it should work fine.

Now, not many curses apps actually use the SLK functions, probably for this very reason. But some do, and it is a standard so I’ve coded NetBSD to support this natively.

Sadly, this has uncovered a bug in the NetBSD curses library where cursor movement in a one line window at the bottom of the screen appears to push the window up. This is very visible with SLK and will have to be fixed for any apps that actually use it, but the SLK implementation itself is sound as resizing the terminal forces a redraw and it looks fine.

We need to do this thing more often, especially in such a nice place where everyone was friendly and the food was just awesome. I think I managed to talk with everyone and have time to play with the kids. Speaking of the kids, they were absolute little gems the whole journey- 4 hours up and 4 hours down. Much better behaved than I was at a similar age no doubt ;)

There is a apparently a thing called PD Curses but I don’t know anywhere that actually uses it by default. If you know, please email me @ roy@marples.name or tweet me. I do know that the pkgsrc curses framework supports it.

And this is the thing I’m currently doing. As NetBSD curses supports more POSIX functions (and a few ncurses extensions, sadly) more and more packages now work alongside NetBSD curses, so I’m plowing through pkgsrc again moving things to the curses framework. This is important, because there is logic in devel/ncurses to use NetBSD curses when passing certain criteria, but sadly this always brings ncurses as a possibility to be linked or it’s header used, both of which are quite bad for NetBSD curses so there is a method to my madness.

One interesting thing though is that since I started comitting my curses patches, it turns out that Sabotage Linux is using NetBSD curses and made it portable. I’m chatting with one of their devs and we’re starting to merge of a few of their patches back into NetBSD.

There were some challenges. Firstly, I couldn’t find a way to get into the BIOS. Neither F12, F2 or Del hammering during the boot process worked. While there is a mouse cursor to move and click, it booted too fast to actually use it. Actually, pressing the BSS on screen button did briefly display a keyboard to enter the SSID, but it then booted instead. This meant I couldn’t enable PXE boot, so a USB install is needed. This lead onto the next problem ….. NetBSD-7 is too old, you need a recent-current for the USB ports to work. No matter, making a NetBSD USB install was quite trivial, the only challenge being to find a pre-built image. Luckily I found http://daily-builds.netbsd.org/pub/NetBSD-daily/HEAD/ and browsed to the dates images folder to get the install img for amd64.

From this point, installation was trivial. But here are some notes:

• Allocating 5G to / is enough to install a-current base and compiler, pkgsrc checkout and build xentools.
• No idea if I need it, but I allocated 1G swap.
• I setup XEN backing using LVM, so the rest of the disk is allocated to that.
• I needed to reboot twice to get LVM/Xen to start recognising the assigned LV.
• The NetBSD Xen How To needs some love in places.

The speed of this box compared to my current Shuttle XS35 is impressive. While I have less disk space (quite a lot less), it should still be enough to replace it entirely and power my email, web and other stuff. As it’s Xen, I am hoping that I can use this box to do my open source work on other OS’s easily and support more OS’s than my exiting laptop allowed (max 4 partitions) which means a faster turnaround time for support.

From 16/0.6mbs to 200/20mbs thanks to Virgin Media! OK, it’s not quite 200 down, it’s quite variable actually. So far, it’s normally around 150mbs at peak times, compared to my old service which was rock steady. Probably because I was on low contention service so my max available was pretty much guaranteed. However, it’s always 20+ up! And lets be fair, even 50mbs down is x3 better than what I had previosuly, but it’s yet to dip under 120mbs. This will be awesome for work conference calls with video :)

Also, one unexpected feature of the Virgin Media SuperHub is the Modem mode (I turned off Router mode as my existing NetBSD router and Unifi wireless AP is better) is that it’s not a PPP based modem. At least, not facing me. So I have to run dhcpcd on it to get my address, which makes me feel all warm and fuzzy as it’s my personal software! A much easier setup than PPP as well as the MTU for routes can stay at 1500 and not be “adjusted” for PPPoE over PPPoA as is the norm in the UK.

Still, this is only day one, the service could go tits up, but so far the experice is very very nice :)

I’m FTTC connected and I get 14mbs down 0.2mbs up …. on a good day. When I took this service up 3 years ago it was sufficient …. but now it’s now. So what’s changed? Well, my site is getting more traffic (yay!) and I have more devices network connected. Also, my family is abusing the network more as well. I am also having been an Amazon Prime subscriber for a year now.

So BT can’t improve it in 3 years. But I need something, so kaboosh, lets look elsewhere. Goscomb technical support while initially good is just simply shocking now they’ve been bought out by hSo group. They don’t actually use their ticket tracker and when I phone up they say file a ticket. Explain patentiently I did ….. you get the idea. Coupled with the fact their IPv4 service has been exceptional, but their IPv6 service has been getting very spotty (and IPv6 was one of the sole reasons for using them) it’s time to look elsewhere again. Also, I’m maxing my download cap and it’s getting annoying and the next tier up is quite pricey….

So where to?

For better or for worse, I will be getting a Virgin Media Vivid Gamer 200 cable connection installed on Saturday. 200mbs down, 20mbs up, no traffic management. Wow! The up is better than my down ever was! Unlimited downloads sounds sexy too. The new deal is cheaper by £22 per month compared with the BT and Goscomb combo too boot. So what’s the catch?

Well, I can’t use the Virgin Media Business account. They can’t get a static IP address working on their business modem in modem mode. At all. Also, there are reports of their static IP service being dog slow and when they change to dynamic the speed gets back up. Both of these are deal breaks for me …. I need it in modem mode as I run my own network. I’m moving to them for the speed, not the slowness and network dropouts. Apparently this is because the static IP is a GRE tunnel over the residential service.

So this means no more 100% self hosting …. I need a static IP address to send mail from. So my solution is to rent a VPS for £3 a month to act as my public nameserver and email gateway. This is now setup …. and in operation! It was fairly challenging, but the added bonus is that a lot more spam doesn’t hit my homenetwork as it’s blocked by spamhaus and postgrey. Even if it gets past that, it still has to defeat spamassassin on my mail mailserver, which is still hosted at home. It also means no more IPv6 …. but I will investigate a Hurricane Electric IPv6 Tunnel. I’ve used them before and it as good but slow … hopefully it will be useable on the new cable! Virgin Media claim to be rolling out IPv6 mid 2017 but I’m not counting on that (but it would be nice).

Anyway, I think I’ve got everything ready to be upgraded now and I’m quite exited!

Abbey made the a lovely teddy bear cake with his name in building blocks … seriously more tasty than any shop bought cake.

Loads of family and friends from both sides of the family made it a wonderful ceremony ….. and social afternoon party.

Well, the kids certainly had a party with a bouncy castle indoors!

Thanks for coming everyone!

Happy Birthday Son!

Tech

My old blog was powered by Serendipity. Before that, it was Trac and before that it was Drupal

Now, Drupal was very …… well, blocky. It worked, and worked well to start. But my poor server at the time could not cope with the resource demands it had, alongside my existing Trac instances to host my project’s wiki and ticketing systems. So I migrated it to the Trac FullBlog Plugin. This was amazing! Where Drupal was complex, Trac was simplicity. Life was good.

But slow. As it turns out Trac itself was getting slower and slower. So I changed from Apache Pre-Fork to Lighttpd. This didn’t work too great for me. Too complex, too crash. So I changed to nginx. and used uWSGI to bridge my fcgi’s. Wow, my multi-purpose server had free memory! Amazing! I used this setup for about 5 years, maybe longer. However, Trac still got slower and slower.

So I migrated it all to Serendipity (the blog that is). I was also getting fed up with using git as my source code repo alongside Trac so I changed that to Fossil. No more Trac? And nginx and uwsgi working great! Life was good, my server now had over 1G free memory … which did slowly get used up as a cache I guess. But swap was never ever hit. Even when compiling the server software. Tech did not get this good.

But then it stopped. Serendipity refused to let me log in to the admin panel. But Life got in the way at this point. So I didn’t blog.

Fast forward to now and my DSPAM filter stopped working. Full stop. No-one knew why, myself and a few NetBSD devs spent a few hours looking at the problem. It only worked outside of my procmail filter, but my setup required it to work inside as I have email lists on the same box that required unfiltered mail to hit their informational funcions. And I hate SPAM. So I installed SpamAssassin as a replacement. It turned out the pkgsrc package needed a small bit of love, but now it’s working as well as DSPAM ever did, but more importantly working. As a result of this, I had to prune spam from my hosted mailing list archives and it turns out hypermail doesn’t allow you to do this, short of manually rebuilding the archive. So I ditched that and rebuilt my mail archives around mhonarc. I also took the time to theme it like my Fossil projects for consistency, which is fine.

Now, a lot of mail archives have a nice search function too so I decided to brush off my PHP skills which I’ve not used since the 1990’s. It turned out my skills weren’t that great as I thought and my first attempt failed. But no error was posted OR logged. It turned out that I couldn’t get uWSGI or NGINX to actually log my PHP errors! I’m sure there was a way, but frankly the setup was hard to manage so instead of bashing my head, I looked to find an easier solution.

Apache had gained a new event MPM which promised the same low memory use as NGINX. It also had really improved mod_fcgid and mod_proxy so it’s very useable easily. PHP had integrated PHP-FPM. As I no longer needed Trac, NGINX or uWSGI i said bye bye to them all and installed Apache. So full circle on the web server, but I now had easy to find PHP error reporting, so my archives search page worked fine after a quick fix!

In summary everything on my server was back to normal, fully working nicely, easily and well reported and diagnosed. Except my blog. I spent more time looking at it, didn’t like it, nor the hard upgrade I was looking at the to latest version. So, after changing all my software, why not my Blog?

After migrating a few times, I know the score. It’s Tricky (TM). So how to make life better? After a lot of research, I have settled on using GRAV for a few reasons listed here:

• Does not force MySQL database (I prefer PostgreSQL)
• No SQL database requirement, uses folders and Markdown files with embedded meta data
• The default theme already looks good
• Can self upgrade
• Because everthing is in markdown files it should be easier to manage and migrate away from later
• I want to self host- this is my data, not someone else’s
• It’s in PHP (so no need to install Node.JS, Ruby or whatever flavour of the month is)
• MIT license (close enough to my favoured 2 clause BSD)

So, this is the first post in a new blog, a new start where my Tech is currently ….. working without issue! This has got to be a first for me :)

Code

I write code. I like to think I’m quite good at it :) And like most people, I have an opinion on what makes good code and bad code …. I hate bad code, even my own! So how did code get in the way?

I mentioned earlier that for some reason I my Serendipity admin login stopped working. Well, I looked at the code …. and it was not nice. (My install was an old version compared to their current 2.0 so this may have changed) There were little comments, there was poor seperation between Model, Controller and View. Now, the actual code in question looked like it should be working …. it hashed my paswsword, compared the hash to the saved DB entry. I even went as far as adding debug code. It all looked perfect, yet it did not work. No-one knew where the error was.

I was also pretty annoyed with Serendipity for other reasons I discovered only after using it, so I decided to ditch it entirely and move on. As you can tell above, I have no problem changing out software I no longer believe in, or fares poorly compared to others. I also enjoy when the software I originally chose makes great strides to tempt me back in the fold, like Apache. I don’t claim to understand web server code, but I did peruse their codebase a while ago and it looks cleaner compared to when I did in the early 90’s trying to make it work on Windows for $DAYJOB.

I’ve also spent a lot of time on my pet coding projects, dhcpcd, dhcpcd-ui, resovlconf and NetBSD.

Anyway, enough ramblings about code. There will be a lot more about that in the future anyway now that I have a working blog again. Well, almost working. I need to write some code to import from Serendipity to here, which will be nice as I enjoy coding and want to preserve the content I’ve lovingly created- alongwith you in the comments- over the years and I would hate to lose that! Speaking of comments, there is currently no comment system on this blog. So Tweet me about it for the time being until I integrate something like ISSO commenting or maybe find a Grav plugin. I also need to find a way of providing top links to my project pages.

Life

So in the past few years, my Family has been very stable and loving :) But a new child got in the way! Ethan Thomas Marples was born 2nd December 2015, so he’s one year old tomorrow! That has taken a lot of my time, and all my children still take a lot of my time, so it’s a very enjoyable getting in the way :D

There’s also been some changes at work. My old company Logos Technologies was bought by Instem. I’m still doing the same job, working on the same product, but the inner dealings of the buy out, company politics and day to day integration have certainly been eye opening.

Playing games got in the way. Far too much time playing MMO’s …. I’m now off that bandwagon. But I don’t regret it, it was a lot of fun :)

I’ve also been on a health roller coaster. I’ve taken the nice Doctors Orders and have invested in a FitBit Charge and FitBit Aria Scales which keep track of my steps, calories burned, sleep and weight. There is no more hiding. The good news is that I’m a lot fitter- I now enjoy walking and take a 40 minute walk each day minimum as well as walking the kids too and from school. I’m drinking a lot lot less, but still enjoy (probably a little too much still) a weekend tipple.

I’ve lost over a stone (21 pounds to be precise) as a result! My resting BPM as dropped from a $HIGHNUMBER to 58, which is healthy :D I should keep track of my blood pressure more, but maybe in the new year ….. but

** GO ME!! **

I’m fitter, stronger, more alert and ready to blog! Although probably shorter blogs next time ;)

During this time I’ve also been working on removing DBus support from libdhcpcd so it talks directly to dhcpcd. I’ve also started a dhcpcd-qt port and written a dhcpcd-online tool, both of which use libdhcpcd. Whilst doing this I discovered that dhcpcd’s control socket was very very flaky so that’s now been vastly improved as well. There is an intermittent SIGPIPE error which I still don’t know why it occurs, but it’s gracefully handled and dhcpcd / libdhcpcd can easily restart things.

The lesson learned from both of the above points is that no matter how well you write your code, on a large project like this there is always an error somewhere and it someones takes someone else to find it!

Boy was I wrong!

Well, not wrong, but a tiny bit annoyed. It did catch some false positives, but only because I know some conditions are impossible but could be better expressed so the pass the analyser. here is a good example of what I mean. The 1st two changes don’t change anything in the real world, but do make the code easier to read and thus easier to analyse.

However, it has caught some real errors which can could problems in the real world. Nothing really earth shattering, but I like being a perfectionist! During this process I also discovered that hard coding CC in Makefiles can be a bad thing. So we now just honour CC as set by the environment and default to using the make default or cc if not set.

So to test clang analyser on the latest dhcpcd head you can do this

./configure
make clean
scan-build make

and check the results!

However ….. how about dhcpcd running in a pure threaded single process environment as found in Real Time OSs? Now it becomes more interesting because you can no longer use signalling and you really are restricted to a single shot dhcpcd controlling all interfaces. It could be argued that not having any global variables is just a better programming practice, but I’m not entirely sold on that one.

Anyway, I’ve recently committed quite a large patch to dhcpcd which passes a context to a functions which would otherwise have none so that every single function and operation can safely work in a threads based environment.

Now, I need to improve the control socket so that the dhcpcd commandline works the same way entirely without signals. fun times!

When I discovered FreeBSD and then NetBSD making dhcpcd work with the same features provided troublesome. For the IPv4 case, we needed to patch the kernel so that IFA_ROUTE remained sane. For the IPv6 case it was a lot more complicated as parts of the IPv6 stack rely on the kernel processing Router Advertisements instead of a 3rd party tool. Working this out was a lot more time consuming and complicated, but we got there with minimal fuss!

So as of now, dhcpcd can fully replace the NetBSD and Linux in kernel Router Advertisement handling code.

But why was I right after all these years? Because quite a few knowledgeable NetBSD folk assured me that the kernel was fine and it was likely dhcpcd at fault. As it turned out, it really was the kernels fault.

roy@uberlaptop ~ $ emerge-1 portage
!!! Unable to parse profile: '/etc/make.profile'
!!! ParseError: Profile contains unsupported EAPI '5': '/usr/portage/profiles/default/linux/x86/13.0/eapi'
!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to--help,--info,--search,--sync, and
!!!--version.
roy@uberlaptop ~ $

Asking for help in a secret Gentoo IRC channel….

10:57 < bonsaikitten> rsmarples: ah, it's so old that you want to update
                      portage, best to use binpkgs
10:57 < rsmarples> *blinks*
10:57 < bonsaikitten> http://packages.gentooexperimental.org/packages/ # my
                      emergency supply of binpkgs
10:57 < bonsaikitten> the minimal set to get portage working again, afair, is
                      glibc+gcc+binutils+python+portage
10:58 < rsmarples> i should just need portage yes?
10:58 < bonsaikitten> yes, but that need python-2.7
10:58 < bonsaikitten> and my python binpkgs are too new and rely on a newer
                      glibc
10:59 < bonsaikitten> if you have python-2.7 there's a good chance that only
                      portage is needed
11:00 < rsmarples> .....
11:01 < bonsaikitten> what :)
11:01 < bonsaikitten> you didn't update for around ... 2 years? maybe 3?
11:01 < bonsaikitten> and our support horizon is currently between 6 months and
                      a year
11:01 < bonsaikitten> the binpkg method is the most painless way to punch a
                      machine forward

Lets just say the the chances of me coming back to working on Gentoo went from small to zero.

In truth, although I had a nice new server upgrade it seems that Trac just keeps on using resource after resource. Now, having the site entirely python powered isn’t entirely possible- thus far I have yet to find a reasonable email web interface other than RoundCube Also, my PostgreSQL interface of choice uses PHP. Speaking of PostgreSQL, that kind of limits my blogging software choices as most of them only want to use MySQL, which I have no wish to use. Here’s a list of blogging software which can use PostgreSQL:

• Trac
• Drupal
• Serendipity

So seeing as I’m moving a way from Trac and I had a bad experience managing my prior Drupal install many years ago, it’s no surprise that I’ve migrated my blog over to Serendipity :) As with the prior migration I’ve preserved comment history and my image gallery. However, I still need to edit all prior posts and correct their links. At this point you maybe wondering what now powers my projects? Well, they recently converted from GIT to Fossil

All of this effort and not much blogging? Well, lets see if I can blog more here as it’s my content which I own entirely on my self hosted site. I think I’ve been giving to much content hosting to Facebook and Twitter. I’ll not stop using them, but I expect to post more here :)

An another note, it’s imperative that everyone knows about IPv6 foo if you run IPv6. Todays lesson is scope- a Link Local IPv6 address needs an interface scope. After fixing dhcpcd to append %eth0 (or your interface name) to the link local IPv6 address that then broke openresolv sending to dnsmasq via dbus. That in turn led me to writing a new DBus interface for dnsmasq and the current one didn’t cater for IPv6 scoped addresses and the new one as used by NetworkManager didn’t work over dbus-send which I need.

So it’s been a productive weekend! I may even produce new releases over the weekend once the fallout has setlled ;)

Anyhoo, since GW2 is a lot more casual than the others the idea is that I have more time for working on open source stuff, like say dhcpcd. Well, I finally found some time over the past few months and put DHCPv6 INFORM support into dhcpcd. This is only triggered when dhcpcd recieves an IPv6 RA with the O flag set. Luckily, I put IPv6 RS support into dhcpcd a few months ago but didn’t feel the urge to blog about it. Well, I do now!

Before the end of the year, I aim to have DHCPv6 IA support triggered via IPv6 RA with the M flag set as well. Do I have any other goals? Well, all my other projects are pretty much complete. One of the reasons why I starting gaming so much is that I’m pretty happy with the current state of affairs and anything else is just a bonus. That being said, there are a lot more DHCPv6 things to do besides IA support such as better separation between IPv4 and IPv6, moving to TAILQ lists intsead of our hand rolled and other misc stuff such as improving dhcpcd-dbus and dhcpcd-gtk.

However, we’re now back! Currently powered by a Shuttle XS35

I took the moment to install NetBSD-6.0RC1 on it with a current snapshot of pkgsrc. The NetBSD-6 install kernel got the disk sizes wrong, but a NetBSD-5 install kernel gets it right. The downside is that the-5 kernel won’t PXE install correctly whereas the-6 kernel will. So a bit of playing around, but I managed a network install in the end. That’s the only real problem I’ve had so far.

Software wise, everything has been updated to the most current in pkgsrc and I’ve just copied my old configs over. Seems to be working fine so far. Shout up if you see something broken!

Will I blog more? Maybe.

I’ve also ditched OpenID for the time being as some spammers were still getting through. I still have trouble getting the AccountManager trac plugin to work well, so I’ve gone back to the trac spam filter plugin, with captcha support enabled. We’ll see how this works.

Firstly, Abbey gave birth to Theo Ray Marples on 3rd April 2010. Secondly, I have joined a World of Warcraft guild that raids Icecrown Citadel on a regular basis (8/12 progression). These two facts have taken up a lot of my free time I normally devote to working on my OSS projects.

But do not fret- the few bug reports I’ve received have all been addressed and I have been releasing new versions when required. However, since I no longer use Gentoo I will not be maintaining OpenRC anymore. That will need a new home. A big thanks to all the people who helped me work on it over the years :)

I may start blogging about Twisted Fire, my Warlock which I play in Warcraft. I may also find the time to work on IPv6 support for dhcpcd, which is hopefully being updated in Debian from the ancient dhcpcd-3 it currently uses.

Evolution suffers two problems on my NetBSD box. One, it takes over 5 minutes to load and two, once it has checked and downloaded mail from my server it closes without error. The 2nd problem is probably a programming error in Evolution or one of its many dependencies, but the first problem could be anywhere! Research led me to this possible soltuion. If you look closely, you’ll find an old post by me in that thread fixing it with ld–as-needed on Gentoo/FreeBSD, but that isn’t a real solution for NetBSD / pkgsrc users. Anyway, at first glance the patch on that thread makes perfect sense. So I came up with better patches (one for glib, one for Evolution) and it now loaded in under 3 seconds! All was well :)

Well, not entirely. Other applications started to show isses as a result. So at least we know we’re in the right area. I then spent a few hours wading through the Evolution build chain and came to the conclusion I still hate autotools as it obscures everything but was no better off fixing it. But I did come up with a test case that opened an evolution library and searched for a symbol that did not exist. This should 30 seconds, which is far too long and is also probably what the poster of the FreeBSD patch did in the mail thread.

Cutting to the chase, the runtime linker on NetBSD was checking the same library for the same symbol over and over again. Going back to the FreeBSD linker, they have a patch which implements a negative cache which should solve this. But it doesn’t, as it doesn’t cache missed weak symbols. Luckily this is easy to rectify and you can find my patch here and an updated one after performance testing elsewhere here. Now Evolution loads in under 3 seconds and crashes right away- but that’s an issue to fix another day :)

So my little angel is now 2 years old :) She had a nice birthday and was very excited by present opening for the first time …. ominous. It only seems like yesterday she arrived!

Anyway, one big party to look forward to next weekend. Should be fun!

No doubt there will be some fallout and flames, but it’s all about bringing some needed modernization into NetBSD!

So NetBSD finally has an up to date replacement for the legacy termcap it currently uses. Well, it will have when merged - I’ve only just asked for testing so it might take some time before it hits-current.

I’m now using nsd as my public authoritative name server and unbound as my private recursive name server. This results in config files much easier to setup and hopefully more security as nsd is very small leaving less room for public exploits. Also, the total memory used after a few days production is 40meg lighter than BIND which is good as my server is really starved for memory right now.

So what’s the catch? Well, there is no way of integrating Dynamic DNS with unbound that I can see. I could use dnsmasq as a DHCP server as that can run a script per lease, which means I can use unbound-control to update DNS. But I dislike this as dnsmasq isn’t entirely the right tool for the job. So I will probably write dhcpsd, a small portable and entirely new DHCP server that can be used to integrate with DNS via scripts. Watch this space!

This is important for NetBSD, as many packages support resolvconf, but only when /sbin/resolvconf exists. This meant that a lot of packages that supported resolvconf, failed to work with any resolvconf implementation from pkgsrc.

PPP users who maintain their own scripts are encouraged to try it out :)

Still, it was very enjoyable. 24-28 deg C with only one day where it was too windy to sunbathe :)

I’ve done almost zero coding since I got back as I’ve been involved in meetings so damned much. This is good though, as we now have a very clear idea on how to achieve World Domination! Actually, it means that we spend less time coding to achieve the same end goal which means a better product delivered faster.

The holiday did hi-light an important need- my aging Motorola PEBL isn’t up to the job of being a phone anymore AND having one that can be a tethered modem cheaply is essential. Or at least send/receive emails and remote desktop. As such I’m hopefully getting the new BlackBerry Storm2 or BlackBerry Bold 9700. As I’ll be making more trips to other parts of the world in the near future, this will become very essential very fast.

So it’s off to the Costa Del Sol for 2 glorious weeks!

To appease the masses, dhcpcd-5.1.3 was released which will hopefully find its way into NetBSD-5.1. Try not to find any new bugs whilst I’m away ;)

Still, it’s a nice city :)

Try not to report too many bugs with my software until I get back ;)

Each terminal type uses different codes to draw the same thing. So there’s a database called termcap that maps a global two letter code to the terminal code to use. However, the API to this was quite limiting and some terminal descriptions don’t fit into the available space. So terminfo was created to remove these limitations. Most systems that use termcap have since changed to terminfo, except for NetBSD and OpenSolaris.

ncurses is a terminfo/termcap/curses solution all rolled into one and nearly everyone uses it. So why roll another one? Well, because I think I write cleaner, faster and smaller code. Also, there should be choice - there are no other modern terminfo implentations. Plus, I like a challenge :)

But I am working once more on integrating it with libcurses. Once that is done, it’s ready to go into NetBSD.

Sounds complicated? It is, so I won’t bore you with any more details! Suffice to say that I’m very happy with how things turned out after months of uncertainty and worry :)

There are quite a few OpenID Providers, so it should be quite easy. You may already have an OpenID! However, due to how the OpenID plugin works with Trac your username will look a little odd. This is because your OpenID is basically a FQDN, just like a web address. So it’s best to use a provider like !http://.provider.com or !http://provider.com/username- in other words, avoid Google for OpenID unless you want a really long username here. This not that bad though, as the plugin strips the !http:// part and the trailing slash.

You can see from this post that my OpenID is roy.marples.name, which shows that having your own domain and OpenID server is important if you want to look good ;)

We’ll see how it goes :)

A few programs in base system needed to be changed due to having their own getline function, most of which aren’t anything like getline(3). Hopefully there won’t be much fallout in pkgsrc as a result.

getline(3) is prefered over over functions such as fgetln(3) and fgets(3) because it’s standards based and you get a dynamic buffer for really really long lines. However, POSIX did drop the ball on making it a standard from the GNU extension- it should return 0 on EOF and more importantly be called fgetline. Oh well.

I shall be rolling getline(3) support into dhcpcd later, but I’ll have to do a link test in the Makefile to see if we can use it. I’m unsure if I want to have a mini configure for dhcpcd or to keep using just make extensions ….

This led to an interesting debtate about my internet connection. In a nutshell my Drayetk Vigor 100 is broken, but apparently by design. The problem is this- for IPv4 goes through my NAT whose public IP has an MTU of 1492 so Path MTU Discovery works fine. However IPv6 does not need NAT and the PPPoE interface does not have a public IPv6 and nor do the internal clients use it even it there was one. So the path MTU is 1500, which is too big for the PPPoE to handle.

The correct solution is to obviously terminate the IPv6 on a PPPoA connection, but only the Cisco 877 router does this and that’s outside of my price range, even on eBay. One solution is to clamp the MSS on the router to 1430 so that all clients work. But this is a hack. The best solution with my hardware is to force the MTU to 1492 for all nodes inside my network, so they share the same MTU as the PPPoE so that Path MTU Discovery works.

You can see this working for yourself on IPv6 now, as this site is IPv6 enabled and sits behind the PPPoE link. You’ll need to query IPv4 servers for the address though as I don’t yet have a glue record for IPv6, but as I’m changing registrar that should change in a few weeks :)

Because of this, I’ve made the dhcpcd default to request and use the MTU value if offered by the DHCP server. You’ll be seeing this in dhcpcd-5.0.5 (now out). dhcpcd-5.0.6 will feature restoring the MTU correctly between leases.

There maybe some more downtime in the near future as I may change registrar as I’ll be getting native IPv6 later today (yay, no more tunnel!) and my current one (1&1)don’t appear to support IPv6 in their domain / dns setup.

So I’m changing to Goscomb Technologies because they are slightly cheaper than Demon, have a 24Mb service NOW and offer native IPv6. I did have cause to phone Goscomb up, and it was nice to hear a native English speaker at the other end on a good quality line instead of a non native speaker on a poor quality line due to the customer services section being out sourced. Another plus.

What does this mean for you? Well, on the 29th of this month, the switch should be made, so there will be some downtime and delay whilst the DNS updates. So this is also a heads up :)

It took some time, but it looks like the NetBSD and Xorg devs have finally cracked ATI X600 laptop cards working :) The final bit on my behalf was enabling EXA acceleration as the older and default XAA caused the screen not to refresh properly.

OpenGL screensavers have been working flawlessly for nearly have the day now, so it looks fairly stable too. However, switching to console and back to X does disable DRI for some reason, so I have to do a full X restart which kinda sucks.

I may now look at getting one of my online games, Eternal Lands working on NetBSD as well :)

Initial reaction is that it is lighter on memory and also faster, but time will tell. What I did like is that installation is very simple and well documented, unlike fastcgi or fcgid. Sadly I still need to use fcgid for the non python parts of my site.

dhcpcd-5 branch. You can see here that I did commit before branching but if you look at the tarball built from svn the commit “Fix existing address detection.” isn’t there.

Now, whilst this ‘‘‘could’’’ be an error on my part, I’m not prepared to take the chance it’s also a subversion error. So I’ve moved all my repositories to git. As I’ll be dealing with more branches in dhcpcd soon, and git is supposed to be renowned for branch management, it’s a move I’ve been considering recently.

Anyway, I’ve installed the trac GitPlugin. It’s not as speedy as the SVN browser for trac, but it works fine. If speed is a concern, gitweb is available as well.

Lastly, Linux has a lof of misc tools- ifenslave, vlanconfig, etc, which should all be wrapped up into ifconfig. The BSD ifconfig is a very good example of how network configuration should be done.

As such, I will attempt to write an ifconfig for Linux which addresses all the above, with a comprehensive man page. This will make the new network script for OpenRC a lot easier to use in turn. It’s not aimed at being an iproute2 replacement as iproute2 handles a lot of other stuff outside of the scope of ifconfig.

Since being back I’ve been very busy and both Abbey and I recovering from various ailments, but we’re now in tip top condition! As such I’ve had time to post some new screen shots.

Now the parents have promised a repeat cruise in 9 years time- can’t wait :)

This looks like a steal to order job as they managed to lift the bonet slightly to get the headlights out, but rushed off as one headligh was left dangling.

I’ve not blogged about it because I was unsure if it was the right thing to do. Now I think it is, as this blog is supposed to be a diary of important life events as well as the standard technical stuff I blog about. Anyway, the oddest thing is that I only realised I loved him after he died. Why it should take a death for me to realise this I don’t know. Does that make me a lesser man for not realising this sooner, or a greater man for gaining insight at the last?

At least now Abbey seems to have recovered from it, although she is mighty tired. She did spend the three weeks prior to this one at her Mums, looking after her and visiting Ray in his final days. I missed her and Robyn a lot in that time and am very glad they are both now home.

It doesn’t have any functional changes or improvements over the 2.0 version. So why the big release? Configuration baby, configuration.

Debian’s resolvconf has plenty of mini configuration files, which openresolv has supported. This is quite problematic from a package distribution and maintainability perspective. Also, the implementation required plenty of forking sed or equivalent to parse them which is inefficient.

So openresolv now sports a shiny configuration file, which I think is more flexible and easier to manage. It has a downside and an upside. The downside is that you now have to configure the files to write resolver configuration to for dnsmasq/named. The upside is that if they are unconfigured (the default) then the subscribers don’t litter /etc with files you’ll never use.

Also, the subscribers themselves have been moved out of /etc and into /libexec as they aren’t really configuration files.

If you have any issues with this and it’s really not spam, then email me or get it in touch via IRC. You will probably need to at least enable cookies and store your email address in the preferences.

I’ve also spent a lot of time polishing it with nicer icons, messages and titles. Unlike the previous version, this dialog works fully! It’s looking so good, I’ve just released dhcpcd-gtk into pkgsrc for your pleasure- enjoy :)

I also ought to get off my lazy ass and get ebuilds for them into Gentoo- maybe someone can do it for me?

The Count Censored. There’s loads off follow ons for this too, and it’s all good :)

All blog posts and comments have been retained, as has my image gallery. There are a few errors in the migration, but they are acceptable.

So welcome to new the site and enjoy :)

• Slow - each net.foo loads every network module
• Hard to extend - only I really know how the network modules work
• Has own dependency code separate to OpenRC

So what’s the future? Well, the future is scrapping the network module idea and just supplying an init script that can be multiplexed for each network component. Yes, I hear your gnashing and grinding of teeth as what is there currently just works out of the box. What I propose will also work out of the box :)The network modules will not vanish over night, as there are some parts that will not be brought over such as the system or arping modules as they don’t make sense as init scripts. You can replicate the system module by configuring dhcpcd-5 per interface/ssid with a static configuration.When will this happen? Well, it will start to happen when dhcpcd-5 is released, which isn’t that far away. Why wait for dhcpcd-5? Glad you asked! dhcpcd-5 takes care of a lot of the network management for you. It now works as one daemon, not one per interface. It manages your routing table for you and allows DHCP created routes to be changed. The last feature is to pickup PTP interfaces and give them a default route to their destination address so that it integrates seamlessly with PPP and VPN clients.But what about the other DHCP clients- I want to use dhclient with ifplugd/wpa_supplicant.You have two options here- keep on using the exiting network modules or script ifplugd/wpa_supplicant yourself. Or use something else like NetworkManager.

At present, dhcpcd-gtk is just an application which sits in the notification area. The icon has several states, showing offline, address negotiation and online. When attempting to negotiate an address you get a nice animation. A notification bubble is also shown per interface state change.

Future versions will have Access Point selection and dhcpcd configuration options. Both are currently available via pkgsrc-wip. Hopefully available in Gentoo soon as well :) EDIT: ebuilds available for dhcpcd-dbus and dhcpcd-gtk from my ftp server.

EDIT: Here’s a screenshot as requested

Woooooooo

I’ll try and get them backported to NetBSD-5 and NetBSD-4 in the New Year.

Proxy ARP Daemon is a new tool I’ve written which complies with RFC 1027. In a nutshell it provides a transparent gateway for your subnet.

Why would you want this? Well, in a modern world you don’t as DHCP clients will configure routing correctly and have more bang for your bug with autoconfiguration. So what’s the use of it? The answer is found in the new buzz-word of the moment- virtualization. To get a virtual host to talk to the host and out into the internet you have to setup some kind of bridge between the two. A Proxy ARP daemon takes away the need for this. Here’s an example of it’s use.

Anyway, version 1.0 has been rushed out the door. Tested on Linux, FreeBSD and NetBSD. Linux does have an in-kernel Proxy ARP daemon, but I think the userland is a better place for it. You can find it right away in pkgsrc-wip with an rc script.

These scripts are for bugzilla-3.0.3 and trac-0.11.1 and assume that no custom fields have been added. They are also coded for specific product id’s and my name- you will need to adjust accordingly.

bugzilla to trac sql script. It simply creates new tables for use in a trac db- ticket_change_status needs to be copied into ticket_change though. bugzilla to trac perl srcript. Extracts attachments from bugzilla and creates them in the current directory in a structure for use in trac.

TODO: attachment filesize is 0, this needs fixing.

Also, my company suddenly had a need for a bug tracking system and a colleague of mine suggested trac which I installed on a server. I only looked at track briefly many years ago, and it had promise but lacked in a lot of places. I was pleased to see that a lot of good progress had been made and it’s now very useable :) So much so, I’ve decided to install it here and it now powers the openresolv project page. Because it’s made to integrate into subversion I used git2svn to convert the openresolv git repo trunk into an svn trunk. It’s now open for business and anonymous users can create and modify tickets and the wiki (well, parts of the wiki).

So is svn better than git or is git better than svn? It’s a hard one to answer, both have their pluses and minuses. Luckily there is a trac addon that works with git, so I’ll give that a try with dhcpcd.

dhcpcd has been able to apply metrics to routes on Linux so that we can prefer to route packets over wired instead of wireless. dhcpcd-git is now able to distinguish wired from wireless and can make a metric accordingly.

But how do we teach configuration files about this? Well, dhcpcd-git has an environment variable sent to each script telling it about the preferred order of interfaces (based on carrier, if we have a lease or not and metric). This works well, and we can now prefer wired nameservers over wireless ones in /etc/resolv.conf. Well, we can at least put them first in the list.

Whilst doing this, it struct me that resolvconf has no means of preferring configurations other than a static interface-order file. This is not good for automatic foo! So openresolv now understands the -m <metric> option and the IF_METRIC environment variable so it can tag resolv.confs by priority. If no metric is specified, it takes priority. If >1 interface on the same metric then we take lexical order.

Whilst putting a mini resolvconf into dhcpcd, I use very similar code to what I already had. But, I ran into a wall- dhcpcd needs to work when /usr is not mounted as it may be network mounted later. On Gentoo and most Linux distros, sed is found in /bin. However, on NetBSD it’s in /usr/bin.

Oh what is a man to do :?

Luckily, we can fall back to shell for simple sed usage. Here’s a simple snippet

key_get_value()
{
       local key="$1" value= x= line=
       shift
       if [$#-eq 0 ](); then
               while read line; do
                       case "${line}" in
                       "${key}"*) echo "${line##${key}}";;
                       esac
               done
       else
               for x; do
                       while read line; do
                               case "${line}" in
                               "${key}"*) echo "${line##${key}}";;
                               esac
                       done < "${x}"
               done
       fi
}

Old call

sed -ne 's/^nameserver //p' /etc/resolv.conf

New call

key_get_value "nameserver " /etc/resolv.conf

According to the time command on dash and bash shells this is ever so slightly faster when doing resolvconf-u. This is probably because we aren’t forking an external command. Also, we’re working on small files- this probably sucks hard for big ones.

Anyway, this has made it into openresolv-1.6 which now works in the root of a BSD system without /usr mounted :)

The main culprit here is ntp.conf, as it can have quite a complex setup and dhcpcd has always imposed it’s own world view on it. However, this lead to an interesting bug being found in NetBSD rc.d script for ntpd- it requires ntp.conf to configure pidfile /var/run/ntpd.pid. Now, dhcpcd can’t put this in as the pidfile location changes from distro to distro, so effectively dhcpcd broke the stock script. Whilst this is a NetBSD bug, some people also asked why dhcpcd could not be more intelligent about things? :?

So I’ve now added a nice framework so that dhcpcd hook scripts can trivially top/tail config files with their own data can clean up after themselves. Here’s a snippet from 50-ntp.conf itself

do_ntp_conf()
{
   local cleaned= added=1 conf= x= 
   clean_conf /etc/ntp.conf
   cleaned=$?
   if ["$1" = "add"-a-n "${new_ntp_servers}" ](); then
      for x in ${new_ntp_servers}; do
         conf="${conf:+\n}server ${x}"
      done
      append_conf /etc/ntp.conf "${conf}"
      added=$?
   fi
   if [${cleaned}-eq 0-o ${added}-eq 0 ](); then
      [-n "${ntpd_restart_cmd}" ]() && ${ntpd_restart_cmd}
   fi
}

Nice and simple :D I’m currently debating if we can use this framework to do more intelligent handling of /etc/resolv.conf if resolvconf is not installed, but that’s not as critical.

But this should be viewed as a non issue as in the real world, leases just are not that long. beta9 did not suffer from this as it used a decreasing timeout, but this proved to be inaccurate. Also for IPv4LL compliance we needed to introduce a second time, and for being able to exit on time for the userland a third time. Moving to a monotonic clock approach is much easier to manage, but did cause the above two bugs introduced in rc1.

NetBSD has also imported rc3 into-current (4.99.71) :) NetworkManager svn (0.7.0) also supports rc3, but it needs to be configured for it if both dhcpcd and dhclient are on the same box.

./configure--with-dhcp-client=dhcpcd

Other highlights

• support for Vendor Encapsulated Options
• use ifname as IAID if less than or equal to 4 bytes
• compiles and works correctly with PCC on NetBSD-4.99.69
• sends LOG_INFO messages to the console by default (suppress with-q)
• all string based config options are now run through the string parser (effectively means you can be stupid and put escape codes in the hostname, but more correctly in the ClientID or Vendor options)

Other bugs have been squashed as well. Since release, I’ve also fixed a cosmetic bug where a new IPv4LL address being probed is reported as 0.0.0.0 and dhcpcd now builds and works correctly on NetBSD for use in / if not using the NetBSD source tree.

EDIT: Just released beta9 with a patch that moves the new option-K in beta8 to-X so that we don’t conflict with a custom SuSE patch if they want to move to dhcpcd-4. This shouldn’t affect anyone as beta8 was only out for a few days.

That picture was taken over 5 years ago now and each and every kiss is new, exciting and very very intoxicating }:) We’re still madly in love with each other, probably more-so than ever.

The last thing todo is decide on the following Do we want per interface conigs in dhcpcd.conf? If so provide an example. We currently trim leading and trailing whitespace in dhcpcd.conf. Should we provide a complex mechanism to allow spaces at the start/end of values? If the hostname is an FQDN should we transmit it as such by default?

Answers on a postcard to the usual address :)

Basically the dhcpcd binary now only configures the interface and routes - everything else is configured by /etc/dhcpcd.sh

Yes, this means dhcpcd will finally install a script! This makes it easier for the end user to change, as scripting in UNIX is common place, whereas hacking C code not so. The script does everything dhcpcd did, bar looking up the hostname in DNS (hopefully add this back soon), so in essence we’ve lost nothing but gained more flexability.dhcpcd is still commandline compatible, and users shouldn’t notice much difference. However, the defaults may change! Probably to something more minimal- the default probably won’t be to update nis/ntp configuration.

Normally I hate massive code drops, but heh.
 The code has been drastically re-arranged.
 Instead of populating a custom structure while parsing dhcp messages, we now pluck what we need right out of the message itself. We have custom functions and a lookup table to make this really easy.
 This makes us more like dhclient and udhcpc, and will enable us to easily add (and remove!) more dhcp options without having to actually change the code (much).
 We now store the real dhcp message we got in /var/db/dhcpcd-$iface.lease, the mtime of the file being used as when we got the lease. This file is read in when re-using an old lease instead of parsing the .info file.
 The benefit of all of this means that we're actually ~15k smaller when compiled with the same features.
 This has been tested for quite some time, and I'm pretty sure most bugs with the 3.2 branch have been fixed whilst making this. Right now, we are 99% command line compatible with the 3.2 branch.

That’s pretty much it :) I’ve made this now, knowing that it will change very soon. Basically we’re going to add knobs such that the default config will just do the basics- configure interface, routes and DNS. This satisfies NetBSD minimalistic requirements (more on this). Then the extra knobs will configure things like NTP, NIS, MTU and the hostname automagically like dhcpcd does at present. Don’t panic though- the command line options to turn those off aren’t going away either!

After this is done, we’ll then add an interface to pull any DHCP option via our new generic interface, maybe something like so `dhcpcd-o LOGSERVERS,99`` Which will pull 2 options, one named which dhcpcd knows about, and the other numeric which dhcpcd doesn’t. When dhcpcd doesn’t know about it, you’ll get the raw data as a hex string I would imagine.

This should make dhcpcd a lot more flexable :) As it’s very small, and already feature rich I’ve put it forward to the NetBSD tech-net mailing list to fullfill a tiny dhcp client to replace the bloated dhclient from ISC. It’s not being done as a SoC project (I don’t feel that I qualify as dhcpcd was already written really) and it’s not a done deal that it will be used in NetBSD, but I am hopeful :)

Before

ubersparc ~ # time rc
real    0m0.215s
user    0m0.036s
sys     0m0.179s
ubersparc ~ # time /etc/init.d/local restart
 * Stopping local ...                                                                 [ok]
 * Starting local ...                                                                 [ok]

real    0m0.331s
user    0m0.086s
sys     0m0.251s

After

ubersparc ~ # time rc
real    0m0.204s
user    0m0.042s
sys     0m0.161s
ubersparc ~ # time /etc/init.d/local restart
 * Stopping local ...                                                                 [ok]
 * Starting local ...                                                                 [ok]

real    0m0.338s
user    0m0.082s
sys     0m0.262s

So we can see that rc itself is quite a bit faster than before, as it did the bulk of string processing. However, services themselves are slightly slower. Why is this? Well, to keep the code similar we have to malloc and return a list head even if the list is empty. valgrind shows this as well, rc has less mallocs, whereas the services have more. So to improve the code futher we need to stop blindly mallocing list heads. It’s hard to time overall speed, and my sparc doesn’t run bootchart so I’ll see if I can make some on my laptop, but my gut feeling is that it’s no faster or slower overall than before.

In other words, using linked lists is only beneficial when dealing with large quantities of data. OpenRC doesn’t deal with large quantities of data and as such is slower for starting individual services. The main speedup is purely due to joining and sorting which is faster due to linked lists.

This code has now been committed to git and the ABI has been broken (obviously- duh). The only things that links to us at this time are splashutils and einit. Here’s a patch to splashutils to make it work again. If I think it has no immediate benefit, why did I commit the code? Well, the answer is purely because it will scale a lot better so hosts with lots of services will see a better speed increase than my slow sparc with few services.

I’ve also re-factored the code to match the BSDs KNF coding style and ditched my own custom one. It was fairly similar, so its not much of a change- the benefit is that hopefully more people will enjoy it :)

More pictures to come later :)

However, during a DHCP RENEW, there is quite a bit of traffic going around and it was all hitting dhcpcd. So I added some code to counter to ensure that we still work correctly during packet floods. However, dhcpcd does burn a lot of CPU during these times. I then saw a post to the busybox mailing list that had a BPF filter- but for Linux! First I knew about this, but apparently Linux has been able to use BPF for a long time. So dhcpcd now uses the same filter for Linux which means that dhcpcd won’t chew CPU time even in packet floods. Or, at least, less CPU.

Get this new feature in dhcpcd-3.2.1- today! :)

So I wanged it on my amd64 and as promised, the default cd worked fine with my trusty RT2500 pci wireless card, and the installed default kernel did too. This was an improvement from NetBSD 3.x which I tried a long while ago as the installed kernel didn’t work with my wireless much is a must have for me these days. To be fair, I started off on the bad foot with NetBSD back then as it was my first BSD experience and it wasn’t good.

Thankfully, my experience with FreeBSD means I now know a lot more, so with NetBSD-4.0 I was very quickly up and running. After getting the network fully working, I pull down the OpenRC repo and tried a compile. Surprise, surprise, loads of warnings and errors.

Now, the gcc that NetBSD ships with has a lot of warnings you just don’t see with the one in Gentoo, or FreeBSD. Luckily, most of the warnings were trivial to fix - and should help matters on archs where the default char is unsigned. The errors were purely on the getmntinfo function and structure being different in NetBSD. Then there is the matter of the init scripts - whilst most work, some won’t or need a slight tweak. The important thing is though that OpenRC now boots NetBSD!

I’m also working on making the misc init scripts work by default through a make install on your arch as well, so moving an existing BSD rc.d system to OpenRC shouldn’t be hard at all. But is NetBSD any better or worse than FreeBSD? Too early to say really…

I strongly dislike autotools after dealing with it for a few years now. I shan’t go into the reasons why here though. But suffice to say that the whole build system required autotools. dhcpcd-3 supported BSD based systems, so it was only fair that we strive to support their base install. This means no autotools. So I set about discovering what I could do that worked on pmake and gmake.

Sadly, the answer is that only simple things work, hence the dhcpcd-3 Makefile being quite long. Especially as it does a little bit of auto config for needed libraries, which used to be in the configure. Well, it sufficed for a long time, and when I started OpenRC I used a similar Makefile as it had to work their out of the box. Eventually my occasional partner in crime, vapier, came across my Makefile and complained about a few things, namely it’s too complicated and good be a lot better. He even went and patched it so that it almost worked on pmake. But still, it needed to fully work.The end result is that I’ve made some .mk stubs that work with both pmake and gmake, using the same variables that the mk stubs pmake ships with. It’s nowhere near as flexable, and we do make many assumptions. Obviously, this is not suitable for everyone, but it suits me very well. And as it’s all BSD licensed, feel free to use it as you wish- just respect my copyright.

Well, OpenRC/URC is pretty much done for a first release in terms of the software. All I’m doing now is writing man pages for everything as we relied a bit on the Gentoo Handbook. The userland applications have been done, including a very long one for runscript. All that’s left are man pages for librc and libeinfo which hopefully I’ll do over Christmas.

Speaking of which, thanks to everyone who bought nearly everything on my wish list. The irony being that the two items I probably wanted the most are the only two items left, and also the cheapest! Ah well, probably treat myself in the New Year.

Or more specifically, it can now query the termcap library to find out if your terminal supports the codes we need to do colour, boldness and cursor movement. We just use tgetent, tgetstr and tgoto, so just a standard libtermcap is all that is needed, but we optionally build against the more popular ncurses as well.Some BSDs termcap database is found on /usr, which is no good for us as we bring the system up, so we do have hardcoded escape codes for cons25 if querying the termcap database fails.

Well, we need an area to save service states, lock files, the dependency tree and a few other goodies even if no physical disks are mounted read-write. This is a little more challenging for BSD as we can only create disks of a fixed size, which is hardly at the cutting edge of efficiency. However, as baselayout used to be entirely shell script based this was the only option, but as we’re now backed by C for the most part I asked myself is there an alternative?

There answer is, of course, yes there is. We can use System V Message Queues. I’ll be looking into making an rc daemon which will handle service states, generate a dependency tree and use that to resolve the dependencies instead of each runscript instance trying to do this itself. This maybe a little complicated, and I have no experience of doing this so it make take a little while. Hopefully I can use semaphores to imitate the existing service exclusive file locks too.

#!/sbin/runscript
command=/usr/sbin/ntpd
pidfile=/var/run/ntpd.pid
command_args="-p ${pidfile} ${NTPD_OPTS}"
required_files=/etc/ntp.conf
depend() {
	need net
	use dns logger
	after ntp-client
}

OK, you don’t get the comment about where the example is, but heh.

But there is one thing it does have which is very nice - and that’s init script templates. Take this sample ntpd init script from Gentoo (line breaks and comments removed, sadly my blog removes the indents too)

#!/sbin/runscript

depend() {
	need net
	use dns logger
	after ntp-client
}

checkconfig() {
	if [ !-f /etc/ntp.conf ]; then
		eerror "Please create /etc/ntp.conf"
		eerror "Sample conf: /usr/share/ntp/ntp.conf"
		return 1
	fi
	return 0
}

start() {
	checkconfig || return $?
	ebegin "Starting ntpd"
	start-stop-daemon--start--exec /usr/sbin/ntpd \
	    --pidfile /var/run/ntpd.pid \
	    ---p /var/run/ntpd.pid ${NTPD_OPTS}
	eend $? "Failed to start ntpd"
}

stop() {
	ebegin "Stopping ntpd"
	start-stop-daemon--stop \
	    --pidfile /var/run/ntpd.pid \
	    --exec /usr/sbin/ntpd
	eend $? "Failed to stop ntpd"
}

Now, here’s a replacement using the template system.

#!/sbin/runscript

command=/usr/sbin/ntpd
pidfile=/var/run/ntpd.pid
command_args="-p ${pidfile} ${NTPD_OPTS}"

depend() {
	need net
	use dns logger
	after ntp-client
}

start_pre() {
	if [ ! -f /etc/ntp.conf ]; then
		eerror "Please create /etc/ntp.conf"
		eerror "Sample conf: /usr/share/ntp/ntp.conf"
	return 1
	fi
	return 0
}

This should make a fair few init scripts smaller, and remove the headache of using start-stop-daemon which some users find.

On another note, the openrc build system has been revamped so we support both FreeBSD make and GNU make 100%. This means it’s easier to slip into FreeBSD as a potential port, and maybe the other BSD’s too.

Anyway, rc6 is now out and features a shiny new logging daemon. This is like blogd from the showconsole package, but has the advantage of being tightly integrated into rc. This allows us to log all runlevel transactions (with the exception of going to single user on FreeBSD and sysinit on Linux).

We also have a new structure for posix shell arrays. Basically instead of quoting each element, they are separted by hardcoded newlines as discussed on-dev. Here are the old and new structures.

array="'1.2.3.4 netmask 5.6.7.8;' \
'\*' \
\"'host.name' netmask 1.2.3.4\" \
\"-I 'option; $FOO with spaces'\"
"
array="1.2.3.4 netmask 5.6.7.8;
\*
'host.name' netmask 1.2.3.4

-I ‘option; $FOO with spaces’ "

See, much easier to read :)

I have no idea who did this, but thanks!

As FuzzyRay wasn’t around to aide in debugging at the time I plunged into it’s source yesterday. My eyes bled from spending time looking through it for the bug :(

Basically revdep-rebuild, like many other Gentoo utilities, uses bash. Well, it uses bash constructs but nothing bash specific that cannot be done trivially in pure sh. It also abuses find, grep, awk and sed. Me being the awkward sort, I took offence to this and did a ground up re-write in pure sh. You can find it here. The crew in #portage are looking over it and it may or may not be used. But they all think it’s cleaner, so I’d say the chances are good.

But how is this about vendor lock-in? Simple really. It’s another demonstration that most of the time you don’t need to force bash. Why should every script on my system force bash? Gentoo is about choice, and I choose not to have bash when possible :) As a login shell it’s the best, but as a scripting shell it’s too big and bloaty. Basically I’m going to make it a little goal - when I find a bash script I’ll de-bashify it and pester upstream to accept it. This makes the script run on more platforms, so unless you’re in bed with the bash maintainer there’s no driving reason not to.

• libc_r
• libphread (really libkse)
• libthr

They should all be ABI compatible with each other and as such are easily interchangeable using libmap.conf. However, that’s not always the case….

In our efforts to fix threaded python on sparc64 and a few other threading weirdness on x86 we shipped a default libmap.conf that defaulted to using libthr for everything. Many moons passed and everything was good, until LavaJoe discovered a little problem with this. Basically applications are built against libpthread but run using libthr. libpthread has some symbols that shouldn’t be used, but are still embedded so even if libmap.conf says use libthr, libpthread is really used. So you could have 2 threading libraries loaded which is the source of the issue.

So our present solution is not to build or install libc_r OR libkse. Instead we set the default threading library to libthr and provide symlinks for libc_r and libpthread. This seems to works very well, but with 1 caveat- applications which embedded private symbols to libkse or libc_r no longer link correctly. So far this is just glib, which has proven to be a most troublesome library over time and is so once more! The solution is to re-emerge the library you’re failing to link to and then it should work.

If anyone is concerned by this you need not be- FreeBSD-7 will ship like this by default.

• Backporting dl_iterate_phdr libc function from FreeBSD-CURRENT to our Gentoo/FreeBSD-6.2
• Syncing FreeBSD gcc-4 build defines with upstream (work in progress still)
• dhcpcd now works with NOMMU systems such as uClinux (work in progress still)
• FreeBSD make now supports-include and sinclude directives to optionally include files
• gcc now includes the correct #defines for FreeBSD/Sparc64
• loads of baselayout-2 fixes
• Many more ebuilds are keyworded ~x86-fbsd

In-case any Gentoo/FreeBSD users are reading this, the recent bumps to-sources,-libexec and-lib are for the dl_iterate_phdr backport. Right now, it does nothing. But when I finish the gcc-4 patchset I' m working on, gcc will use it to determine if it really needs to link to libgcc_s.so.1 just like it does for our glibc users. This is important, as currently it’s linked against every binary and library built, which isn’t really that great. :O

Hopefully when this is all done, –as-needed linker should work for all you ricers out there as it currently doesn’t on Gentoo/FreeBSD. }:)

Nerd Test says I’m an Uber-Dorky High Nerd. What are you?

The Marples line will continue, against all the odds! Needless to say, both sets of parents were delighted! And yes, my Mum did the predictable thing and burst into tears- awww, bless her cottons!

We’ve already had the first scan and Baby (working title) is due on the 25th of Jan 2008. It was a shock seeing something on the screen moving around and give a little wave. Makes it very real. What made is seem even more real was I felt it give a little kick last night! Abbeys tummy doesn’t kick so either one of two things are true

1. Baby said “hi” to me 8)
2. Alien said “rarrrrrrrrr” to me :O

baselayout-2 now buffers stdout and stderr to a file which means the errors are aligned too, so the output should be pretty much perfect.

This is not without any downsides. Firstly, the buffer files take space. Secondly, the buffered output can look a little strange when a long running init script, like say a wireless interface scanning for AP’s, suddenly appears in one go.

ciaranm proposed an alternative- prefixing each line with the init script name. He was even kind enough to point to some sample C++ code which did just this for paludis. So after writing a C equivalent (not that hard) baselayout can now do the same. We even colour code for extra prettiness as you can see below :D

This is toggled through RC_PREFIX="yes" and works independently of RC_PARALLEL.

But this still probably the nicest release as bash-3.2_p3 mostly works. The big caveat is the =~ semantic has changed. Take this snippet

[[ $(/proc/filesytems)$'\n' =~ " tmpfs"$'\n' ]]

That works fine for bash-3.0 and 3.1 Not so for 3.2! Here’s how it’s now done

[[ $(/proc/filesytems)$'\n' =~ \ tmpfs$'\n' ]]

Basically, we need to stop using quotes on the RHS when using =~ unless we want to match against the quote exactly. We also need to escape spaces.

As it’s totally incompatible for most parts of baselayout, the 1.12 branch has had all the =~ instances removed where it didn’t work with either. For the 1.13 branch we’ve changed to the new semantic and as such, alpha5 will require bash-3.2_p3. If you’re using FreeBSD then you’ll probably need this little patch for bash too.

Whilst writing it, I discovered that we no longer needed the RC_STICT_NET_CHECKING variable as we can now have >1 service providing something- in this case “net”. A discussion then took place about removing it and I think I’ve answered everyones queries.

Hopefully I’ll find the time to adapt this to our net modules too as that’s the last real bottle neck in our bash code in baselayout :P

One reboot and wanging my new stage3 onto my sparc I discovered a few missing files- heh. Most of them were the fault of baselayout not installing system FreeBSD files (bad me) and one was a gcc-config bug where mv/cp is linked against libgcc_s.so as such can’t copy libgcc_s.so from the specific gcc lib dir to /lib. Now vapier thinks that things like mv linking against libgcc_s are toolchain bugs: I’m inclined to agree only Gentoo gcc-4.x seems todo this on BSD as gcc-3.4.x doesn’t. However, until we can find out why gcc wants todo this, we really need gcc-config fixed. Besides, other archs may have this issue too, so it’s not like it’s bad or anything ;)

But the really good news is that emerging Python now works :D This means I can make a puka stage 3 using catalyst once Flameeyes emails me the spec files.

I rebooted and get kernel panics soon after loading rc and I thought eeek! wtf! :sick: So after chrooting of a livecd and swapping FreeBSD rc other with Gentoo baselayout I rebooted again …. and it worked! wtf? Why should this be?

Turned out it was openvpn crashing the kernel. Not sure why, so I’ll need to investigate. Probably due to gcc dropping the __sparc64__ keyword, but I’m sure my patch is puka :/

But it’s getting closer, soon Gentoo/FreeBSD/Sparc64 will be real and here for you to use :D

EDIT: Turns out kernel crashing is being caused by kldload loading ANY module. This means I can document this as a Known Issue and continue. Obviously it’s something we need to resolve before making anything stable.

My SPARC has now got to the stage where all base system packages + a few extras I need (openvpn, dhcp, keychain, etc) have emerged successfully :) I’ve even managed to compile and install a custom kernel. The one thing I’ve not done yet is rebooted to check that the new kernel and boot loader actually work.

So all I have todo is emerge-e world twice to make 100% sure everything is peachy, rebuild the kernel again and reboot. If all goes well then I’ll swap from FreeBSD rc over to Gentoo rc and reboot for the first time into a fully working Gentoo/FreeBSD on SPARC :jawdrop:

Not bad for a weeks work ;)

Actually, most of the work had already been done for me by the x86-fbsd team, all I had todo was patch binutils to brand elf binaries as FreeBSD when using FreeBSD on Sparc (FreeBSD patches elf.c as used by everything- we cannot do this, patch is here) and ensure that all __sparc64__ defines were changes to __sparc__ & & __arch64__. So it was mainly setting up the base profile and keywording ebuilds. Gentoo makes this very very easy 8)

Anyone wanting to look at the portage overlay for this can download it here. If either of the two links disappear it means they are now in the main portage tree, which means Mission Complete :D

sparc is a little wierd- here’s a quick list

• __sparc__
• __sparc64__
• __sparc_v9__

Now, FreeBSD only supports Sparc64 and has lots of checks using #ifdef __sparc64__. At first glance, this looks OK and there appears to be nothing wrong with it. However, gcc likes to define __sparc64__ as a CPU type (as I understand it) and is no guarantee of 64-bit code. As such, if you use -mcpu=ultrasparc (which is valid) then it drops the __sparc64__ define as we’re targeting a specific CPU model- namely the v9. A FreeBSD developer submitted a gcc patch to always define __sparc64__ but this was rejected, and looks like for good reasons too.

Of course, we can always say “dude, don’t use-mcpu in your CFLAGS” and the issue magically goes away. Not so! In gcc-4, FreeBSD defaults to using-mcpu=ultrasparc which means that it never defines __sparc64__ which in turn means a vanilla gcc cannot compile FreeBSD code! Which explains why the default compiler on FreeBSD is gcc-3.4.6. Interesting we cannot use that version in Gentoo as our 3.4.6 never exports the __arch64__ define (or any other 64-bit defines) but still compiles 64-bit code. Weird. Anyway, our gcc-4.1.1 is fine for us.

So I’m busy writing patches to change __sparc64__ to __sparc__ as most of the time the code isn’t 64-bit specific anyway. If it is, then it becomes __sparc__ && __arch64__ which works fine :) I dunno if upsteam will accept it, but I’ll email it to them anyway when I’m done :P

• Wrapped e* output. Ensures that we never go over terminal width- if we do it’s another e* call
• Working [ vserver ]support
• FreeBSD init support (not networking scripts yet)
• Critical services can use dependencies (you can no longer set svcdir)
• BASH TEXTDOMAIN support, so people can translate init scripts I only have two things on my todo list before I’ll put an alpha into portage- FreeBSD network support (which may take a while as I don’t have a FreeBSD box yet) and the ability to have >1 init script to provide the same thing.So there we go, something to look forward too :)

So I finally got of my lazy rear end and filed a kernel bug. With Francois Romieu I wrote a patch that has been accepted into the mainline kernel :) OK, he wrote the actual fix, but still. It gives a very big warm fuzzy feeling that you’ve contributed something to the kernel as every Linux user uses it :D

I’ve also filed a bug requesting the inclusion of the patch in gentoo-sources so hopefully Gentoo users will get the fix a bit quicker :)

The morale of this story is to file a bug if something doesn’t work for you as it gets fixed much quicker!

grep-q '^envID:[/proc/self/status

Did you know that bash can do this without the use of grep? Here’s how …

[[ $'\n'$(/proc/self/status) =~ $'\n''envID:[[:space:]([:space:]]*[1-9]')]*[](1-9]')]

(NOTE: everything that matches the regex is now in the array BASH_REMATCH)

One important point is that the file is read as one big string, so the ^ $ anchors don’t work in the same way as they now point to the start and end of the file. The work around is to match on \n instead. As such we always put one at the start of the file incase the match is on the first line.

Now, which is faster? Well, I ran all our existing is_foo_sys checks in /sbin/functions.sh 500 times in a row. grep was around 1 whole second slower than bash! :jawdrop:

That’s impressive, but then again I’ve already shown bash to be faster than some other C programs ;)

What I don’t know is how it performs on large files …. I would guess that grep is faster as it can return once a match is found whereas bash has to read the entire file, but as all files checked should be quite small there shouldn’t be a problem :)

[drm:r300_emit_3d_load_vbpntr] *ERROR* Offset failed range check (k=0 i=2) while processing 3D_LOAD_VBPNTR packet.
[drm:r300_emit_packet3] *ERROR* r300_emit_raw_packet3 failed
[drm:r300_do_cp_cmdbuf] *ERROR* r300_emit_packet3 failed

Which is not good.

Luckily, the culprit turned out to be mesa and this issue is fixed upstream in cvs- the error is apparently exclusive to PCIE r300s cards (which my X600 is) :) I’ll see if I can cut a patch against 6.5, but I doubt I have the skills or the time :(

So now I have working 3D and no 3rd party binary crap (ok, ATI’s fglrx driver isn’t crap, but it sure is bloated and has a few issues- see below) :D. Well, I do, but that’s just firmware for my ipw2200 and prism54 cards.

Although the 3D works it’s not exactly error free- there are artifacts with some KDE screen savers and my favourite game Eternal Lands doesn’t work with a few options- however it’s very playable.

The big win for me is that it works with vesa-tng, although sadly not radeonfb. Also, I no longer need to force KDM to restart X on logout and while 3D is slower and obviously spotty 2D is faster and smooter. Which is important for me as I mainly code on my laptop and play games on my AMD64.

I may even try ut2004-demo and Quake4 on my lappy just to see how they fair :jawdrop:

But wait ….. it had actually loaded my modules and started my network scripts in the background. Excellent- life was good :)

After I became more involved with baselayout I discovered that coldplug was starting things too early so we put some crazy fudges in place like not being able to add net scripts to the BOOT runlevel and other such trickery. Now coldplug has been moved into udev itself it’s starting even earlier, which forced us to radicaly rethink our coldplug handling and we think we’ve got it just right now.

You will now be informed which services will be initiated at boot if the service isn’t in the boot or default runlevels. Here’s the configuaration for it in /etc/conf.d/rc

RC_COLDPLUG="yes|no"
RC_HOTPLUG="yes|no"
RC_PLUG_SERVICES="net.wlan !net.*"

Hopefully Greg KH can soon write a udev that can work off these variables so that udev won’t even coldplug or hotplug by itself which will finally shut up the moaners once and for all! 8)

Can’t believe that it’s been so long …. sometimes it feels like I’ve just met her and fallen in love and other times it’s like we’ve known each other all our lives.

I can’t imagine life without her …… but that’s why she agreed to marry me :D

If you are ever in the Hordle area, near Lymington then I heartly recommend the Mill At Gordleton for dining out. Very very nice food, excellent service and unforunately a high price tag! But worth every penny :)

Well, it has now. Well, it will when I upload it (don’t ask when as it depends on other stuff being done first).

More importantly it also supports the up-coming resolvconf I blogged about earlier. This is all well and good, but libc won’t work like we want - if we want a vpn name and the first server that responds isn’t on the vpn then libc won’t even try the other servers. Infact, all dns servers/caches/whatever work like this so it’s not a new limitation.

But there is a workaround - dnsmasq can forward queries to a specific nameserver for a specific domain. So we inform dnsmasq of this like so using a resolvconf plugin. If an interface resolv.conf file contains the “domain” directive, then forward requests for that domain to the specified nameservers only. Otherwise, treat it as a generic nameserver.

Of course, this requires a little change to the dhcp clients (dhcpcd, udhcpc, pump and dhclient) to only fill out the search part. EDIT: bind can do this too, as such I’ve written a bind plugin that works in the same way. The setup is a little more complex, but do-able.

So along comes openvpn with its ability to use an unspecified tun/tap interface (if tun0 is in use it will use tun1, etc) and that wanting to setup it’s own resolv.conf too (not that current portage one does this, but next one will ;) ). The nice thing about openvpn as it is right now is that it doesn’t rely too much on baselayout, and I would like to keep it like this so that when other packages come along that want to say “here, use this dns information now” we have an easy way for them not to stamp on each others toes.

So I got thinking. Gentoo can’t be the only distro that has this problem, so I looked around some. I found Debians resolvconf which does pretty much what baselayout did with resolv.conf but provided a generic framework for managing it. This is what I’ve been looking for as it also provided a framework for subscribers to be notified of a resolv.conf change, something that my solution did not. This means that no-one should have to do a /etc/init.d/nscd restart in their conf.d/net postup anymore :)

So I’m writing a resolvconf for Gentoo at the moment, taking the best bits of baselayout-1.12 (better merging and interface selection based on metrics) and Debians resolvconf (command line compatibility and subscriber framework). Once I get that into portage, I’ll be patching baselayout, our dhcp clients and things like openvpn to use it.

I’m also thinking that instead of patching dhclient for dbus support for up and coming NetworkManager support it’s probably a better idea just to write a !NetworkManager subscriber for resolvconf which means !NetworkManager could in theory work with any dhcp client :)

The client part however is a little more complex …… So what I’ve been working on is a set of scripts that mimic baselayout-1.12’s inactive status so that the openvpn daemon can instruct the init script that authentication was successfull and mark it as started. This is great as it means you don’t have to know the ip at the other end and ping it like the old init scripts did. So if you missed that feature, don’t panic it’s coming back :)

It works in a very simple way too, simply by using --up--down <script>--up-delay--up-restart flags if we’re a client (detected by specific options in the selected config).

Bah.

So I found a reasonably priced dentist near work and went along. The good news - my filling has been restored. The bad news, I need two crowns. This is gonna cost be just short of a grand :jawdrop:. “Bugger”, said I. I’m not too suprised that I need 2 crowns as there’s been hardly anything left of the 2 teeth in question for a few years now and have been mainly just filling.

Well, I now have my temporary crown on one of my teeth (doing one at a time to spread payment across months) and I get my proper crown in 2 weeks time. Moral of the story - get dental insurance!

So Abbey has been busy repainting our bedroom this nice fawn suede colour whilst I’ve been stripping paint from the bannister. I’ve helped out in the bedroom by doing some skimming :)

Bedroom was finished last night so we grandly moved all our stuff back in and it was …… almost perfect. The yellow curtains just HAVE to go and we may change the carpet. But that’s all money - curtains we can get cheaply we thing as they need replacement anyway as they simply don’t fit. We’ll put up with the carpet until after we’re married though as it’s not that bad …… years of life left in it :P

This time, I think I’ve cracked it. What’s more is that other people have verified my patches, so all reported services sticking, not responding, not starting when they said they did, etc should now have been resolved :)

I also pushed out a udev/sysinit fix to all baselayouts in preparation for fixing bug #118419. We just await a udev version with the required patch now ;)

Some other noticeable fixes …..

• We no longer hotplug non ethernet interfaces.
• Tell the user that a suitable DHCP client is not installed instead of a more cryptic message about modules ;)
• Tell the user to check hardware/kernel module if “interface is not found” ;)

Too many clueless lusers on the forums these days….. :(

Well, those have been solved whilst keeping our nice defaults :) Several other important issues have also been resolved 8) The bad news is that all of this has forced quite a few changes in the code, but everything seems peachy on all my boxes ;)

I’m fairly confident about this new release though- and if it proves trouble free then hopefully a stable candidate? :jawdrop:

And with recent ego stroking from fellow Gentoo dev brix about how easy and flexable the networking config is in Gentoo on IRC yesterday I really do have the Warm Fuzzy Feeling of a job well done :D

Of course, it’s not done yet- baselayout-1.12 isn’t stable (but it’s very very close- you have been warned) and we already have some nice features in our trunk which will appear in baselayout-1.13. So if anyone thinks that we’re not being innovative enough, simply checkout what baselayout did 2 years ago and what it does now. Whole load more! And there’s more to come- we’re not done yet 8)

However, it’s not without it’s cons either. Some pages don’t display well (compared to other browsers) when the brower is in a window smaller than ~800x600, including this page when it has a wide picture. It doesn’t appear to work with the Mozilla css ad blocker which was a godsend. Looks Like I’ll have to use Konquerors internal ad-block which is a simple URL filter by the looks of it. :(

But my main gripe is that some things Just Don’t Work. Well, they do, but you have to tell Konqueror to impersonate another browser. Like First Direct, which blocks any browser they haven’t “tested”. Using the same id as Apples Safari Browser the site works 100% fine.Another guilty one is Draytek. Well, more specifically their routers. They make use of javascript for their buttons and you have to tell Konqueror to impersonate Firefox or it won’t work. Luckily Opera users have to do the same thing, but the bad news for Opera users is that they can only set which browser to impersonate on a global basis where Konqueror can do it on a site basis :)

So with a small tweak here and there, Konqueror is now in daily use. My hope is that Konqueror never gets ported to Windows as I see that as a lot of bloat that Firefox has. Konqueror compiles faster than Firefox and is a fair bit smaller than Firefox, and almost as good. 8)

So I’ve closed all outstanding bugs (some marked NEEDINFO, UPSTREAM or TEST-REQUEST) bar one which should be solved when 3.0.4 goes into portage. I may put 3.0.4b2 in when 3.0.3-r1 goes stable as it’s very good for me so far. Heck, I may even put it into ~ARCH but p.masked. :evil:

After actually taking time going through the code, it’s not as badly written as I last thought- although it was almost a year ago when I last looked. There’s a distinct lack of comments in places and it’s hard to read, which makes it difficult for me to implement a cracking idea I’ve had. Namely passing a string of setup commands like the config file OR an extra config file so that you can stack configs.

This idea stems from these issues I have with dhclient:

• You have to put the hostname into a config file to send the host name
• I don’t like messing with config files in /etc with sed/grep
• Every other dhcp client can send hostname on the command line

Well, hopefully I can find the time to write a patch for that over the comming Christmas/New Year break 8)

But recently, life has not been to good for me in GNOME land. Things changed, mostly for the better, but some for worse as well. The software got slower and more unreliable. Thankfully the 2.10 release solved most of the instability issues I had. So I thought why not take a look at KDE and see where it’s at?

So, after emerging the split and newly into portage 3.5.0 ebuilds with gcc-4 (and it killed my laptop with the kdeendablefinal USE flag as it really does need 512 meg memory- my laptop as 464 useable after graphics card nicks some - even X took too much!) I gave it a whirl. Very nice, very taseful. The default theme for KDE looked swish even if it resembled my Windows XP box a little ;) The first thing that made me go “Wow” was konsole - as it loads very fast compared to gnome-terminal (I use the console a lot). Plus it has a nicer shade of colours, more pastel compared to gnome-terminal boldness.

Konqueror is nice too - again it loads much much much faster than Firefox, which is important for use on this slow laptop. It may not render as well as Firefox and it’s a little slower and klunkier in use but it also compiles a lot lot faster whereas Firefox takes ages. Swings and roundabouts - I’ll give Konq a whirl for a while ;)

I’m not going to blather for ages about KDE vs GNOME- each has their merits, flaws (bug #114141) and target audience and there’s plently of room for each of them in the NIX world. But this GNOME user is going to try KDE for a while :)

With-r1 which fixes a few critical bugs, I think we’re finally stable across the board. The only issue to be resolved (which is just changing default settings) is #112088 which defines default timeouts for ifplugd/netplugd and wpa_supplicant. baselayout-pre_10-r1 changed the defaults to-1 means means “return instantly and mark inactive” instead of 10 which means “wait 10 seconds and mark started/stopped accordingly”.

Personally, I like the-1 default which makes the most sense from a workstation/laptop perspective especially when you use udev/hotplug to start interfaces and don’t have any interfaces aside from lo in your runlevels. Infact this whole issue seems to stem from having interfaces in a runlevel. We could alleviate this by making the default for RC_STRICT_NET_CHECKING="lo" which solves it another way, but could mess some server configs.

Not many people have complained so far, so we’ll see how we go :)

Flanders fields the poppies blow
Between the crosses, row on row
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.

We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.

Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.

• John McCrae

I have no memory of Wing Commander Roy Marples, but I have his name. I will never, ever forget what he did. He fought in the 2nd World War, and rememberance day is about the 1st World War. But war is war, 1st, 2nd … 3rd?

History teaches us one thing …. we do not learn!

And yet we must otherwise every sacrifice made is meaningless ….

For example, I fully expect to insert a card, load the driver and there’s the wireless interface waiting to be configured. But no, I have to tell it what sort of interface I want. Phooey! :(

So the quest for something new! At first I wanted an Intel chipset based card, but 10 mins research shows they are laptop only - sucks! Then I find that RaLink rt2x00 cards have native linux drivers with no firmware required and they get full use of the kernel as the drivers are pure GPL. So I placed the order and got to work on an ebuild for the rt2x00 beta driver which is a new all singing all dancing driver for all rt2x00 chipsets. RaLink open-sourced all their drivers (rt2400, rt2500, and a USB one) and the new rt2x00 driver is a rewrite combining all three + the generic ieee80211 stack.

Today I got my hardware - a pci and a pcmcia card. All working fine from my new ebuild.OK, there are a few caveats - SMP is broken, 4K stacks can’t be enabled and low latency pre-emptive kernel don’t work either (all I can live without). But the beta driver works. 8) It has a few issues, the most important one being the connectivity isn’t quite there (it drops from time to time, but comes back automatically) and wpa_supplicant support is broken.

I’m confident that this driver will shape up to be something good, it’s already better than the prism54 driver and provided that they never go down the “need to use our tools” road like madwifi they’ll do well :) Go and try it out!

C tsort stopping

real 0m0.343s
user 0m0.196s
sys 0m0.144s

C tsort starting

real 0m0.243s
user 0m0.132s
sys 0m.112s

bash tsort stopping

real 0m0.353s
user 0m0.248s
sys 0m0.108s

bash tsort starting

real 0m0.239s
user 0m0.164s
sys 0m0.076s

So there we have it! But what does this mean? Why is C tsort faster at stopping than bash? Why is bash tsort faster at starting than C? More importantly does this so that I’m a really good coder? :evil:

This is easily explained- when we “stop” we load all modules available. When we start we filter out duplicates based on user and then system preference, which means we’re dealing with less modules. My bash code is more efficient, purely because I don’t have to rebuild the entire PROVIDES array at the end of the sort as it gets changed during the sort- something which I cannot do in the C tsort. However, bash loops are notoriously slow, hence the more modules you throw at it the C tsort will overtake.

Attached is a patch (will apply to baselayout-1.12.0_pre10) that enables the C tsort. This is not being comitted to the tree as it adds more code, complexity and the C tsort may not be available all the time (/usr may be net mounted) and as proved, can actually be slower. What’s more, it’s unlikely that everyone will use the number of modules that I use (I have all of them “loaded” bar adsl) by the time we reach the sort, so bash is faster. :P So there. :P

However, there’s not stopping you from seeing if you can improve it and tempt us into comitting it …. ;)

First, we cached a lot of repeatedly called functions (mainly _provides, _before and _after). Then we removed a function which checked to see if a module over-wrote any functions (nice idea, but very very slow and didn’t always work as you could nest functions in bash). This brought us from a stop time of around 15 seconds to the current stop time of around 5 seconds on my laptop (1.6 Celery for reference).

Ever since I’ve been bugged with how slow the sort routine agriffis wrote (which replaced my even slower bubble sort) as it’s the slowest point of the whole process. Infact, sorting the modules when stopping normally took around 2.5 seconds on my laptop, the only part of the process which takes over 0.3 seconds.

So I had a wild idea, lets try and implement a topological sort routine. Luckily I had made one earlier for the main rc system, and written in bash (the C tsort command is in /usr/bin and may not always be available).

This has reduced our times to as follows with lan_config=( "dhcp" )

# time /etc/init.d/net.lan stop
real     0m3.005s
user     0m1.740s
sys      0m1.000s

# time /etc/init.d/net.lan start
real     0m2.662s
user     0m1.612s
sys      0m0.888s

Now, thats fast- almost cut the times in two! :) We get even faster with less modules, and of course there’s still the option to force modules.

Experience it when baselayout-1.12.0_pre10 hits a portage near you!

Once you have a tunnel, you can set it up like so in baselayout

iptunnel_sit1="mode sit remote $REMOTE_IPV4 ttl 64"
config_sit1=( "$TUNNEL_IPV6" )
routes_sit1=( "2000::/3" )

Then setup radvd for your allocation block (don’t forget to advertise the router this time) and you’re good to go :)

The problem as I see it is that it’s too complicated for end users. Even though I’m the maintainer of Gentoo networking scripts, I know nada about IPv6. It’s too complicated. Well it’s actually quite easy, it’s the documentation thats too complicated.

So here’s a quick and easy for IPv6 on the home network :)

1. Enable IPv6 in the kernel
2. Add “ipv6” to your USE flags
3. emerge-uaDvN world to get everything recompiled for ipv6
4. emerge iproute2 and baselayout-1.12.0 as you’ll get much better ipv6 support :)

Now we have a Gentoo IPv6 system, let’s get to the meat! :D

If you look at the output from ip addr you get something like this inet6 fe80::230:48ff:fe54:45a6/64 scope link

That shows an IPv6 address encoded with your NICs MAC address.

In simple terms, the network prefix is fe80::/64- this is akin to a subnet (255.255.255.0 or /24) 230:48ff:fe54:45a6 is based off the mac address.

Now fe80 is non routeable anywhere- it’s just on your PC. So we need a routeable private IPv6 prefix- like 192.168. in IPv4 terms. Prefixes you can use are fecx, fedx, feex and fefx. For this example we’ll use fec0.

Now, we combine these two bits of information to create a routeable IPv6 address fec0::230:48ff:fe54:45a6/64. Lets add this to conf.d/net

config_eth0=( "196.168.0.1/24" "fec0::230:48ff:fe54:45a6/64" )

Restart net.eth0 and you have a site local IPv6 address :)

Lastly, we need to inform other machines about the IPv6 subnet. Todo that, we use radvd, so emerge it. Then slap this into /etc/radvd.conf

interface eth0
{
    AdvSendAdvert on;
    prefix fec0::/64
    {
        AdvOnLink on;
        AdvAutonomous on;
        AdvRouterAddr off;
    };
};

Start radvd (/etc/init.d/radvd start) and hey presto, all IPv6 capable machines will now have their own IPv6 address as radvd configures a stateless network.

Thats it :D

So what’s missing? Well, DHCP information for starters. IPv6 stateless config does not cater for this. You could emerge dhcpv6 and put a dhcp6s server and and get dhcp6c clients in use, but they don’t do any dyanmic dns and I can’t find an easy way to enable that. You could write a script on the server to parse the leases file and update a hosts file with the ipv6 info and then hup dnsmasq which can read host files but that’s klunky. Especially as the hostname has to be coded into the dhcp6c.conf (just like dhclient- it cannot be specified on the commandline for some obscure reason).

But as we’re using IPv6 addreses based on MAC addresses, they won’t change. And they’re unique. So the best solution is to put the addresses into a /etc/hosts file or install a DNS server to translate the name into an IPv6 address- dnsmasq is very good for this.

Lastly, because radvd is just broadcasting subnet information there is no gateway/default route. This is because my ADSL router and ISP don’t handle IPv6 so it’s pointless setting one. You’ll probably be in the same boat for a while, so ……

I’ll look into a 6to4 tunnel so I can use IPv6 across my IPv4 only adsl router and access IPv6 on the internet. I’ll even blog about it when I get it working :)

Thanks to johnm, tigger^ and brix in #gentoo-dev on irc.freenode.net for helpful stuff :)

Resources

• TLDP Linux IPv6 HOWTO
• Gentoo IPv6 Router Guide

My first thought was “WTF?” as wpa_supplicant is just for associating with an AP and then doing encryption / authentication, so I told him so! Lo and behold, the cheeky git had the nerve to tell me I was wrong. :? So I checked out the latest wpa_supplicant and it did indeed have an option to work over a wired interface :O

So I set about the task of fixing up the wpa_supplicant module, which was much tricker than I thought as you could mix ifplugd or netplug into the equation. I’ve always worked under the assumption that you’ll only ever have one daemon controlling the interface (netplug, ifplugd or wpa_supplicant- dhcp daemons don’t), hence everything worked fine with the IN_BACKGROUND env var. Now we had two daemons controlling the same interface :jawdrop:

Luckily, thanks to a recent feature addition, we know which daemons are started by which init script- but more importantly the order they are started in. This means that when netplug detects that a cable has been removed, wpa_supplicant can now stop even though the init script was stopped from the background. We know this, because wpa_supplicant stops when not IN_BACKGROUND or it wasn’t the first daemon started 8)

If you need to try out this wonderous beast, you can download the patch for baselayout-1.12.0_pre9-r1.

EDIT: Patch updated 19:24 BST today- it works on wireless cards again!

The new script is 3 quarters the size of the existing script as I’ve taken advantage of knowing baselayout :)

However, I’ve changed the config layout to match upstreams, so beware! Here’s the default layout

• init.d/openvpn requires /etc/openvpn/openvpn.conf
• init.d/openvpn.foo requires /etc/openvpn/foo.conf
• init.d/openvpn.bar requires /etc/openvpn/bar.conf

openvpn.foo and openvpn.bar are symlinks to openvpn. Recognise this? net.ethx scripts work in a similar fashion. This allows us to start/stop each vpn.

Here’s a list of the openvpn bugs I’ve fixed today:

• #98772 - optional iproute2 support
• #100943 - install all documentation
• #103711 - minimal support (don’t build plugins)
• #105439 - start/stop specific vpns
• #105479 - enable static linking
• #107831 - don’t error on files in /etc/openvpn
• #108945 - fix init script for baselayout-1.12

Apologies to anyone this causes trouble to (maybe luckyduck, the openvpn maintainer - but he’s away!) but the new config, init script and ebuild has solved every current outstanding openvpn bug reported in our bugzilla 8)

Why did I change? Well, apache changing things around annoyed me and it takes a long time to emerge it. lighttpd is smaller, so less time emerging! Plus it’s got a smaller memory footprint, so my server doesn’t complain too much :)

This means that you have to use baselayout-1.11.x as older versions used the-z “release lease but cache IP” no longer works. It didn’t work anyway from the DHCP servers perspective so you loose nothing :)

The main difference from the 1.3 gentoo patchset - aside from loosing the above - is the integration of the Debian and Gentoo patchsets. Yay.

Upstream is dead, so we forked - Simon Kelly (Debian dhcpcd maintainer) and I merged patchsets and created dhcpcd-2.0.0 :)

Which you now have 8)Woooooooooooo. Means absolutely didly aside from one small detail- no more 25k patches because the new upstream listens!

Enough - I need a smoke, more port and some more victims in ET :P

To make things more consistent with other baselayout modules, you have to specify the exact options you want over the defaults like so ifplugd_eth0="--api-mode=wlan"

Still, I recommend netplug over ifplugd as it’s smaller, lighter and more simple. Oh, and if anyone can get wlan mode to work with ifplugd, please let me know so I can get it working for wep on my prism54 ;)

Anyway, here’s a sample config which tries to find a gateway and sets up the interface for it

#configure for dhcp, then arping
config_eth0=( "dhcp" )
fallback_eth0=( "arping" )

# List gateways to arping
arping_eth0="192.168.1.1 10.1.1.1"

# Map found gateways to config's
config_192168001001=( "192.168.1.240/24" )
fallback_192168001001=( "apipa" )
routes_192168001001=( "default via 192.168.1.1" )
config_010000000001=( "10.0.0.254/8" )
fallback_010000000001=( "apipa" )
routes_010000000001=( "default via 10.0.0.1" )

Those that want to try it out, can grab it from here. Drop it into /lib/rcscripts/net.modules.d and delete the apipa module that ’s currently there. Requires baselayout-1.12.0_pre8 (will ship with pre9).

One thing I’ve never liked about ifplugd is that it never seems to work well with the default settings and it currently runs from it’s own init script- which makes it hard to integrate into baselayout automatically. netplug does not suffer from this as I deliberately didn’t write or use the packages init script and it has no configuration so to speak.

Here’s a rough diagram as to how it works

insert kernel module-> hotplug event-> hotplug starts net.eth0-> net.eth0 starts-> netplug launched-> net.eth0 goes inactive netplug gets a link beat-> net.eth0 configures and goes started or inactive depending on result

This means that all the user has to do is create a link from net.lo to net.eth0, which is something I’d like hotplug to do at some point so the user has nothing to configure in a DHCP based network. Well, they have to configure their NIC driver as an external module and trigger it via coldplug, but we’re starting to get to the stage where everything “Just Works” (tm) 8)

Of course, you can disable netplug from running on specific interfaces via the modules variable modules_eth0=( "!netplug" )

If you want to try it out, you’ll need baselayout-1.12.0_pre8 So what are you waiting for? GO TRY IT :P

EDIT: If you have more than one interface regardless of type, you’ll need the below patch otherwise netplugd will shut down

--- netplug     (revision 1468)
+++ netplug     (working copy)
@@-114,9 +114,14 @@
# Returns 0 (true) when successful, non-zero otherwise
netplug_post_stop() {
	${IN_BACKGROUND} && return 0
-       ebegin "Stopping netplug on $1"
+       local iface="$1"
+       local pidfile="/var/run/netplug.${iface}.pid"
+
+       [!-e ${pidfile} ]([)] && return 0
+
+       ebegin "Stopping netplug on ${iface}"
        start-stop-daemon--stop--exec /sbin/netplugd \
-	    --pidfile "/var/run/netplug.$1.pid"
+	    --pidfile "${pidfile}"
            eend $?
     }

eth0 has

nameserver 1.2.3.4 nameserver 1.2.3.5 domain foo

eth1 has

nameserver 4.3.2.1 nameserver 4.3.2.2 search wibble wobble

This turns into a merged resolv.conf of

nameserver 1.2.3.4 nameserver 4.3.2.1 nameserver 1.2.3.5 search foo wibble wobble

However, if eth1 has a lower metric than eth0 (eth0 wireless, eth1 wired for example) then it becomes

nameserver 4.3.2.1 nameserver 1.2.3.4 nameserver 4.3.2.2 search wibble wobble foo

Duplicates are also removed :) Incase you’re wondering why not all the information was taken across, well the answer is due to libc limitations. If you consult the resolv.conf man page then you’ll discover that you’re limited to 3 nameservers and 6 search domains. Great!

It’s a trivial excercise to change the numbers we limit too, but they’ll have no effect with current libc. But if anyone uses a libc/resolver with greater limits I’m happy to accept patches :P

As it took a lot less time todo than I first thought, I’ll merge ntp.conf information too- hopefully in time for pre6 8) EDIT: ntp.conf merging took 10 minutes and it is in :D

If anyone used the resolvconf package which Debian uses, then this does a similar thing but transparently and without the need for a daemon.

Fans of the baselayout-1.12.0_pre releases so far will be pleased to know that we now default RC_AUTO_INTERFACE to no ;)

So I hacked in a toposort function which is faster then the “sort until nothing” approach we used previously. We can’t use tsort (C toposort program) as that’s in /usr/bin and isn’t available at boot time, so it’s in bash! The toposort function is also non-infinite by definition, so circular deps should be a thing of the past- well, for that function anyway! ;)

Here’s a good reference to toposort.

I’ve also been chastised for my non-use of tabs (blame my vim settings :P ), so I’ve now been a good boy and added # vim:ts=4 to all my stuff in baselayout and re-indented with proper tabs.

Lastly, I’ve removed all the

e=$( iwconfig ${some_command} 2>&1) [-n ${e} ]([)] && eerror "dufus"

and replaced it with

if ! iwconfig ${some_command} ; then eerror "dufus"; fi

which is much nicer and should now work as the affected wireless-tools isn’t in portage anymore. Or was it a driver problem? Either way, it’s cleaner, better and here to stay. It works on my madwifi and prism54 so if it doesn’t work for you then isn’t that just teh sux0r! Bitch about it to the driver maintainers for your hardware }:)

1. pump
2. udhcpc
3. dhcpcd (although it can easily be the slowest, it’s normally better than dhclient)
4. dhclient

So if you’re a ricer, or just want a faster DHCP connection then use pump! It does everything 99% of DHCP users need to - the only thing it doesn’t do that the others do is dump it’s DHCP information into env vars so a script can parse them (maybe useful for samba WINS settings), but as I said not many users would need that functionality.

I’m hopefull that I can write another pump patch so I can append a DNS search path option to specify the “search foo bar” option in /etc/resolv.conf. This means that the only thing the pump helper script needs to do is mark the net.eth0 service as inactive/started - which guess what, makes us faster! Debians dhcpcd patchset inspires me todo this - although hopefully more on dhcpcd later ….

I’m not sure why this is, as everything is as stable as rock (except for my horde + php5 combo which causes iconv utf8 assert errors in my apache logs). It’s not like I’m using any ricer flags either- I was however using reiser4.

So I moved the entire server over to reiserfs - still kept segfaulting. Hmmmmmz went I. xfs and jfs were out of the question (too slow for my other needs) so I tried ext3 for the first time. I knew that ext3 had a rep for being slow, but also knew that dir_index helps greatly. So after moving to ext3 last night the server actually seems slightly more responsive, but some i/o does take longer and the same stuff takes up around 20% more space (reiser is really good at packing).

Touching wood, the 3 emerges I’ve done haven’t segfaulted on me :)

But I suspect it’s something in the hardware/environment as my main PC running reiserfs is perfect …… :?

….. because we have to …. and you must accept this :puzzled:

Thanks for making me laugh Kumba :D

//me thinks that the people who matter about software patents will not be reading my blog or Planet Gentoo

So, I wanged a new net-scripts module into baselayout svn today which should remedy the situation. net-tools (ifconfig) ships with nameif and I finally discovered that ip (iproute2) can do the same with the “ip link set name” command.

You’ll be able to do the following in the next alpha release of baselayout 8)

rename_001122334455="foo1" # Rename based on MAC address
rename_eth1="foo2" # Rename based on current name

Please note that it is preferable to ensure the device has it’s correct name before any net.xxx scripts are called- you can do this by writing udev rules.

The reason for this is that we have to create a net.$newname link to net.lo in /etc/init.d if it doesn’t exist, mark ourselves as stopped and then start net.$newname - which of course takes a lot longer (in CPU terms) than using a udev rule or similar.

Of course, if you have wireless-tools installed then you get a 3rd way of renameing things - ifrename which can be used by hotplug automatically.

Hopefully I’ve now satisified all our grumpy users :p

Due to changes in baselayouts networking (basically moving /etc/{resolv,ntp,yp}.conf to /var/lib/net-scripts/state/${iface} and symlinking) I’ve had to pour over pumps code and make up some patches to enable this and route metrics. The code base is actually very clean - much like udhcps (in terms of readability).

So what does this mean for you? Well, probably not much - but it does mean that pump gets to stay in portage a while longer :P

All that remains is to add a --script option to pumps command line so that I can specify a script to call on dhcp state change (I could fudge /etc/pump.conf but that sucks) and add an --keep-up option so that pump does not shut the interface down on dhcp release. That is needed for ifplugd/wpa_supplicant or similar as they need the interface up to work.

So after that happens, I think I’ll prod the guys for a baselayout-1.12.0-alpha3 release and see just what havoc the last few months of coding have caused :evil:

But Abbey has moaned royally! So here we go!

Here’s to two fantasic years with Abbey!

She’s consistenly made me very happy in many wonderful and different ways ;) We’ve accomplished a lot- moved down south with my work, bought a house and we’re now spending loads of hard earned pennies improving it :jawdrop:

Hopefully we can complete the back garden before this time next year so we can get a puppy which we both really want :)

config_eth0=( "192.168.0.{1..20}/24 brd +" ) gives you 20 IP addresses.

config_eth0=( "192.168.{1..20}.{1..20}/24 brd +" ) gives you 400 IP addresses. (20x20)

config_eth0=( "192.168.{1..20}.{1..20}/24 brd 192.168.{1..20}.{1..20}" )
Currently doesn’t work as expected- it tries to add 160000 IP addresses with most of them duplicates thanks to a sequence in the broadcast field. Not sure how to solve this at present- but I don’t see it as that much of a problem at this time.

The problem is that to get the first two examples working correctly, I need to replace spaces with _ (any char will do as long as it’s not space) so that expansion works just by clever use of quoting and arrays- no loops.

Here’s the current code- which accepts a string and returns an array

expand_parameters() {
    local x="$( eval echo ${@// /_} )"
    local-a a=( ${x} )
    a=( "${a[)
    a=( "${a[@](@]/#/\"}")/%/\"}" )
    echo "${a[*]//_/ }"
}

Any ideas about how to fix it are welcome :)

Well, Emma Pitt finally became Emma Hatton- you can find pictures of Emma getting married to Richard in the image gallery- which has been slightly re-arranged. I’ve known Emma since she was about 6 months old and I wish her a very happy marriage to Richard :)

And yes, Kat had to wear a dress for the first time in years as she was a bridesmaid! Bless :lol:

If anyone knows how to use the anti red-eye function on a Canon IXUS 330 then please let me know as the manual instructions don’t appear to work! Also if anyone knows how to do a manual focus as well- by all means let me know.