Not having a spare 12V 1A adapter with the right barrel made testing this pretty darn hard and desperate times convinced me to buy the USG (which is still working like a champ, no regrets here!). So I waited until everyone was in bed (not hard due to my medication as I don’t sleep much) and used the USG power supply. Still didn’t boot, but performing a hardware reset did!

Unsure what the hardware reset does, maybe update some internal flash a little, unsure if it touches the USB stick or relies on EdgeOS being installed rather than say NetBSD, but it’s done the trick! I bought a cheapo 12V 1A AC adapter with the right barrel size and plugged it in today and it still boots.

This means I can keep the USG powering my internet at blazingly fast speeds and the ERLite for NetBSD development as it doesn’t support any of the network hardware offload the ERLite has, so throughput suffers as a result.

If anyone from Ubiquiti reads this, please do something about your power adapters as it seems this has happened to quite a few customers after reading past community posts on your forums!

Thanks to the wonderful folks over at the NetBSD port-mips mailing list for their valuable assistance.

Since I was the main protagonist in bringing terminfo to NetBSD, I had a reasonable grasp on how our curses worked and I had looked at this bug before, but left scratching my head over it.

Since I looked at it, Valery Ushakov added a simple test case to demonstrate the issue.

#include <curses.h>
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
	const char *errmsg = NULL;
	int err;

#define CHECKERR(_msg)					\
	do {						\
		if (err != OK) {			\
			errmsg = _msg " failed";	\
			goto out;			\
		}                                       \
	} while (0)


	WINDOW *scrw = initscr();
	if (scrw == NULL)
		errx(EXIT_FAILURE, "initscr failed\n");

	if (!has_colors()) {
		errmsg = "no colors";
		goto out;
	}

	err = start_color();
	CHECKERR("start_color");

	err = init_pair(2, COLOR_WHITE, COLOR_RED);
	CHECKERR("init_pair");

	int maxy, maxx;
	getmaxyx(stdscr, maxy, maxx);

	WINDOW *lborder = newwin(maxy- 1, 1, 0, 0);
	if (lborder == NULL) {
		errmsg = "newwin failed";
		goto out;
	}

	wbkgdset(lborder, COLOR_PAIR(2));

	werase(lborder);

	mvwaddstr(lborder, 0, 0, "A");
	mvwaddstr(lborder, maxy- 2, 0, "Z");

	wnoutrefresh(stdscr);
	wnoutrefresh(lborder);
	doupdate();

	sleep(5);
out:
	endwin();

	if (err != OK) {
		if (errmsg == NULL)
			errmsg = "unknown error";
		fprintf(stderr, "%s\n", errmsg);
		return EXIT_FAILURE;
	}

	return EXIT_SUCCESS;
}

He then noted that using wbkgd instead of wbkgdset makes it work.

Now, the difference between the two is that wbkgd forces new attributes on the whole window background, whereas wbkgdset sets new attributes for new actions on the window. Keen observers will notice that the following function called is werase, which does what it says on the tin and erases the window but also resetting the attributes of the contents. So, I looked at this function and noticed the attribute checking code was faulty and a simple fix was enough to solve it!

Valery then pointed out that wclrtoeol probably needed fixing to and as it turned out, wclrtobot likewise. As it transpired, all 3 had different logic in working out when to erase something! We went through a few iterations of patches to harmonise the code and have settled on a macro that all 3 now share, so hopefully no more erasing issues in our curses.

As a follow on task, I should go through the NetBSD bug backlog and see if this fixes any other reported issues.

Infact, such a terminfo description was imported where colour pairs for screen-256color went up to 65536 which exposed a bug in the existing implementation where it was set to zero. Because the number might mean something more than a range, we need to be able to store it accurately.

This requires a version bump because whilst the API hasn’t changed thanks to C int promotion, the ABI has. Also the underlying database structure has changed as well- we now store the numeric parameter inside a uint32_t field rather than a uint16_t one. Whilst this change can still read the old style database, the old one cannot read the new one and thus we now maintain the database as terminfo2.cdb, leaving the old library and database alone so old programs still work fine.

libcurses, libfrom, libmenu and libpanel have also been bumped to accomoate this change.

Many improvements have been made with regards to compability with NetBSD and the others, but there are the following caveats:

• NetBSD will only resize stdscr, curscr and virtscr windows (like PDCurses)
• NetBSD clears the window when resized
• NetBSD will re-size and re-position the ripped off lines (like ncurses)
• NetBSD will not change LINES when ripoffline is called and may offset stdscr to it’s relative position on the screen. This means that portable application should check getbegyx and getmaxyx for the size and location of stdscr.

The expectation is that the client will re-draw all windows, including ripped off windows on a KEY_RESIZE event. It’s also expected that the client will re-size any other windows as it has a clearer idea of where things should be on the screen to it’s proportion rather than curses guesstimating.

Now, the very colourful Roguelike game Ascent into the Depths of Beyond now works on NetBSD with a minor patch

• gcc toolchain is broken for some packages which affected running any PHP build
• clang toolchain was broken for my config (USE_SSP= yes and CPUFLAGS+=-march=core2)
• clang compiles as a whole were broken due to a recent efiboot import

In hind-sight, I could have had the box up and running a lot sooner if I used NetBSD-7 guests (or maybe just a NetBSD-7 build box), but no, I just had to get-current running. It offers more than-7 and prior exerience told me that tracking-7 was very problematic … but that could have been due to my settings and wanting to compile everything with clang. I’ve currently got 3-current images there now … the Xen DOM0, the actual server itself as Xen DomU (without any compiler or tools) and another Xen DomU which just builds stuff for other guests to use.

Anyway, the box itself is now up and running and all relevant services have been moved to it. During this move, I decided to modernise things a little and setup HTTP/2. If you’re reading this then it’s working :) As such, I’ve re-directed all basic HTTP traffic to HTTPS and it does seem to load a lot faster. One side effect of this is that I’ve stoped using my own self signed certificates and I’m now using the nice Let’s Encrypt service. pkgsrc users should use the py-certbot package and don’t bother searching for any letsencrypt package as it’s been renamed. I found this quite confusing as there was no reference to the rename in pkgsrc I could find and wasted a little time on it.

But now it’s live I can finally look into bringing online some other development utilities to play around with as the server has a lot more room to grow :)

Currently we lack Soft Label Key functions as specified by POSIX. What are these? Glad you asked! Soft Label Keys ripoff a line from the bottom of the screen and allow for up to 8 keys to be displayed there. You can define labels for them, colours, etc.

“BIG WHOOP!" I hear you say. “My app already does this, why should I care?"

Because SLK also works with the terminfo database to work out if the terminal natively supports labels (via plab_norm, label_on, etc)! This means a line doesn’t have to be ripped off from the screen so no real-estate is lost. Sadly, no terminal actually seems to support these markings. But if one is ever made, it should work fine.

Now, not many curses apps actually use the SLK functions, probably for this very reason. But some do, and it is a standard so I’ve coded NetBSD to support this natively.

Sadly, this has uncovered a bug in the NetBSD curses library where cursor movement in a one line window at the bottom of the screen appears to push the window up. This is very visible with SLK and will have to be fixed for any apps that actually use it, but the SLK implementation itself is sound as resizing the terminal forces a redraw and it looks fine.

There is a apparently a thing called PD Curses but I don’t know anywhere that actually uses it by default. If you know, please email me @ roy@marples.name or tweet me. I do know that the pkgsrc curses framework supports it.

And this is the thing I’m currently doing. As NetBSD curses supports more POSIX functions (and a few ncurses extensions, sadly) more and more packages now work alongside NetBSD curses, so I’m plowing through pkgsrc again moving things to the curses framework. This is important, because there is logic in devel/ncurses to use NetBSD curses when passing certain criteria, but sadly this always brings ncurses as a possibility to be linked or it’s header used, both of which are quite bad for NetBSD curses so there is a method to my madness.

One interesting thing though is that since I started comitting my curses patches, it turns out that Sabotage Linux is using NetBSD curses and made it portable. I’m chatting with one of their devs and we’re starting to merge of a few of their patches back into NetBSD.

When I discovered FreeBSD and then NetBSD making dhcpcd work with the same features provided troublesome. For the IPv4 case, we needed to patch the kernel so that IFA_ROUTE remained sane. For the IPv6 case it was a lot more complicated as parts of the IPv6 stack rely on the kernel processing Router Advertisements instead of a 3rd party tool. Working this out was a lot more time consuming and complicated, but we got there with minimal fuss!

So as of now, dhcpcd can fully replace the NetBSD and Linux in kernel Router Advertisement handling code.

But why was I right after all these years? Because quite a few knowledgeable NetBSD folk assured me that the kernel was fine and it was likely dhcpcd at fault. As it turned out, it really was the kernels fault.

Evolution suffers two problems on my NetBSD box. One, it takes over 5 minutes to load and two, once it has checked and downloaded mail from my server it closes without error. The 2nd problem is probably a programming error in Evolution or one of its many dependencies, but the first problem could be anywhere! Research led me to this possible soltuion. If you look closely, you’ll find an old post by me in that thread fixing it with ld–as-needed on Gentoo/FreeBSD, but that isn’t a real solution for NetBSD / pkgsrc users. Anyway, at first glance the patch on that thread makes perfect sense. So I came up with better patches (one for glib, one for Evolution) and it now loaded in under 3 seconds! All was well :)

Well, not entirely. Other applications started to show isses as a result. So at least we know we’re in the right area. I then spent a few hours wading through the Evolution build chain and came to the conclusion I still hate autotools as it obscures everything but was no better off fixing it. But I did come up with a test case that opened an evolution library and searched for a symbol that did not exist. This should 30 seconds, which is far too long and is also probably what the poster of the FreeBSD patch did in the mail thread.

Cutting to the chase, the runtime linker on NetBSD was checking the same library for the same symbol over and over again. Going back to the FreeBSD linker, they have a patch which implements a negative cache which should solve this. But it doesn’t, as it doesn’t cache missed weak symbols. Luckily this is easy to rectify and you can find my patch here and an updated one after performance testing elsewhere here. Now Evolution loads in under 3 seconds and crashes right away- but that’s an issue to fix another day :)

No doubt there will be some fallout and flames, but it’s all about bringing some needed modernization into NetBSD!

So NetBSD finally has an up to date replacement for the legacy termcap it currently uses. Well, it will have when merged - I’ve only just asked for testing so it might take some time before it hits-current.

This is important for NetBSD, as many packages support resolvconf, but only when /sbin/resolvconf exists. This meant that a lot of packages that supported resolvconf, failed to work with any resolvconf implementation from pkgsrc.

PPP users who maintain their own scripts are encouraged to try it out :)

A few programs in base system needed to be changed due to having their own getline function, most of which aren’t anything like getline(3). Hopefully there won’t be much fallout in pkgsrc as a result.

getline(3) is prefered over over functions such as fgetln(3) and fgets(3) because it’s standards based and you get a dynamic buffer for really really long lines. However, POSIX did drop the ball on making it a standard from the GNU extension- it should return 0 on EOF and more importantly be called fgetline. Oh well.

I shall be rolling getline(3) support into dhcpcd later, but I’ll have to do a link test in the Makefile to see if we can use it. I’m unsure if I want to have a mini configure for dhcpcd or to keep using just make extensions ….