NetBSD curses erase functions fixed

NetBSD PR lib/23910 was filed over 16 years ago. It describes how NetBSD curses fails to work with Vifm, a Vim interface for a curses GUI file manager. It’s quite a nice idea after playing around with it some as Vim is my favourite text editor.

Since I was the main protagonist in bringing terminfo to NetBSD, I had a reasonable grasp on how our curses worked and I had looked at this bug before, but left scratching my head over it.

Since I looked at it, Valery Ushakov added a simple test case to demonstrate the issue.

#include <curses.h>
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main()
{
	const char *errmsg = NULL;
	int err;

#define CHECKERR(_msg)					
	do {						
		if (err != OK) {			
			errmsg = _msg " failed";	
			goto out;			
		}                                       
	} while (0)


	WINDOW *scrw = initscr();
	if (scrw == NULL)
		errx(EXIT_FAILURE, "initscr failed\n");

	if (!has_colors()) {
		errmsg = "no colors";
		goto out;
	}

	err = start_color();
	CHECKERR("start_color");

	err = init_pair(2, COLOR_WHITE, COLOR_RED);
	CHECKERR("init_pair");

	int maxy, maxx;
	getmaxyx(stdscr, maxy, maxx);

	WINDOW *lborder = newwin(maxy- 1, 1, 0, 0);
	if (lborder == NULL) {
		errmsg = "newwin failed";
		goto out;
	}

	wbkgdset(lborder, COLOR_PAIR(2));

	werase(lborder);

	mvwaddstr(lborder, 0, 0, "A");
	mvwaddstr(lborder, maxy- 2, 0, "Z");

	wnoutrefresh(stdscr);
	wnoutrefresh(lborder);
	doupdate();

	sleep(5);
out:
	endwin();

	if (err != OK) {
		if (errmsg == NULL)
			errmsg = "unknown error";
		fprintf(stderr, "%s\n", errmsg);
		return EXIT_FAILURE;
	}

	return EXIT_SUCCESS;
}

He then noted that using wbkgd instead of wbkgdset makes it work.

Now, the difference between the two is that wbkgd forces new attributes on the whole window background, whereas wbkgdset sets new attributes for new actions on the window. Keen observers will notice that the following function called is werase, which does what it says on the tin and erases the window but also resetting the attributes of the contents. So, I looked at this function and noticed the attribute checking code was faulty and a simple fix was enough to solve it!

Valery then pointed out that wclrtoeol probably needed fixing to and as it turned out, wclrtobot likewise. As it transpired, all 3 had different logic in working out when to erase something! We went through a few iterations of patches to harmonise the code and have settled on a macro that all 3 now share, so hopefully no more erasing issues in our curses.

As a follow on task, I should go through the NetBSD bug backlog and see if this fixes any other reported issues.