Roy's Blog

A Hacker's musings on Code | Tech | Life

Here's the changelog:

  • dhcp: fixed classless static routes
  • prefix delegation: build routes after assigning addresses
  • dhcp: on lease expiration, discover only when carrier
  • ip6: fix potential segfault when lifetime overflows
  • bsd: compile on recent OpenBSD platforms
  • dhcp: fix reporting of DNS encoded SIP servers
  • dhcp6: fix unicast in non master mode

All the critical issues people have highighted have now been resolved, so hopefully this is the last rc before release :)

Continue reading...

So dhcpcd has supported a shared IP address for a long time. It did this by removing the address from the non preferrred interface and then adding it to the preferred interface.
Easy!

But this came with some issues:

  • There is a window where the IP address doesn't exist, and the kernel may wipe out the subnet route at that point also.
  • DHCP renews didn't come through to the right interface.
  • Some kernels didn't like the address moving interfaces.

Still, to the best of my knowledge, no other product has this feature and for the most part, it did work well allowing almost seamless switching of wired -> wireless and back again with both using the same IP address. But that wasn't good enough - I was challenged to do better!

So I took up the bat and cooked up this changeset to change the behaviour to this:

  • Each applicable interface will have the shared ip address.
  • Whenever the address is added, the most preferred address will be ARP announced.

And lo - IT WORKS!!! The changeover when plugging/removing the wired interface is 100% seamless for me. ssh, ping, etc get zero interuption. Of course, YMMV ;)
But there are some costs:

  • Thanks to ARP, only the primary interface will receive DHCP unicast messages for other interfaces.
    As such we need to re-direct them to the correct interface by examining xid and chaddr.
    This means we have to relax the BPF filters to allow more through.
  • Kernels supporting RFC5227 will double ARP announce the address.
  • NetBSD-8 kernels needed some love to get it to work and there's still an issue with it not working when an address is deleted from the interface.

Only the last bullet is really important, which is mainly why the changeset hasn't hit the master branch yet. But that should be fixed soon. The other points can be fixed as and when.

Continue reading...

It's been a while in the making, but dhcpcd-7.0.0-beta1 is finally here! I have been using this a lot on all supported platforms bar Solaris and it's been very trouble free, so hopefully not many changes (if any? famous last words!) before a RC and final release.

Summary of changes since dhcpcd-6.11.5:

  • source file locations reworked: dhcpcd source is in src dhcpcd hooks are in hooks compat is in compat
  • README split into README.md and BUILDING.md
  • internal routing is now protocol agnostic
  • avoid using __packed and use compile time asserts instead
  • addresses some alignment issues
  • disable some ARP code on kernels which support RFC5227
  • BSD IPv6 kernel settings are now updated to reflect dhcpcd config
  • custom logger has been removed, syslog handles everything as such, the --logfile option has been removed as well. If you need better/earlier logging, get a better syslogger!
  • distinfo and signed distinfo files are now available alongside release taraballs from this point onwards
  • default DBDIR has changed from /var/db to /var/db/dhcpcd
  • /etc/dhcpcd.duid moves to DBDIR/duid
  • /etc/dhcpcd.secret moves to DBDIR/secret
  • lease file names have dhcpcd removed from them as they are now inside a directory of the same name
  • fixed issues with reject routes not working on some platforms
  • improved nl80211 support on Linux for working out the SSID
  • no longer request NTP by default in dhcpcd.conf
  • fix detecting IPv6 DAD on OpenBSD
  • remove custom Solaris DLPI filtering in favour of BPF (note there seems to be a kernel issue where the DHCP fd receives ARP's as well, the only side effect is a noisy syslog)
  • BPF filtering vastly improved so dhcpcd only wake up on ARP or DHCP packets destined for it
  • support for MUD URL (draft-ietf-opsawg-mud-05)
  • if the kernel isn't doing DAD, don't insist on waiting for it to actually do it
  • fix a potential crash where the DHCP or ARP states could be freed before the packet processing loop naturally breaks
  • removed gateway and nogateway options (these can be controlled by the nooption directive which works for more than just gateways)
  • removed ipv6ra_own and ipv6ra_own_default options (these can be controled by the ipv6rs/noipv6rs directive)
  • fix a memory leak on systems where posix_spawnattr_init allocates memory by calling posix_spawnattr_destroy afterwards
  • fix a crash receiving SIGUSR1

I've not done everything I've wanted to, but I feel that many issues have now been addressed and on the whole dhcpcd is in a very good state right now.

Let me know of any issues you find!

Continue reading...

I've been using Fossil for quite a while now as my SCM. I like Fossil. But Fossil is not Git, and most people seem to like Git. It could be better to say that most people like GitHub because it's the first hosted SCM that's free for open source with good social interaction I'm aware of. And GitHub is huge. Some might say that if you're not on GitHub, you don't exist as a project. Well this obviously isn't true, but you get the idea. You can get free Fossil hosting at Chisel, which is nice, but it's also not GitHub. Plus I like to be 100% self hosted.

So, to get myself on GitHub (as a mirror only), there needs to be a bridge between Fossil and Git. Fossil documentation implies this is quite easy. Sadly, this isn't the case as the <=Fossil-1.37 releases (note there is no guaranntee that future versions of follow will not have these flaws - my branch may not be comitted to trunk) have the following flaws:

  • Branch and Tag name mangling (dhcpcd-6 becomes dhcpcd_6)
  • Silent master branch renaming into trunk on inport, but not on export
  • No tag comments (Fossil lacks the feature) which means syncing tags back and forth results in tag conflict due to signature change

I submitted some initial patches the the Fossil mailing list and I now have a Fossil commit bit! You can find my branch here to fix the Fossil Git bridge.

But that's not the end of the story. A bridge has two ends. With my initial setup, the Git end was bare bones repository which I pushed to GitHub. This is no longer the case - I now need a staging repository to pull both ends. And this requires a script because Git needs a little more hand-holding to completely track a remote. The below script is tailored for my needs, yours may differ. It also reflects the above initial design and the subsequent change - as such it it may need editing if you need to create a git clone from fossil. This comes with no support, just as an idea of how you might implement such a bridge.

#!/bin/sh

fossildir=/var/fossil
# Cannot be a bare directory for git as we cannot write to the host directly.
# So we have a staging directory instead.
# This requires a bit of hand-holding to track all the branches.
gitdir=/var/git-staging

marksdir=/var/scm-marks

# Respect default naming at either end
fossil_export_opts="--rename-trunk master"
fossil_import_opts="--rename-master trunk"

# Only used when creating a git bare bones repo from Fossil.
export_fossil_to_git_new()
{

        rm -f "$fossilmarks" "$gitmarks"
        git init
        fossil export --git \
                --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --export-marks="$gitmarks"
}

export_fossil_to_git()
{

        fossil export --git \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --import-marks="$gitmarks" --export-marks="$gitmarks"
}

export_git_to_fossil()
{

        git fast-export --all \
                --import-marks="$gitmarks" --export-marks="$gitmarks" | \
                fossil import --git --incremental \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_import_opts "$fossildir/$fossilrepo"
}

pull_git()
{
        local remote

        git fetch --all
        # Track all remote branches
        git branch -r | grep -v '\->' | while read remote; do
                if [ -z "$(git branch --list "${remote#origin/}")" ]; then
                        git branch --track "${remote#origin/}" "$remote"
                fi
        done
        git branch --list | sed -e 's/^\* //' | while read branch; do
                git checkout "$branch"
                git merge --ff-only
        done
}

push_git()
{

        git push --all
        git push --tags
        # Reset the current branch checkout.
        # If we don't, the next run will complain about unstashed changes.
        # This maybe a bug in git, but maybe not because the live checkout
        # *is* behind at this point as we just fast-imported.
        git reset --hard
}

echo "Syncing git and fossil."
for repo in "$fossildir"/*.fossil; do
        fossilrepo=${repo#${fossildir}/*}
        repo=${fossilrepo%.fossil}
        gitrepo="$repo"
        fossilmarks="$marksdir/$repo.fossil.marks"
        gitmarks="$marksdir/$repo.git.marks"

        # We just sync old fossil repos to new phab clones
        if [ -d "$gitdir/$gitrepo" ]; then
                cd "$gitdir/$gitrepo"
                pull_git # staging only
                export_git_to_fossil
                export_fossil_to_git
                push_git # staging only
# Enable the below if pusing to a bare git repo from fossil
#       else
#               export_fossil_to_git_new
        fi
done

Direct download to script

Continue reading...

So, I've not blogged for a long long long time. The last time was over 2 years ago. There are various reasons for this, involving Tech, Code and Life.

Tech

My old blog was powered by Serendipity. Before that, it was Trac and before that it was Drupal

Now, Drupal was very ...... well, blocky. It worked, and worked well to start. But my poor server at the time could not cope with the resource demands it had, alongside my existing Trac instances to host my project's wiki and ticketing systems. So I migrated it to the Trac FullBlog Plugin. This was amazing! Where Drupal was complex, Trac was simplicity. Life was good.

But slow. As it turns out Trac itself was getting slower and slower. So I changed from Apache Pre-Fork to Lighttpd. This didn't work too great for me. Too complex, too crash. So I changed to nginx. and used uWSGI to bridge my fcgi's. Wow, my multi-purpose server had free memory! Amazing! I used this setup for about 5 years, maybe longer. However, Trac still got slower and slower.

So I migrated it all to Serendipity (the blog that is). I was also getting fed up with using git as my source code repo alongside Trac so I changed that to Fossil. No more Trac? And nginx and uwsgi working great! Life was good, my server now had over 1G free memory ... which did slowly get used up as a cache I guess. But swap was never ever hit. Even when compiling the server software. Tech did not get this good.

But then it stopped. Serendipity refused to let me log in to the admin panel. But Life got in the way at this point. So I didn't blog.

Fast forward to now and my DSPAM filter stopped working. Full stop. No-one knew why, myself and a few NetBSD devs spent a few hours looking at the problem. It only worked outside of my procmail filter, but my setup required it to work inside as I have email lists on the same box that required unfiltered mail to hit their informational funcions. And I hate SPAM. So I installed SpamAssassin as a replacement. It turned out the pkgsrc package needed a small bit of love, but now it's working as well as DSPAM ever did, but more importantly working. As a result of this, I had to prune spam from my hosted mailing list archives and it turns out hypermail doesn't allow you to do this, short of manually rebuilding the archive. So I ditched that and rebuilt my mail archives around mhonarc. I also took the time to theme it like my Fossil projects for consistency, which is fine.

Now, a lot of mail archives have a nice search function too so I decided to brush off my PHP skills which I've not used since the 1990's. It turned out my skills weren't that great as I thought and my first attempt failed. But no error was posted OR logged. It turned out that I couldn't get uWSGI or NGINX to actually log my PHP errors! I'm sure there was a way, but frankly the setup was hard to manage so instead of bashing my head, I looked to find an easier solution.

Apache had gained a new event MPM which promised the same low memory use as NGINX. It also had really improved mod_fcgid and mod_proxy so it's very useable easily. PHP had integrated PHP-FPM. As I no longer needed Trac, NGINX or uWSGI i said bye bye to them all and installed Apache. So full circle on the web server, but I now had easy to find PHP error reporting, so my archives search page worked fine after a quick fix!

In summary everything on my server was back to normal, fully working nicely, easily and well reported and diagnosed. Except my blog. I spent more time looking at it, didn't like it, nor the hard upgrade I was looking at the to latest version. So, after changing all my software, why not my Blog?

After migrating a few times, I know the score. It's Tricky (TM). So how to make life better? After a lot of research, I have settled on using GRAV for a few reasons listed here:

  • Does not force MySQL database (I prefer PostgreSQL)
  • No SQL database requirement, uses folders and Markdown files with embedded meta data
  • The default theme already looks good
  • Can self upgrade
  • Because everthing is in markdown files it should be easier to manage and migrate away from later
  • I want to self host - this is my data, not someone else's
  • It's in PHP (so no need to install Node.JS, Ruby or whatever flavour of the month is)
  • MIT license (close enough to my favoured 2 clause BSD)

So, this is the first post in a new blog, a new start where my Tech is currently ..... working without issue! This has got to be a first for me :)

Code

I write code. I like to think I'm quite good at it :) And like most people, I have an opinion on what makes good code and bad code .... I hate bad code, even my own! So how did code get in the way?

I mentioned earlier that for some reason I my Serendipity admin login stopped working. Well, I looked at the code .... and it was not nice. (My install was an old version compared to their current 2.0 so this may have changed) There were little comments, there was poor seperation between Model, Controller and View. Now, the actual code in question looked like it should be working .... it hashed my paswsword, compared the hash to the saved DB entry. I even went as far as adding debug code. It all looked perfect, yet it did not work. No-one knew where the error was.

I was also pretty annoyed with Serendipity for other reasons I discovered only after using it, so I decided to ditch it entirely and move on. As you can tell above, I have no problem changing out software I no longer believe in, or fares poorly compared to others. I also enjoy when the software I originally chose makes great strides to tempt me back in the fold, like Apache. I don't claim to understand web server code, but I did peruse their codebase a while ago and it looks cleaner compared to when I did in the early 90's trying to make it work on Windows for $DAYJOB.

I've also spent a lot of time on my pet coding projects, dhcpcd, dhcpcd-ui, resovlconf and NetBSD.

Anyway, enough ramblings about code. There will be a lot more about that in the future anyway now that I have a working blog again. Well, almost working. I need to write some code to import from Serendipity to here, which will be nice as I enjoy coding and want to preserve the content I've lovingly created - alongwith you in the comments - over the years and I would hate to lose that! Speaking of comments, there is currently no comment system on this blog. for the time being until I integrate something like ISSO commenting or maybe find a Grav plugin. I also need to find a way of providing top links to my project pages.

Life

So in the past few years, my Family has been very stable and loving :) But a new child got in the way! Ethan Thomas Marples was born 2nd December 2015, so he's one year old tomorrow! That has taken a lot of my time, and all my children still take a lot of my time, so it's a very enjoyable getting in the way :D

There's also been some changes at work. My old company Logos Technologies was bought by Instem. I'm still doing the same job, working on the same product, but the inner dealings of the buy out, company politics and day to day integration have certainly been eye opening.

Playing games got in the way. Far too much time playing MMO's .... I'm now off that bandwagon. But I don't regret it, it was a lot of fun :)

I've also been on a health roller coaster. I've taken the nice Doctors Orders and have invested in a FitBit Charge and FitBit Aria Scales which keep track of my steps, calories burned, sleep and weight. There is no more hiding. The good news is that I'm a lot fitter - I now enjoy walking and take a 40 minute walk each day minimum as well as walking the kids too and from school. I'm drinking a lot lot less, but still enjoy (probably a little too much still) a weekend tipple.

I've lost over a stone (21 pounds to be precise) as a result! My resting BPM as dropped from a $HIGHNUMBER to 58, which is healthy :D I should keep track of my blood pressure more, but maybe in the new year ..... but

GO ME!!

I'm fitter, stronger, more alert and ready to blog! Although probably shorter blogs next time ;)

Continue reading...