Roy's Blog

A Hacker's musings on Code | Tech | Life

I've been using Fossil for quite a while now as my SCM. I like Fossil. But Fossil is not Git, and most people seem to like Git. It could be better to say that most people like GitHub because it's the first hosted SCM that's free for open source with good social interaction I'm aware of. And GitHub is huge. Some might say that if you're not on GitHub, you don't exist as a project. Well this obviously isn't true, but you get the idea. You can get free Fossil hosting at Chisel, which is nice, but it's also not GitHub. Plus I like to be 100% self hosted.

So, to get myself on GitHub (as a mirror only), there needs to be a bridge between Fossil and Git. Fossil documentation implies this is quite easy. Sadly, this isn't the case as the <=Fossil-1.37 releases (note there is no guaranntee that future versions of follow will not have these flaws - my branch may not be comitted to trunk) have the following flaws:

  • Branch and Tag name mangling (dhcpcd-6 becomes dhcpcd_6)
  • Silent master branch renaming into trunk on inport, but not on export
  • No tag comments (Fossil lacks the feature) which means syncing tags back and forth results in tag conflict due to signature change

I submitted some initial patches the the Fossil mailing list and I now have a Fossil commit bit! You can find my branch here to fix the Fossil Git bridge.

But that's not the end of the story. A bridge has two ends. With my initial setup, the Git end was bare bones repository which I pushed to GitHub. This is no longer the case - I now need a staging repository to pull both ends. And this requires a script because Git needs a little more hand-holding to completely track a remote. The below script is tailored for my needs, yours may differ. It also reflects the above initial design and the subsequent change - as such it it may need editing if you need to create a git clone from fossil. This comes with no support, just as an idea of how you might implement such a bridge.

#!/bin/sh

fossildir=/var/fossil
# Cannot be a bare directory for git as we cannot write to the host directly.
# So we have a staging directory instead.
# This requires a bit of hand-holding to track all the branches.
gitdir=/var/git-staging

marksdir=/var/scm-marks

# Respect default naming at either end
fossil_export_opts="--rename-trunk master"
fossil_import_opts="--rename-master trunk"

# Only used when creating a git bare bones repo from Fossil.
export_fossil_to_git_new()
{

        rm -f "$fossilmarks" "$gitmarks"
        git init
        fossil export --git \
                --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --export-marks="$gitmarks"
}

export_fossil_to_git()
{

        fossil export --git \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --import-marks="$gitmarks" --export-marks="$gitmarks"
}

export_git_to_fossil()
{

        git fast-export --all \
                --import-marks="$gitmarks" --export-marks="$gitmarks" | \
                fossil import --git --incremental \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_import_opts "$fossildir/$fossilrepo"
}

pull_git()
{
        local remote

        git fetch --all
        # Track all remote branches
        git branch -r | grep -v '\->' | while read remote; do
                if [ -z "$(git branch --list "${remote#origin/}")" ]; then
                        git branch --track "${remote#origin/}" "$remote"
                fi
        done
        git branch --list | sed -e 's/^\* //' | while read branch; do
                git checkout "$branch"
                git merge --ff-only
        done
}

push_git()
{

        git push --all
        git push --tags
        # Reset the current branch checkout.
        # If we don't, the next run will complain about unstashed changes.
        # This maybe a bug in git, but maybe not because the live checkout
        # *is* behind at this point as we just fast-imported.
        git reset --hard
}

echo "Syncing git and fossil."
for repo in "$fossildir"/*.fossil; do
        fossilrepo=${repo#${fossildir}/*}
        repo=${fossilrepo%.fossil}
        gitrepo="$repo"
        fossilmarks="$marksdir/$repo.fossil.marks"
        gitmarks="$marksdir/$repo.git.marks"

        # We just sync old fossil repos to new phab clones
        if [ -d "$gitdir/$gitrepo" ]; then
                cd "$gitdir/$gitrepo"
                pull_git # staging only
                export_git_to_fossil
                export_fossil_to_git
                push_git # staging only
# Enable the below if pusing to a bare git repo from fossil
#       else
#               export_fossil_to_git_new
        fi
done

Direct download to script

Continue reading...

So finally I've moved all services from my old server to my Christmas Xen box! This was not without problems due to the fact it had to run NetBSD -current

  • gcc toolchain is broken for some packages which affected running any PHP build
  • clang toolchain was broken for my config (USE_SSP= yes and CPUFLAGS+= -march=core2)
  • clang compiles as a whole were broken due to a recent efiboot import

In hind-sight, I could have had the box up and running a lot sooner if I used NetBSD-7 guests (or maybe just a NetBSD-7 build box), but no, I just had to get -current running. It offers more than -7 and prior exerience told me that tracking -7 was very problematic ... but that could have been due to my settings and wanting to compile everything with clang. I've currently got 3 -current images there now ... the Xen DOM0, the actual server itself as Xen DomU (without any compiler or tools) and another Xen DomU which just builds stuff for other guests to use.

Anyway, the box itself is now up and running and all relevant services have been moved to it. During this move, I decided to modernise things a little and setup HTTP/2. If you're reading this then it's working :) As such, I've re-directed all basic HTTP traffic to HTTPS and it does seem to load a lot faster. One side effect of this is that I've stoped using my own self signed certificates and I'm now using the nice Let's Encrypt service. pkgsrc users should use the py-certbot package and don't bother searching for any letsencrypt package as it's been renamed. I found this quite confusing as there was no reference to the rename in pkgsrc I could find and wasted a little time on it.

But now it's live I can finally look into bringing online some other development utilities to play around with as the server has a lot more room to grow :)

Continue reading...

A new year, a new website!

In truth, although I had a nice new server upgrade it seems that Trac just keeps on using resource after resource. Now, having the site entirely python powered isn't entirely possible - thus far I have yet to find a reasonable email web interface other than RoundCube Also, my PostgreSQL interface of choice uses PHP. Speaking of PostgreSQL, that kind of limits my blogging software choices as most of them only want to use MySQL, which I have no wish to use. Here's a list of blogging software which can use PostgreSQL:

So seeing as I'm moving a way from Trac and I had a bad experience managing my prior Drupal install many years ago, it's no surprise that I've migrated my blog over to Serendipity :) As with the prior migration I've preserved comment history and my image gallery. However, I still need to edit all prior posts and correct their links. At this point you maybe wondering what now powers my projects? Well, they recently converted from GIT to Fossil

All of this effort and not much blogging? Well, lets see if I can blog more here as it's my content which I own entirely on my self hosted site. I think I've been giving to much content hosting to Facebook and Twitter. I'll not stop using them, but I expect to post more here :)

Continue reading...

I've upgraded to trac-0.12 and all the plugins to the latest available. Everything still seems to work :)

I've also ditched OpenID for the time being as some spammers were still getting through. I still have trouble getting the AccountManager trac plugin to work well, so I've gone back to the trac spam filter plugin, with captcha support enabled. We'll see how this works.

Continue reading...

First off, ISC must be given credit for supplying the BIND DNS and DHCP servers. They've been powering a lot of the internet and private networking for many years, including this server. But the time has come to change!

I'm now using nsd as my public authoritative name server and unbound as my private recursive name server. This results in config files much easier to setup and hopefully more security as nsd is very small leaving less room for public exploits. Also, the total memory used after a few days production is 40meg lighter than BIND which is good as my server is really starved for memory right now.

So what's the catch? Well, there is no way of integrating Dynamic DNS with unbound that I can see. I could use dnsmasq as a DHCP server as that can run a script per lease, which means I can use unbound-control to update DNS. But I dislike this as dnsmasq isn't entirely the right tool for the job. So I will probably write dhcpsd, a small portable and entirely new DHCP server that can be used to integrate with DNS via scripts. Watch this space!

Continue reading...