Roy's Blog

A Hacker's musings on Code | Tech | Life

I've been using Fossil as my SCM for quite a few years now and it has served me well. It replaced my aging Trac (which I've now really retired in the recent server move ... it didn't move) + GIT setup. There is nothing inherently wrong with it and upstream are quite quick to resolve any issues. So lets start with a list of Fossil plus points, in no particular order:

  • BSD license.
  • One binary, easy installation, very low maintainence cost.
  • Integrated CGI web front end.
  • Integrated Wiki, Tickets - which are also distributed.
  • Sane command line UI.
  • Stores everything in a SQLite database.
  • Repository is not joined to the checkout, supports different checkout from the same cloned repository.
  • I have a Fossil commit bit - my change allows a near perfect Fossil <> GIT bridge.

And naturally, after many years of use, there are some negative points:

  • The ticketing system is very basic and has no email support - you're expected to use each tickets RSS feed, but this is not clear.
  • It's not extendable.
  • It's possible for an admin in the upstream repo to wipe out parts or the whole of your cloned repo.
  • It's not social.

That's actually a very small list of negative points. It shows that Fossil is a great product, with a great team behind it. Let's address these these negatives in more detail though.

The ticketing system is is poor

Yes, the tickets are distributed, but that's the only good point. The UI to progress the ticket needs a lot of work and is not intuitive to use. Tickets don't support markdown. It's not clear to the end user that the only feedback they get is a RSS feed. My initial attempt to fix this was about 3 years ago but was met with silence. I could try and improve this by creating a fossil branch just to add RSS icons to the ticket UI.

Fossil is not extendable

This isn't actually that bad, what it does have works well enough (aside from the ticketing). And to be fair, there is a 3rd party library to extend fossil but it doesn't seem to be used by anything I can find. However, based on recent experiences at my day job (where I don't use Fossil), code reviews are turning out to be quite critical and the tools we were using suck. Well, Fossil doesn't have any code review feature nor any easy way of hooking it into an automated build system for continuous integration So we're left reviewing changes via pastebin where links expire or diffs via email. Now diffs via email have been standard on many open source projects, and still are in many. Most of the time I can read them fine, but sometimes they are hard to review in an email. Sometimes I end up using a code review tool, loading the dhcpcd source code, copy and paste the diff into it, reviewing and then copying my review comments back into the email. This is hardly ideal and quite time consuming.

Fossil history can be wiped out

Fossil has an ability to delete anything from your cloned repo - it's called shunning. While this is an awesome feature for corporations (I work for one, I understand the problems and wish my day job had this feature), but equally I believe it's entirely un-suitable for open source use. This is my PC, it contains my contributions to a project which could wipe out my copy of said project and published contributions. History, gone. Now, it's entirely likely that this will never happen, I like to believe in the good in people, but the possibilty remains someone could push the button. OK, there's a bit more to it than that - the default fossil setting is to auto-sync the shun list. However, the code is disabled for auto-sync (ie sync on commit) but is enabled for the manual pull/push commands. While that removes the item from your checkout (if it's there), it won't actually remove it from the repository itself until the repository is rebuilt, which is sometimes forced on you when upgrading fossil. Yes, this is probably a knee-jerk reaction to a none-issue, but it still grates. This is also a reason why I love to self host and would never consider having GitHub or similar being the one sole place where I publish my work. I have always, and always will do, self host.

Fossil is not social

By it's very nature, you can't contribue to the club unless you're in the club - at least not using just Fossil. It's designed (from my perspective anyway) to be a distributed CVS/SVN + wiki + tickets. By this, I mean there is one master repository everyone clones from and pushes to. This makes it impossible to have my own branch outside of the main repo and publish it to others (equivalent of GIT Fork and Pull). It can also be argued that this is a good thing because it encourages people to work together and just like the prior point, this is a good feature for corporate setups. But equally sometimes someone needs to maintain a patchset unsuitable for upstream for valid reasons. This is rare, but it has happened. And I hate losing users for any reason. Could they have a branch they maintain in my repo? Quite possibly, but Fossil's security isn't that granular AFAIK and I would dislike someone messing around in the other branches. Maybe that's anti-social of me, but equally no-one has ever asked for commit access to my repos either.

EDIT: Dr Richard Hipp pointed out privately that this is The Cathedral and the Bazaar. Fossil is the pre-eminent solution for The Cathedral, while others are more suited to the Bazaar.

In summary

Taking the above into account, I can no longer justify the use of Fossil in my Open Source projects. For other projects, Fossil is still an awesome tool if that's all you need.

Discuss this Post

I've been using Fossil for quite a while now as my SCM. I like Fossil. But Fossil is not Git, and most people seem to like Git. It could be better to say that most people like GitHub because it's the first hosted SCM that's free for open source with good social interaction I'm aware of. And GitHub is huge. Some might say that if you're not on GitHub, you don't exist as a project. Well this obviously isn't true, but you get the idea. You can get free Fossil hosting at Chisel, which is nice, but it's also not GitHub. Plus I like to be 100% self hosted.

So, to get myself on GitHub (as a mirror only), there needs to be a bridge between Fossil and Git. Fossil documentation implies this is quite easy. Sadly, this isn't the case as the <=Fossil-1.37 releases (note there is no guaranntee that future versions of follow will not have these flaws - my branch may not be comitted to trunk) have the following flaws:

  • Branch and Tag name mangling (dhcpcd-6 becomes dhcpcd_6)
  • Silent master branch renaming into trunk on inport, but not on export
  • No tag comments (Fossil lacks the feature) which means syncing tags back and forth results in tag conflict due to signature change

I submitted some initial patches the the Fossil mailing list and I now have a Fossil commit bit! You can find my branch here to fix the Fossil Git bridge.

But that's not the end of the story. A bridge has two ends. With my initial setup, the Git end was bare bones repository which I pushed to GitHub. This is no longer the case - I now need a staging repository to pull both ends. And this requires a script because Git needs a little more hand-holding to completely track a remote. The below script is tailored for my needs, yours may differ. It also reflects the above initial design and the subsequent change - as such it it may need editing if you need to create a git clone from fossil. This comes with no support, just as an idea of how you might implement such a bridge.

#!/bin/sh

fossildir=/var/fossil
# Cannot be a bare directory for git as we cannot write to the host directly.
# So we have a staging directory instead.
# This requires a bit of hand-holding to track all the branches.
gitdir=/var/git-staging

marksdir=/var/scm-marks

# Respect default naming at either end
fossil_export_opts="--rename-trunk master"
fossil_import_opts="--rename-master trunk"

# Only used when creating a git bare bones repo from Fossil.
export_fossil_to_git_new()
{

        rm -f "$fossilmarks" "$gitmarks"
        git init
        fossil export --git \
                --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --export-marks="$gitmarks"
}

export_fossil_to_git()
{

        fossil export --git \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_export_opts "$fossildir/$fossilrepo" | \
                git fast-import \
                --import-marks="$gitmarks" --export-marks="$gitmarks"
}

export_git_to_fossil()
{

        git fast-export --all \
                --import-marks="$gitmarks" --export-marks="$gitmarks" | \
                fossil import --git --incremental \
                --import-marks "$fossilmarks" --export-marks "$fossilmarks" \
                $fossil_import_opts "$fossildir/$fossilrepo"
}

pull_git()
{
        local remote

        git fetch --all
        # Track all remote branches
        git branch -r | grep -v '\->' | while read remote; do
                if [ -z "$(git branch --list "${remote#origin/}")" ]; then
                        git branch --track "${remote#origin/}" "$remote"
                fi
        done
        git branch --list | sed -e 's/^\* //' | while read branch; do
                git checkout "$branch"
                git merge --ff-only
        done
}

push_git()
{

        git push --all
        git push --tags
        # Reset the current branch checkout.
        # If we don't, the next run will complain about unstashed changes.
        # This maybe a bug in git, but maybe not because the live checkout
        # *is* behind at this point as we just fast-imported.
        git reset --hard
}

echo "Syncing git and fossil."
for repo in "$fossildir"/*.fossil; do
        fossilrepo=${repo#${fossildir}/*}
        repo=${fossilrepo%.fossil}
        gitrepo="$repo"
        fossilmarks="$marksdir/$repo.fossil.marks"
        gitmarks="$marksdir/$repo.git.marks"

        # We just sync old fossil repos to new phab clones
        if [ -d "$gitdir/$gitrepo" ]; then
                cd "$gitdir/$gitrepo"
                pull_git # staging only
                export_git_to_fossil
                export_fossil_to_git
                push_git # staging only
# Enable the below if pusing to a bare git repo from fossil
#       else
#               export_fossil_to_git_new
        fi
done

Direct download to script

Discuss this Post

So finally I've moved all services from my old server to my Christmas Xen box! This was not without problems due to the fact it had to run NetBSD -current

  • gcc toolchain is broken for some packages which affected running any PHP build
  • clang toolchain was broken for my config (USE_SSP= yes and CPUFLAGS+= -march=core2)
  • clang compiles as a whole were broken due to a recent efiboot import

In hind-sight, I could have had the box up and running a lot sooner if I used NetBSD-7 guests (or maybe just a NetBSD-7 build box), but no, I just had to get -current running. It offers more than -7 and prior exerience told me that tracking -7 was very problematic ... but that could have been due to my settings and wanting to compile everything with clang. I've currently got 3 -current images there now ... the Xen DOM0, the actual server itself as Xen DomU (without any compiler or tools) and another Xen DomU which just builds stuff for other guests to use.

Anyway, the box itself is now up and running and all relevant services have been moved to it. During this move, I decided to modernise things a little and setup HTTP/2. If you're reading this then it's working :) As such, I've re-directed all basic HTTP traffic to HTTPS and it does seem to load a lot faster. One side effect of this is that I've stoped using my own self signed certificates and I'm now using the nice Let's Encrypt service. pkgsrc users should use the py-certbot package and don't bother searching for any letsencrypt package as it's been renamed. I found this quite confusing as there was no reference to the rename in pkgsrc I could find and wasted a little time on it.

But now it's live I can finally look into bringing online some other development utilities to play around with as the server has a lot more room to grow :)

Discuss this Post

After starting the year with curses work, I decided to do some more!

Currently we lack Soft Label Key functions as specified by POSIX. What are these? Glad you asked! Soft Label Keys ripoff a line from the bottom of the screen and allow for up to 8 keys to be displayed there. You can define labels for them, colours, etc.

"BIG WHOOP!" I hear you say. "My app already does this, why should I care?"

Because SLK also works with the terminfo database to work out if the terminal natively supports labels (via plab_norm, label_on, etc)! This means a line doesn't have to be ripped off from the screen so no real-estate is lost. Sadly, no terminal actually seems to support these markings. But if one is ever made, it should work fine.

Now, not many curses apps actually use the SLK functions, probably for this very reason. But some do, and it is a standard so I've coded NetBSD to support this natively.

Sadly, this has uncovered a bug in the NetBSD curses library where cursor movement in a one line window at the bottom of the screen appears to push the window up. This is very visible with SLK and will have to be fixed for any apps that actually use it, but the SLK implementation itself is sound as resizing the terminal forces a redraw and it looks fine.

Discuss this Post

Long trip up to Nunsmere Hall for a long over due but fantastic family lunch. The entire Marples and Coulbeck tribes actually managed to make it - not a single person missing!

Marples vs Coulbeck

We need to do this thing more often, especially in such a nice place where everyone was friendly and the food was just awesome. I think I managed to talk with everyone and have time to play with the kids. Speaking of the kids, they were absolute little gems the whole journey - 4 hours up and 4 hours down. Much better behaved than I was at a similar age no doubt ;)

Discuss this Post