Phabricator is written in PHP which means I don't have to install Yet Another Framework. I use quite a few things that depend on PHP on this site already, such as Grav and RoundCube. So of course, it allows me to self host. Or you can rent a Phabricator VPS @ Phacility.
The sign up process (to my Phabricator instance, not somewhere else) is very straight-forward, allowing email/password with ReCaptha or use a OAuth2 provider such as Google. So this is very socially acceptable and should be secure from spambots.
The core work is based around the ease of code auditing and review of patches. There is even a pastebin so users can upload config files and logs for analysis. Doing all this in a mailing list over the years results in things being here, there and everywhere .... and then expiring. Having it all centralised means nothing is lost. But more importantly, it's much easier to look at and work with, so this is a massive quality of life improvement.
Tickets (or tasks in Phabricator) very user friendly, showing a collapsable history with full links to related objects such as commits, reviews, logs, etc. Infact the linking is extremely easy, one can reference some more of the popular objects by using a single letter follows by the id. Such as
T1. Tickets can be related to one or more Projects and in turn Projects can display Tasks on a KanBan Board.
Phabricator can host your code in your SCM of choice for you and defaults to not allowing destructive changesets by default which saves me from messing around with custom hooks. This allows the same feature as Fossil's immutable history on the server - you can do what you like to your own clone still.
It's fast! No, it's not as fast as Fossil, but it's still more than fast enough especially when you consider the extra toys you get - syntax highlighting, desktop notifications (on supported browsers, which is most recent ones), user icons, in-depth tooltips. It's certainly faster than other solutions I've looked at recently and bar Fossil, probably the fastest.
You get a chat room (does require a NodeJS server on the host for automatic updates though it seems) and a wiki. I still use IRC on FreeNode, but the advantage here is that this is web based and persistent so you don't loose anything if you get disconnected. Still, unsure how useful either be as I don't recall users editing any publically editable wiki pages I've had over the years - are my man pages really that good? Heh.
Phabricator is written in PHP. Now I did say that was a good thing earlier, but it's a double edged sword. PHP does have a bad reputation for both security and language structure. I would argue that this is no different from how C is today. This is also bad, because my site ran on PHP-7.0 and that was soooo much faster than earlier versions it was silly. But Phabricator didn't support PHP-7 until PHP-7.1 in early Feb this year. Something to think about for long term support, but this equally applies to other languages, especially the Python-2 vs Python-3 issue as my box has two Python versions due mainly to certbot needing Python-2.7
Phabricator requires MySQL (I installed MariaDB, the fork from MySql). I was very happy with PostgreSQL but my box does not have the resources to run both. Pretty much all other software I use allows the choice of DB, so this actually took me by surprise. And just like the PHP reaction others have, I was concerned by using MySQL, but as I'm not really into being a DBA I'm quite happy with MySQL so far.
The linking is really bad for DHCP, because we always talk about
T2 as timers. This is important, because my main product is of course dhcpcd.
T2 are shorthand to link to Task 1 and Task 2.
You can fix this by stopping Phabricator from linking via a matchig regex, but I quite like the ease of use and solved the problem via changing the
AUTO_INCREMENT value in some tables from 1 to 101. This reduces the potential collision with other things, such as
Z1 and allows the same workflow. Upstream rejected my change and even went as far as to remove me posting my fix if anyone else has the same issue claiming this would make support hard. As it turns out, something with my change isn't quite right - either Phabricator or MySQL resets the
AUTO_INCREMENT value. I don't know which one, or what action I did or if it's a general Garbage Collection going on. This could be why they didn't like the change, but heh ho most of the important tables now have values in at 101 and higher so it shouldn't be a problem anymore.
Because Phabricator is based on and developed in a DevOps fashion, there is practically no support for managed releases or milestones. This isn't a problem for me as such, but I would like a feature to track important things that went into a release better.
Phabricator is NOT ugly. It's quite visually appealing. However, it is quite possibly the most complex installation I've ever done as it uses many databases and as many configuration options as sysctl on a good BSD. This wasn't helped by running on NetBSD -current and a gcc built PHP with Phabricator just didn't work and I spent a long time working out why. My fix was to build everthing with clang which required a lot of personal effort from me at the time due to the recent UEFI booting support breaking the build and a the new clang-4 compiler not working with the NetBSD build knobs I was using. On the plus side, the Phabricator docmentation is good and about 95% of the issues I had were easily searchable on StackOverflow or the (mostly) friendly Phabricator community helped me out in their chat channel - which oddly enough is also a Phabricator application.
Phabricator workflow with more than one dev, or the best way of submitting patches, is to use the Arcanist tool. They admit it's not great and things should be manageable directly through the SCM. We'll see how that progresses. In the meantime, posting patches to the Differential application is quite easy and allows easy patch review.
I had to stop using Fossil because Fossil is more than just a SCM - it strives to be a complete one stop solution. Obviously that won't work for the desire to use Phabricator for all the good reasons, so I needed to pick a SCM to use. Luckily Phabricator quite a few - GIT, Mercurial and SVN.
But what about the source code control?
It's importance cannot be understated - the code is everything, the history of the code is everything. This has been known since the dawn of time. At this point though, the SCM just becomes a tool in the box, just like sed.
Every SCM solution out there has pretty much the same set of basic features you need - atomic checkins (ok, CVS lacks this), changesets, branching, tagging. That's all you pretty much need at a basic level - the rest of the features are predominently driven by workflow.
Tools exist to export data from one to the other, and tools are being created to allow a more transparent bridge again making the choice of SCM even less important than it was before. The only real issue is the importance of meta data that has no place-holder in the other SCM you want to use. A good example of this would git the Author vs Commiter git attribute on the commit.
Then, you need to understand that the SCM is only for developers. End users don't care a hoot about it - what they do care about is an easy to use system which handles the lifetime of their issue where dicussion, patches, logs, reviews and audits can happen. Hopefully they can even get a fixed build at the end. This is basically part of Application Lifecycle Management.
I've been using Fossil as my SCM for quite a few years now and it has served me well. It replaced my aging Trac (which I've now really retired in the recent server move ... it didn't move) + GIT setup. There is nothing inherently wrong with it and upstream are quite quick to resolve any issues. So lets start with a list of Fossil plus points, in no particular order:
- BSD license.
- One binary, easy installation, very low maintainence cost.
- Integrated CGI web front end.
- Integrated Wiki, Tickets - which are also distributed.
- Sane command line UI.
- Stores everything in a SQLite database.
- Repository is not joined to the checkout, supports different checkout from the same cloned repository.
- I have a Fossil commit bit - my change allows a near perfect Fossil <> GIT bridge.
And naturally, after many years of use, there are some negative points:
- The ticketing system is very basic and has no email support - you're expected to use each tickets RSS feed, but this is not clear.
- It's not extendable.
- It's possible for an admin in the upstream repo to wipe out parts or the whole of your cloned repo.
- It's not social.
That's actually a very small list of negative points. It shows that Fossil is a great product, with a great team behind it. Let's address these these negatives in more detail though.
The ticketing system is is poor
Yes, the tickets are distributed, but that's the only good point. The UI to progress the ticket needs a lot of work and is not intuitive to use. Tickets don't support markdown. It's not clear to the end user that the only feedback they get is a RSS feed. My initial attempt to fix this was about 3 years ago but was met with silence. I could try and improve this by creating a fossil branch just to add RSS icons to the ticket UI.
Fossil is not extendable
This isn't actually that bad, what it does have works well enough (aside from the ticketing). And to be fair, there is a 3rd party library to extend fossil but it doesn't seem to be used by anything I can find. However, based on recent experiences at my day job (where I don't use Fossil), code reviews are turning out to be quite critical and the tools we were using suck. Well, Fossil doesn't have any code review feature nor any easy way of hooking it into an automated build system for continuous integration So we're left reviewing changes via pastebin where links expire or diffs via email. Now diffs via email have been standard on many open source projects, and still are in many. Most of the time I can read them fine, but sometimes they are hard to review in an email. Sometimes I end up using a code review tool, loading the dhcpcd source code, copy and paste the diff into it, reviewing and then copying my review comments back into the email. This is hardly ideal and quite time consuming.
Fossil history can be wiped out
Fossil has an ability to delete anything from your cloned repo - it's called shunning. While this is an awesome feature for corporations (I work for one, I understand the problems and wish my day job had this feature), but equally I believe it's entirely un-suitable for open source use. This is my PC, it contains my contributions to a project which could wipe out my copy of said project and published contributions. History, gone. Now, it's entirely likely that this will never happen, I like to believe in the good in people, but the possibilty remains someone could push the button. OK, there's a bit more to it than that - the default fossil setting is to auto-sync the shun list. However, the code is disabled for auto-sync (ie sync on commit) but is enabled for the manual pull/push commands. While that removes the item from your checkout (if it's there), it won't actually remove it from the repository itself until the repository is rebuilt, which is sometimes forced on you when upgrading fossil. Yes, this is probably a knee-jerk reaction to a none-issue, but it still grates. This is also a reason why I love to self host and would never consider having GitHub or similar being the one sole place where I publish my work. I have always, and always will do, self host.
Fossil is not social
By it's very nature, you can't contribue to the club unless you're in the club - at least not using just Fossil. It's designed (from my perspective anyway) to be a distributed CVS/SVN + wiki + tickets. By this, I mean there is one master repository everyone clones from and pushes to. This makes it impossible to have my own branch outside of the main repo and publish it to others (equivalent of GIT Fork and Pull). It can also be argued that this is a good thing because it encourages people to work together and just like the prior point, this is a good feature for corporate setups. But equally sometimes someone needs to maintain a patchset unsuitable for upstream for valid reasons. This is rare, but it has happened. And I hate losing users for any reason. Could they have a branch they maintain in my repo? Quite possibly, but Fossil's security isn't that granular AFAIK and I would dislike someone messing around in the other branches. Maybe that's anti-social of me, but equally no-one has ever asked for commit access to my repos either.
EDIT: Dr Richard Hipp pointed out privately that this is The Cathedral and the Bazaar. Fossil is the pre-eminent solution for The Cathedral, while others are more suited to the Bazaar.
Taking the above into account, I can no longer justify the use of Fossil in my Open Source projects. For other projects, Fossil is still an awesome tool if that's all you need.
So finally I've moved all services from my old server to my Christmas Xen box! This was not without problems due to the fact it had to run NetBSD -current
- gcc toolchain is broken for some packages which affected running any PHP build
- clang toolchain was broken for my config (USE_SSP= yes and CPUFLAGS+= -march=core2)
- clang compiles as a whole were broken due to a recent efiboot import
In hind-sight, I could have had the box up and running a lot sooner if I used NetBSD-7 guests (or maybe just a NetBSD-7 build box), but no, I just had to get -current running. It offers more than -7 and prior exerience told me that tracking -7 was very problematic ... but that could have been due to my settings and wanting to compile everything with clang. I've currently got 3 -current images there now ... the Xen DOM0, the actual server itself as Xen DomU (without any compiler or tools) and another Xen DomU which just builds stuff for other guests to use.
Anyway, the box itself is now up and running and all relevant services have been moved to it. During this move, I decided to modernise things a little and setup HTTP/2. If you're reading this then it's working :) As such, I've re-directed all basic HTTP traffic to HTTPS and it does seem to load a lot faster. One side effect of this is that I've stoped using my own self signed certificates and I'm now using the nice Let's Encrypt service. pkgsrc users should use the py-certbot package and don't bother searching for any letsencrypt package as it's been renamed. I found this quite confusing as there was no reference to the rename in pkgsrc I could find and wasted a little time on it.
But now it's live I can finally look into bringing online some other development utilities to play around with as the server has a lot more room to grow :)
After starting the year with curses work, I decided to do some more!
Currently we lack Soft Label Key functions as specified by POSIX. What are these? Glad you asked! Soft Label Keys ripoff a line from the bottom of the screen and allow for up to 8 keys to be displayed there. You can define labels for them, colours, etc.
"BIG WHOOP!" I hear you say. "My app already does this, why should I care?"
Because SLK also works with the terminfo database to work out if the terminal natively supports labels (via plab_norm, label_on, etc)! This means a line doesn't have to be ripped off from the screen so no real-estate is lost. Sadly, no terminal actually seems to support these markings. But if one is ever made, it should work fine.
Now, not many curses apps actually use the SLK functions, probably for this very reason. But some do, and it is a standard so I've coded NetBSD to support this natively.
Sadly, this has uncovered a bug in the NetBSD curses library where cursor movement in a one line window at the bottom of the screen appears to push the window up. This is very visible with SLK and will have to be fixed for any apps that actually use it, but the SLK implementation itself is sound as resizing the terminal forces a redraw and it looks fine.
So over the holidays, I managed to get in some good quality family time and find some time to work on some Open Source stuff. I meant to work mainly on dhcpcd, but it turned out I spent most of my time working on NetBSD curses library so that Python Curses now works with it. Now, most people reading this blog may not care much for NetBSD or for curses, but it's important because ncurses is used pretty much everywhere by default ..... aside from NetBSD. And most readers should know, I go out of my way to promote interoperable software even when it's at a detriment to my own projects. That's just how I roll.
There is a apparently a thing called PD Curses but I don't know anywhere that actually uses it by default. If you know, please email me @ email@example.com or tweet me. I do know that the pkgsrc curses framework supports it.
And this is the thing I'm currently doing. As NetBSD curses supports more POSIX functions (and a few ncurses extensions, sadly) more and more packages now work alongside NetBSD curses, so I'm plowing through pkgsrc again moving things to the curses framework. This is important, because there is logic in devel/ncurses to use NetBSD curses when passing certain criteria, but sadly this always brings ncurses as a possibility to be linked or it's header used, both of which are quite bad for NetBSD curses so there is a method to my madness.
One interesting thing though is that since I started comitting my curses patches, it turns out that Sabotage Linux is using NetBSD curses and made it portable. I'm chatting with one of their devs and we're starting to merge of a few of their patches back into NetBSD.