FreeBSD.org - What went wrong; and a way out

Few would disagree that we, as a project, are in a spot of bother. This is a short version of my own rwatsonbomb at FreeBSD-ng-detail

We began, circa 1993, as a pair of organizations. FreeBSD.org handled making a 'damn fine unix' that aimed for technical excellence. Walnut-Creek CDROM took that and made a product out of it and supported it.

Since WC-CDROM has gone, we tricked ourselves into taking on everything. We spread ourselves too thin. It's not for the lack of effort or work. We've come up with some extraordinary aberrations to try and maintain the illusion. One of which is code freezes. We used to freeze because of cvs problems. Then we realized that was a way to force developers to help with releases. Now, there are no scm reasons for it but we maintain it to force people to help re@.

There is something horribly wrong with that picture. We've lost focus on being technically excellent. As a result, we've fallen behind in technical excellence AND our releases and release process is somewhat archaic.

We've become a “Jack of all trades, master of none” project. There is no future in that.

If I had a magic wand, or a time machine, and could ignore logistics, hurt feelings and egos and freely change a few key things, here's what I'd do and why.

• We decide that “head” will be required to be functionally stable and our development workflows would be changed to support that.

This means.. no more WIP (work in progress) or half-baked stuff in head. Implementation work is done in feature branches somewhere and ONLY they get merged when they are ready. This is becoming more common, the change is that we complete the process and require it.

The expectation that “head” is functional and stable is vital. I didn't say “release quality” because that implies all the beta testing etc. Any given time should be a reasonable starting point for a release cycle without requiring a freeze, except perhaps to bump accumulated ABI changes etc.

• No more “head” freezes unless head gets seriously destabilized.

The most destructive things we do to our tree are the mad rush to beat the freezes and the aftermath of it being lifted. We stop that - the code is either ready or it doesn't go in. Period.

• I would split freebsd.org development and release engineering into two separate groups with their own repositories.

This is important because it clearly separates goals and responsibilities. There would be a significant people overlap between them of course.

Think of the Film Industry.. The Director's job is to produce the best film they can. The Producer's job is to make a Product to sell. Both know clearly, unambiguously what they're supposed to be doing. There is a heathly push-pull between the two.

The developer (think director) group's job is to go for technical excellence to feed the downstream release group, along with the likes of PCBSD, pfsense, netapp, juniper, freenas, yahoo, netflix, sandvine, etc, etc.

Once “head” is in a perpetual state of being close to being ready to start a release from, then suddenly the re@ burden is reduced.

• The “head” src repository would no longer have long lived “release” branches.. It would have feature development branches only where the goal is to merge to head if/when it passes muster.

And for “head” to be reliable and stable need practical, collaborative feature development branches. Potentially destabilizing changes get thrashed out in a feature branch, like what happens in perforce.freebsd.org and people's private and public git trees.

If you look at perforce.freebsd.org, you'll discover that probably 3/4 of the feature branches are abandoned or dead ends because they didn't work out. This is a good thing, working as intended. It is 100000 times better for a feature that doesn't pass muster to bit-rot in a branch rather than to bit-rot in head.

• re@ as it exists right now does a very good job of stabilizing trees. A similar function would be applied to “head” but they'd get a Big Stick(TM) to back it up.

Potentially destabilizing patches get sorted out in feature branches and go in when they're ready. If something can't be quickly trivially fixed that otherwise impairs usability of “head” then head-re-2.0 gets to promptly back it out.

• core@, or some other flag-waver/cat-herder/progress tracker entity would take an active part in tracking long term project goals to try and keep people's eyes on the ball.

We've got to keep pulling in the same direction. Duplication of work is more easily avoided when people know what's going on and what the progress is. We bump version numbers based on what has successfully reached head, not because of arbitrary release cycle timelines.

ABI changes become a bigger issue if we expect head to be functional, usable and stable because people will use it more. Indeed that's the point - make sure it is well exercised. That means we have to exercise significantly more care with regard to ABI changes.

Agile feature branching is necessary to support a feature-branch driven workflow. This probably rules out svn for this purpose, leaving p4 or a dcvs like git/hg.

We have been doing feature branch based development on the side or quite some time. The entire perforce.freebsd.org repository exists because of this. If you look at things like the AMD64 port, lots of the SMPng, network stack renovation etc.. those lived as p4 feature branches before being merged into head as a functioning, going concerns.

However, I don't think p4 is an option given their second rate command line interface and unwillingness to fix 10 year old regressions or make basic quality of life improvements, not to mention being closed source and the scalability of keeping client state on the server.

As for dvcs's it seems fairly clear that the community at large is falling behind git. Yes, there are a few high profile exceptions, but in general the convergence seems to be on git.

Git wasn't designed with our freebsd.org-1.0 workflow in mind. However, the question is whether freebsd.org-2.0 and git can put us in a better position. I think it does.

Assuming that we tear up the rulebook for “the way things are done” in freebsd.org universe, we could have a git repo as the root repository to hold head, and everything else is downstream from that, including release engineering. Use the best available tools to support each job at hand.

This isn't a “What VCS?” thread. Its a “What should our workflow be going forward?” followed by “What tools support that best?”

Changing our development workflow goes a long way towards helping us get back towards having a vibrant 'head' development process and frees us from many of the self imposed burdens.

Maybe, just maybe, getting back to what we're really good at and making a top notch, kick-ass, take-no-prisoners technically excellent unix will help our downstream users (PCBSD, pfsense, netapp, juniper, freenas, yahoo, sandvine, etc etc) and ensure a long term niche for us.

• “head” is no longer WIP and is required to remain stable on a day-to-day basis.
• Workflow changed to support that. (feature branches, commit when finished, not before)
• Split development and release systems and processes, ala Director/Producer in film.
• No more “head” freezes unless head gets seriously destabilized.
• Releases dont happen from “head” repo. Release process imports head+history to start a new branch in a release repo.
• “Big stick” created and given to Somebody(TM) to keep head “stable”, like what re@ does now.
• Somebody(TM) picks up the flag-waiving/cat-herding/keep-eyes-on-the-ball-people task for “head”.

And finally.. pick the right tools to support the changed workflow, rather than fight the tools and processes we've come up with to date.