Warner's Useful Mercurial Work Flow

I've been asked by several people to document my current Mercurial setup and work flow. While some people love git, others find it tedious and cumbersome to deal with. If you love git, use git. If not, consider this work flow. If you wish to turn this page into a handbook chapter, please contact the author and remove the folksy, chatty style.

This works well on new development pushed into head. It works less well when I have to MFC. To do that, I still follow the handbook using raw svn directly. I'd love to find a way to not have to do this where mergeinfo is correctly done, but so far I got nothing.

Ideally, this section would be “Install the devel/mercurial port” and then I'd move on. However, this port doesn't have the subversion upstream support baked into it. I use hgsubversion. While there is a FreeBSD port, as of this writing (June 5, 2014), it was too old (1.6.1) to support pulling from the FreeBSD repo due to FreeBSD's repo layout. So this section is a little longer…

Install the following ports. While one might be able to do clever things with linking svn to svnlite, I've had mixed results trying that and have always reverted to the actual svn port, since the SWIG bindings work a lot better than the command line bindings, and SWIG works a lot better with the full subversion port. While all this might not be strictly necessary, it is what I know works.

• devel/mercurial
• devel/subversion
• devel/py-subversion

I put this under $HOME/src like so after installing the above ports:

cd $HOME/src
hg clone http://bitbucket.org/durin42/hgsubversion

The author knows that the following checkout works for him. There are plenty of newer revisions that might also work. Caveat Emptor.

% hg sum
parent: 1146:4cdb0e95604f tip
 svncommands: call util.dump instead of manually writing file
branch: default
commit: (clean)
update: (current)
mq:     (empty queue)

It has also been suggested that you go ahead and install the devel/py-hgsubversion port (to get all the dependencies right) and then pull a second copy as just described so that the next section works (and also so you get the version that's known working). I may have done this on some machines while I was discovering how to set things up, but others I puzzled out errors from running hg with the hgsubversion extention, so on the whole I'd say this is a good suggestion.

You need to enable the mq (mercurial queues) and hgsubversion extentions in your $HOME/.hgrc file:

[ui]
verbose=true
username="Warner Losh <imp@freebsd.org>"

[extensions]
mq=
hgsubversion=~/src/hgsubversion/hgsubversion

Note, I also set my username to my FreeBSD email address, and turn on verbose. While I normally like commands to be silent, I find that verbose tells me just enough about what's going on to know something has happened. But this may be because its verbosity is similar to cvs and svn's default settings and I've used those tools for a decade and a half…

Create a patch queue repo on freefall. I place mine under my public_html directory so anybody can pull from it, but I'm also careful about what I work on to make sure that it is all public (even if it all isn't going into FreeBSD or needs a lot of love before I commit it to FreeBSD). I store it on freefall in case my local machine's hard drive goes to the great storage hunting grounds… Also, it is a convenient pivot point to allow development on multiple machines…

local% slogin freefall.freebsd.org
freefall% cd public_html
freefall% mkdir public-patch-queue
freefall% cd public-patch=-queue
freefall% hg init
freefall% logout
local%

While this works equally well on ports as src, all my examples are on src. I've used this technique for the tiny number of ports commits I've done, as well as a hundred or two commits to the repo. It works well for a single commit, or when you have a dozen small chunks that are closely related that you'd like to push upstream at the same time to allow for easy bisection in case there's an issue.

When pulling the FreeBSD tree, you get to decide how much of a conversion you want to do. hgsubversion converts the FreeBSD history into local hg history. I've never wanted/needed much locally (since I use svnweb for much of it), so I tell hgsubversion to start with the tip. You can substitute your own version, or omit it entirely for all the mainline history if you want. It does seem to slow down the creation process to do this, however. Since I've not done it, I don't know how much it slows down the repo actions, but my experience with hg on other projects suggests the slowdown will be minimal.

% cd FreeBSD
% hg clone --startrev HEAD svn+ssh://svn.freebsd.org/base/head
% cd head

OK. Now that you've checked out the FreeBSD tree, there's only one more step left. You need to link it to the patch queue repo that you created a couple of steps ago. While not strictly necessary, keeping your patches in a repo means you get versioned history of them, protection from OOPSes and all the other benefits of version control. It can be a bit of a pain to do, but it is well worth it in the long run if it even saves you having to recreate one patch…

% pwd
.../FreeBSD/head
% cd .hg
% hg clone ssh://user@freefall.freebsd.org/public_html/public-patch-queue patches
% cd ..

That should be it. This step always trips me up, so please let me know if I documented it wrong. You'll know that if future steps don't work :)

This section assumes that you have things setup as described in the previous section. Let's get started. I'll assume some minor familiarity with patch queues. You might want to give http://mercurial.selenic.com/wiki/MqTutorial or http://mercurial.selenic.com/wiki/MqExtention a spin if you have trouble understanding the following.

In this example, we're going to edit UPDATING to add a new note. We'll create a patch on the queue, look at its status, edit its commit message, commit the patch to your repo, push it out, etc. Not all these steps are always necessary, but it makes a good example to do them all, even if you iterate over only some of the steps.

% hg diff
# should be no output here
% vi UPDATING
% hg diff
diff -r 9fa9e4914606 UPDATING
--- a/UPDATING
+++ b/UPDATING
@@ -9,7 +9,8 @@ handbook:
     http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
 
 Items affecting the ports and packages system can be found in
-/usr/ports/UPDATING.  Please read that file before running portupgrade.
+/usr/ports/UPDATING.  Please read that file before running portupgrade or
+portmaster.
 
 NOTE: FreeBSD has switched from gcc to clang. If you have trouble bootstrapping
 from older versions of FreeBSD, try WITHOUT_CLANG and WITH_GCC to bootstrap to
% hg qnew -e update
% hg diff
% hg qdiff
diff -r 9fa9e4914606 UPDATING
--- a/UPDATING
+++ b/UPDATING
@@ -9,7 +9,8 @@ handbook:
     http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
 
 Items affecting the ports and packages system can be found in
-/usr/ports/UPDATING.  Please read that file before running portupgrade.
+/usr/ports/UPDATING.  Please read that file before running portupgrade or
+portmaster.
 
 NOTE: FreeBSD has switched from gcc to clang. If you have trouble bootstrapping
 from older versions of FreeBSD, try WITHOUT_CLANG and WITH_GCC to bootstrap to
% hg qhead
Tell users about portmaster too.
%

The above sequence will check to make sure you have a pure tree, then edit UPDATING, and then create a new patch named update. The -e on the qnew command kicks you into the editor for your commit message. hg diff tells you what is different and uncommitted. hg qdiff combines the most recently saved patch with any local changes, as shown above. So, in this example, we've made a change to UPDATING, and have created a commit message “Tell users about portmaster too.”

Now, before we push this upstream, let's say we've thought about the change and now want to reword the UPDATING things a bit. We also don't like the commit message. So we go in and edit things. Here's a typical sequence of events.

% vi UPDATING
% hg diff
diff -r 2a6b5ffa63c9 UPDATING
--- a/UPDATING
+++ b/UPDATING
@@ -9,8 +9,8 @@ handbook:
     http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
 
 Items affecting the ports and packages system can be found in
-/usr/ports/UPDATING.  Please read that file before running portupgrade or
-portmaster.
+/usr/ports/UPDATING.  Please read that file before running portupgrade,
+portmaster or other port updating utility.
 
 NOTE: FreeBSD has switched from gcc to clang. If you have trouble bootstrapping
 from older versions of FreeBSD, try WITHOUT_CLANG and WITH_GCC to bootstrap to
% hg qdiff
diff -r 9fa9e4914606 UPDATING
--- a/UPDATING
+++ b/UPDATING
@@ -9,7 +9,8 @@ handbook:
     http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
 
 Items affecting the ports and packages system can be found in
-/usr/ports/UPDATING.  Please read that file before running portupgrade.
+/usr/ports/UPDATING.  Please read that file before running portupgrade,
+portmaster or other port updating utility.
 
 NOTE: FreeBSD has switched from gcc to clang. If you have trouble bootstrapping
 from older versions of FreeBSD, try WITHOUT_CLANG and WITH_GCC to bootstrap to
% hg qrefresh
% hg diff
% hg qrefresh -e
% hg qhead
Wordsmith advise on port upgrading.
% hg qseries
0 A updating
% 

As you can see, we've tweaked the UPDATING file. You can also see the difference between hg diff and hg qdiff here, now that we've been working on a change for a while. The first hg qrefresh regenerates the patch we're working on. The second one (with the -e) allows us to edit the commit message, which you can see we've completely rewritten. We could keep up this all day, but let's say we're happy with this.

We could go and directly commit it. However, to instill good habits, let's save a copy of the patch and push that copy to freefall first.

% hg commit --mq -m"Save updating patch"
updating
series
committed changeset 70:2e92a0e95b31
% hg push --mq
pushing to ssh://freefall.freebsd.org/public_html/patch-queue
running ssh freefall.freebsd.org 'hg -R public_html/patch-queue serve --stdio'
searching for changes
1 changesets found
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 2 files
%

Notice that there's changes to two files, and this weird file 'series' gets pushed as well. series is where hg keeps track of the order of the patches in this directory. There's a second file that you'll find if you look in the patches directory too called status. It tells the order the patches are applied right now. This file is never saved. These files are 'special' and so you can't name any of your patches series or status. Sorry. Otherwise you can name them anything. If you include / in the names, it will put them in subdirectories. I never thought this too useful, but some people might.

Now it is time to get serious and push these changes in. Since this is an example I don't want to commit, I've pasted an actual commit below.

% hg qfinish 'mq()'
% hg out
comparing with svn+ssh://svn.freebsd.org/base/head
changeset:   1475:d239c74b6b6a
user:        "Warner Losh <imp@freebsd.org>"
date:        Thu Jun 05 16:36:35 2014 -0600
files:       sys/amd64/conf/GENERIC sys/arm/conf/ZEDBOARD sys/i386/conf/GENERIC sys/i386/conf/XEN sys/ia64/conf/GENERIC sys/sparc64/conf/GENERIC
description:
Restore comments accidentally removed.

MFC after: 3 days

changeset:   1476:f50eda80d522
tag:         tip
user:        "Warner Losh <imp@freebsd.org>"
date:        Thu Jun 05 22:07:35 2014 -0600
files:       bin/ed/Makefile lib/libtelnet/Makefile libexec/telnetd/Makefile release/picobsd/bridge/crunch.conf release/picobsd/qemu/crunch.conf usr.bin/telnet/Makefile usr.sbin/ntp/ntp-keygen/Makefile usr.sbin/ntp/ntpd/Makefile usr.sbin/ppp/Makefile usr.sbin/sendmail/Makefile usr.sbin/tcpdump/tcpdump/Makefile
description:
When building picobsd, define WITHOUT_OPENSSL and WITHOUT_KERBEROS and
remove the now-redundant checks for RELEASE_CRUNCH. This originally
was defined for building smaller sysinstall images, but was later also
used by picobsd builds for a similar purpose. Now that we've moved
away from sysinstall, picobsd is the only remaining consumer of this
interface. Adding these two options reduces the RELEASE_CRUNCH
special cases in the tree by half.

% hg push
pushing to svn+ssh://svn.freebsd.org/base/head
searching for changes
committing d239c74b6b6a
[r267134] gjb: Document r266463, newsyslog.conf(5) includes in conf.d.
M head/release/doc/en_US.ISO8859-1/relnotes/article.xml
release/doc/en_US.ISO8859-1/relnotes/article.xml
 committed to "default" as 5e47f969d8bb
[r267141] allanjude: Style cleanups on ifconfig.8
M head/sbin/ifconfig/ifconfig.8
sbin/ifconfig/ifconfig.8
 committed to "default" as 63a749057781
[r267142] zont: Use mtx_lock_spin/mtx_unlock_spin primitives on spin lock
M head/sys/dev/netmap/netmap.c
M head/sys/dev/netmap/netmap_mbq.c
sys/dev/netmap/netmap.c
sys/dev/netmap/netmap_mbq.c
 committed to "default" as a6acb044c8fd
[r267145] brd: - Fix the keyfile being cleared prematurely after r259428
M head/sys/geom/eli/g_eli.c
sys/geom/eli/g_eli.c
 committed to "default" as f011200641fe
[r267146] imp: Restore comments accidentally removed.
M head/sys/amd64/conf/GENERIC
M head/sys/arm/conf/ZEDBOARD
M head/sys/i386/conf/GENERIC
M head/sys/i386/conf/XEN
M head/sys/ia64/conf/GENERIC
M head/sys/sparc64/conf/GENERIC
sys/amd64/conf/GENERIC
sys/arm/conf/ZEDBOARD
sys/i386/conf/GENERIC
sys/i386/conf/XEN
sys/ia64/conf/GENERIC
sys/sparc64/conf/GENERIC
 committed to "default" as 74a80bdb8e36
pulled 5 revisions
committing f50eda80d522
[r267147] imp: When building picobsd, define WITHOUT_OPENSSL and WITHOUT_KERBEROS and
M head/bin/ed/Makefile
M head/lib/libtelnet/Makefile
M head/libexec/telnetd/Makefile
M head/release/picobsd/bridge/crunch.conf
M head/release/picobsd/qemu/crunch.conf
M head/usr.bin/telnet/Makefile
M head/usr.sbin/ntp/ntp-keygen/Makefile
M head/usr.sbin/ntp/ntpd/Makefile
M head/usr.sbin/ppp/Makefile
M head/usr.sbin/sendmail/Makefile
M head/usr.sbin/tcpdump/tcpdump/Makefile
bin/ed/Makefile
lib/libtelnet/Makefile
libexec/telnetd/Makefile
release/picobsd/bridge/crunch.conf
release/picobsd/qemu/crunch.conf
usr.bin/telnet/Makefile
usr.sbin/ntp/ntp-keygen/Makefile
usr.sbin/ntp/ntpd/Makefile
usr.sbin/ppp/Makefile
usr.sbin/sendmail/Makefile
usr.sbin/tcpdump/tcpdump/Makefile
 committed to "default" as 0b26ea42a4e8
pulled 1 revisions
resolving manifests
getting release/doc/en_US.ISO8859-1/relnotes/article.xml
getting sbin/ifconfig/ifconfig.8
getting sys/dev/netmap/netmap.c
getting sys/dev/netmap/netmap_mbq.c
getting sys/geom/eli/g_eli.c
5 files updated, 0 files merged, 0 files removed, 0 files unresolved
2 changesets found
saved backup bundle to /usr/home/imp/FreeBSD/head/.hg/strip-backup/d239c74b6b6a-backup.hg
6 changesets found
adding branch
adding changesets
adding manifests
adding file changes
added 6 changesets with 22 changes to 22 files

So hg qfinish 'mq()' takes the commit from the patch queue and finishes it up and places it in the hg default branch. The 'mq()' bit says do all pushed changes in your queue. In this case, there were two. “hg out” gives you a list of what will be pushed upstream. In this case, there's two patches. One that touches some kernel config files, and one that does some build stuff. You'll also see a bunch of other stuff. When using hg with subversion, the hg push command is also doing a bit of a rebase. It pulls in all the changes from upstream that happened while you were editing, and then pushes your changes out on top of that. It never does the usual hg thing of creating multiple heads. I consider this a feature, but others disagree. Such is life. In this case, it pulled down 4 changes, added my 2 and gave a summary at the end. I was lucky that these 4 commits didn't touch anything that I'd messed with, or I'd get a merge conflict, which would need to be resolved.

So those are the raw basics. I'll add more about multiple patches, moving patches around, and resolving conflicts. I'll also talk about multiple machines and resolving conflicts in the actual patch queue. My patch queues typically have a dozen entries, 2-3 are approaching ready to commit, and a bunch of experimental stuff, some of which is just debug crap, others are diamonds in the rough, and others are lumps of coal in the rough that will never amount to anything.

Do all the above stuff, but before the commit install devel/arcanist port.

% arc diff

I haven't found a way to have arc do the commit, so I do the commit as above. I then mark it close with

% arc close-revision d1234

Sometimes as you develop a series of patches, you find that some trivial patches you did second are ready to push upstream before other changes you might have started first. It is a simple matter to “hg qpop -a” the entire queue, and then move these fixes to the front of the queue with “hg qpush –move patch-name”. If all the patches are popped off, you can also edit the .hg/patches/series file directly. After moving patches around, each new hg qpush will reapply the patch in the patch queue. This may result in some fuzz or in some cases conflicts. To resolve these, you'll need to do “hg qrefresh” to regenerate the patch.

Speaking of “hg qrefresh” you sometimes need to use this command when you rebase. If it has been a while since you've pulled sources from upstream, you'll need to go through a rebasing operation. There's an hg rebase command, but I never use it. It is useful when you are using hg's branching stuff, which isn't used in this workflow. Instead, a rebasing operation is simple a replaying of the patches. Since the history of the development of the patches is usually uninteresting to the upstream repo, if the patches are constructed right and have good commit messages, this works out well. I usually hg qpop all my changes off. I then hg commit –mq and hg push –mq to save the current state (using the current upstream revision of your working tree in the commit message lets you get back more easily). Then I do the hg pull -u to update the upstream sources. I then hg qpush the patches one at a time, doing hg qrefresh for those that didn't apply 100% cleanly (after checking the diffs and maybe cleaning up any mismerges). Once I've rebased, I'll often save a copy with the same hg commit –mq, this time with the revision of the newly pulled tree. It is a little bit of overkill, but when it saves your butt, it suddenly becomes well worth it.

First off, these are best avoided. Three way merge for patch files is at best confusing and at worst a nightmare.