Version Control Blog

Brian Harry shares an inside story about how the branches are organized in Developer Division at Microsoft. They use many branches (called “virtual build labs”), with strict rules for moving changes between labs. Private branches and release branches are also used extensively. The post is quite old (November 2005), and they seem to be already switching to feature branches. But nevertheless, it’s a good read. Some arithmetics: I’d estimate that we have on the order of 800 developers[…]. Let’s imagine for a minute that any given developer has a brain fart and checks in a build break, serious bug, etc once every 6 months (which in my opinion would be a team of 800 of the best developers I’ve ever met). There’s about 250 working days in a year and simple math says 800 * 2 / 250 > 6 serious bugs checked in every single working day. The fact is it’s higher than that. Read more on how they cope with that scale of bugs at Brian Harry: “Branch structure in Developer Division at Microsoft”. (via notgartner blog)

Dec 17 2006: Mercurial 0.9.3 was released. Bug fixes, documentation fixes, fixes in mq and gpg extensions. Read more at Matt Mackall: “Mercurial 0.9.3 released!”

20 Dec 2006: GIT 1.4.4.3: This release contains merge-recursive corner case fix; it also fixes git-cvsserver (when used with newer Perl) and Mac OS build (when you use config.mak), among other things. Junio also added:

To let people who only follow the ‘maint’ releases know what’s happening in the larger picture… We have just started talking about the next feature release v1.5.0 on the ‘master’ branch side. If we are lucky we could do a -rc1 around Christmas, emperor’s birthday in Japan, or perhaps emperor’s birthday in the Penguin land, but in any case the real release is not expected to happen by mid January. The new release will have many end-user level changes since the last feature release v1.4.4, both at the UI level and at the documentation level, based on previous discussions on the list. It is strongly encouraged and very much appreciated to review and to fill gaps you would find in today’s ‘master’ and what’s cooking in ‘next’, if you were involved in the discussions and/or if you are interested in the theme of v1.5.0: “usability and teachability”.

09 Dec 2006: GIT 1.4.4.2: contains a handful fixes since 1.4.4.1.

Karl Fogel, one of the lead developers behind Subversion and CVS, wrote a book on Open Source software development. A large part of text is dedicated to the social side of software development: communication, finances, managing volunteers, licenses. Technical side is also strong, as could be expected from a developer of such caliber. Of course, Karl has quite a few words about version control as applied to topic. The book is available freely under the open license. You may browse it online at Producing Open Source Software: How to Run a Successful Free Software Project. Particularly, the version control chapter is at Chapter 3. Technical Infrastructure: Version Control. You may also buy the book at Amazon: Karl Fogel: “Producing Open Source Software: How to Run a Successful Free Software Project”. More books at Amazon aStore.

Joel Spolsky forwarded more information about source control in Windows team:

“… prior to the restart effort of Longhorn, there were about seven [branches], reverse-integrating into one main branch every two or three weeks perhaps. Now, imagine several thousand developers checking in directly into seven branches. This will lead to two things: “1. you check in frequently, and there’s a very high chance of either breaking the build, or breaking functionality in the OS, or 2., as a counter-reaction, you don’t check in very often, which clearly is bad, since now you don’t have a good delta history of what you did. “So this clearly didn’t scale. As part of the restart effort, we decided that each team would get its own feature branch, each feature area (multiple teams) would go up to an aggregation branch, and those would lead up to the final main branch. (As such there’s now north of a hundred branches in tiers, leading up to about six aggregation branches.) Teams were free to choose how many sub-feature branches they wanted, if any, and they were free to choose how often they wanted to push up their changes to the aggregation branch. As part of the reverse-integration (RI, i.e. pushing up) process, various quality gates had to pass, including performance tests. Due to how comprehensive those gates ended up being, this would take at least a day to run, plus perhaps a day or two to triage issues if any cropped up; so there was a possibly considerable cost to doing an RI in the first place. However, these gates were essential in upholding the quality of the main branch, and had they not existed, the OS would have never shipped. I suppose it’s one of those ‘what doesn’t kill you…’ type deals.

Joel adds:

When you’re working with source control on a huge team, the best way to organize things is to create branches and sub-branches that correspond to your individual feature teams, down to a high level of granularity. If your tools support it, you can even have private branches for every developer. So they can check in as often as they want, only merging up when they feel that their code is stable. Your QA department owns the “junction points” above each merge. That is, as soon as a developer merges their private branch with their team branch, QA gets to look at it and they only merge it up if it meets their quality bar.

Read all at Joel Spolsky: “Using source control tools on huge projects”.

It is well known that Google uses Perforce as its internal source management system (it has a source license). Niall Kennedy writes:

Google uses a company-wide Perforce depot with almost no developer branches. Each developer has their own NFS workspace readable by anyone in the company, including automated processes. An administrative process takes snapshots of each developer workspace including local development environments accessed over SSH. Files within these snapshots can be compared to checked-in data, encrypted, and archived.

Dan Bloch did a presentation at Perforce European User Conference, called “Performance and Database Locking at Large Perforce Sites”. It contains statistics on Google Perforce Depot, such as:

• More than 3000 users and 100Gb of metadata on one primary server;
• Hardware is an HP DL585 4-way Opteron with 128Gb of memory;
• Depot is on a NetApp filer;
• Metadata and journal on RAID-10 local disk;

(via Niall Kennedy: “Google Mondrian: web-based code review and storage”)