#2782

Continuous Integration…

Someone at Microsoft asked for some thoughts on CI…

• I guess my first thought on CI is why would anyone not do it? A CI server takes less than a morning to setup, an hour or so a month to maintain – depending exactly how you use it – and it saves your devs from running a full set of suites and doing buddy builds for each checkin. In my last team this saved us at least an hour per checkin!
• Of all the agile practices CI is the easiest to adopt and provides lots of gain for minimal effort.
• Your CI build should be fast, sub 10mins, and ideally sub 5mins. All it should really have to do is build and run (unit) tests. This is what people really mean when they say “CI” but I rarely do just this.

The concept is too powerful to just leave at that. Here’s some other things I do or have done using a CI server and had success with:

• Don’t be afraid to use you CI as a tool for forcing quality. We’re just about to change our CI build to not build MSIs but to run FxCop instead. Why? We were having a lot of quality issues around our MSI setup breaking, now we’re more worried about introducing FxCop issues. We also wanted to reduce the CI time as it was creeping up.
• If you don’t have lots of unit tests (and of course you should) you can use your CI to run click tests. We did this in when we setup CI for the Visual Studio Tools for Office team and had our CI machine build and run a series of automated build verification suites that launched VS etc. This probably isn’t going to meet your <10mins bar but I’d rather have some tests than no tests. You can also use option #3 below.
• I’m also a big fan of defense in depth when it comes to use of a CI Server. What I mean by this is apply progressively higher bars using the server to drive this. Currently our process looks like this: 
  1. Devs are “encouraged” to do a clean build and run at least some unit tests locally before checking in. I don’t mandate this but most people don’t like to feel foolish when they break the build.
  2. CI build builds on a clean tree and runs tests in a clean environment to catch missing files and other local dev environment related issues. Our CI machine may also do some other things if they are light weight enough; FxCop, installer and setup testing.
  3. There would be nothing to stop us having another intermediate build that ran every hour or so and did other, lengthier, tasks around quality. For example I could be running a much larger test suite every hour or so. The name of the game if finding defects as soon as possible, ideally a few minutes after they’re checked in.
  4. Every evening we use the same server to build a drop. At this point we do some additional checking around the setup – for example; are all the DLLs being added to setup signed? This build gets sent to QA for testing
  5. Every night we run static analysis; FxCop, NDepend and VSTS code coverage which all generate reports.
• You have to promote a culture within the team that CI build breaks are BAD. Nobody checks in when the build is broken and somebody – preferable the person who broke it – fixes the build. Immediately. If you can have your CI server send email when this happens and generally make a stink until it thinks it’s fixed.

Frequency of CI build breaks is a somewhat useful metric for “do my devs have their act together?”. The question “Is it always the same person who breaks the build?” is also interesting in some cases. I’m not suggesting a witch hunt but it may highlight either people who are struggling to understand the overall application architecture, an application with pieces of horribly coupled functionality, or someone who simply needs to be more careful.

A Story… I mentioned my last team (part of Visual Studio Tools for Office) earlier. During Visual Studio 2005 we required each dev to do a buddy build and run build verification tests (BVTs)on each checkin, this would take at least an hour. As we added more BVTs this time grew and we adopted a CI system. The allowed us to remove this requirement, devs would do a local build and run a smaller set of sanity check suites and check in. This practice saved us many hours of dev time.