Continuous Integration with Hudson and .NET

Recently I was tasked with setting up a development environment for a .NET shop. I’m a great fan(atic) of Continuous Integration (CI) and set about thinking of a good way of doing this (and a fun/cool way as well).

For those of you new to Continuous Integration, I think Martin Fowler describes it best on his Continuous Integration site:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

I’d been using CruiseControl for a while now, but for Java projects, and I had a recommendation to look at Hudson as an alternative. I evaluated it and decided it would be a much better solution for the following reasons:

• It’s easy to use web based interface makes setup, configuration and use a breeze (over the XML driven environment of CruiseControl). This, combined with…
• Hudson’s extensible plugin architecture makes it easy to create custom functionality. Finally add to this…
• A large community support/development base

and you have a great, easy to use, well supported product with numerous extensions that will inevitably cover everything you need to do.

Ok, I know, CruiseControl has lots of this as well, but come on, who want’s to struggle with XML all their lives?

Installation

Installation is pretty easy, follow the instructions here based on your setup / choices. Personally we installed Tomcat 6 and deployed the Hudson WAR via the Tomcat Management interface. One additional thing we did was to setup a HUDSON_HOME environment variable to point to a disk with lots of space, I like to keep LOTS of build information (I love metrics) so space was an issue.

Configuration

Configuration is a breeze, once Hudson is up and running, access it via a web browser (normally http://<server address>:<port>/hudson) and select Manage Hudson. Before configuring Hudson itself (Configure System) I suggest you install any plugins you want to use. Do this by selecting Manage Plugins.

Plugins

The plugin interface is divided into four sections, Updates, Available, Installed and Advanced, all pretty self explanatory (right?). Use one of the following methods to install your plugin(s):

1. Select the plugin(s) in the Available tab, scroll  down to the bottom and select Install.
2. Download the plugin(s) (generally from Hudson plugin repository), select the Advanced tab, fill in any proxy settings (if needed) and upload the plugin.

either way your plugin will be available once you restart Hudson.

For our installation these were some of the key plugins we installed, and why:

• Hudson Backup – Allows you to backup/restore your Hudson configuration, we found this invaluable for obvious reasons.
• Hudson build timeout – This plugin allows you to set a timeout threshold on builds, in case it goes into a “hang” state.
• Claim – This plugin allows a team member to claim responsibility for fixing a broken or unstable build. Broken builds are BAD, and should be fixed immediately; it’s always good if someone takes ownership.
• Green Balls – Green Balls instead of Blue Balls. People respond better to the Green/Red colour combination than Blue/Red.
• Hudson GIT – We chose GIT as our repository and this plug-in enables access to it. No need to stress the importance of good Source Code Management (right?)
• Hudson Doxygen – This plugin enabled the generation of Doxygen documentation from the source code. It’s a good tool to (visually) check the overall quality of the in-code documentation and I believe it encourages people to write coherent understandable documentation (although it’s only truly effective if someone periodically goes over it.)
• Hudson MSBuild – MSBuild is used to build our projects from the command line. This plugin enables this functionality.
• NCover – This plug-in enables the gathering and presentation of NCover results within Hudson.
• Hudson NUnit – This plug-in enables the gathering and presentation of NUnit statistics within Hudson
• Hudson Seleniumhq – This enables running and gathering of Selenium tests. This exercises our front end to ensure we don’t break anything already in place.
• Task Scanner – This plugin scans for open tasks in a specified set of files in the project modules and displays the results.
• Warnings – This plugin collects the compiler warnings of the project modules and visualizes the results.
• Hudson instant-messaging – This enables instant messaging capabilities in Hudson. Used by the Jabber plugin.
• Jabber – Sends build notifications to jabber contacts and/or chatrooms. Also allows control of builds via a jabber ‘bot’.
• Twitter – This allows the sending of (basic) tweets after builds have completed to a specific twitter account. Visibility is important on any project.
• Hudson Violations – This allows the gathering of data from various “violation checking” tools (we use it for StyleCop and Simian specifically) and presents it in Hudson. Also allows for limits to be set on “violations” forcing builds into an unstable state if necessary.
• Hudson Release – This plugin allows you to configure pre and post build actions that are executed when a release build is manually triggered. We use this plugin to do our “releases”
• The Continuous Integration Game – Enables scoring of builds, encourages good behaviour and is generally a good fun plugin to have.
• Chuck Norris – A “fun” plugin, fun is important. Chuck get’s upset when the build breaks, plus a few cool Chuck lines to boot.

Hudson

Ok, now for Hudson itself. Hudson’s configuration is pretty self explanatory and well documented (click on the question marks for context sensitive help). I’ll run through what we did to get our server ready for jobs (some of these options only appear once the plugins are installed, which is why I recommend doing them first):

• Home Directory – This is picked up from the HUDSON_HOME environment variable.
• System Message – Appears on your landing page, write something cool here about your CI environment.
• Enable Security – checked, with these options:
  • Random TCP Ports, Access Control using Hudsons own Database, Logged in users can do anything
• Ant – added an Ant installation for use by our Release Plugin.
• MSBuild Builder – Configured for our builds, needed for .NET building
• Selenium Remote Control – Configured the htmlSuite runner for running our Selenium tests.
• Git – Configured the location of the Git executable (include the actual exe as part of the path).
• Email Notification – Pretty standard SMTP information, remember to set the Hudson URL to something people who receive the email can use to access the build machine (ie, don’t use something like localhost). Also, set the System Admin Email address.
• Continuous Integration Game – Activate it.
• Jabber Notification – Enable it. Use the context help to understand the options and complete the need stuff. One suggestion, change the Bot command prefix to something other than !! to prevent outside people accidentally accessing your build bot and messing with your system.
• Global Twitter Settings – Setup twitter access, Hudson will tweet based on the options you give, configurable per project as well.

Done, click Save

Creating Jobs

We were constrained to use .NET so we chose to build all our projects as freestyle projects, and the following describes how and what we configured using this option:

Click New Job, fill in a job name and choose Build a freestyle-software project.

Again, most of the options are well documented in the context sensitive help. Here are some of the options that we configured (I’ve left out most of the the common ones that are self explanatory):

• Main Section
  • Discard Old Builds – We chose to only keep 100 builds, for metrics purposes (did I mention I like metrics?), may change this later.
• Advanced Project Options Section
  • Quiet Period – Changed this to 15 secs to prevent problems with synchronization and building.
• Source Code Management Section
  • Git – We use git as out SCM of choice, we have a “central point” where everyone synchronizes to and the build server monitors for change. Just added a file based location (was on the same server as the Hudson so this was the easiest option) to point to the git repository. The rest of the options were left as defaults.
• Build Triggers Section
  • Poll SCM – checked, polls SCM every minute for changes. This triggers the build.
• Build Environment Section
  • Configure Release Build
    • Release Version Template – This label is used by the Release Manager to label builds. Can use the environment variables defined in the next section.
    • Release Parameters – Define parameters that you need during the release build here. We defined RELEASE_VERSION to allow for a release number to be passed through to the ant build script, defined later. These parameters will be prompted for when selecting Release Build from the projects menu.
    • Before Release Build – Steps to run before the build. Multiple steps can be defined.
    • After Release Build – Steps to run after the build. Multiple steps can be defined. We define a single Ant Build step to package, compress and add the build number to the relevant artifacts for the particular project. This compressed package is then uploaded to a FTP “release” site for consumption by other parties (QA people, release people etc).
  • Abort the build if stuck – We set this at 10 min, fail if it gets stuck, just in case.
• Build Section
  • Here you can define a number of build steps to construct your solution and prepare it for release. These are the steps we performed, in order (for most projects, some were excluded for particular projects):
    • Windows Batch Command to pre-configure the environment. Essentially we modified the existing app.config and web.config and merged in environment specific values to override default values for things like databases and file paths. The tool we used for this was the MergeConfiguration tool supplied with Microsoft Enterprise Library 4.1. It takes a delta file (xxx.dconfig) and merges it with the existing file (app.config / web.config) and creates a new web.config / app.config. One thing to note here was that we had to move the existing app.config / web.config file to a temporary file (eg base.config) BEFORE merging and merge base.config with xxx.dconfig to create  app.config / web.config

    move “<path>\app.config” “<path>\base.config”
    “<path>\MergeConfiguration.exe” “<path>\base.config” “<path>\ci.dconfig” “<path>\app.config”

    • Windows Batch Command to run in the database scripts using osql.exe. All databases are recreated from scratch using supplied scripts each build. We did this as a failsafe to make sure the SQL scripts matched the code that was being deployed.

    osql -S<server> -U<user> -P<password> -i<path>\ResetCIDatabase.sql
    osql -S<server> -U<user> -P<password> -d<database> -i<path>\Initialise.sql

    • MSBuild step, no parameters, just build the solution from the root.
    • Windows Batch Command to run the Doxygen command line utility to generate the code docs for the project using a custom doxyfile.config file.

    “<path>\doxygen.exe” doxyfile.config

    • Windows Batch Command to run the StyleCop command line utility (StyleCopCmd) to check the code for style violations. Some things to note about the command line version were that the settings file MUST be called Settings.Stylecop and be referenced without a path (ie, put it in the root of the project). Run the command without parameters to see what else you can do with it. This generates an XML output of violations that is used by the Violations plugin.

    “<path>\StyleCopCmd.exe” -of stylecop-output.xml -d . -r -sc Settings.Stylecop -ifp AssemblyInfo*

    • Windows Batch Command to run the Simian command line utility to check the code for duplication. A very handy tool to catch you cut and paste warriors out there, generates an XML file used by the Violations plugin.

    “<path>\simian-2.2.24.exe” **/*.cs -formatter=xml:simian-output.xml -failOnDuplication-

    • Windows Batch Command to execute Nunit and run Ncover against the results to generate a report on Unit Test coverage. Nunit executes the unit tests, NCover takes the XML output of Nunit and generates reports with trends. Very handy tool. One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though). One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though)

    “<path>\ncover.console.exe” //x coverage.xml “<path>\nunit-console.exe” /noshadow /nologo   <full path to Unit Test DLL’s, space seperated> //h coveragedir //at ncover3.trend

    • Windows Batch Command to release the built artifacts to the internal environment. We “release” out files onto the CI server (running IIS) for Selenium to run against. Currently we do not have a need to run Selenium but when the need does come up, this will facilitate it.

    xcopy PrecompiledWeb <path of built artifacts> /I /R /V /E /Y

  That’s it as far as building the project for now. There are a few more things that we have not implemented yet, but will in time, like Selenium tests, but at the time of this blog they were not done.

• Post Build Actions Section

This section handles all the reporting back of builds. Here are some of the important things we enabled/configured:

• Archive the artifacts – We chose to archive the entire build for reference purposes. Maybe a bit overkill.
• Activate the Chuck Norris Plugin – Chuck is watching you!
• Scan workspace for open tasks – This plugin scan the workspace for TODO’s, FIXME’s and NOTE’s (the tags we use to signify something to do in the code). Reports on the occurence of these. These are bad, putting TODO’s in code is pointless, rather just DO it.
• Report Violations – gathers all the input XML files and creates a single Violations report. Used to pull in the Simian and StyleCop violations into a report.
• Scan for Compiler Warnings – checks the standard out for compiler warnings, we set it to scan only for MSBuild parser.
• Publish Doxygen – publish the Doxygen generated files, using the Doxygen HTML directory.
• Email notifications – enabled the email notifications to send mail to everyone in the team whenever an event happened on the build (fail, fix, build etc.)
• Activate the Continuous Integration Game (see the plugin link above for more details on this)
• Allow broken build claiming – Allow people to claim broken builds. Ownership is important, someone must take responsibility for fixing the broken build. Remember, it’s not about blame, it’s about fixing it.
• Jabber notification – activate and entered everyones Jabber accounts. This allows instant notification of build events to developers. Again, everyone gets notified of everything.

That’s it, your job is created and ready to build.

I hope this proves useful to some of you out there, I didn’t have too many issues with configuring Hudson in a .NET environment and those I did have were solved eventually. There are still a few issues/concerns that we need to iron out like:

• How do we control versions and releasing versions?
• How do we deal with build dependencies and releasing dependencies with projects? (Hudson supposedly handles this well, yet to be proven though)
• How do I get NUnit to run the Web Service Unit tests?
• Automating releases into a QA environment.

Other than that things are running pretty smoothly, will post more (shorter) entries as we make more progress.