Implementing an Effective Build Dashboard (for Hudson) (Part 3)

Not really part of the dashboard configuration but more of a cool thing I thought I’d share the underlying layers (hardware, OS etc) behind our monitor.

I decided to go with full portability and hands free start-up, I did this as follows:

  1. Installed Puppy Linux on a 2gb USB stick.
  2. Installed Firefox 3.x (needed for the Autohide extension).
  3. Installed Autohide extension (auto hides parts of Firefox and forces full screen on start-up).
  4. Found an old derelict PC and plugged it in, started up, configured the video display (once off on start-up).
  5. Configured the build monitor as per previous posts.
  6. Changed the start-up of Puppy Linux to auto start Firefox.

And there you go:

Build monitor, on a steeeck!
Build monitor, on a stick.

Implementing an Effective Build Dashboard (for Hudson) (Part 2)

This is a follow on of my previous post and details how to implement the code available in the original post by Fabio Parera here, together with the modifications I made for our environment. I suggest you look at the original post first though to understand the extra stuff that I did not implement (I removed “expected duration” and “pinging servers”).

  • To start, modify Hudson to generate a code coverage percentage file, in our environment (.NET) we used NCover and NUnit but adapting your tools to do the same shouldn’t be a problem, the concept is simple:
  1. Generate the coverage percentage file(s). See this post on setting up NCover and Nunit in Hudson
  2. Extract the percentage from the file(s) and put into a single file in the root of your project. This is now accessible as http://<ci-server>/jobs/<project>/<coverage-percentage-file> after every build.

  • You need to install the Greasemonkey Add-on for Firefox for the script to work, do that now. After you’ve installed, you need to modify Firefox’s config and set greasemonkey.fileIsGreaseable to true:
    • Open up Firefox and open up the page about:config (type it in the link box).
    • Find the greasemonkey.fileIsGreaseable setting and, if it’s not set to true, double click it to set it.
  • Now extract the contents of this file to a working folder. Please note that this code is neither pretty nor up to date but is completely functional, feel free to modify at will.
  • Edit build-dashboard-radiator.html. Each project is defined in a section like this:

Create a section like this for each project you want to monitor, changing the href to your projects base directory.

  • (Optional) Edit build-dashboard-radiator.css and modify the style-sheet so it’s suitable for your display, the sizes for the fonts are located towards the end of the script. The current sizes are great for a 40″ screen.
  • Edit build-dashboard-radiator.user.js
    • Change coverage_minimum to whatever value you want, anything more than this will be displayed as green, less, red.
    • Find the section that looks like this:

you need to link the names of commiters to your project(s) to pictures, do this here. The names are what’s extracted from the version control system you use and should be set by every team member before checking code in. To find out what it is in Hudson, find a build and have a look at the json object generated from it, you can find it here (look for the value of “fullName”):

http://<server>/hudson/job/<project>/lastBuild/api/json

Note: This is, in fact, the source for all the information in the script, if you’re thinking of extending the monitor this is the first place to look for information on your build. Another place to look would be

http://<server>/hudson/job/<project>/api/json

which lists information related to the project as a whole (not build specific). Also, you can substiture lastbuild in the previous link for any build number.

  • Next, after you’ve saved all your changes, open Firefox and load the html file. Drag and drop the js file into the browser and this should now install if you have Greasemonkey installed correctly

You’ll know it’s installed correctly if you see this in your status bar

That should be it, hit refresh on your browser and bask in a cool dashboard monitor.

I’d like to thank the original poster (Fabio Parera), and of course all the people he thanks, for linking this awesome monitor, it was exactly what I was looking for and I hope it turns out that way for you too. Let me know how it goes and if you can think of any way of improving it.

Implementing an Effective Build Dashboard (for Hudson) (Part 1)

Information radiators are one of the core building blocks of any agile environment, you should never hide or feel the need to hide anything you’re doing. Information radiators encourage (or at least they should) a feeling of pride in teams allowing them to show off what they’re doing. They should be freely accessible for everyone to see what’s going on and share in the joy of what’s being created. They also encourage (if used correctly) good behaviour in teams: no-one likes public shame.

In line with this and following on my post of setting up a Continuous Integration environment with Hudson and .NET I decided to go about putting in place an effective Build Dashboard. For me an effective dashboard (or information radiator) is one that gives as much relevant information as possible with the least amount of effort to read. So, lots of flash, none of the boring stuff, easily accessible by as many people as possible.

Of the various methods out there I went for displaying information on a big screen, this gives you both the high visibility and the information all in one. Some of the other methods like lights and sound are nice, but limited in some way.

In our environment Hudson is our CI tool of choice and it has a pretty nice built in default dashboard:

.. which works great at short distance but not really great at long distance. There are also lot of plugins available to customize this dashboard and make it better viewable but none of them appealed to me as fit for purpose.

Looking around I finally found a post by Fabio Pereira that looked promising. Essentially the monitor is an HTML page with backing style sheet and javascript to extract information from your CI Server (like Hudson, using it’s API facilities). The standard one that he supplies in his blog looks like this (sorry for the bad quality of the pic) (when implemented in our environment):

Dashboard Version 1

For each project is should the following:

  • Name of project.
  • Build result (green for passed, red for failed, orange for building).
  • Time since last build.
  • How long the last build took.

I made the following modifications:

  • Added in a unit test coverage percentage (with colors indicating above/below agreed minimum).
  • Changed the font size based on how long ago the project last built. In other words, old projects shrink, lots of tiny font projects = not much being done.
  • Added a picture displaying who checked in code that triggered the build. Encourage responsibility for what’s done.
  • Added a color (yellow) to indicate an unstable (but not failed) build.

And now it looks like this:

Dashboard Version 2

Also, it’s strategically located:

Location of Dashboard

In the end it turned out to be pretty effective. It was fully automated, highly visible, everyone (team and customer) had access to it and could see it clearly and understand at a glance what was happening on each project. Broken builds and low unit test coverage are quickly exposed and no activity on projects can be seen quite clearly. Only thing I need to figure out now is how not to burn the image into the screen (no screensaver :) ) but I guess if there’s enough churn on a project that shouldn’t happen, more motivation to continuously check in and build I guess…

As usual I’m always looking for ways to improve so feedback is more than welcome. Can you think of ways to improve what I’ve done?

In part 2 of this I’ll got through the code behind this dashboard and try and explain it a little so that implementing it becomes easier, I’ll also attach the actual code (not pretty, but effective nevertheless 🙂 )

Continuous Integration with Hudson and .NET

Recently I was tasked with setting up a development environment for a .NET shop. I’m a great fan(atic) of Continuous Integration (CI) and set about thinking of a good way of doing this (and a fun/cool way as well).

For those of you new to Continuous Integration, I think Martin Fowler describes it best on his Continuous Integration site:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

I’d been using CruiseControl for a while now, but for Java projects, and I had a recommendation to look at Hudson as an alternative. I evaluated it and decided it would be a much better solution for the following reasons:

  • It’s easy to use web based interface makes setup, configuration and use a breeze (over the XML driven environment of CruiseControl). This, combined with…
  • Hudson’s extensible plugin architecture makes it easy to create custom functionality. Finally add to this…
  • A large community support/development base

and you have a great, easy to use, well supported product with numerous extensions that will inevitably cover everything you need to do.

Ok, I know, CruiseControl has lots of this as well, but come on, who want’s to struggle with XML all their lives?

Installation

Installation is pretty easy, follow the instructions here based on your setup / choices. Personally we installed Tomcat 6 and deployed the Hudson WAR via the Tomcat Management interface. One additional thing we did was to setup a HUDSON_HOME environment variable to point to a disk with lots of space, I like to keep LOTS of build information (I love metrics) so space was an issue.

Configuration

Configuration is a breeze, once Hudson is up and running, access it via a web browser (normally http://<server address>:<port>/hudson) and select Manage Hudson. Before configuring Hudson itself (Configure System) I suggest you install any plugins you want to use. Do this by selecting Manage Plugins.

Plugins

The plugin interface is divided into four sections, Updates, Available, Installed and Advanced, all pretty self explanatory (right?). Use one of the following methods to install your plugin(s):

  1. Select the plugin(s) in the Available tab, scroll  down to the bottom and select Install.
  2. Download the plugin(s) (generally from Hudson plugin repository), select the Advanced tab, fill in any proxy settings (if needed) and upload the plugin.

either way your plugin will be available once you restart Hudson.

For our installation these were some of the key plugins we installed, and why:

  • Hudson Backup – Allows you to backup/restore your Hudson configuration, we found this invaluable for obvious reasons.
  • Hudson build timeout – This plugin allows you to set a timeout threshold on builds, in case it goes into a “hang” state.
  • Claim – This plugin allows a team member to claim responsibility for fixing a broken or unstable build. Broken builds are BAD, and should be fixed immediately; it’s always good if someone takes ownership.
  • Green Balls – Green Balls instead of Blue Balls. People respond better to the Green/Red colour combination than Blue/Red.
  • Hudson GIT – We chose GIT as our repository and this plug-in enables access to it. No need to stress the importance of good Source Code Management (right?)
  • Hudson Doxygen – This plugin enabled the generation of Doxygen documentation from the source code. It’s a good tool to (visually) check the overall quality of the in-code documentation and I believe it encourages people to write coherent understandable documentation (although it’s only truly effective if someone periodically goes over it.)
  • Hudson MSBuild – MSBuild is used to build our projects from the command line. This plugin enables this functionality.
  • NCover – This plug-in enables the gathering and presentation of NCover results within Hudson.
  • Hudson NUnit – This plug-in enables the gathering and presentation of NUnit statistics within Hudson
  • Hudson Seleniumhq – This enables running and gathering of Selenium tests. This exercises our front end to ensure we don’t break anything already in place.
  • Task Scanner – This plugin scans for open tasks in a specified set of files in the project modules and displays the results.
  • Warnings – This plugin collects the compiler warnings of the project modules and visualizes the results.
  • Hudson instant-messaging – This enables instant messaging capabilities in Hudson. Used by the Jabber plugin.
  • Jabber – Sends build notifications to jabber contacts and/or chatrooms. Also allows control of builds via a jabber ‘bot’.
  • Twitter – This allows the sending of (basic) tweets after builds have completed to a specific twitter account. Visibility is important on any project.
  • Hudson Violations – This allows the gathering of data from various “violation checking” tools (we use it for StyleCop and Simian specifically) and presents it in Hudson. Also allows for limits to be set on “violations” forcing builds into an unstable state if necessary.
  • Hudson Release – This plugin allows you to configure pre and post build actions that are executed when a release build is manually triggered. We use this plugin to do our “releases”
  • The Continuous Integration Game – Enables scoring of builds, encourages good behaviour and is generally a good fun plugin to have.
  • Chuck Norris – A “fun” plugin, fun is important. Chuck get’s upset when the build breaks, plus a few cool Chuck lines to boot.

Hudson

Ok, now for Hudson itself. Hudson’s configuration is pretty self explanatory and well documented (click on the question marks for context sensitive help). I’ll run through what we did to get our server ready for jobs (some of these options only appear once the plugins are installed, which is why I recommend doing them first):

  • Home Directory – This is picked up from the HUDSON_HOME environment variable.
  • System Message – Appears on your landing page, write something cool here about your CI environment.
  • Enable Security – checked, with these options:
    • Random TCP Ports, Access Control using Hudsons own Database, Logged in users can do anything
  • Ant – added an Ant installation for use by our Release Plugin.
  • MSBuild Builder – Configured for our builds, needed for .NET building
  • Selenium Remote Control – Configured the htmlSuite runner for running our Selenium tests.
  • Git – Configured the location of the Git executable (include the actual exe as part of the path).
  • Email Notification – Pretty standard SMTP information, remember to set the Hudson URL to something people who receive the email can use to access the build machine (ie, don’t use something like localhost). Also, set the System Admin Email address.
  • Continuous Integration Game – Activate it.
  • Jabber Notification – Enable it. Use the context help to understand the options and complete the need stuff. One suggestion, change the Bot command prefix to something other than !! to prevent outside people accidentally accessing your build bot and messing with your system.
  • Global Twitter Settings – Setup twitter access, Hudson will tweet based on the options you give, configurable per project as well.

Done, click Save

Creating Jobs

We were constrained to use .NET so we chose to build all our projects as freestyle projects, and the following describes how and what we configured using this option:

Click New Job, fill in a job name and choose Build a freestyle-software project.

Again, most of the options are well documented in the context sensitive help. Here are some of the options that we configured (I’ve left out most of the the common ones that are self explanatory):

  • Main Section
    • Discard Old Builds – We chose to only keep 100 builds, for metrics purposes (did I mention I like metrics?), may change this later.
  • Advanced Project Options Section
    • Quiet Period – Changed this to 15 secs to prevent problems with synchronization and building.
  • Source Code Management Section
    • Git – We use git as out SCM of choice, we have a “central point” where everyone synchronizes to and the build server monitors for change. Just added a file based location (was on the same server as the Hudson so this was the easiest option) to point to the git repository. The rest of the options were left as defaults.
  • Build Triggers Section
    • Poll SCM – checked, polls SCM every minute for changes. This triggers the build.
  • Build Environment Section
    • Configure Release Build
      • Release Version Template – This label is used by the Release Manager to label builds. Can use the environment variables defined in the next section.
      • Release Parameters – Define parameters that you need during the release build here. We defined RELEASE_VERSION to allow for a release number to be passed through to the ant build script, defined later. These parameters will be prompted for when selecting Release Build from the projects menu.
      • Before Release Build – Steps to run before the build. Multiple steps can be defined.
      • After Release Build – Steps to run after the build. Multiple steps can be defined. We define a single Ant Build step to package, compress and add the build number to the relevant artifacts for the particular project. This compressed package is then uploaded to a FTP “release” site for consumption by other parties (QA people, release people etc).
    • Abort the build if stuck – We set this at 10 min, fail if it gets stuck, just in case.
  • Build Section
    • Here you can define a number of build steps to construct your solution and prepare it for release. These are the steps we performed, in order (for most projects, some were excluded for particular projects):
      • Windows Batch Command to pre-configure the environment. Essentially we modified the existing app.config and web.config and merged in environment specific values to override default values for things like databases and file paths. The tool we used for this was the MergeConfiguration tool supplied with Microsoft Enterprise Library 4.1. It takes a delta file (xxx.dconfig) and merges it with the existing file (app.config / web.config) and creates a new web.config / app.config. One thing to note here was that we had to move the existing app.config / web.config file to a temporary file (eg base.config) BEFORE merging and merge base.config with xxx.dconfig to create  app.config / web.config

      move “<path>\app.config” “<path>\base.config”
      “<path>\MergeConfiguration.exe” “<path>\base.config” “<path>\ci.dconfig” “<path>\app.config”

      • Windows Batch Command to run in the database scripts using osql.exe. All databases are recreated from scratch using supplied scripts each build. We did this as a failsafe to make sure the SQL scripts matched the code that was being deployed.

      osql -S<server> -U<user> -P<password> -i<path>\ResetCIDatabase.sql
      osql -S<server> -U<user> -P<password> -d<database> -i<path>\Initialise.sql

      • MSBuild step, no parameters, just build the solution from the root.
      • Windows Batch Command to run the Doxygen command line utility to generate the code docs for the project using a custom doxyfile.config file.

      “<path>\doxygen.exe” doxyfile.config

      • Windows Batch Command to run the StyleCop command line utility (StyleCopCmd) to check the code for style violations. Some things to note about the command line version were that the settings file MUST be called Settings.Stylecop and be referenced without a path (ie, put it in the root of the project). Run the command without parameters to see what else you can do with it. This generates an XML output of violations that is used by the Violations plugin.

      “<path>\StyleCopCmd.exe” -of stylecop-output.xml -d . -r -sc Settings.Stylecop -ifp AssemblyInfo*

      • Windows Batch Command to run the Simian command line utility to check the code for duplication. A very handy tool to catch you cut and paste warriors out there, generates an XML file used by the Violations plugin.

      “<path>\simian-2.2.24.exe” **/*.cs -formatter=xml:simian-output.xml -failOnDuplication-

      • Windows Batch Command to execute Nunit and run Ncover against the results to generate a report on Unit Test coverage. Nunit executes the unit tests, NCover takes the XML output of Nunit and generates reports with trends. Very handy tool. One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though). One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though)

      “<path>\ncover.console.exe” //x coverage.xml “<path>\nunit-console.exe” /noshadow /nologo   <full path to Unit Test DLL’s, space seperated> //h coveragedir //at ncover3.trend

      • Windows Batch Command to release the built artifacts to the internal environment. We “release” out files onto the CI server (running IIS) for Selenium to run against. Currently we do not have a need to run Selenium but when the need does come up, this will facilitate it.

      xcopy PrecompiledWeb <path of built artifacts> /I /R /V /E /Y

    That’s it as far as building the project for now. There are a few more things that we have not implemented yet, but will in time, like Selenium tests, but at the time of this blog they were not done.

  • Post Build Actions Section

This section handles all the reporting back of builds. Here are some of the important things we enabled/configured:

  • Archive the artifacts – We chose to archive the entire build for reference purposes. Maybe a bit overkill.
  • Activate the Chuck Norris Plugin – Chuck is watching you!
  • Scan workspace for open tasks – This plugin scan the workspace for TODO’s, FIXME’s and NOTE’s (the tags we use to signify something to do in the code). Reports on the occurence of these. These are bad, putting TODO’s in code is pointless, rather just DO it.
  • Report Violations – gathers all the input XML files and creates a single Violations report. Used to pull in the Simian and StyleCop violations into a report.
  • Scan for Compiler Warnings – checks the standard out for compiler warnings, we set it to scan only for MSBuild parser.
  • Publish Doxygen – publish the Doxygen generated files, using the Doxygen HTML directory.
  • Email notifications – enabled the email notifications to send mail to everyone in the team whenever an event happened on the build (fail, fix, build etc.)
  • Activate the Continuous Integration Game (see the plugin link above for more details on this)
  • Allow broken build claiming – Allow people to claim broken builds. Ownership is important, someone must take responsibility for fixing the broken build. Remember, it’s not about blame, it’s about fixing it.
  • Jabber notification – activate and entered everyones Jabber accounts. This allows instant notification of build events to developers. Again, everyone gets notified of everything.

That’s it, your job is created and ready to build.

I hope this proves useful to some of you out there, I didn’t have too many issues with configuring Hudson in a .NET environment and those I did have were solved eventually. There are still a few issues/concerns that we need to iron out like:

  • How do we control versions and releasing versions?
  • How do we deal with build dependencies and releasing dependencies with projects? (Hudson supposedly handles this well, yet to be proven though)
  • How do I get NUnit to run the Web Service Unit tests?
  • Automating releases into a QA environment.

Other than that things are running pretty smoothly, will post more (shorter) entries as we make more progress.