Author: Rick Tonoli

Implementing an Effective Build Dashboard (for Hudson) (Part 1)

Information radiators are one of the core building blocks of any agile environment, you should never hide or feel the need to hide anything you’re doing. Information radiators encourage (or at least they should) a feeling of pride in teams allowing them to show off what they’re doing. They should be freely accessible for everyone to see what’s going on and share in the joy of what’s being created. They also encourage (if used correctly) good behaviour in teams: no-one likes public shame.

In line with this and following on my post of setting up a Continuous Integration environment with Hudson and .NET I decided to go about putting in place an effective Build Dashboard. For me an effective dashboard (or information radiator) is one that gives as much relevant information as possible with the least amount of effort to read. So, lots of flash, none of the boring stuff, easily accessible by as many people as possible.

Of the various methods out there I went for displaying information on a big screen, this gives you both the high visibility and the information all in one. Some of the other methods like lights and sound are nice, but limited in some way.

In our environment Hudson is our CI tool of choice and it has a pretty nice built in default dashboard:

.. which works great at short distance but not really great at long distance. There are also lot of plugins available to customize this dashboard and make it better viewable but none of them appealed to me as fit for purpose.

Looking around I finally found a post by Fabio Pereira that looked promising. Essentially the monitor is an HTML page with backing style sheet and javascript to extract information from your CI Server (like Hudson, using it’s API facilities). The standard one that he supplies in his blog looks like this (sorry for the bad quality of the pic) (when implemented in our environment):

Dashboard Version 1

For each project is should the following:

  • Name of project.
  • Build result (green for passed, red for failed, orange for building).
  • Time since last build.
  • How long the last build took.

I made the following modifications:

  • Added in a unit test coverage percentage (with colors indicating above/below agreed minimum).
  • Changed the font size based on how long ago the project last built. In other words, old projects shrink, lots of tiny font projects = not much being done.
  • Added a picture displaying who checked in code that triggered the build. Encourage responsibility for what’s done.
  • Added a color (yellow) to indicate an unstable (but not failed) build.

And now it looks like this:

Dashboard Version 2

Also, it’s strategically located:

Location of Dashboard

In the end it turned out to be pretty effective. It was fully automated, highly visible, everyone (team and customer) had access to it and could see it clearly and understand at a glance what was happening on each project. Broken builds and low unit test coverage are quickly exposed and no activity on projects can be seen quite clearly. Only thing I need to figure out now is how not to burn the image into the screen (no screensaver :) ) but I guess if there’s enough churn on a project that shouldn’t happen, more motivation to continuously check in and build I guess…

As usual I’m always looking for ways to improve so feedback is more than welcome. Can you think of ways to improve what I’ve done?

In part 2 of this I’ll got through the code behind this dashboard and try and explain it a little so that implementing it becomes easier, I’ll also attach the actual code (not pretty, but effective nevertheless 🙂 )

Performance appraisals, do we really care?

I often wonder why companies think the current way of doing Performance Appraisals are actually effective (you know, a few times a year, you sit with your manager and he tells you what a good/bad boy you were). For me they’re unnecessary and this is why: Firstly, “appraising” someones “performance” is something that should have the goal of INDIVIDUAL self improvement (for the appraised) in whatever areas he feels would improve his ability to deliver work, it shouldn’t be some blanket template applied to a group of employees (ok, you lot are “Developers” and you lot are “Senior Developers”) but rather a personal negotiated path to improvement. Secondly it should be something that occurs continuously, not a few times a year; self improvement doesn’t happen in leaps and bounds, it happens in tiny steps, learning from your mistakes and building on your successes.

I have a theory though as to why they still occur. Traditional “managers” exist to micromanage and ensure maximum productivity out of employees. Their focus is on creating the “perfect employee” that will obey orders and sacrifice all for “the company” without little or no concern for the employee as a person. What they’ve then done is pawned off the responsibilty of dealing with “human” issues (self improvement etc) to the dreaded Human Resources department, a set of people divorced from the actual work environment of the employee who apply templates to people in order to shape them into a perfect little workforce. What a crock of shit. How about managers step up to the plate and do what they should be doing: creating the environment for people to excel in, helping employees find paths to individual self improvement and backing off and letting people do what they were HIRED to do? Then we can finally get rid of the HR department and bring the responsibility back to where it should be, firmly in the hands of your leader.

Continuous Integration with Hudson and .NET

Recently I was tasked with setting up a development environment for a .NET shop. I’m a great fan(atic) of Continuous Integration (CI) and set about thinking of a good way of doing this (and a fun/cool way as well).

For those of you new to Continuous Integration, I think Martin Fowler describes it best on his Continuous Integration site:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

I’d been using CruiseControl for a while now, but for Java projects, and I had a recommendation to look at Hudson as an alternative. I evaluated it and decided it would be a much better solution for the following reasons:

  • It’s easy to use web based interface makes setup, configuration and use a breeze (over the XML driven environment of CruiseControl). This, combined with…
  • Hudson’s extensible plugin architecture makes it easy to create custom functionality. Finally add to this…
  • A large community support/development base

and you have a great, easy to use, well supported product with numerous extensions that will inevitably cover everything you need to do.

Ok, I know, CruiseControl has lots of this as well, but come on, who want’s to struggle with XML all their lives?

Installation

Installation is pretty easy, follow the instructions here based on your setup / choices. Personally we installed Tomcat 6 and deployed the Hudson WAR via the Tomcat Management interface. One additional thing we did was to setup a HUDSON_HOME environment variable to point to a disk with lots of space, I like to keep LOTS of build information (I love metrics) so space was an issue.

Configuration

Configuration is a breeze, once Hudson is up and running, access it via a web browser (normally http://<server address>:<port>/hudson) and select Manage Hudson. Before configuring Hudson itself (Configure System) I suggest you install any plugins you want to use. Do this by selecting Manage Plugins.

Plugins

The plugin interface is divided into four sections, Updates, Available, Installed and Advanced, all pretty self explanatory (right?). Use one of the following methods to install your plugin(s):

  1. Select the plugin(s) in the Available tab, scroll  down to the bottom and select Install.
  2. Download the plugin(s) (generally from Hudson plugin repository), select the Advanced tab, fill in any proxy settings (if needed) and upload the plugin.

either way your plugin will be available once you restart Hudson.

For our installation these were some of the key plugins we installed, and why:

  • Hudson Backup – Allows you to backup/restore your Hudson configuration, we found this invaluable for obvious reasons.
  • Hudson build timeout – This plugin allows you to set a timeout threshold on builds, in case it goes into a “hang” state.
  • Claim – This plugin allows a team member to claim responsibility for fixing a broken or unstable build. Broken builds are BAD, and should be fixed immediately; it’s always good if someone takes ownership.
  • Green Balls – Green Balls instead of Blue Balls. People respond better to the Green/Red colour combination than Blue/Red.
  • Hudson GIT – We chose GIT as our repository and this plug-in enables access to it. No need to stress the importance of good Source Code Management (right?)
  • Hudson Doxygen – This plugin enabled the generation of Doxygen documentation from the source code. It’s a good tool to (visually) check the overall quality of the in-code documentation and I believe it encourages people to write coherent understandable documentation (although it’s only truly effective if someone periodically goes over it.)
  • Hudson MSBuild – MSBuild is used to build our projects from the command line. This plugin enables this functionality.
  • NCover – This plug-in enables the gathering and presentation of NCover results within Hudson.
  • Hudson NUnit – This plug-in enables the gathering and presentation of NUnit statistics within Hudson
  • Hudson Seleniumhq – This enables running and gathering of Selenium tests. This exercises our front end to ensure we don’t break anything already in place.
  • Task Scanner – This plugin scans for open tasks in a specified set of files in the project modules and displays the results.
  • Warnings – This plugin collects the compiler warnings of the project modules and visualizes the results.
  • Hudson instant-messaging – This enables instant messaging capabilities in Hudson. Used by the Jabber plugin.
  • Jabber – Sends build notifications to jabber contacts and/or chatrooms. Also allows control of builds via a jabber ‘bot’.
  • Twitter – This allows the sending of (basic) tweets after builds have completed to a specific twitter account. Visibility is important on any project.
  • Hudson Violations – This allows the gathering of data from various “violation checking” tools (we use it for StyleCop and Simian specifically) and presents it in Hudson. Also allows for limits to be set on “violations” forcing builds into an unstable state if necessary.
  • Hudson Release – This plugin allows you to configure pre and post build actions that are executed when a release build is manually triggered. We use this plugin to do our “releases”
  • The Continuous Integration Game – Enables scoring of builds, encourages good behaviour and is generally a good fun plugin to have.
  • Chuck Norris – A “fun” plugin, fun is important. Chuck get’s upset when the build breaks, plus a few cool Chuck lines to boot.

Hudson

Ok, now for Hudson itself. Hudson’s configuration is pretty self explanatory and well documented (click on the question marks for context sensitive help). I’ll run through what we did to get our server ready for jobs (some of these options only appear once the plugins are installed, which is why I recommend doing them first):

  • Home Directory – This is picked up from the HUDSON_HOME environment variable.
  • System Message – Appears on your landing page, write something cool here about your CI environment.
  • Enable Security – checked, with these options:
    • Random TCP Ports, Access Control using Hudsons own Database, Logged in users can do anything
  • Ant – added an Ant installation for use by our Release Plugin.
  • MSBuild Builder – Configured for our builds, needed for .NET building
  • Selenium Remote Control – Configured the htmlSuite runner for running our Selenium tests.
  • Git – Configured the location of the Git executable (include the actual exe as part of the path).
  • Email Notification – Pretty standard SMTP information, remember to set the Hudson URL to something people who receive the email can use to access the build machine (ie, don’t use something like localhost). Also, set the System Admin Email address.
  • Continuous Integration Game – Activate it.
  • Jabber Notification – Enable it. Use the context help to understand the options and complete the need stuff. One suggestion, change the Bot command prefix to something other than !! to prevent outside people accidentally accessing your build bot and messing with your system.
  • Global Twitter Settings – Setup twitter access, Hudson will tweet based on the options you give, configurable per project as well.

Done, click Save

Creating Jobs

We were constrained to use .NET so we chose to build all our projects as freestyle projects, and the following describes how and what we configured using this option:

Click New Job, fill in a job name and choose Build a freestyle-software project.

Again, most of the options are well documented in the context sensitive help. Here are some of the options that we configured (I’ve left out most of the the common ones that are self explanatory):

  • Main Section
    • Discard Old Builds – We chose to only keep 100 builds, for metrics purposes (did I mention I like metrics?), may change this later.
  • Advanced Project Options Section
    • Quiet Period – Changed this to 15 secs to prevent problems with synchronization and building.
  • Source Code Management Section
    • Git – We use git as out SCM of choice, we have a “central point” where everyone synchronizes to and the build server monitors for change. Just added a file based location (was on the same server as the Hudson so this was the easiest option) to point to the git repository. The rest of the options were left as defaults.
  • Build Triggers Section
    • Poll SCM – checked, polls SCM every minute for changes. This triggers the build.
  • Build Environment Section
    • Configure Release Build
      • Release Version Template – This label is used by the Release Manager to label builds. Can use the environment variables defined in the next section.
      • Release Parameters – Define parameters that you need during the release build here. We defined RELEASE_VERSION to allow for a release number to be passed through to the ant build script, defined later. These parameters will be prompted for when selecting Release Build from the projects menu.
      • Before Release Build – Steps to run before the build. Multiple steps can be defined.
      • After Release Build – Steps to run after the build. Multiple steps can be defined. We define a single Ant Build step to package, compress and add the build number to the relevant artifacts for the particular project. This compressed package is then uploaded to a FTP “release” site for consumption by other parties (QA people, release people etc).
    • Abort the build if stuck – We set this at 10 min, fail if it gets stuck, just in case.
  • Build Section
    • Here you can define a number of build steps to construct your solution and prepare it for release. These are the steps we performed, in order (for most projects, some were excluded for particular projects):
      • Windows Batch Command to pre-configure the environment. Essentially we modified the existing app.config and web.config and merged in environment specific values to override default values for things like databases and file paths. The tool we used for this was the MergeConfiguration tool supplied with Microsoft Enterprise Library 4.1. It takes a delta file (xxx.dconfig) and merges it with the existing file (app.config / web.config) and creates a new web.config / app.config. One thing to note here was that we had to move the existing app.config / web.config file to a temporary file (eg base.config) BEFORE merging and merge base.config with xxx.dconfig to create  app.config / web.config

      move “<path>\app.config” “<path>\base.config”
      “<path>\MergeConfiguration.exe” “<path>\base.config” “<path>\ci.dconfig” “<path>\app.config”

      • Windows Batch Command to run in the database scripts using osql.exe. All databases are recreated from scratch using supplied scripts each build. We did this as a failsafe to make sure the SQL scripts matched the code that was being deployed.

      osql -S<server> -U<user> -P<password> -i<path>\ResetCIDatabase.sql
      osql -S<server> -U<user> -P<password> -d<database> -i<path>\Initialise.sql

      • MSBuild step, no parameters, just build the solution from the root.
      • Windows Batch Command to run the Doxygen command line utility to generate the code docs for the project using a custom doxyfile.config file.

      “<path>\doxygen.exe” doxyfile.config

      • Windows Batch Command to run the StyleCop command line utility (StyleCopCmd) to check the code for style violations. Some things to note about the command line version were that the settings file MUST be called Settings.Stylecop and be referenced without a path (ie, put it in the root of the project). Run the command without parameters to see what else you can do with it. This generates an XML output of violations that is used by the Violations plugin.

      “<path>\StyleCopCmd.exe” -of stylecop-output.xml -d . -r -sc Settings.Stylecop -ifp AssemblyInfo*

      • Windows Batch Command to run the Simian command line utility to check the code for duplication. A very handy tool to catch you cut and paste warriors out there, generates an XML file used by the Violations plugin.

      “<path>\simian-2.2.24.exe” **/*.cs -formatter=xml:simian-output.xml -failOnDuplication-

      • Windows Batch Command to execute Nunit and run Ncover against the results to generate a report on Unit Test coverage. Nunit executes the unit tests, NCover takes the XML output of Nunit and generates reports with trends. Very handy tool. One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though). One thing we haven’t figured out yet is how to run the unit tests that require a running web service, although supposedly the //iis parameter does it (not for use though)

      “<path>\ncover.console.exe” //x coverage.xml “<path>\nunit-console.exe” /noshadow /nologo   <full path to Unit Test DLL’s, space seperated> //h coveragedir //at ncover3.trend

      • Windows Batch Command to release the built artifacts to the internal environment. We “release” out files onto the CI server (running IIS) for Selenium to run against. Currently we do not have a need to run Selenium but when the need does come up, this will facilitate it.

      xcopy PrecompiledWeb <path of built artifacts> /I /R /V /E /Y

    That’s it as far as building the project for now. There are a few more things that we have not implemented yet, but will in time, like Selenium tests, but at the time of this blog they were not done.

  • Post Build Actions Section

This section handles all the reporting back of builds. Here are some of the important things we enabled/configured:

  • Archive the artifacts – We chose to archive the entire build for reference purposes. Maybe a bit overkill.
  • Activate the Chuck Norris Plugin – Chuck is watching you!
  • Scan workspace for open tasks – This plugin scan the workspace for TODO’s, FIXME’s and NOTE’s (the tags we use to signify something to do in the code). Reports on the occurence of these. These are bad, putting TODO’s in code is pointless, rather just DO it.
  • Report Violations – gathers all the input XML files and creates a single Violations report. Used to pull in the Simian and StyleCop violations into a report.
  • Scan for Compiler Warnings – checks the standard out for compiler warnings, we set it to scan only for MSBuild parser.
  • Publish Doxygen – publish the Doxygen generated files, using the Doxygen HTML directory.
  • Email notifications – enabled the email notifications to send mail to everyone in the team whenever an event happened on the build (fail, fix, build etc.)
  • Activate the Continuous Integration Game (see the plugin link above for more details on this)
  • Allow broken build claiming – Allow people to claim broken builds. Ownership is important, someone must take responsibility for fixing the broken build. Remember, it’s not about blame, it’s about fixing it.
  • Jabber notification – activate and entered everyones Jabber accounts. This allows instant notification of build events to developers. Again, everyone gets notified of everything.

That’s it, your job is created and ready to build.

I hope this proves useful to some of you out there, I didn’t have too many issues with configuring Hudson in a .NET environment and those I did have were solved eventually. There are still a few issues/concerns that we need to iron out like:

  • How do we control versions and releasing versions?
  • How do we deal with build dependencies and releasing dependencies with projects? (Hudson supposedly handles this well, yet to be proven though)
  • How do I get NUnit to run the Web Service Unit tests?
  • Automating releases into a QA environment.

Other than that things are running pretty smoothly, will post more (shorter) entries as we make more progress.

The chatty daily standup…

One of the “ceremonies” of Scrum (man that sounds so culty) is the “Daily Standup”, a daily planning meeting where teams realign and coordinate their efforts for the upcoming day. It’s a concise, no frills no fluff zero waste of time meeting that answers three questions: what have I done, what am I doing and what are my issues, and is generally over quite quickly…but what if it isn’t, what if it degenerates into a technical discussion about the implementation of widget-x or how the build script is not building something correctly or which framework to use to solve some problem…

I’ve found this to happen quite often and all too often I see people coming down on the team with statements like “cut the meeting off at 15 min” and “remind the team to stick to the 3 questions”… all good advice but you’re fixing the wrong problem. I believe this behavior to be an indicator of a dysfunction within the team itself. In a truly collaborative environment teams should be discussing and solving problems constantly and not just at certain points through the sprint (like the daily stand up). If they’re only collaborating around the daily stand up then something is wrong and the team has a dysfunction preventing constant collaboration…

What is code quality ?

So starting the new jobs has given me the opportunity to take a fresh look at software and how it should be approached.

One of the questions that I have been pondering on alot is “What is code quality ?”  Is it code that’s well documented ? Is it code that follows some sort of coding convention, code that is uniform throughout ? Well named variables ? Test cases ?

Obviously we all have our own perceptions, but in order to write good software I think its important to look at code from multiple perspectives, in order to do this I need your guys feedback!

Please tell me what you think makes good code ?

But haven’t we always been working this way?

When I was a kid I had a Commodore 64 with a cassette player, awesome little machine with cool games! I decided I had to write my own game, this is where it all started some 25 odd years ago. I decided I would write a “maze-like” game where an “avatar” would navigate through a maze picking various “gems” and scoring points, that was my goal anyway.

I started by skilling myself up in the basics of, well, BASIC (from books and magazines I had), once I knew enough to do the basics I put code to screen and started churning out lines of code, every now and then I would take a checkpoint and verify that everything still matched my goal…

draw the sprite, done, run and check the results, looks cool, figure out how to move it, done, write to code to make it move, done, run the code and see if it moves as desired, bugs!, change the code to work properly, sweet!

Excellent… time for a break, have something to eat and think about if I could do anything better, nah, the results looked good, was working as well as I had hoped.

I repeated this process until I had a working “game” knocked out. Every time I had a problem I couldn’t overcome I would simply change what I wanted, as long as it matched what I ultimately wanted, a (cool?) game with an avatar moving around a maze and scoring points by picking up stuff…

Sound familiar?

Interviews

Having recently been on a round of interviews looking for that ideal job I thought I’d write down some of my thoughts on the whole process of interviewing the interviewer, or, knowing what you’re getting yourself into!

As a start let me first say that the word interview, in my opinion, is not the correct term to use as all:

interview ( ) n. A formal meeting in person, especially one arranged for the assessment of the qualifications of an applicant.

…implies a one way assessment of interviewer(s) over interviewee. This was (and in most cases still is) the traditional way of looking at this process, essentially you (as an interviewee) get grilled by one to many people to determine if you’re an acceptable candidate. You generally get time to fire off one or two irrelevant questions about pay and leave and that’s about it. I think this is ridiculous, that’s like signing a contract without reading or at least understanding it!

I prefer to use the term negotiation:

negotiation ( ) n. mutual discussion and arrangement of the terms of a transaction or agreement…

…implies a two way assessment, they interview you, and you interview them. Yes, interview, as described above, just like they interviewer is there to determine if you’re a good candidate, you should be there to determine if the company is a good candidate for you, the “sale” should come from both sides.

Entering an interview prepared for me is key, and I don’t mean reading up on the company and understanding their business etc etc… that’s just common decency (or common sense?). What I mean is having a thorough list of what you think are important criteria for you to accept the position they’re offering, and don’t be afraid to ask the important questions!

As an example during a recent interview I asked this question of the manager who would ultimately lead me:

“You seem like a manager that has his head in the right place, what are the chances you will resign and/or leave within the first few months I’m employed?”

A very relevant question, the company was strongly corporate (heirarchical, dictatorial, classic waterfall) but this manager worked in an agile, collaborative way. Everything would change if he were to leave.

During my interview for a ScrumMaster position I asked this question of the CEO:

“So what do you think of scrum?”

For me extremely relevant, resistence to agile methodologies is a surefire way to fail, change must be driven from, and embraced at, the highest level.

A while ago I read an interesting excerpt from an article by Bruce Eckel, it was titled: Interview Questions You Wish You Had Asked:

  • If I want to buy something like a book or a tool, how does the process work (how hard is it?).
  • What’s the cost limit before the approval must go up the management chain?
  • What’s the noise level like during the day?
  • How many meetings am I expected to attend, and how long do they usually last?
  • Is there a dress code?
  • Can I work from home sometimes?
  • Does it matter when I work, as long as I come to meetings?
  • How many projects have succeeded/failed in the last five years? To what do you attribute the failures?

Again, all relevant questions to ask!

What I’m trying to get at here is don’t short-sell yourself, have enough respect in yourself to know that you’re the right person for the job, and take the attitude that you have all the opportunities in the world available, it is up to the interviewer to prove to you that their opportunity is the best!

My questions on agile processes

Something I have found very interesting in today’s “agile environments” is the adoption of processes.

I’m caught in an endless loop on my thoughts around the place of processes and how they should fit in a team.

Generally processes are put in place in order to protect the team, sort of like a safety net.  Given this chain of thought people should be interested in following the process, and should try following processes as close as possible. At the end of the day, Its almost like wearing a safety belt.

This is where it gets tricky though, in many cases developers, testers or any other team members may not not see the need for this process, they may believe it is more of an impediment than anything else.

How should this be dealt with ?

Should the team be wholly responsible for the decision to keep or lose the process ?

Should the team be educated every time a process is put in place ? ( Its too often I have noticed that people simply follow processes mindlessly)

In true scrum nature it suggests that the team make these decisions. What if some of these processes have been put in place by parties outside of the team ? Furthermore  these parties that may have a good understanding for the need of the processes ?

So I go back to my safety belt analogy, sure its a pain to put it on, but if you are unlucky enough to be involved in an accident I can assure you that you will wish you had been wearing it. Unfortunately in scenarios like this it seems its only when its to late when one may value the process ?

Another question  I have is around how often should the necessity of a process be revisited ? Is agile not about constant improvement, about constantly looking at what went well and what could have been done better ?

Here is my take, but please add comments im really excited to hear other opinions on this…

I believe that  processes should be revisited constantly, possibly with each scrum retrospective, its important to remember when doing this to consider the impact of failing to implemt and follow the process.

If the process is enforced by external party, and you feel that you dont see the need for it, discuss this with the party responsible and try to understand why it is they deem the process necessary, it may turn out that you have already covered the risk that this very process is trying to prevent. After all is this not what scrum is all about ? COMMUNICATION ?