Continuous integration and best practices

Is it not an interesting idea to see a red light flashing in the development room, when there is a build failure or when some one make a code change which breaks some other's code?


Is it not an interesting idea to see a red light flashing in the development room, when there is a build failure or when some one make a code change which breaks some other's code?

Mike Swanson, my project architect, who is a senior consultant from Microsoft, made it work. When any developer check-in a piece of code that is not compilable or if it breaks some other functionality, our Green ORB will turn RED and alert every one in the team!

Continuous integration - How it works?

Before I go deep into the exciting Green/Red ORB details, let me give an introduction to why we use it and how it can be helpful to a project.

Several years back when I was a junior developer, I used to work for a large project where around 20 developers used to work on a very tight schedule. Everything went through fine during the initial stages of the project. Each developer had his own setup in his machine and developed each task almost independently. And after couple of months, the project leader announced - it is time to deliver the first phase of the project... so let us put the pieces together.

I am leaving it to your imagination. I don't want to remember those sleepless nights and how the team members blamed each other for changing the database structure without informing the other guy!

After couple of days sleepless nights, we were able to make all code changes in everybody's code and successfully integrate all pieces together in one machine and give a demo to the customer and he was happy.

Well, but we had to get back to work for the next phase after the demo. By this time everybody found there is lot of dependency between tasks and so we decided everybody will take a snapshot of the project from the demo machine and continue to work in this copy in their local machine. People started using the source control and checking in changes to the source control often.

One of the big mistakes many junior developers did was, they never checked out the files from source control. They used to simply take latest version and made it writable. After making all changes, they will show it to the team leader and make sure what they have done is correct. Then they will simply overwrite it to the source control assuming that nobody has made any changes to the same file after they took a snapshot of it to make their changes.

But this assumption was wrong most of the time. Many other developers did the same and happened to modify the same files, resulting in losing the integrity and ended up in total mess again. When other developers take the latest files from the source control, it never compiled.

The project leader had a tough time going to each developer’s desk and help him compile the whole project in his computer by taking the ‘appropriate version’ of the files from the source control. So, the project leader came up with the worst decision - he instructed everyone not to take the latest files from source control ever. Every one should take only the files which are really needed for him to continue his task!

Learn from the mistakes

We did few big mistakes in this project:

1. No control over the source control.

Team members should not be allowed to "play" with the source control. If anybody want to make any change in the code, he has to first checkout the file from the source control. Never assume that nobody else would have changed "my file".

Moreover, nobody in the team should claim a file as "my file". If a file is in the source control, the file belongs to the team. Any member in the team should be able to check out any file and make the changes.

You may ask "what if someone who doesn't understand my code make some changes and break my stuff?"

Well, my first argument is, a matured developer will not go and change code written by another developer without properly understanding it.

And the second argument is, if you have well written Automated Unit Tests(like NUnit) that covers all the functionality of the project, it is very easy to catch the change which breaks anything in the project. (I will talk about Unit Tests in the last part of this document.)

2. There was no continuous integration.

It is very important to do continuous integration of the project, instead of waiting for the demo day to put all pieces together. This will ensure that:

- All pieces will work together and everybody is in sync with others in the team.

- If there is any major changes like database schema change and changes in public methods of your classes which are used by other developers, it can be immediately passed on to other team members. So, you won't hear the complaint "if you had told me earlier about this change, I need not have wasted so much time. Now I have to re write my code due to your changes."

- It is nice to see the whole application running in each machine instead of seeing each piece of it in different machines.

3. Just look at the code and I can say who wrote it!

It is quite usual to hear experienced programmers claim "it is my style of coding...". In the above mentioned project, it was easy for any one to look at any file and easily figure out who wrote it, just by seeing the coding style.

It is very important for any project to follow certain standards. There is no need to strictly follow some standards provided by others. But we can extract the best from many other sources and prepare a customized standards document for each project. It is very important for all developers to strictly follow the standards, irrespective of their personal preferences.

Here is a small coding standards document I prepared from my experience and also by consolidating from several other sources:

Best practices in NxOpinion

I would like to discuss some good practices we followed in my current project. Even though each project is different and requires different approaches, the following practices can be followed for most of the large enterprise development.

1. Weekly code review meeting

We choose a file every week for our weekly code review. Each member of the team will review the file in advance and come to the review meeting with their comments and feedback. In the review meeting, we will review the code using a projector displaying the code in a white board. Everyone will go through the code and provide their feedbacks. The feedbacks will be recorded for later reference and original developer of the code is responsible for fixing the issues found in the review.

We may not be able to fix 100% of the issues we find in the reviews, but the reviews help in several ways:

- All developers know that their code will be displayed in the screen in front of everyone and so they will be embarrassed to write 'bad code' (we all write bad code blaming the time constraints and for other thousand reasons, right!)

- This gives an opportunity to see how others are approaching the problem and learn from them.

- Helps all members understand other features of the project.

- When reviewing other's code, everyone will be eager to find issues in code. This results in more secure and stable application.

- Code reviews ensure that everyone is following the coding standards.

- helps to identify any design flaws in the system, when several people review and discuss together.

2. Continuous Integration

In our current project, we followed the "continuous integration" approach which helped us a lot in finding problems earlier and avoid many last minute head-aches. We never waited for the demo day to put all pieces together. Continuous Integration is a process involving several steps, but the core concept is this:

"Build and test your software at regular intervals of time and make sure it still works as expected."

The common steps involved in continuous integration process are:

- Use source control to manage source code.

- Automate the build process so that build happens automatically at regular intervals of time.

- Automate the testing so that you can run a suite of tests on the system at any time.

- Save the executables of the system with version numbers, so that anyone can go back and get a copy of the application executables of any version.

We use Cruise Control (the continuous integration system) and NAnt (scripting tool to perform required tasks for continuous integration) for our ‘continuous integration’ process. The integration server is responsible for monitoring the changes in the source control. When a change is detected, the process will become active and execute the NAnt scripts.

The process includes the following steps:

- Retrieve all latest files from the source control.
- Rebuild the entire project.
- Run all unit tests.
- Save a build report.
- Send emails to members who have made any modifications to the code in the most recent build process.

CruiseControl.NET also provides a web site and system tray application to monitor the status of the continuous integration server. We can visit the web site any time to view the build reports of all previous builds. The system tray icon can be installed by all the developers in their machines, which will show the build status by turning icon to various colors (Green - success, Yellow – Build in progress, Red – Build failure)

3. Unit Testing

I wrote my first unit tests for a small project for Microsoft Corp, USA. When we started writing the unit tests using NUnit, we all cursed the guy who invented it! We had to write several hundred lines of code to implement the unit tests for a small project.

But it paid off in the end. We had to make several changes to the project during the ongoing development. And it was a matter of few seconds to identify if any of our change broke the existing code.

My current project ‘NxOpinion’ takes advantage of the NUnit a lot. We have a pretty big suite of unit tests – around 600 tests ensure that our project is still healthy during continuous integration process.

We have configured the continuous integration server such that if a unit test fails, it will be treated as ‘build failure’. This will send a ‘build failure’ report and the members will become immediately active to fix the issue. This helps a lot in keeping the project healthy.

If you are not familiar with automated unit testing, search for ‘automated unit testing’ in google. You can find several good articles. Also visit – a good tool for automated unit testing in .NET development.

And finally the fun part - the Ambient ORB showing the health of the project

You may be curious to know about the ‘ambient ORB showing the health of the project’. In our current project ‘NxOpinion’, our project architect Mike Swanson, installed an ORB in the development room. This ambient device has some built-in wireless system and it can receive signals from internet. Based on the signals it receive, it can change into a Red light or green light (or, any other color we configure)

Our continuous integration process will call some URL in the provider’s site ( with some query strings indicating success or failure in the build process. Based on the parameters we set indicating our build result, the ORB will turn the color. That means, if there is a build failure, we all see a red light in the development room, alerting all members of the team.

If any of these steps in our continuous integration process fail, the build script will call the ORB provider’s URL with the query string indicating the ‘failure’. The provider will transmit appropriate signals to the ORB and the ORB will change the color to RED or GREEN.

So, if any one check-in a piece of code which breaks the build process (which includes compiling all files and running the unit tests), we will see a red light in the room and all developers shout ‘who broke the build?’

Ref: - NxOpinion - Video case study about NxOpinion in Microsoft website - Continuous integration - Ambient devices - Mike’s blog about continuous integration and ambient ORB - Cruise Control for continuous integration – NAnt for build scripts. - Perforce, the source control.


Author: kavitha chowdhary11 Oct 2004 Member Level: Silver   Points : 0

Thanks Mr.Tony! This is a very useful article.
Thank U

Author: Ashwini B Thakur12 Oct 2004 Member Level: Bronze   Points : 0

I found this article very useful.
Thanks a lot.

Author: Ravi Makwana12 Oct 2004 Member Level: Bronze   Points : 0

Thanx 4 such a valuable information which gave a new angle of view to the software professionals.

Author: Venkat Avunoori18 Oct 2004 Member Level: Bronze   Points : 0

Thanks for talking in detail about continuous integration....useful article

  • Do not include your name, "with regards" etc in the comment. Write detailed comment, relevant to the topic.
  • No HTML formatting and links to other web sites are allowed.
  • This is a strictly moderated site. Absolutely no spam allowed.
  • Name: