CI dot Pivotal Labs dot com

At Pivotal Labs we take Continuous Integration (CI) seriously. Every project has a dedicated machine that serves as a CI environment. Each checkin on the project causes a build to be kicked off. A “build” means checking out the code from scratch and running of all the project’s tests, which, for a Rails project, means unit and functional tests, JavaScript tests and Selenium tests. For the JavaScript and Selenium tests, we run multiple browsers on multiple OS’s (e.g. IE 7 on Windows XP, FF 3 on OS X, etc).

We consider it critically important to keep each project’s build green (i.e. successful) at all times. A build is the heartbeat of the project: if it’s green, everything is healthy; if it turns red (i.e. fails), the team is encouraged to jump on the problem and get it back to green right away. We want red builds to go away quickly; the longer a build stays red, the longer it takes to track down the problem and the more likely it is that additional tests will be broken (the “broken windows effect”).

In order to facilitate this level of discipline, we’ve learned over the years that making the status of our CI environments obvious and visible to the team is critical. If a team isn’t acutely aware of the status of its build, it’s unlikely that a red build will get noticed and fixed quickly. You can have the CI server email the team, but that doesn’t work very well when the whole team is pairing all day: it might be a few hours before someone notices the email. You can install plugins in your browser or system tray that show build status, which helps, but still, they’re not always obvious enough. The best way we’ve found to keep the team informed is to display the status of the build high on a wall near the team as a big red or green indicator. That way, even when you’re busy coding, it’s easy to notice the build going red. These days we use 2 wide screen TVs, positioned in the office so that one is easily seen from any developer station.

When there’s only a single project going on, we’ve found that a screen that’s simply all red or all green is effective. At Pivotal Labs, though, we have many projects going on at once. Rather than putting numerous TVs up on the wall, we’ve created an application that aggregates each project’s CI status into one page. It’s only visible internally, of course. It displays the build status of each of our projects – all the client projects, internal projects, and open source projects that Pivotal Labs is involved with.

Recently we decided to bring up an external instance of the aggregator that shows the status of our open source projects. We’ve also pulled in the status of some of the open source projects that Pivotal depends on (e.g. Rails, CCrb). It’s available at ci.pivotallabs.com. The idea is to provide the same level of visibility into build status for open source developers, or teams that rely on their products, as we have internally at Pivotal Labs. Feel free to display the page on a monitor/TV/projector in your office! It refreshes itself every 30 seconds.
If you have an open source project and you’d like us to run your build and display its status, or if you already have a build and you’d like us to add it to the page, there’s no charge – just let us know (email contact@pivotallabs.com).

9 Comments

Hah, yes, those are cool. We’ve had some success in the past also with the [Ambient Orb](http://www.ambientdevices.com/cat/orb/orborder.html), and I’ve seen people using lava lamps, Christmas lights, etc. Whatever is super-visible, impossible to ignore, and updates as soon as the build fails is great.

I’m curious as to if you have you experimented with distributing this at all? We’re seeing build times in the 45 minute range (unit, integration, selenium, screw-unit) which is obviously too long of a feedback loop. Do you feel a similar pain?

Maybe I missed it but what CI tool do you use at Pivotal for your Rails projects? CruiseControl?

March 27, 2009 at 1:40 pm

Chad Woolley says:

@Sam – Yes, we use cruisecontrol.rb (you can see by clicking through the “visit” link for each project hosted on our server.

I’ve looked at Integrity some, but the main thing it doesn’t offer is polling. This is an important feature, both to monitor repositories we don’t own and can’t set up commit hooks, and to ensure that builds still happen even if the CI server happens to be down when the commit hook fires.

March 27, 2009 at 4:29 pm

Edward Hieatt says:

@Bruno Michel – We don’t have plans to open-source it right now. The current goal is to put it out there to help the community get visibility into the status of open-source projects’ builds. If you have an open-source project that you’d like us to add, do get in touch.

March 27, 2009 at 8:44 pm

Chad Woolley says:

@Josh

About the distributed builds – I’ve thought about that a lot. I think
we don’t have issues at Pivotal mainly because:

* We have shorter-term projects with fast builds
* We work to keep our builds fast
* We have small teams, so slow feedback isn’t too bad, we just deal
with any redness and communicate the workarounds. On larger teams,
with more integration (and communication) issues, redness is more of a
problem.

As Chief Operating Officer, Edward Hieatt oversees all operational aspects of Pivotal Labs’ consulting business including engineering, client services and the company’s open source product strategy. In this role, he drives revenue growth and performance, and is responsible for managing all regional offices across the US and overseas.

Edward ensures that Pivotal Labs' development teams identify, anticipate and exceed client expectations for their products. His engineering organization is acclaimed for delivering rapid, high-quality, sustainable, iterative development services.

With experience as a software developer and an engineering leader responsible for delivering web and mobile applications as well as large-scale enterprise infrastructure projects, Edward brings deep insights to the COO position. After joining Pivotal Labs as Principal in 2003, Edward was responsible for leading dozens of client engagements, and was instrumental in defining the company's development process and methodologies. Edward was promoted to Vice President, Engineering in 2008, and subsequently oversaw Pivotal Labs' expansion into the New York and Boulder, CO, markets. Edward was also a key contributor to the development of Pivotal Tracker, the agile project management and collaboration tool used by thousands of software developers worldwide.