Monthly Archives: July 2009

Is BDD the solution to the tester’s dilemma?

We, testers, are coming late to the agile party. I would argue that the role of a programmer is first class in Extreme Programming, the role of a team lead first class in Scrum, and the role of the architect first class in Feature Driven Development (FDD). I know, I am oversimplifying and of course each of these roles requires a diversity of skills to fit an always changing job description, and some even got a name change (Scrum Master).

But what about testers…

I know of a team in which every member is called a programmer, not a developer or a team member. Do you believe it? It is true, but isn’t that strange?

What if said “I know of a team in which every member is called a tester”. Would you believe it?

Brian Marick was one of the original authors of the Agile Manifesto and the lone self-defined tester in the group.  He also made big contributions to the redefinition of our role, and others follow the path, but we have had what I consider (re)foundational book, Agile Testing, for less than 6 months now.

We are still in transition as a community . This transition is sometimes caused by a lack of understanding of Agile Development and sometime it is caused by the lack of breakthrough innovations.  There is some re-thinking going on, both in the broader sense (see Brian Marick on Micro-Scale Retro-Futurist Anarcho-Syndicalism), and in a more restricted sense (there is a group, AA-FTT – Agile Alliance Functional Test Tools, that is trying to advance the state of the art, you can see the discussions on the mailing list and a there are a number of workshops [1][2]).

So it might be too early to have a solution to the tester’s dilemma, but anyway, we have to continue to test and deliver products today. What is the best we can do right now?

Let’s discuss a promising proposition: BDD and Cucumber, as presented in The RSpec book: Behaviour Driven Development with RSpec, Cucumber and friends.

But first we need a little of history and background.

Test driven development (TDD)

TDD was a wonderful idea that came to us in the XP package. It is widely accepted as a good practice but isn’t so widely used.  Why?  TDD had the positive effect of pulling up quality, both as a concern of the team and in the product result. But the four letter word (test) caused a lot of friction and misunderstanding.  The friction often appears as a resistance of programmers to do “tester’s job”.  The misunderstanding could be caused by the way people learn new concepts. If the words are known, they usually assume that the concept is the same (see Sapir-Whorf Hypothesis).

That’s why, instead of Agile Development, Brian turned to Micro-Scale Retro-Futurist Anarcho-Syndicalism (reading that, you most certainly won’t have a clue what he is talking about, so you will go and check) and Kent Beck turned to Responsible Development (he prefers the connotation of responsible; being responsible has some cost attached; being agile, well … who doesn’t want to be agile?)  As a result of getting TDD wrong, some practitioners focus on testing the code, while they should be paying attention to the design. This loss of focus leads to bad practices; for example, when a developer defines tests that depends on the implementation (internal structure) instead of the behaviour of the object.

Acceptance Test Driven Development (ATDD) …

You might like to know that software development and theater companies have many things in common: “Good theater companies routinely produce innovative products under extreme deadline pressure (opening night) ”. Or at least, they could have. To get those results (reliable innovation), you have to build products through what Rob Austin and Lee Devin called Artful Making, instead of Industrial Making. This is not a case of wishful thinking, you need some precondition to use Artful Making. One of them: Quick, low cost and high quality iteration, and a product at the end of each iteration (yes, this sounds very Lean).

One impediment to Artful Making is the language/conceptual gap between the user and the development team. If we have different language, we incur in translation time and cost and we risk to inject error every time that we cross the gap… in any direction. For instance, during requirement elicitation and during acceptance test.

Would it be nice if we could have requirement stored in such a way that:

  • They are understandable for both the users and the development team (communication).
  • They could be defined (written) before the product is build (test).
  • The product could be quickly and easily (automatically) validated against them (test).

Enter Fit: Framework for Integrated Test. What if we let the user write the requirements as examples tables in Word, Excel or HTML pages? And then, with a minimum gluing code and a test runner, we could test the application.

It was a great leap forward, both in communication and test. It has been followed by FitLibrary and Fitnesse which add a wiki as the front end (instead of HTML).

A lot of agile teams use some incarnation of FIT/FitLibrary/Fitnesse.  Some problems where reported:

  • Many end user don’t take ownership of the tests. Team members (usually the testers) act as scribes, and translate user / team conversation to HTML /wiki.
  • When the number of test cases scale up, test case development become awkward. A wiki or a set of html pages are not a good development environment. As a developer, you have to manage two models of thinking (FIT/Fitnesse and your application programming language and environment)

…And Behaviour Driven Development (BDD)

BDD is an evolution of TDD and ATDD proposed by Dan North (video).

We can get rid of Test in TDD, and call it Behaviour Driven Development. But why Behaviour? Because it is better to focus the programmer in the more important aspect of the objects, their behaviour. It is also built upon the idea of a ubiquitous language as in Domain Driven Design.

To complete the name/concept switch away of Test, the executable specification are called examples. So specifications of code behaviour are called code examples and the higher level specification are called application examples.

At the requirement/acceptance level, we define application behaviour, at the design/programming level, we define object behaviour. In both cases you follow the red/green/refactor cycle, having two, concentric cycles. The outer cycle is the ATDD part, the inner cyle is the TDD part, but the both work together. These are the steps, number between () refer to the figure extracted from The RSpec Book from The Pragmatic Bookshelf.

  1. Write an scenario with Cucumber (1)
  2. Write a step definition
  3. Run and Watch it fail (2)
    1. Write a failing code example with RSpec (3)
    2. Get the example to pass (4)
    3. Refactor (5)
    4. Repeat 3.1 – 3.3 until Cucumber step is passing
    5. Repeat 2 – 3 until Cucumber scenario is passing (6)
    6. Refactor (7)

Building the application in this way promotes an Outside-in development. Models appear as solutions to the acceptance criteria. This has a few implications: it is easier for the client to write acceptance test (as examples), the model is more relevant to everyone on the team, and it requires the use of techniques to support incremental development (like mocks).  The bottom line of BDD is “writing software that matters”, and it follows three principles:

  • It’s all behaviour. This behaviour should be described in a common (ubiquitous) language, that can be understandable by the business and technical people on the team.
  • Deliver stockholder value. Brian Marick will say “focus on the cheese”.
  • Enough is enough. This is true for upfront planning, design, etc, but also for automation, don’t try to automate everything.

There are many tools implementing BDD ideas with different approaches and contexts (i.e. languages). In Ruby, the tool stack includes RSpecCucumberWebrat. I will only drill into Cucumber which is the BDD tool most relevant to end user and (non technical) testers.


I will not try to explain the use of Cucumber. Following the excellent Ruby/Rails tradition of web/podcast you can check some Cucumber resources here and of course, check the Cucumber site and The RSpec Book.

Disclaimer: Please consider that I am reading a beta version of The RSpec Book, a nice feature of Pragmatic Bookshelf, and I haven’t research Cucumber as much as I would like, so I might be missing many features. I just want to share some impressions as a newbie user.

Just to agree on vocabulary, this is a short example (login.feature):

Each feature file contains:

Feature: Login
In order to use the site functionality
As a guest user
I want log in

Scenario: I try to access without logging in
Given I am not logged in
When I go to the taskboard number 1
Then I should be on the login screen

  • One feature, with a free description (not parsed), that could take traditional story format (As… I want … So that I can …) or another format that change the order to put more emphasis on the value obtained (In order to… As a … I want …). The last format is closer to the order of the steps.
  • One or more Scenario, each scenario has a name and a number of steps. Each step begins with any of five keywords: GivenWhenThenAnd, and But.
    • Given: define the context, the preconditions.
    • When: is the action or event that happen to the application.
    • Then: is the expected result.
    • And/But: could be used after Given/When/Then to add steps with the same meaning of the previous one.

Feature files are written in a very simple DSL (Feature/Scenario/Given/When/Then/And/But) called Gherkin. Like Python and YAML, Gherkin is a line-oriented language that uses indentation to define structure.

My first impression, coming from FIT/Fitnesse, is that it is not so different, I mean, you can also have a Do fixture that can be read almost as English. Almost.

Examples written in natural language (like English, but many languages are supported) is a big selling point. And you don’t lose the table (data driven) approach (Scenario Outlines).

It is far easier to edit, just plain text and it seems to be more integrated to the development environment. I really like the ability to switch more easily between application examples (feature file) and code.

Working in a Ruby environment also allows a more quick and simple cycle that I think could help when pairing with a user.

It’s sad to say that I don’t have experience with large set of features. One of the problems with FIT/Fitnesse is that it is more difficult to refactor examples than to refactor code. And sometimes, you unwillingly break the examples by changing an apparently innocuous html. I don’t know how scaling and refactoring is managed in Cucumber (except the Scenario Outlines).

One downside of plain text is the loss of styling and hyperlinking. The first one could be replaced by convention over configuration and some viewer, but without hyperlinking, we lose the ability to arrange examples as a nice snippet of documentation.

Final comments

In the BDD double cycle, you can extend the use of Cucumber to make code examples or extend RSpec to make application examples.

The questions that come to my mind is, why developers will not choose to use RSpec, so they keep the toolset simpler. Anyway, they don’t need English if they can read Ruby.

In my experience, the cost to maintain the gluing code is only acceptable when there are full time team members that are not fluent in the main programming language (say Java or Ruby). It could be the user or somebody working as the user scribe (an Analyst or Tester).

I see that Cucumber has many good point, and a superficial use is not enough to judge it. I would like to give it a deeper try. Do you have a project Rails? Call me 😀

Will testers role in agile team become the user scribes of functional tests? Yes, probably. But not only that. Exploratory testing is much needed, and a devil’s advocate is useful for the team. For instance, related to coverage, test automation and risk analysis.  The Agilar Taskboard team (Francisco Tufró & Marcos Navarro) allows me to use the product as a test bed for Cucumber.  Thanks!

About the author: Juan Gabardini has more than 20 years of experience in IT management, IT product development, teaching and consulting in financial services, retail, telecommunications, medical equipment and manufacturing sectors.

Juan is currently focused on testing and coaching the Agile way, and building an Argentinian and Latam Agile community (Agile Open Tour and Ágiles 200x).

Juan is a member of IEEE, Agile Alliance and Scrum Alliance, and holds degrees in System Analysis and Electronic Engineering from the Buenos Aires University, where he also is teaching. You can find him (in Spanish) in and

Tagged ,