During our BDD Testing Skills webinar last week, a number of questions were asked. Kudos to Jeff for taking time out to answer as many of them as he could, below:
We release product versions via a waterfall-ish 3 month full dev/test cycle for new versions.
Can BDD be applied in this eco-system / release management? [Ben Balot]
Following BDD would likely add value in this environment. Its biggest value is in helping negotiate the specifics *and* the scope of a product for the next release cycle.
Would it be right to assume BDD is applied to Unit Testing only? [Anon]
Let’s clarify the terminology first. “Unit testing” is a somewhat overloaded term. Most agile teams characterize unit testing as a programmer activity where tests focus on a tiny, isolated logical unit of code. Using this definition, then, BDD does not drive the creation of unit tests, but instead tests that demonstrate end-to-end functionality. Classically, I believe “unit tests” were typically larger units. In an attempt to resolve the ambiguity, others call these focused tests “microtests.”
If we take the older possible definition of unit tests to mean “functional tests” that verify user end goals, then I’d answer “mostly yes” to your question — BDD is primarily used to negotiate these unit behaviors. That’s the “B” in Behavior-Driven Development. But there’s nothing that says we can’t also define other tests up-front using a BDD-like collaborative process.
Get TestRail FREE for 30 days!
For the company that had too many tests – what happened? Are they OK now? [Mark Cole]
They were on a path to clean up their act as I left this team (c. 2013). Last I heard–a couple years ago–they’d been able to pare down the number of tests, partly by simply deleting a considerable number of them. They also introduced some mechanisms to halve the average test time, by using a more direct manner (than UI driving) to create policies. This team had also not fully embraced the “collaborative” aspect of BDD–it was typically more efforts that derived from the QA folk, and no one else had a lot of visibility into the process. Things picked up a bit once teams learned it was everyone’s job to ensure they returned on value.
Do we need BDD only for the cases which requires user interaction in UI? [Selamawit Demewoz]
We want to use BDD to define any interesting behaviors that the business is interested in. That might involve transforming a datasource in an ETL process, for example. The level at which the behavior is interesting might also change. Think about a system with a reasonably small front end–let’s say an airline reservations system where you shop for and book flights from multiple carriers–but considerably complexity behind the scenes. (An “iceberg” system, perhaps). There are tens of thousands of rules that combine to govern what flights/fare are available, so most of the interesting behavior we’d need to describe would be at the level of the individual rule. But the aggregation of rules is simple — a flight at a certain fare is either available or not.
When do we stop writing BDD scenarios for a single user story? sometimes there are 1 or 2 or 3 scenarios sometime 10. do we need to write all the scenario or we need to breakdown the story? [Selamawit Demewoz]
I view deriving the scenarios as part of the process required to understand just what a story means–what’s its scope? Part of building a successful system is to use this as an exploratory process–let’s think outside the box about some crazy things that a user might try, or a creative manner in which we can add more business value. Obviously there’s also the value of making sure we cover all typical boundary and exceptional behavior. This is also why I recommend not detailing the scenarios until we have a sense of the boundaries that we’re willing to tackle for now.
I sometimes think of BDD as a process that replaces a conversation–a story–with a set of artifacts (scenarios) that capture the details of the story, by example.
What are some tips when trying to get good feedback from the Business Analyst while reviewing the scenarios? [Dwayne Green]
To me it’s just a process of asking questions, and treating this as a team sport, not an “us vs them” thing. Our job as a team delivering product is to ask enough questions to help firm up the PO/BA’s idea for the product. What are they not thinking of–from a system’s perspective or a users’ perspective–that we can help with?
If their story isn’t making sense to us, that’s the whole point of using examples. Initially the BA’s/PO’s might find it different or even awkward to think in terms of examples, but they’ll start to get it. Once they give you one example, turn around and say, “OK, now, does the behavior change when I change *this* value? What happens if I tweak it–omit this step, for example?”
I think the more we do this as a team, and the more incrementally (i.e. this needs to at least be a part of what we do in a sprint or story planning session), the more we iterate our learning. It’s a continual conversation, and I think we have to learn to get good at negotiating things; doing this in short cycles will help.
Is it good practice to include all scenarios (scenario titles only) in the user story to help devs get a better understanding on what to look out for? [Nabeel Ibrahim]
At the point where we get a feature to the iteration or day on which the team should start building it, the information should start shifting from JIRA or wherever we capture some of the conversation, and moving into the primary tool we’ve chosen for capturing the scenarios / examples that come out of BDD. Maybe think of the set of scenarios as your primary artifact for capturing documentation (and tests!) about the behaviors of the system. So as soon as you start understanding what the scenario titles should be, capture them in your feature files (if you’re using Gherkin / Cucumber) or Fitnesse pages, or whatever tool you use.
If it’s not clear, everyone should have access to this source of record, and should feel empowered to improve them as we go.
Join 34,000 subscribers and receive carefully researched and popular article on software testing and QA. Top resources on becoming a better tester, learning new tools and building a team.
How long typically does it take for a company to be practicing BDD? [Sarah Kingswell]
We can usually get something running in the first sprint we decide to tackle this. It’s an easy concept for most POs/BAs (and really everyone); the challenge is getting everyone to commit to supporting it. I’ve sat with non-technical folks, showed them a few examples and talked for an hour or so with them, and they came back a few hours later with a pile of their own examples from their system. “Great, let’s clean up this,” I’d tell them; they’d come back with dozens of decent looking starter examples the next day.
Sometimes the challenging part is automating them, which is a secondary (though valuable) goal. Depending on how an existing system is designed, this can take a few days or even a sprint or two to shake out.
How to link BDD execution results (automated) and manual tests in a testrail test plan? [Christophe Lemoine]
See comments and links in the previous BDD Testing Skills Webinar post.
What is the difference between acceptance tests and acceptance criteria put in user stories [Solomon Lingala]
Usually “acceptance criteria” means a brief English description of what needs to happen before the business can accept the feature. Acceptance tests are the concrete realization of this–the actual examples that we automate. At some point, the notion of acceptance criteria goes away. We have a conversation around a feature–a story–and we shape that into the actual executable examples, which remain our artifact. Do this well and we minimize the vague summaries of behavior that we capture in, for example, JIRA.
How many testers required per developer for a product of high complexity to support BDD process ? What is normally testers to developers rate for simple, meduim, complex, extream software complexity? [Danylo Prokopiv]
If this ends up being a full-team effort, i.e. POs, BAs, devs, testers, then everyone is part of being responsible to ensure we capture all needs in the form of a set of appropriate scenarios/examples. I don’t see it as increasing the number of QAs at all over what you already do–it is a different view on testing, and in some ways the effort shrinks a little for QA, allowing you to spend more time on other forms of testing.
- Who should be writing user stories (is it BA only)?
- Who should be creating feature files (is it QA only)?
- How do you convince the IT/SMT to adapt BDD environment?
- Should BDD be automated? If yes, what tool should be used to automate bdd?
- Is ATDD and BDD the same?
- Did I get this right that BDD in general is “”Use the Test to Drive the Development””?
- Can we do Data Driven Testing in BDD?” [Qureshi,Zaigham]
Most stories should be coming from the business side of the fence, expressing a customer interest in what the product should do. Other parties can (and should) certainly introduce stories for consideration, but hopefully your organization has established a policy for who gets to prioritize / decide the work actually done.
It is a process built around continual negotiation.
In a similar vein, QA often spearheads initiatives around creating feature files, but all on the team should be contributing. Sometimes it’s the PO/BA side that initiates the implementation of these, and I think that’s an even better approach.
IT/SMT: In a healthy organization, the PO is defining the product and criteria for its acceptance. They have every right to decline something built if they do not trust its ability to meet their needs. So in some senses, IT buy-in shouldn’t be a question.
There are enough benefits in doing BDD from their stance, the least of which is another level of confidence that they’re going to ship another embarrassing defect. As a developer myself, I love it because it answers the question for me, “Are we done yet.”
Usually you should approach BDD with the mentality that you’ll automate what you can possibly automate. It’s not always essential but it is where things can take off–without automation, continuous delivery (for example) is not possible.
ATDD and BDD are similar enough that I don’t worry about the nuances. The notion of driving development with tests (examples that we automate) that are defined up front is the same. BDD brings some mentalities around collaboration and domain-driven design to the mix.
Correct, I think you can view BDD in some sense as test-driven development; the preferred way to phrase this nowadays is “specification by example.” (See the book by Gojko Adzic.)
Yes you can do data-driven testing in BDD. Tools other than Cucumber might be more appropriate. Fitnesse, for example, has you express your examples in the form of decision tables.
I’m new to BDD. So I scenario desc before is like a Story. How is one BDD Story ties to another dependent story . And how do you manage size of scenario or story? [Ali Rizvi]
To clarify, a scenario is effective a test case showing a specific example; the example exists to describe how a goal is accomplished in the system.
One scenario could potentially cover aspects of various stories, so there’s not a right answer around dependencies. Think about the various ways people can use your software to accomplish goals (or not)–the set of these ways should be the set of scenarios in your system.
That aside, scenarios in general should be to the point, focusing on one aspect of behavior as much as possible. You’ll know they’re too large if the scenario description can’t be stated concisely in less than about ten words. Use of “and” in the scenario description or even the steps can be a hint that the scenarios are too long.
Presenter must love meetings… [Martin Schrick]
I actually find most meetings a waste of time. With the notion of “getting everyone together,” the idea is to avoid the inevitable waste when the right people are not there. I view the BDD three-amigos sessions more as collaborative work, not meeting.
I’ve also been doing a good deal of whole-team mobbing in the past couple years, something I thought was a waste of time not long before. I’m fascinated about how it generally goes faster than other modes (pairing or solo work) and is more enjoyable. One reason is that we’re not wasting time on mis-interpretation, or weak attempts to avoid it by writing things down. In turn that means we’re also not wasting time on ineffective post-facto reviews (too little too late).
I find it best to start with everyone for at least a few sprints. You’ll figure out who doesn’t need to be there as you go, and sometimes that tells you about who is providing the true value on the team.
If we are building detailed scenarios later, in that case the PBI is open however the developer may move to another PBI. Which affectively makes it a mini waterfall. [Rohit Sharma]
You can turn anything into a mini-waterfall. This should be “JIT,” and even better, think about BDD as a way of doing testing + documentation + specification + design all simultaneously.
You mentioned about Cucumber. What are other preferable tools which can be used for BDD? [Madhav Shelke]
Beyond Cucumber / Gherkin, Fitnesse is a tool that many BA’s appreciate for its resemblance to spreadsheets.
What advice would you give for a team in the beginning steps of implementing BDD? [Luke Skinner]
Enjoy. Don’t over word-smith, but do start to learn how to be compulsive about phrasing. In other words, it doesn’t have to be perfect up front to proceed, but you should always be reflecting on things that are there and seeking to continuously improve them. Learn when to stop the up-front speculation about things. Treat it as a just-in-time process.
Retrospect on it. Don’t hesitate to have a brief chat at day-end to discuss something you feel is amiss with the process or scenarios / examples.
In a team of 5 developers and 2 qa guys, who do you think has the responsibility to automate the bdd scenarios? The qa or the developer? [Gabi Udrescu]
I’ve seen both, though in most teams it’s the developers. If QA has the technical capability, great. Even better: Have them pair up on it. This will help ensure that the tests are bona fide (see my article on the ABCs of acceptance test design).
What about Exploratory Testing? Where does that fit in the pipeline? [David Vydra]
By definition, actual manual exploratory testing has to occur after the software is built. Still, parts of the “test plan” for exploratory testing can be built up-front as part of the BDD process–it can be valuable for QA to collaborate and get the insights of others about things to try. Remember, automating all the other tests frees up more time for exploratory testing.
Is it a good idea having BDD tests as Gate Keeper on CI? [Aleksey Zevalich]
Absolutely. The set of BDD tests represent the behaviors needed for the system. If they’re not passing, we shouldn’t be shipping our system or even adding additional features to it.
Who should be writing BDDs? 1)Testers 2)PO/BAs or 3) Client? [Atiq Kamran]
All of the above should be involved in writing and honing the examples (scenarios) involved in BDD. Often it’s the PO or BA who initiate it, but it is best accomplished as a joint effort.
I have seen teams that often put any kind of values (test data) in an excel instead of in the gherkin to work with their automation frameworks. is it a right approach to write ur gherkins this way? [Saunak Tarafdar]
I would take a look at Fitnesse if this is the tendency. Cucumber also provides support for table-driven tests.
Don’t chose a tool that doesn’t mesh well with the way you work or like to express things. There are plenty of additional tools out there; you can also build your own if that’s best for your product. That’s what we did in 2000 before these tools existed.
Which style of bdd is preferred- imperative or declarative? [Saunak Tarafdar]
Declarative, absolutely. Tests should describe examples without bogging down in implementation details. They should tell a story about the business needs, not re-create stepping through the specifics of an application’s implementation.
Imagine having built a web application in a very imperative fashion–rote-wise steps to emulate clicking through a UI. Now imagine deploying a new client to mobile. The business behaviors are the same, yet the imperative set of scenarios are no longer applicable. That’s not what you want.
Should api tests be implemented in BDD? [Saunak Tarafdar]
Do you have a customer for your API? Do you want to publish up-to-date documentation for it?
I would likely not use cucumber for this, but collaborating on API design up-front is a great thing to do–these are the interface points between clients and your system, so all the more reason to get agreement on them.
Post by Simon Knight, Gurock Product and Community Manager. Follow him on Twitter or LinkedIn.
Additional content by Jeff Langr. Jeff has spent more than half a 35-year career successfully building and delivering software using agile methods and techniques. He’s also helped countless other development teams do the same by coaching and training through his company, Langr Software Solutions, Inc.
In addition to being a contributor to Uncle Bob’s book Clean Code, Jeff is the author of five books on software development:
- Modern C++ Programming With Test-Driven Development
- Pragmatic Unit Testing
- Agile in a Flash (with Tim Ottinger)
- Agile Java
- Essential Java Style
- He is also on the technical advisory board for the Pragmatic Bookshelf.
Jeff resides in Colorado Springs, Colorado, US.
Test Automation – Anywhere, Anytime
