Basic expectations for acceptance test framework

In previous post I have described a story of working with acceptance tests and how I encountered problems that motivated me to create LightBDD framework. In this post I have written about different types of tests, especially describing nature of acceptance and end-to-end tests. Now, I would like to focus on my observations regarding requirements for a framework allowing developers to work on behavioral tests effectively.

Basic requirements

During work in different companies I have realized that the expectations for acceptance tests and testing framework depends on the company size and its culture. The first team in which we started looking on improving our testing tools was a part of a small company with a very informal culture. Product Owner and Quality Assurance were dedicated to our team and were pairing with us in order to formulate scenarios that fulfill their expectations but also fit to the system architecture. They were interested how our acceptance and end-to-end tests look like. Both tests were having only one purpose at that time – to ensure that our software works fine.

That was a time when we have realized that tests written in SpecFlow were too difficult to maintain (I have described reasons previously). We have started asking questions what we really need from a testing framework.

Clear tests

The first set of questions were related to the fact that we were receiving requirements from PO/QA in form of business scenarios. We wanted to be able to quickly response PO/QA questions like:

Is this scenario already covered by tests?

What is this test checking exactly?

We thought that the best option would be to model our tests to allow to preserve a nice given-when-then that PO/QA were preparing for us. If our tests would reflect provided scenarios, as close as possible, they would be easy to present to PO/QA but also would be easy to read and understand by developers.

Maintainability

With the knowledge about maintenance problems related to tests written in frameworks like SpecFlow / Fitnesse, we realized that it was a crucial requirement for a testing framework. At that point we knew that it is a tricky problem, because maintainability issues reveal after a longer period, when project grows a bit. It is safe to say that project consisting of 1 scenario written in any testing framework looks easy to maintain, but would it be the same if there are 30 different scenarios? What if there are even more? All projects evolve (unless they are dead), so do the tests. Some of the scenarios become no longer applicable and are removed, some are added, while others are extended or shrunk with few steps. Finally some scenarios may become more precised or generalized, so their steps would just be altered.

All of those changes brought a following questions that we started considering in our design decisions:

How easy would it be to add a new scenario?

How easy would it be to add or remove steps to any given scenario?

How easy would it be to rename scenarios or steps?

If scenarios are removed, how easy would it be to clean methods that are no longer used by any scenarios?

How easy would it be to restructure and reorganize test suite?

If project has 5, 30, 100 scenarios, how long it would take to apply those changes to all of them?

By how easy we mean:

• how many manual steps have to be taken by developer / PO / QA in order to apply change?
• are all of those steps have to be applied into one place / project / location / repository, or they have to be made in different places?
• how long it would take to apply such change?

Clean code

Maintainability does not refer only to changing code. It is also about:

• understanding existing tests by new people in a team,
• investigating why they are failing,
• checking which scenarios are still valid after changed requirements.

It brings a following questions to be answered:

How easy would it be to understand how given scenario works?

Is it possible to analyze scenario flow, without debugging it?

How easy would it be to debug given scenario?

We wanted to have a framework that:

• does not require using literals with regular expressions everywhere,
• does not generate a bunch of files with unreadable code,
• does not use loose binding between scenarios and underlying methods,
• does not require usage of static contexts or any complex constructs to pass state between scenario methods,
• has an intuitive behavior,
• is easy to navigate with Visual Studio.

Traceability

Previously, I have mentioned that acceptance tests covers much wider scope than unit tests. During investigation of failed acceptance or end-to-end tests we have often been asking questions like:

What was the test stage when scenario has failed at?

Which operation performed on GUI failed the scenario?

Which component on end-to-end journey behaved incorrectly?

We wanted a framework that would allow those questions to be easily answered at the first glance, without spending minutes on analyzing logs and stack trace.

Execution progress monitoring

Acceptance tests are slow. End-to-end tests are even slower. All of us has spent so much time staring at Teamcity, waiting for tests to finish in order to close a ticket, to release project on production or to finally go home leaving board green. So many times it occurred that some of those tests were broken, causing whole build to fail. Those failing builds were taking much more time to execute than the ‘normal’ builds, making waiting even worse (I have described reasons for this behavior in Test characteristics section of this post)… If we only knew what was happening with those tests, we could immediately detect issue, stop the tests, fix it, rerun them and go home… Of course, during fix we were adding more Console.WriteLine() or _log.Debug() statements to the test methods to detect those problems much faster next time, but there were always some places where such logging was missing. Also, the practice itself was not good, because it made whole tests code less clear to read and required additional typing.

So, what we really wanted was a framework which would allow to answer a following questions without the need of any additional developer intervention:

What is the progress of tests that are currently being executed on CI?

Why current execution takes 2 minutes more than normally?

What are currently executed tests doing now?

Are those tests just slower but still passing, or is something horrible happening with them?

Simple solution is the best one

All of those requirements that I have just described could make an impression that we wanted to have a very complex, sophisticated framework and it would take at least a year to build it – it was exactly opposite! The first version of a testing framework that fulfilled all of those requirements, consisted of a class with 1 public method in total. It was quite difficult to call it even a framework…

Within a week, after a few design meetings we came with the idea to use a standard NUnit Framework with a few conventions to write our acceptance tests:

• reflect a Given-When-Then scenario name as a test method name,
• represent each scenario step as a method call in test,
• name each step method the same as step in scenario (replace spaces with underscores),
• wrap all steps with a RunScenario method, so step methods could be passes as delegates that would allow to omit brackets and will allow to display execution progress,
• separate all test implementation details from test by using partial classes.

An example scenario taken from Wikipedia page:

Story: Returns go to stock

Scenario 1: Refunded items should be returned to stock
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock

would look like as follows:

[TestFixture]
public partial class Returns_go_to_stock
{
    [Test]
    public void Refunded_items_should_be_returned_to_stock()
    {
        _runner.RunScenario(
            Given_a_customer_previously_bought_a_black_sweater_from_me,
            And_I_currently_have_three_black_sweaters_left_in_stock,
            When_he_returns_the_sweater_for_a_refund,
            Then_I_should_have_four_black_sweaters_in_stock);
    }
}

with the example implementation as follows:

public partial class Returns_go_to_stock
{
    private Item _sweater;
    private Shop _shop;
    private BDDRunner _runner = new BDDRunner();

    [SetUp]
    public void SetUp()
    {
        _shop = new Shop();
    }

    private void Given_a_customer_previously_bought_a_black_sweater_from_me()
    {
        _sweater = new Item(ItemType.Sweater, Color.Black);
    }

    private void And_I_currently_have_three_black_sweaters_left_in_stock()
    {
        _shop.Stock.Add(new Item(ItemType.Sweater, Color.Black));
        _shop.Stock.Add(new Item(ItemType.Sweater, Color.Black));
        _shop.Stock.Add(new Item(ItemType.Sweater, Color.Black));
    }

    private void When_he_returns_the_sweater_for_a_refund()
    {
        _shop.Refund(_sweater);
    }

    private void Then_I_should_have_four_black_sweaters_in_stock()
    {
        Assert.That(
            _shop.Stock.Count(i => i.Type == ItemType.Sweater && i.Color == Color.Black),
            Is.EqualTo(4));
    }
}

The BDDRunner.RunScenario() method was responsible for doing two things only:

• executing step delegates in provided order,
• printing step name before it’s execution.

That’s it!
So, how all requirements were fulfilled? – Lets see:

Requirement	Solution
Clear tests	Used conventions allowed PO/QA easily understanding tests, even if they were written purely in code. We were still able to pair and work together on them. We were also able to quickly browse our existing tests to check if given scenario is already in place or not.
Maintainability	We decided to place all our tests directly in code, representing all feature elements (features, scenarios, steps) with corresponding code constructs like classes or methods. This allowed us to use all standard developers’ tools (IDE, Resharper) and methods (refactoring, static analysis, running tests from IDE) to maintain our test code effectively.
Clean code	Instead of reinventing the wheel, we decided to use existing tools to do things that they are doing well. Everybody knew NUnit framework, how to write tests with it and what behaviors can be expected from it. We went with this well known test structure. The convention that we used for structuring our tests gave us better clarity of what given test is doing. Explicit steps execution allowed us to analyze them quickly and effectively (after all it is only a matter of navigating to step method implementation).
Traceability	Representing each step as a method with self describing name and printing step name before it’s execution allowed us to localize and understand scenario failures quicker, by analyzing exception/assertion stack trace or checking a execution console output in both, CI and Visual Studio.
Execution progress monitoring	Again, because each step name was printed before it’s execution, we have got an execution progress monitoring for free. It finally allowed us to track on CI what is the current stage of executed tests and quickly determine that some of the steps are executing longer or failing. Also, because the Teamcity was using time stamps for printing console logs, we could analyze which steps are executing longer and focus on their optimization.

LightBDD

I have noticed that the small BDDRunner class become very helpful for our team to develop both, acceptance and end-to-end tests, so I decided to create an opensource project and share it with others. The class that I have described above became a first version of LightBDD – there is a first commit showing how it looked then.

Thank you.

PS. In the upcoming post, I will describe how requirements changed when I joined a larger company with a corporation-like environment, and how LightBDD evolved into a current form.