Reliable end-to-end tests are AAA Atomic

Have you ever looked at the code for an automated test and wondered "What is this code actually testing?" Or maybe you have seen giant tests that try to cover every single test case, take forever to run, and break after the slightest app change?

How we design a test strongly affects our ability to run tests quickly and maintain them over time. Automated tests gain value the more often they are run, so maintaining automated tests to keep them running is arguably even more important than writing new tests.

Given how important test maintenance is over the long run, yet how little priority test maintenance tends to receive compared to new features, how can we write tests that are easy to understand and thus easier to maintain?

Arrange, Act, Assert: the blueprint for successful tests

Arrange-Act-Assert (AAA) is a simple, yet powerful pattern for designing tests that are easy to understand and maintain. This pattern is most often associated with unit testing, but it works just as well for end-to-end tests. In Behavioral Driven Design (BDD) using Gherkin, the "Given-When-Then" keywords work similarly.

• Arrange

  • During the "Arrange" step, we set up the initial conditions for the event to be tested.

• Act

  • During the "Act" step, we describe the event to be tested.

    • A test should have only one "Act" step, so we only test one event.

• Assert

  • During the "Assert" step, we perform a check called an "assertion" to compare the actual results of a test against one or more expected results.

    • Numerous code libraries exist to make assertions easier such as AssertJ for Java, Mocha and Chai for JavaScript, and AssertPy for Python.

I like Arrange-Act-Assert because it encourages us to design test cases that have a start, middle, and end. When writing in Gherkin, people can go overboard with the "And" keyword and chain "And-Then-And-Then..." steps together, making tests that run on forever. Avoid this temptation!

Atomic tests: test one thing at a time

Every test should have at least one assertion, and ideally not more than one.

Why must every test have an assertion? For each test, an assertion is a step where we test a hypothesis. Without a hypothesis, that "test" we just wrote is just some code enclosed in a function. And no, adding an "@Test" annotation so TestNG runs your code doesn't make it a test!

Without comparing an actual result to an expected result, do we really test anything? Or are we just getting a green check mark since our code ran without throwing an exception?

We can usually tell that a test is atomic when:

• The test only has one assertion, or two assertions at most.

  • Sometimes in the "Arrange" step, we might want to assert that our test state is correctly established before continuing a test.

  • It occasionally makes sense to assert a few related outputs at once if:

    • the outputs are caused by the same event

    • the event and outputs follow the same workflow or path

• The test keeps UI interactions to a minimum.

  • When possible, try to set up the test state in the "Arrange" step by using database and API steps. Each UI step that the end-to-end test performs increases the chances that a step might fail due to flakiness or reasons unrelated to the event in the "Act" step.

Consistency leads to reliability

By consistently using the Arrange-Act-Assert and Atomic test principles across our automated tests, other testers on our team can quickly become accustomed to how our tests work. Maintaining a consistent structure also makes tests easier to understand, helping newer testers learn how to write tests sooner. This collective knowledge will then allow our teams to quickly isolate and fix problems, thus improving the maintenance and reliability of our automated tests.