The Automation Pyramid

The Automation Pyramid

Published: October 10, 2013 by Brian Kitchener

Across the world, in many automation models, the concept of the ‘automation pyramid’ is spoken about. The idea is fairly simple. There are multiple layers of automation, each focusing on different areas of the application, and offering a different degree of coverage. The base of the pyramid are the Unit Tests, these are executed against the code. Next is the API test, executing against a service layer. Finally, at the top of the pyramid sits the UI tests that actually validate the application as a whole. Each of these layers offers a different level of coverage. Unit Tests are fast and can test many different permutations, but don’t test any integrations between components. API tests can validate our units integrate, but can’t test an end-to-end user scenario. UI tests validate multiple components in a single test, but are typically very fragile and take a long time to run. So each layer provides a pivotal role, and needs to be tested thoroughly.

The Base of the Pyramid

At the base of the pyramid are the Unit Tests. As our foundation, they will have the broadest coverage, and should test as many permutations as possible. Typically they will test a specific “Unit” of code. They will instantiate a class, call some functions, and verify the functions return the correct results. Unit Tests are built in the same language as the code and will usually be stored and executed with it. Because they are executing a piece of code, it’s extremely easy to test for many different permutations. For example, if I have a function that does some simple arithmetic, I could test it 100 different ways in a couple seconds. It could take hours to verify the logic using the UI, because of all the navigation and setup required. Because of this, our unit tests should provide extremely broad coverage, testing as many permutations as possible. They should attempt to test and validate important business logic at a component level. However remember that they are unit tests, and will not test how components work together, only that they individually work as expected.

The Middle of the Pyramid

The next level of our pyramid is made up of API/Service Layer Tests. These should fit somewhere in between UI and Unit tests in complexity and scope. This layer is where we first start to test how our components integrate together, and how they handle real data. The advantage of API testing is that a lot of logic can be validated without being dependent upon the UI. For example, let’s suppose we are building a new web site. Even before the UI is finished, the service layer and authentication mechanisms could be finished. I can build a set of API tests that verify the login service works as expected. And while the tests aren’t quite as fast as Unit tests, you can expect most API’s to return a result in less than a second. That means that testing for a variety of conditions such as invalid credentials, timeouts, or special characters is very easy and very fast. This frees up our UI tests to focus on testing the UI and end-user functionality, instead of trying to validate every piece of business logic. Without our API tests, we wouldn’t even be able to start testing until the UI is finished.

The Top of the Pyramid

The highest level, and therefore the smallest layer, is comprised of the UI tests. It is at this level that we actually launch our application and perform an end-to-end test through it. It is at this point when we finally verify that all the individual pieces work together as a cohesive whole. But since much of the business logic has been covered in the lower levels, the tests can focus on validating the UI looks and behaves appropriately. It’s important to remember that UI tests are extremely slow in comparison with our UI and API level tests. A UI test will take usually a minute or so, when a API test takes several seconds and a unit test takes less than a second. But a single UI test will validate hundreds of components in a single test, so its coverage is much deeper than the other layers. However the UI is constantly changing and being updated by new functionality, which means that the UI tests are inherently less stable, and will require more maintenance than the other layers. This is why we try to limit our testing in this layer to as little as possible.

By breaking the application apart into these layers we are able to systematically and logically break apart the application and test it in components. Without a strong foundation of Unit and API tests, we are forced to automate every possible scenario in the UI layer. And while it’s still possible to automate only UI tests, our resulting automation suite ends up being expensive, slow, and fairly fragile. And while it’s possible to balance a pyramid on its end, it certainly won’t be stable. It is for this reason that many companies fail at efforts to implement effective automation strategies. If we take a systematic approach, and all the levels are automated to an appropriate degree, we can build and maintain a cohesive suite of automated tests with much less effort.