Breakdown Your Costly E2e Tests into Efficient Low Level Tests Through Hermetic Testing

End-to-end tests are really good testing solutions that provide greater confidence in the system that it tests. They are also the most commonly used form of tests when it comes to test automation. It is also pretty straightforward to get started with them. But as software systems grow and become complex, your tests also become complex. Most of all, they become… costly, very soon.

Well not really. In my opinion, e2e tests provide a good level of smoke test coverage. You can test out your core workflows and make sure they are not affected. But they aren’t the most efficient when it comes to feedback times. Any kind of errors it might catch comes at the last point of your development workflow. Ie. after all of your services have been deployed to a dev or staging environment. By that time it is a bit too late. The unplanned work for product teams becomes tenfold. 

So instead of relying heavily on end-to-end tests, we now take a systematic approach to achieving the same level of confidence, but by taking a step down. We now try to break down our e2e tests into low-level tests. And to achieve this, we try to use the concept of hermetic testing. 

A hermetic test is a test that is completely self-sufficient. It is fully independent, therefore every time a test is run, you are absolutely sure that a failure is a real failure, and has nothing to do with a dependency going wrong.

In order to use hermetic testing to achieve what we want, we first split our e2e into meaningful chunks. The most common breakdown is to split it into frontend and backend systems. 

We then run our tests within the context of the front end and back end independently. We then add additional levels of tests to make sure we make up for skipping the e2e tests like contract tests, API tests, visual tests and improving existing unit testing coverage. 

Now we know what journey we went through in our organisation, in this talk I will be presenting the strategy and implementation of this concept. I will: