Hermetic Testing

A hermetic test is a test that is completely self-sufficient. It is fully independent, therefore every time a test is run, you are absolutely sure that a failure is a real failure, and has nothing to do with a dependency going wrong.

In order to use hermetic testing to achieve what we want, we first split our e2e into meaningful chunks. The most common breakdown is to split it into frontend and backend systems.We then run our tests within the context of the front end and back end independently. 

We then add additional levels of tests to make sure we make up for skipping the e2e tests like contract tests, API tests, visual tests and improving existing unit testing coverage.

Hermetic testing to me is a method for ensuring that tests run consistently and offer a consistent value. If we apply the idea of Hermetic Testing to only the data element of a tests requirements, we can understand what this means.

Imagine there are three ways to handle data in our testing:

The third bullet point is how I imagine Hermetic Testing. Test-001 expects [item]to already exist, it navigates directly to the UID of [item] and performs the test.

This is a more scientific method of testing when compared to 1. and 2., because they inherently have risk built into them.

This, to me, is the purpose of Hermetic Testing: to build a regression pack that executes exactly the same way, every single time, so that when a test fails, we can trust that something really broke.