Hermetic Test Pattern & You3 min read
I’ve never heard the Hermetic test pattern defined better than by the phrase “Test is an Island“, a play on the adage of no man is an island.
The gist of the Hermetic test pattern is to write tests in such a way that each test is completely self-sufficient and independent of others. Any dependency on other tests or a third-party service should be ruthlessly avoided.
To do that with any test, a test needs to do the following :
- Handle its Setup
- Handle its Teardown
- Handle its Environments
In broad terms, the advantages of doing so are the following :
1. Speed Up the Tests:
By removing the dependencies, you are removing the need to wait for those dependencies to return with a response. Admittedly this is more of a concern with UI tests than any other, but this quickly can either become a major headache or a major relief depending on the way you chose to go, as even small differences in test times add up.
A simple back-of-the-napkin calculation can help us to get this point across.
Let’s say you have a test harness of 200 test workflows, that make network calls. Let’s say on average each network call takes 400 milliseconds and there are on average 10 such calls per test.
If you were to remove these network calls by mocking them, you might only be reducing each test by 4 seconds, but overall you have shaved off over 13 minutes from your testing. This can be a godsend if you have placed the tests in your CI Pipeline (as you should), speeding up your build process.
2. Prevent Test Environment Pollution:
You don’t want the state of your unit/app for one test, interfering with another. Let’s see this with an example :
Let’s say, Joe, our automation guy working for the famous taxi sharing app, Super, has decided to test two workflows, one being when the user has not paid for the previous ride and is trying to book another ride, the other being the normal ride-booking workflow that the user goes through. Being the slick guy he is, he kept the tests in the CI pipeline and created a script to log bugs in the company’s Slack channel whenever any of those two tests fails.
The only problem is Joe, due to his love for spaghetti, has gone with the Spaghetti Anti-Pattern and created end-to-end tests that have common setups and teardowns and share an environment.
A short while later everybody in the company is very angry with Joe due to the deluge of bug messages in Slack everyone has been receiving.
You see, he placed the Unpaid workflow before the normal one, and since he didn’t make the test hermetic, the second workflow will always fail and the state of the app will indicate that the user has not paid for the previous ride, and prevent any booking before the user has done so.
Don’t be like Joe, keep your tests hermetic.
3. Reduce Test Flakiness:
This is the most important point. The bedrock of good testing is determinism, such that everything works when it meets functional requirements and if it does not, it is easy to debug why.
However, whenever you have tests that are dependent on third-party dependencies, you are introducing more things that might lead to the scenario of a test failing even though the Unit/App under test meets all the functional requirements, because something went wrong with those dependencies.
By making the test hermetic you can be sure that the test fails only when the thing being tested fails to meet the test criteria.
Hopefully, this post will motivate you to start the journey towards making your tests hermetic, and avoid the pitfalls of the spaghetti anti-pattern that most people end up struggling with.
This blog is a Guest Contribution from Atin Agnihotri (frontend QA).
