In-Sprint Test Automation On Agile Teams: Yes You Can!

¨My bold ask to the stakeholders was to support “a safe space to fall down and get back up again.” I made it clear that a period of trial and error was essential, since it would offer priceless lessons that would only make our process more resilient and reliable over time.¨

If you're like me, you may have read or heard about in-sprint test automation… and wondered why such an elusive goal was high priority. After all, the goal is just about impossible for testers alone to reach. 

But even if there’s only one dedicated tester on your team, in-sprint test automation is in fact within reach! You can make it work when the entire development team, including the product manager and scrum master, set in-sprint test automation as a goal for the entire team, not just the tester. 

I had to try in-sprint test automation myself before I realized any of this. And I'm happy to report that with the right understanding and support in place, it's more than possible; it's a great way to work. 

Here, I share my experience report: There have been lows, there have been highs, and it was never an easy sail. I’ll share successes, lessons learned, and even a couple of silly true confessions!

My First Words Of Advice: Learn Your Test Automation Craft Well

When my team first began to consider in-sprint test automation, I was already comfortable with test automation principles in general, and Cypress and continuous integration / development (CI / CD) specifically. If I'd had to come up to speed with that knowledge as we moved into in-sprint test automation, the result would likely not have been optimal, to put it mildly. So if you're new to test automation or the automation toolset you'll use, take the time to learn the ropes before you attempt in-sprint test automation. 

In-Sprint Test Automation: Shift-Left Testing In Practice

We probably all have heard the term "shift-left¨ with respect to what ¨testing¨ means in the Agile context. (Be assured that if you put in-sprint test automation into practice, your team will exemplify good shift-left principles!) 

Below are a few good habits to get into as you adopt a shift-left mindset:

• If a question occurs to you: ask it. Don’t worry if it seems trivial at first.
• There is no good reason to be idle while the developers are writing code. I’ll explain how I kept VERY busy during active coding. 
• If a user story seems unclear in any respect: see the first bullet in this list! Ask a question.
• Embrace testability every step of the way.

Surveying The Territory

From the outset, I was actively involved in the project, which was a complete refresh of legacy software. The complexity of the old code and the number of improvements envisioned would require all new code to be written. The developers got busy right away with proofs of concept.

I had at least one clear advantage: the legacy project docs and test automation framework set me up perfectly to understand the crux of the project, who the end users were, what the key product features were, and what overall improvements to quality I could suggest. I made a list of all possible test oracles I could get and made the utmost use of all of them. I watched the knowledge transfer videos from the legacy project and learned about the various layers and integration points between the features. And I met with the senior test automation engineer for the legacy project, whose explanation and guidance were excellent. 

The availability of all of these resources inspired me to make the most of my early engagement. 

Making A Roadmap 

I have always been a huge fan of mind maps, so I used Lucidchart to make a diagram of the application user journey. I adapted a basic concept map by using shapes available from the Lucidchart gallery. I have drawn my inspiration from a Lucidchart blog which shows the various ways of building mind maps efficiently. 

Fig 1: A GIF of a Lucidchart mind map bringing ideas into action

The mind map (see Fig. 2) helped me to understand what each application page offered to the user, how it drove the user’s actions, and how each user action was reflected in the various sections of the application. Diving deep into an application like this gives testers a clear idea of the business purpose and the value the application offers to the users. And it only boosted my confidence, which I would need to advocate for quality as an important member of the team. 

Fig 2: Lucidchart showing the breakdown of the application user journey (blurred due to independent checks)

I presented the mind map to the team in a sprint demo. One unanticipated win: the stakeholders were delighted because they could use the mind map to compare the product we were developing with the legacy product. 

Sharing My Ideas And Plans With The Team

As I am the only tester in the team, I believe it's my responsibility to explain to the team my ways of working and how the QA tasks flow across the sprint. 

So for this first go at in-sprint test automation, I created a Lucidchart diagram (see Fig. 3) that showed how QA tasks flow in parallel to developer tasks and illustrated ideas to improve communication between testers and developers. My team appreciated that I made my process transparent to them, which enhanced communication going forward. It also helped everyone understand that we had a common goal. 

I created the diagram to explain and help the team visualize testing activities. The key points that I illustrated were:

• When a user story in the backlog is added to the current sprint or iteration, developers and testers work on the user story in parallel. So it's good practice to have dev and QA tasks added to the same user story. 
• When a user story is not clear or team members differ in their interpretation of acceptance criteria, it’s time for a Three Amigos session. (My team’s product owner calls them De-Fuzz Sessions!) These time-boxed meetings clarify the story from the viewpoints of the developer, product owner and the tester.
• While a developer is implementing the user story, testers can work on the test planning and prepare the test data and test scenarios. To ensure developer and tester are working in sync, we adopted the practice of peer-reviewing the test cases with developers. Often, potential bugs come to light in those sessions. And if that sounds like test-driven development, that’s because it is!

Fig 3: A diagram of QA tasks in parallel to developer tasks

Time For A Pep Talk

Since we were attempting in-sprint test automation for the very first time as a team, I figured that some psychological preparation was in order. So I gave a presentation to the team, and invited discussion. Our conversation was rewarding. 

For each activity and goal, we talked about: 

• Why is this needed? 
• How can this be achieved? 
• What does it take from each one of us to try this out? 
• What are the benefits and also the challenges? 

My bold ask to the stakeholders was to support “a safe space to fall down and get back up again.” I made it clear that a period of trial and error was essential, since it would offer priceless lessons that would only make our process more resilient and reliable over time. 

Key To Success: A Solid Proof Of Concept

I knew there were some areas where I personally needed to upskill. I identified at least two:

• Cypress BDD integration with Cucumber
• Accessibility principles and automated accessibility testing

Then I created a proof of concept using the Cypress BDD framework with Cucumber. I read about accessibility testing from various resources, and I learned that Cypress has its very own accessibility testing plug-in: Cypress-Axe. I incorporated that into my proof of concept.

So I had a skeleton framework all ready to go, awaiting the delivery of the first product code. I also made sure to advise the developers of the toolset that would be used.

Overall: I had a simple vision: automate tests in-sprint, build a pipeline, and publish results in a chart to the stakeholders. I prepared (upskilled) myself in the best way possible to get this set up in the right time. And I adopted a standard practice to automate the positive, negative, and null scenarios for each feature wherever applicable. Finally, I put my log-in scenario into a global ‘Before’ hook. 

Educating My Team On Solid Practice

Together with the developers, we standardized some best practices, which we memorialized in a document for future reference (and for onboarding purposes should any other developers join the team). 

Among the best practices we agreed on were:

• Developers would include data test IDs on all the UI elements that needed validations or user interactions in test automation.
• We also agreed on a tear-down strategy, so the automated tests would not leave behind any unwanted data.

There Is No "I" In Team: Asking For Help

Last but not least, I had my team to support me. I asked for help every time I needed the support. I felt I was heard by them and I had an awesome team lead who gave me the spotlight every time I communicated. For the first time, I felt like I could shoot webs like Spiderman. Sorry to be dramatic, let me put it this way: you don't always need superpowers to be a superwoman, just a super team to work with.

Keeping Testing In Motion While Product Code Is Written

I didn't wait around for product code to be ready. I couldn't afford to; I had a lot to do. 

• Information exchange prior to API publication. As the developers were getting ready to publish the first set of APIs, I met with them to understand what each endpoint did in terms of functionality and what the validation rules were. I got the Swagger doc before it was hosted and wrote up valid and invalid inputs for each endpoint, such as the minimum and maximum values of each property in the payload.
• Getting regression test automation ready in advance. Before the APIs were published, I got the manual regression test suite ready for tests and wrote the test scenario templates for Cypress automation. As a result, once the endpoints were deployed, I was ready for the actual testing. 
• Ensuring quick updates to tests for authentication tokens. While the developers were working on the user authentication and authorization flow, I worked on the strategy to fetch the bearer token so this could be passed as the authorization header in all the API calls. When the feature was deployed in the following sprint, I was able to update all the existing tests straight away. We achieved this thanks to collaboration between developers and testers: simply put,  we were ONE STEP AHEAD.
• Helping developers build testability into UI product code. As the developers were building the front end, the documentation we created to standardize best practices for automation came in handy. I made sure a data test ID was added for all necessary UI components, so we didn't have to go back and forth with requests for its addition after each UI feature was deployed and ready to test.
• Optimizing test runs and reporting in the pipeline. The automation tests were configured to run on the pipeline early on. This gave me ample time to fine-tune the tests for issues faced on the pipeline agent. I optimized the tests to run in parallel by reducing the pipeline execution time. I generated test reports in Junit format so HTML test artifacts were published. We also reconfigured the yaml file to make pipeline triggers for every pull request and merge to the base branch. These and other optimisations saved a lot of time and effort especially when the pipeline was running frequently.
• Automating screenshots for quicker root cause analysis. Screenshots of failures were captured and published as artifacts on the pipeline. This helped the team to analyze failures and identify the root cause quickly. 

Feeling The Fear And Writing Test Code Anyway 

We all know a tester's role is a niche skill. I’m not sure about all of you but I've had some challenging lows along with the rewarding highs. Thankfully, over time, I've grown far more confident to speak about the fears I've had during my career. I saw that if I worked on my fears, putting myself out there despite them, those fears turned into my strengths. 

The road to in-sprint test automation was pretty scary at times! But I processed my fears, spoke about them, turned them into action points to upskill, dealt with a few bumps, and learned from it. It was not easy but now I feel it made me stronger. 

Some of those fears follow….

• Before, I'd always maintained a separate repository for automation tests, keeping test code very much apart from product code. I always imagined the product code repository as a lion's den :) But this time around, the stakeholders wanted a single repository for the project, to include the test automation code too. At first I balked, but then I realized that my tests would simply reside in a folder within that repository. We had already standardized on how the code from each component was going to be organized. The immediate upside: my test code had a lot more visibility, and the developers reviewed test results frequently. Even better: with each merge of product code to the base branch, the test automation pipeline was triggered. How cool is that !!
• Stakeholders like to have information on the test automation suite available at a glance. They also want to see whether something is failing and how we fixed it.

And so I was challenged to create graphics and chart widgets in Azure DevOps. This is a task I never dreamt of doing. I didn't think I'd be any good at creating attractive reports on project pages, but, then again, I'd never done it before. So I figured out how to add a widget for test results on Azure Devops. In doing so, I collaborated with developers to understand what they expected from such an artifact. I tailored the widget settings in such a way that the team had some useful metrics to work from.        

Below, you can see the graphic produced by the widget I configured. The image was edited to hide sensitive company information. But I promise this is an original screenshot from our Cypress testing dashboard.

Fig 4: Azure Dashboard Widget for Cypress CI/CD Pipeline

What We've Learned: The Good and the Oops!

As I advised my team: trial and error would show us what experiments worked and what didn't go so well. Here are some highlights. 

The joy of code review. I was terrified when I submitted my first test code for developer review in a pull request. I expected to be sent back to the drawing board entirely. That did NOT happen. What I got instead was specific and useful feedback, which led me to: 

• Remove unwanted outputs in my pipeline yaml file.
• Create payloads as new instances of JavaScript constructor functions
• Successfully decode the mystery acronym ¨LGTM¨ that appeared in one of my pull request reviews. It means: ¨Looks Good To Me¨!

Ready machine one. When I joined the team, I was unable to review the onboarding guide. So I delayed downloading SQL Server Management Studio. Having to wait for IT approval, then having to download, install, and configure the software while the sprint was in motion, held up my testing. Lessons learned: 

• Never wait to get IT approvals for the applications and platforms you'll need for testing.
• Document links for downloads, installations, and procedures related to configuring a tester's machine. Maintain these in a wiki page for testers, so new team members will come on board smoothly. 

Pay attention during those Agile ceremonies. At one point, a feature was ready to test, but a potential blocker made testing impossible. I didn't help matters by waiting to ask my team for clarification. Lesson learned: I had to pay closer attention to discussions that come up during Agile ceremonies. 

To Wrap Up 

I hope this article is a good primer for establishing in-sprint test automation on your Agile team. I have found it invaluable for the entire development team to be educated on the value of in-sprint test automation (as part of a suite of shift-left testing practices). And I've learned how important it is for testers to master the art of persuading their teams to try something new and worthwhile.

To reiterate: I could not have attempted this if I had been new to Cypress or test automation, as timeline is critical in in-sprint test automation. So if you want to incorporate in-sprint test automation and you don't yet know how to plan for it or code it, start learning how to do so now. 

Let me know your experiences in the comments. What went great? What was your biggest challenge? 

For More Information

• Roadmap To Becoming A Test Automation Engineer, Julia Tekin
• Sustainable Test Automation: From Garden to Ecosystem, Anaïs van Asselt
• Embracing Leadership: My Journey to Stop Fearing and Start Leading, Pedro González