By Josh Grant
You’ve been working hard with your team on your software product. Developers have created new features and patched up existing bugs. Testers helped any way they can to improve the quality of the team and the product. Managers spent time coordinating developers, testers, operations staff, stakeholders and perhaps even customers before and during a release. How is your team going to show all of this great work to the world & customers?
Developing A Good Release Testing Practice
One of the most critical parts of any software project is releasing it out into the world, and customers. It can also be underappreciated. Sometimes releases go badly and even good quality software has problems because of how it’s released.
Improving the release process is one area a modern tester can greatly help their team. Software testers often have a good overarching knowledge of a product, including how the latest features or fixes will affect the product overall. Testers are also a good point of contact between managers, developers, operations, sysadmins, and other members of an organization which can be essential when coordinating releases.
Good release testing practices often lead to smoother, more regular releases. Since testers excel at exploratory and confirmatory testing, release testing is a place that testers can directly contribute their skills to releases. Â
Teams that want to have good releases should follow these three pieces of advice:
- Have a release plan.
- Inform all stakeholders.
- Follow good release testing practices.
Testers can help their team by taking on some or all of these responsibilities where needed by executing some or all of these pieces of advice. Another approach testers can take is advocating for having these aspects of a release covered in some way, whether undertaken by testers themselves or other team members.
Have A Release Plan
A simple approach to help improve a release is to have a plan for each release. A plan could involve any team members and have any kind of process. One such example of a release plan could look like this example. In the linked example, there’s a structured document that has sections outlining steps for three stages of a release process: pre-release, release, and post-release; and starts with a description of the release and which staff are involved. The document shows a step-by-step process which can be signed off at each step. This shows who is included in the release, what is included in a given release, and how a release is structured step-by-step.
In general, a release plan could be:
-
highly formal: This might require several sign-offs in the document before a next step could be executed in the release. Using something like Docusign to timestamp signatures would be a very formal process.
-
informal: This might be an email or in-person conversation between two or three people who discuss and decide when to release.
-
process heavy: This might mean having many several sign-off steps with multiple levels of teams and/or management, with individuals validating each step in Jira tickets or an internal CMS.
-
process light: This may involve a single document with checkboxes that are ticked as each step is finished, perhaps by a single person.
Your release plan is dependent on the needs of your team and your product’s stakeholders. As a software tester, one way you can greatly help your team is to lead efforts to collaborate within your team and between your team and others. Release efforts are one place where several individuals and several teams may be involved with a common goal, and testers can act as leaders and coordinators to determine how to accomplish releases or release plans in an effective way.
A release plan should be available to all individuals involved for the entire release. This may include developers, release engineers, testers, managers and system admins, or operations personnel. Smaller groups might be able to do more informal, lightweight plans while larger enterprise groups need formal documentation and approvals. The details of how a release plan is created depends greatly on the size and composition of a team.
Stakeholder Approvals Are Important
Make sure to get all necessary individuals and stakeholders to be aware of the release plan and how it will be executed. Even in smaller organizations, or teams, getting everyone on the same page is crucial for a good release plan and a good release. Sometimes having everyone agree to even a simple release plan can be the most difficult part of the release process.
After Collaborating On A Release Plan, What’s Next?
As mentioned, a release plan could involve any kind of process, but two things every release plan should have are:
-
Step-by-step instructions for deploying and releasing the application.
-
A fall-back/roll-forward strategy if the release doesn’t go well.
The step-by-step instructions should clearly communicate what needs to happen at each step and what each individual needs to be aware of at each step. Based on the previously mentioned release plan document, below is a visual representation of what the release process looks like.
Sharing Your Test Plans
Some ideas for sharing release plans could include:
-
A shared document that is created and updated for each release. One member of the releasing team creates a single document accessible to everyone needed for the release. This document may contain each step of the release plan, dates and time of each step, and who is responsible for each step. Sharing this document to a cloud location can make it easy for everyone to update or review the plan in real time. The release document can be saved as a reference or release artifact.
-
A common chat channel/room for the release. If everyone on a release team has access to a chat application like Skype or Slack, creating a common place for coordinating a release can work well. This particularly helps if some members of a release team are remote. The team can communicate in real time and confirm that each step of a release plan is underway or completed.
-
Using a whiteboard with sticky notes for release planning and progress. An efficient and low-tech solution for coordinating a release is to have a whiteboard with markers and sticky notes. Sections on the board can be drawn to denote each stage and members can move notes, or draw sketches, as the release progresses. This approach is good because it can provide a quick visual representation of how a release is progressing.
-
Some combination of the above. Â For example, a team that has a shared document may also have a release-only Slack channel. As teams communicate in the Slack channel which stage of the release they are completing based on the shared document, the document can be updated alongside the chat.
After setting a release plan, it’s important to decide how to communicate your excellent documentation to stakeholders.
Inform All Stakeholders
Releases can cause different reactions from different stakeholders. Some stakeholders, like customers or product owners, might be excited to see new features added or critical bugs fixed. Other stakeholders, like support staff or operations engineers, may be anxious because a new release may bring about new issues to solve. Regardless of how different stakeholders feel about a new release, it is important to inform all stakeholders with as much information as needed so that they have the full picture of what is going on with their product.
Even if a team has a good plan for releasing, informing stakeholders of all or parts of this plan is equally as important. It’s key for setting expectations and for allowing the entire release to go smoothly. Shareholders that experience problems during, or soon after, a release may take some actions which may end up reducing frequency and/or quality of future releases if they perceive a major issue. Often, this occurs due to a communication failure instead of a technological failure. Solid information to all stakeholders can prevent these communication breakdowns from happening.
Knowing who your stakeholders are is key here. Think about who they are, what they care about, and how best to communicate with them. As well, consider if they’re internal or external to your team or organization.
Technical stakeholders (support engineers or third-party developers) may prefer to receive technical updates that include precise hardware or software changes like a database version being updated.
Product stakeholders such as end users or product owners may like to have updates on what’s changed in the product, such as new features or revisions to existing features.
All stakeholders usually want to know when a release is happening. Sometimes communications need to be sent in multiple formats like email and chat, and sent multiple times before, during, and after a release. Often more communication is preferable to less communication.
Providing Valuable Information At The Right Time to Stakeholders
Basic information about release timing and dates is a good starting point. Begin with trying to answer the questions of when, why, what and possibly how a release going to happen.
When a release is going to happen is critical and also fairly simple to communicate. Telling stakeholders when a release is scheduled can often be accomplished in a single message or email. It’s important to be clear and precise when it comes to times, which may include timezone for distributed teams or multiple DateTime formats for teams with different localizations. Also include other related events, such as multiple servers getting updated at slightly different times. Often it’s better to be more descriptive than less. Telling stakeholders the release will happen at “8:00 PM Friday the 15th” is better than telling stakeholders “Friday evening”, “Friday after work hours” or even “sometime late on the 15th”. Clarity is king in this case.  Absolute times are better than relative times.
Why a release is going to happen can also be helpful for stakeholders. Releasing on a known timetable is important for stakeholders. Giving them the ability to have input or make known issues that could hinder a release process is important. Even if it’s a hotfix release, having a known timetable for post release fixes is also valuable to communicate. If a release is set to occur every month on the 15th, for example, then saying so will help new stakeholders and remind existing stakeholders of why they may see some changes in the application. If a release is not scheduled, an explanation of why the release is happening can benefit stakeholders. As an example, the release may be to fix a mission critical bug, an important piece of information for many stakeholders. Sometimes even a short description along the lines of “Updates are needed in this application” is sufficient.
What is getting released is also important. Telling stakeholders specifics of what’s going into a release can help them decide how to react to new changes. Including release notes is a popular way of packaging up these changes into an easy-to-understand document. Release notes often contain the date of release, the version being released (if there is one) and notes with exactly what changes are going into that release. The descriptions of the changes don’t necessarily need to be overly technical depending on your audience. You could also include specific links to other documentation or issues in your issue tracker as well. The goal here is to include enough information to stakeholders so they can be knowledgeable about what changes are coming up.
How a release is occuring can be helpful to inform stakeholders. A release may include maintenance, downtime, or some slow or non-standard behaviour in the app during or after the deployment of a release. This may be important to inform stakeholders in case it causes any issues. Some stakeholders may have to take some actions after a release based on the changes introduced. Some post-release behaviors might include end users logging in and then logging out of the app for changes to take place, or as a requirement of using the newly released version of the software. Stakeholders should be aware of details that may affect their work during a release process. Customers should also be aware of changes that might impact their work as well. Notifying a customer of a planned outage or steps they will need to take after an update, ahead of the release, should be considered critical communication.
Release Testing
After setting a plan, and following up with stakeholders, the time will come to actually release the software. One thing software testers enjoy doing is testing, and testing a release can be helpful to ensure quality software.
Verifying The Release
The first thing a tester should do when testing a release is verify that the new software has in fact been released as expected. This may sound trivial, but verifying the release is an easy way to avoid unexpected rollbacks or complicated patches after the release window.
One way to do this kind of testing is to have a list of one or more of newly released changes, visit the newly released app and verify that these changes are present. If they aren’t present then the release can be rolled back or forward as dictated by the release plan. If they are present as expected, then the release can be called a success.
On one team I’ve worked on, we were able to get to a point where only a single new feature needed to be verified on each release for our team to be confident new code changes were released as expected. This only took a software tester a couple of minutes to verify on each release. Pairing with an operations staff or product owner may be helpful for verifying new changes, depending on the changes being released.
Basic Release Testing
After verifying that newly released features are present, a tester should do basic diagnostics to confirm the released software is still operational. This may include logging in/logging out, doing confirmatory CRUD actions (creating an item, reading an item, updating an item, deleting an item) as well as basic workflows within the app.
Some testers refer to this kind of testing as acceptance or smoke testing, but it can be as simple as following the “happy path” workflows within the software to confirm there are not high priority bugs. Approaching release testing like this is a good idea but it’s important to have clear priorities and goals when planning this testing approach.
Finding issues that are trivial or definitely unrelated to the released code may be noted but not fixed until after the release is complete. Testers should have a clear understanding of what constitutes a show-stopper issue versus a lower priority issue. Exploration is a great way to test a product but the goal of any release is to ship software. Keep release testing light and straightforward, saving deeper exploration for pre-release testing.
Using Automation In Production
One approach that can work well for release testing is test automation. Based on the above ideas, a lot of release testing comes down to validating software still works as expected. This sounds like a good job for automation, which can efficiently run checks on software to validate expected behaviour.
Writing test scripts to repeat release testing can be a big win in terms of efficiency when done well. Automated tests on release can also be stressful and cause needless delays when done badly. When considering whether to automate tests on release, do consider re-using automated tests from other environments such a development or staging environments. This may require some overhead in automated test configuration or your application configuration but the benefits can be impressive.
Testing that a release hasn’t severely broken anything could be as easy as confirming automated test results are the same on a production instance as on an internal environment instance. It also means you can get test results against different environments on demand if required.
Use Automation Wisely
Be careful when running or using automation in production. One approach I’ve seen not work well is having automated tests specifically and only for releases. This sounds reasonable but in practice, these tests only get run at release time without any intermediate maintenance. Since they can’t be easily run against other environments, you end up with automated tests that have false failures or which need just-in-time updates as the release is happening. This causes stress and frustration for everyone involved in releases, particularly in cases where releases are infrequent and have several changes.
Running automated tests regularly is a good practice to keep such tests in good health, and allowing tests to run against multiple environments can help smooth the progression from development to production in a release.
Let’s Ship It!
Creating software is good but releasing it to end-users is even better. Software testers can be good advocates for release practices because testers are often at the intersections of code, users’ needs, and management. By improving the release process, testers can provide benefits to their team(s) and to their users.  Practices such as having a release plan, informing all stakeholders about release information, and good release testing approaches contribute to testers improving quality for releases, and in turn, software products overall.
References:
NEW Testing AMA on Modern Testing with Alan Page! It's free and it's happening on Tuesday 10th July, don't miss out!
Author Bio
Joshua Grant is a software automation specialist who is into software, automation, quality, and big ideas. He is currently a Solutions Architect at Sauce Labs, helping testers get the most out of their test automation frameworks. Joshua currently lives in Toronto, Canada and enjoys it quite a bit.
