Most of us testers know by now that Postman is one of the most sought-after testing tools for all things API. Of all the API testing tools out there, I have used Postman the most, and I simply love its GUI. In my experience, the most loved features of Postman are: pre-request scripts, authorization methods, the Faker library integration for random test data, and one that I'm curious to learn and explore: Postman Flows.
In this article, I will list and describe the different ways we can execute our tests from Postman Collections. I researched this topic for my close friend, a junior test engineer who was keen to learn about Postman. And I figured I would share what I learned with the software testing community at large!
The method of execution you should choose varies depending on the needs of your project. When learning something new, we always start in a granular fashion with the simplest methods, then gradually move on to more complexity. This gives us a crystal clear understanding, similar to following a recipe in the kitchen!
Clearly, there are ways of executing Postman tests other than what I describe below. But I have chosen to cover a finite set of scenarios, since they are based on what Postman can offer us on its own without the need for external integrations.
Here are some ways of running Postman Collections:
- Manually in Collection Runner
- Schedule the collection in Postman
- Monitor the collection in Postman
- Manually from the command line using Postman CLI
- Manually from command line using Newman and the HTML reporting plug-in
- GitHub Actions using Postman CLI within a continuous integration / delivery (CI / CD) framework
Method 1: Run A Collection Manually in Collection Runner
We can never underestimate the complexity of any testing task. And so it's advisable to run a new or changed Postman collection manually first to help us understand the stability of the tests.
By default, the tests in a collection or folder will run sequentially. But we can also execute the tests in various sequences by dragging and rearranging the requests within the Collection runner window. The tests can be run in various sequences through Postman CLI too, as you'll see below.
Cool Fact for you: The postman.setNextRequest() method in the test tab can affect the sequence of the run from the Collection Runner.
Collections come in handy to group requests based on type, functionality, or test scenario. You can also use them to cover an end-to-end test scenario.
You can execute your API test collections within Postman by selecting the Run collection option from the collection menu.
From the Test Runner, you choose configuration settings like Run manually, Schedule runs, or Automate runs via CLI and the number of iterations. For this section, we will select Run manually. Then select the Run button to proceed to execution.
A cool alternative to this is choosing the Runner menu from the bottom of the Postman UI and dragging the collection you wish to run into the Test Runner window!
Results and Reporting
Postman neither generates any HTML reports nor allows any plug-in integrations within its GUI to get it done. But the display of the results is clear, with all the information we need to trace what has gone wrong in case of any failures.
On the other hand, I'm not a big fan of the Export Results feature in Postman. All it shows is a simple pass or fail, although we can extract the detailed results as JSON. You could do some coding to push the request and responses to the results, but I still think Postman could add some new features around this to grab reports easily.
The Run results panel gives a detailed view of each test: the request body, headers, and response body.
Test Summary gives an overview of the test results without much detail.
Method 2: Schedule Collection Runs And Get Notifications Of Results
The scheduling feature in Postman is very helpful if your team has decided to maintain a steady and growing regression suite in Postman but you do not yet use CI / CD. There are many valid reasons why this could be the case. On one of the projects I worked on, we were in a very early stage of development and a few APIs were deployed each sprint iteratively. So I wanted to maintain a growing API suite which would help me to run tests regularly, but I also wanted to avoid going overboard by running them on CI / CD.
Scheduling tests will send a notification straight to our inbox if there are any failures, and we can set it to run for any desired setup. Isn't that cool!
After you choose the collection to be executed, use the Collection Runner to schedule the collection and select the frequency of the run, for example Monday - Friday at 8 a.m. This frequency setting came in handy for me once, since I had a morning status report to give on scrum calls. Likewise, you can choose the frequency that best favors your stakeholders’ expectations. For example, you can set it on the day you have your sprint review so you can share information about the health of the APIs. Most importantly, the Advanced Settings in schedule configuration allow you to specify delays between requests and to provide for retries if any run fails. Most testers will find these settings handy.
After you configure the schedule, it can be viewed under the Runs tab in the collection details page. When you select the View button, you can view the results of the schedule or trigger the collection manually.
Now let's check if I have received an email notification (in case of any failures) and view the results on the Schedule page. The results say the APIs are healthy and they also give information about the response times.
Cool Fact for you: URLs must be publicly accessible endpoints for scheduled and monitored runs. Endpoints on localhost cannot be used.
Method 3: Monitor The Collection In Postman
In the previous method, we scheduled our collection and viewed our results in the Schedule UI. So you might think: what does it mean for running the collection and how does it differ from scheduling?
The difference is simple, Scheduling the collection allows you to run the collection at specific times on selected days. But monitoring helps to run the collections automatically not just on specific days and times but also with granular configurations like regular hour and minute intervals. These configurations help us to assess the performance of the APIs.
Choose Monitors from the left pane of your workspace and select Create Monitor.
Configuration of monitors is very similar to that of schedules. Give the monitor a name and choose the collection, region, and email notifications. Here, you also specify advanced settings like time delay.
Note: With monitors, you cannot inherit any global variables. You have to pass a local variable instead.
In this example, I have chosen the run frequency to be every hour.
Let’s see the results. We can see that a few runs have failed. The failed tests reveal that the response time of one of the requests has been greater than the expected time. This could point to an inconsistency in the behavior of the API endpoints.
The results can lead to a worthy discussion with the developers so that any issues can be resolved at an early stage. This is a great example of shift-left principles.
Cool Fact for you: Monitors can incorporate integrations to push the results to a variety of applications, for example, a Slack channel. This way you won't have to look in the Postman UI for the results all the time.
Method 4 : Run Tests Manually From The Command Line Using Postman CLI
Postman CLI (Command Line Interface) allows you to run any of your collections through the command line. So for any CI / CD solution, Postman CLI comes handy.
From the Collection Runner window, choose Automate Runs via Postman CLI. This gives you the option to download Postman CLI via curl.
When you download the CLI on a Mac with Apple silicon, you get results like those shown below. You can refer to the Postman CLI installation instructions for other devices.
From Collection Runner, generate an API key and copy the command to run the collection from the command line.
Note: Store the API key in a secure location. Do not hard-code it.
The result will look like the screenshot below.
Cool Fact for you: Postman CLI can also run locally exported collections. However, these will not be linked back to the parent collection in the UI. There are a few different CLI flags that allow us to change the sequence of the run order.
Method 5: Run Collections Manually With Newman And Create Great-Looking HTML Reports
Newman is an open-source command line collection runner for Postman. It allows the execution of Postman collections on CI / CD. You can choose between Postman CLI and Newman based on the project needs.
I personally prefer Newman over Postman CLI because Newman has a plug-in called HTML Reporter that can create an impressive HTML report dashboard. This HTML report can be generated for each collection and it is a great way to showcase the API test reports to stakeholders. Let's dive into this now.
Download Newman on your machine using the command below:
npm install -g newman
Export your collection from Postman. You must also export any environment variables present. In Newman, install the newman-reporter-htmlextra - npm plug-in.
Let us see how to run our Postman Demo collection and view our results dashboard. As discussed above, start by exporting the collection and the environment variables associated with the collection. Save the exported files to a folder.
Remember to run the Newman command below in the same location as the Postman collection file.
newman run collectionName.json -r htmlextra
Once the command is successfully run, a folder called newman will be generated in the same location as the collection file. The newman folder will contain an appealing interactive report, as you can see below.
This report can be generated using Newman in a CI / CD setup, but I'll save that for another exciting article specifically about Newman and its reporting features. For now, we will cover Postman CLI and GitHub Actions for CI / CD.
Method 6: Run Collections With The Postman CLI And GitHub Actions
If your organization has adopted CI / CD, you're already aware of its benefits. And you've probably noticed how CI / CD can elevate the impact a quality engineer can make in an Agile environment.
When you set up a pipeline using GitHub Actions, your tests will trigger for any desired setup against the preferred branch. This setup gives iterative feedback about the tests for each sprint and enhances visibility. This is especially helpful when tests reside on the same repository as the product code: it helps ensure that bugs don’t get merged into the main branch.
It will never hurt you as a tester to dip a toe into the CI / CD side of things and gain experience in running the tests on a pipeline. Postman provides a very friendly integration setup of CI / CD options with various providers, but we will deal only with GitHub Actions in this article.
Let's dive into how we can run the collections on GitHub Actions using Postman CLI. We have various ways of accomplishing this. In this section we will see how to back up our collection from Postman to GitHub and configure a pipeline on GitHub Actions.
Integrate Postman With GitHub
Go to the Home section of your Postman UI and select Integrations.
Select GitHub.
Note: There are two types of Github access tokens: classic and fine-grained. Classic Token gives access to all repositories of that profile, while a fine-grained token gives an option to choose which repositories can be accessed. So what I mean here is, if it's classic , the user can see all the repositories listed. But if fine grained, users will see only repositories that are selected under the token.
In the window below, you choose a name for the integration for easy reference later. Then choose the workspace and the collection. Finally, choose the repository from your GitHub list. Voila! Our integration Demo_Postman_GitHubActions has been added.
The outcome of this integration can be checked by looking at your repository. You should see your folder and collection inside the repository. From now on, any changes you make to your collection within Postman will be committed automatically. This serves as the perfect backup for your collection.
After doing this, you can check the backup by making a change to your tests, and then seeing if you have a new commit in your repository.
I added an extra line to an assertion in Postman, and can confirm that it was updated in the GitHub repository too.
Integrate Postman with GitHub Actions
Now that we have finished syncing our Postman collection to GitHub, we can add these tests on GitHub Actions. Once we have done this, the collection will run on a pipeline every time a push has been made to the preferred branch.
In the Collection Runner, choose Postman CLI and select Configure CI / CD.
- You can choose GitHub Actions and generate the API key. The API key generated in Postman can be stored in GitHub repository secrets and passed during runtime instead of hard-coding it in the execution script.
- Copy the configuration file for GitHub Actions and add it as a yaml file for the action in the GitHub repository. An auto-generated template yaml file is shown below. You would copy and paste this code into the yaml file in the .github/workflows folder.
- Now that the pipeline is set up, we can customize the pipeline file based on project needs. For example, you can add steps to generate reports and upload your artifacts to a folder in the repository. But that's a more advanced topic that might be covered in another article.
The configuration above has run and its results can be seen in the Actions tab of my repository.
The runs shown above can also be viewed in the Runs tab of your collection.
You could also use the power of GitHub Actions syntax to help customize a collection run by setting different run configurations. For example, you could set the number of iterations.
Note: Always remember to store sensitive info like the Postman API key in your GitHub repository secrets.
To Wrap Up…
I hope you enjoyed the roller coaster ride of executing Postman collections, from basic to advanced methods !
Please comment on the feature of Postman that most enhances your API tests, and don't be shy about suggesting features you think Postman can improve or introduce for testers.
For More Information
- Test Data Generation in Postman with Nick Relkov
- API Challenge Using Postman with Trisha Chetani
- Postman gamification - Save The Postman Space Station Game, forum post by wadders88 (James)
