The Many Ways of Executing Postman Collections

The Many Ways of Executing Postman Collections

Collection running in Postman has seen some changes recently. Here's Swathika's advice on how to get the most out of them.

Most of us testers know by now that Postman is one of the most sought-after testing tools for all things API. Of all the API testing tools out there, I have used Postman the most, and I simply love its GUI. In my experience, the most loved features of Postman are: pre-request scripts, authorization methods, the Faker library integration for random test data, and one that I'm curious to learn and explore: Postman Flows.

In this article, I will list and describe the different ways we can execute our tests from Postman Collections. I researched this topic for my close friend, a junior test engineer who was keen to learn about Postman. And I figured I would share what I learned with the software testing community at large! 

The method of execution you should choose varies depending on the needs of your project. When learning something new, we always start in a granular fashion with the simplest methods, then gradually move on to more complexity. This gives us a crystal clear understanding, similar to following a recipe in the kitchen! 

Clearly, there are ways of executing Postman tests other than what I describe below. But I have chosen to cover a finite set of scenarios, since they are based on what Postman can offer us on its own without the need for external integrations. 

Here are some ways of running Postman Collections:

  • Manually in Collection Runner
  • Schedule the collection in Postman
  • Monitor the collection in Postman
  • Manually from the command line using Postman CLI
  • Manually from command line using Newman and the HTML reporting plug-in 
  • GitHub Actions using Postman CLI within a continuous integration / delivery (CI / CD)  framework

Method 1: Run A Collection Manually in Collection Runner 

We can never underestimate the complexity of any testing task. And so it's advisable to run a new or changed Postman collection manually first to help us understand the stability of the tests. 

By default, the tests in a collection or folder will run sequentially. But we can also execute the tests in various sequences by dragging and rearranging the requests within the Collection runner window. The tests can be run in various sequences through Postman CLI too, as you'll see below. 

Cool Fact for you: The postman.setNextRequest() method in the test tab can affect the sequence of the run from the Collection Runner. 

Collections come in handy to group requests based on type, functionality, or test scenario. You can also use them to cover an end-to-end test scenario.

You can execute your API test collections within Postman by selecting the Run collection option from the collection menu. 

Context menu for a postman collection

From the Test Runner, you choose configuration settings like Run manuallySchedule runs, or Automate runs via CLI and the number of iterations. For this section, we will select Run manually. Then select the Run button to proceed to execution.

Test runner window with configuration set to run the collection

A cool alternative to this is choosing the Runner menu from the bottom of the Postman UI and dragging the collection you wish to run into the Test Runner window!

Results and Reporting

Postman neither generates any HTML reports nor allows any plug-in integrations within its GUI to get it done. But the display of the results is clear, with all the information we need to trace what has gone wrong in case of any failures. 

On the other hand, I'm not a big fan of the Export Results feature in Postman. All it shows is a simple pass or fail, although we can extract the detailed results as JSON. You could do some coding to push the request and responses to the results, but I still think Postman could add some new features around this to grab reports easily.

The Run results panel gives a detailed view of each test: the request body, headers, and response body.

Test runner report window showing all tests passing.

Test Summary gives an overview of the test results without much detail.

Test summary view showing all tests passed


Method 2: Schedule Collection Runs And Get Notifications Of Results

The scheduling feature in Postman is very helpful if your team has decided to maintain a steady and growing regression suite in Postman but you do not yet use CI / CD. There are many valid reasons why this could be the case. On one of the projects I worked on, we were in a very early stage of development and a few APIs were deployed each sprint iteratively. So I wanted to maintain a growing API suite which would help me to run tests regularly, but I also wanted to avoid going overboard by running them on CI / CD. 

Scheduling tests will send a notification straight to our inbox if there are any failures, and we can set it to run for any desired setup. Isn't that cool!

After you choose the collection to be executed, use the Collection Runner to schedule the collection and select the frequency of the run, for example Monday - Friday at 8 a.m. This frequency setting came in handy for me once, since I had a morning status report to give on scrum calls. Likewise, you can choose the frequency that best favors your stakeholders’ expectations. For example, you can set it on the day you have your sprint review so you can share information about the health of the APIs. Most importantly, the Advanced Settings in schedule configuration allow you to specify delays between requests and to provide for retries if any run fails. Most testers will find these settings handy.

Advanced settings for scheduling a run in Postman including configuration items such as name and frequency.

After you configure the schedule, it can be viewed under the Runs tab in the collection details page. When you select the View button, you can view the results of the schedule or trigger the collection manually.

View of scheduled postman collection run.

Now let's check if I have received an email notification (in case of any failures) and view the results on the Schedule page. The results say the APIs are healthy and they also give information about the response times.

Results of a collection run that was scheduled previously. It shows the results in a date order.

Cool Fact for you: URLs must be publicly accessible endpoints for scheduled and monitored runs. Endpoints on localhost cannot be used.

Method 3: Monitor The Collection In Postman

In the previous method, we scheduled our collection and viewed our results in the Schedule UI. So you might think: what does it mean for running the collection and how does it differ from scheduling? 

The difference is simple, Scheduling the collection allows you to run the collection at specific times on selected days. But monitoring helps to run the collections automatically not just on specific days and times but also with granular configurations like regular hour and minute intervals. These configurations help us to assess the performance of the APIs. 

Choose Monitors from the left pane of your workspace and select Create Monitor.

Postman create a monitor configuration page.


Configuration of monitors is very similar to that of schedules. Give the monitor a name and choose the collection, region, and email notifications. Here, you also specify advanced settings like time delay.

Note: With monitors, you cannot inherit any global variables.  You have to pass a local variable instead.

In this example, I have chosen the run frequency to be every hour. 

Let’s see the results. We can see that a few runs have failed. The failed tests reveal that the response time of one of the requests has been greater than the expected time. This could point to an inconsistency in the behavior of the API endpoints. 

Results view of monitoring in Postman showing runs by date order with colour cues for passed or failed monitoring.

The results can lead to a worthy discussion with the developers so that any issues can be resolved at an early stage. This is a great example of shift-left principles.

Cool Fact for you: Monitors can incorporate integrations to push the results to a variety of applications, for example, a Slack channel. This way you won't have to look in the Postman UI for the results all the time.

Method 4 : Run Tests Manually From The Command Line Using Postman CLI

Postman CLI (Command Line Interface) allows you to run any of your collections through the command line. So for any CI / CD solution, Postman CLI comes handy. 

From the Collection Runner window, choose Automate Runs via Postman CLI. This gives you the option to download Postman CLI via curl.

When you download the CLI on a Mac with Apple silicon, you get results like those shown below. You can refer to the Postman CLI installation instructions for other devices. 

Installation of Postman CLI

From Collection Runner, generate an API key and copy the command to run the collection from the command line.

NoteStore the API key in a secure location. Do not hard-code it.   

Configuration page to generate an API key in Postman

The result will look like the screenshot below. 

A view of the CLI runner report.

Cool Fact for you: Postman CLI can also run locally exported collections. However, these will not be linked back to the parent collection in the UI. There are a few different CLI flags that allow us to change the sequence of the run order.

Method 5: Run Collections Manually With Newman And Create Great-Looking HTML Reports

Newman is an open-source command line collection runner for Postman. It allows the execution of Postman collections on CI / CD. You can choose between Postman CLI and Newman based on the project needs. 

I personally prefer Newman over Postman CLI because Newman has a plug-in called HTML Reporter that can create an impressive HTML report dashboard. This HTML report can be generated for each collection and it is a great way to showcase the API test reports to stakeholders. Let's dive into this now.

Download Newman on your machine using the command below:

npm install -g newman

Export your collection from Postman. You must also export any environment variables present.  In Newman, install the newman-reporter-htmlextra - npm plug-in.

Let us see how to run our Postman Demo collection and view our results dashboard. As discussed above, start by exporting the collection and the environment variables associated with the collection. Save the exported files to a folder.

View of exporting a postman collection


Remember to run the Newman command below in the same location as the Postman collection file.

newman run collectionName.json -r htmlextra  

Once the command is successfully run, a folder called newman will be generated in the same location as the collection file. The newman folder will contain an appealing interactive report, as you can see below.          

Collection dashboard showing passes and failures

This report can be generated using Newman in a CI / CD setup, but I'll save that for another exciting article specifically about Newman and its reporting features. For now, we will cover Postman CLI and GitHub Actions for CI / CD. 

Method 6: Run Collections With The Postman CLI And GitHub Actions 

If your organization has adopted CI / CD, you're already aware of its benefits. And you've probably noticed how CI / CD can elevate the impact a quality engineer can make in an Agile environment. 

When you set up a pipeline using GitHub Actions, your tests will trigger for any desired setup against the preferred branch. This setup gives iterative feedback about the tests for each sprint and enhances visibility. This is especially helpful when tests reside on the same repository as the product code: it helps ensure that bugs don’t get merged into the main branch. 

It will never hurt you as a tester to dip a toe into the CI / CD side of things and gain experience in running the tests on a pipeline. Postman provides a very friendly integration setup of CI / CD options with various providers, but we will deal only with GitHub Actions in this article.  

Let's dive into how we can run the collections on GitHub Actions using Postman CLI. We have various ways of accomplishing this. In this section we will see how to back up our collection from Postman to GitHub and configure a pipeline on GitHub Actions.

Integrate Postman With GitHub

Go to the Home section of your Postman UI and select Integrations

Integration link highlighted in the Postman UI


Select GitHub.

Github highlighted in the Postman integration page


Backup a collection highlighted in the GitHub integration page


Authentication page to add in API key

Note: There are two types of Github access tokens: classic and fine-grained. Classic Token gives access to all  repositories of that profile, while a fine-grained token gives an option to choose which repositories can be accessed. So what I mean here is, if it's classic , the user can see all the repositories listed. But if fine grained, users will see only repositories that are selected under the token.

In the window below, you choose a name for the integration for easy reference later. Then choose the workspace and the collection. Finally, choose the repository from your GitHub list. Voila! Our integration Demo_Postman_GitHubActions has been added. 

Add integration configuration page to setup where the collection is pushed to in GitHub


Completed GitHub integration view

The outcome of this integration can be checked by looking at your repository. You should see your folder and collection inside the repository. From now on, any changes you make to your collection within Postman will be committed automatically. This serves as the perfect backup for your collection. 

After doing this, you can check the backup by making a change to your tests, and then seeing if you have a new commit in your repository.

View of GitHub with Postman collection pushed to GitHub repository

I added an extra line to an assertion in Postman, and can confirm that it was updated in the GitHub repository too.

Tests being updated in Postman with new code.
View of the commit in GitHub showing test code that has been updated.


Integrate Postman with GitHub Actions

Now that we have finished syncing our Postman collection to GitHub, we can add these tests on GitHub Actions. Once we have done this, the collection will run on a pipeline every time a push has been made to the preferred branch.

In the Collection Runner, choose Postman CLI and select Configure CI / CD.

Collection runner with run in CI/CD section highlighted
  1. You can choose GitHub Actions and generate the API key. The API key generated in Postman can be stored in GitHub repository secrets and passed during runtime instead of hard-coding it in the execution script.
  2. Copy the configuration file for GitHub Actions and add it as a yaml file for the action in the GitHub repository. An auto-generated template yaml file is shown below. You would copy and paste this code into the yaml file in the  .github/workflows folder. 
  3. Now that the pipeline is set up, we can customize the pipeline file based on project needs. For example, you can add steps to generate reports and upload your artifacts to a folder in the repository. But that's a more advanced topic that might be covered in another article.
GitHub Actions script put together by Postman for copying to GitHub

The configuration above has run and its results can be seen in the Actions tab of my repository.

View of GitHub actions with Postman runs shown within GitHub

The runs shown above can also be viewed in the Runs tab of your collection.

View of GitHub actions run within Postman

You could also use the power of GitHub Actions syntax to help customize a collection run by setting different run configurations. For example, you could set the number of iterations. 

Note: Always remember to store sensitive info like the Postman API key in your GitHub repository secrets.

To Wrap Up…

I hope you enjoyed the roller coaster ride of executing Postman collections, from basic to advanced methods !

Please comment on the feature of Postman that most enhances your API tests, and don't be shy about suggesting features you think Postman can improve or introduce for testers.

For More Information

Swathika Visagn's profile
Swathika Visagn

Senior Test Engineer at PwC UK

I am a very curious Senior Quality Engineer who is more driven towards automation and promotes shifting left. I have proven experience as an agile tester having strong fundamentals in manual and automation testing principles. I enjoy the entire journey from setting the automation framework from scratch to building the pipelines onto continuous integration tools like Jenkins. My framework adds more flavor by incorporating service layer (APIs) calls with UI layer automation which we call it 'Seaming' in automation terms. I communicate with stakeholders about risks, accessibility and pain points rather than number of passes/fails. I test with a purpose by automating business flow and add in appropriate plug-ins to make the automation reports/metrics readable for stakeholders. I also love to take part in agile ceremonies and volunteer to run retrospectives/daily scrums to keep the team self thriving in temporary absence of the scrum master. When I'm not scripting, I love to binge on Netflix, indulge in testing communities, read about Web3 and all things Quality :-) I am a yogic person too. If anything that calms me that's a cup of chai and a morning walk in the park.

Taking an Agile Approach to Test Automation by TestRail

0h 26m 55s

Ask mabl Anything!

0h 43m 26s

Prioritizing Your Tests by Considering Impact and Value!

0h 42m 27s

Web Services
Cross-browser Tools
Memory Test Tools
Automated Security Testing Using ZAP Python API
Test Data Generation in Postman

0h 30m 5s

The Power of Mocking APIs - Shivani Gaba

0h 46m 27s

Is this on your radar?

Learn more with MoT