Postman API Tests Collection: Run With Docker Compose

Create a Postman tests collection

We typically use Postman for sending an API Request and receiving the Response. Postman also offers a powerful runtime based on Node.js which gives its users the ability to write scripts. These scripts can be written in the JavaScript language.

To add tests to a request, open the request, go to the Tests tab and enter your code. Postman includes code snippets you can click to add and then edit to suit your logic if needed.

Tests will execute after the request runs. The output can be seen in the Test Results tab. Tests can be added to requests, folders and collections. A collection can be run to execute all the tests.

The guidelines for writing Postman scripts have been well documented at Scripting in Postman.

Debugging

One can use the console statements in Pre-Request Scripts and Tests to log helpful messages.

Run a collection automatically

A postman collection can be run directly from the command line using the Node.js-based Newman command line tool.

Install Newman using npm

npm install -g newman

Export a Postman collection

To export a collection from Postman, click Collection -> Export. This will export all the tests into a JSON file.

Execute a collection using Newman CLI

To execute the collection file using Newman, run the following command:

newman run FastRestAPITests.postman_collection.json

If there is an accompanying set of environment variables JSON file, use the following command:

newman run postman_collection.json -e environmentVariables.json

This will run all the tests in the collection and the results will be shown on the Terminal:

Let's write a few test scripts:

Next, let's find out how we can use the Newman CLI capability to run the Postman Collection inside a Docker container.

Docker and Docker Compose

Our Goals

• Running the tests should be easy. New developers should be able to join the development process without spending too much time and effort setting up the test environment.
• Test runs should be isolated and repeatable. We don't want to have that one flaky test that fails only on your machine. A failing test should fail when it's run by anyone in the team and on CI as well.
• The test environment should be as close to the production environment as possible.

Enter Docker

Using Docker to run tests can help achieve these goals. All developers will have the same isolated test environment setup, which can be used for running tests on CI as well. New developers won't need to spend half a day just to set up everything required to run tests.

Containers help ensure that an application runs as expected on any machine. By using containers, we can help solve the familiar "works on my machine" problem. We isolate the services into their environment and can test them independent of local environment configuration. This gives us confidence that our application will behave as expected both for new developers and production deployments.

For regression testing, particularly in a continuous integration (CI) environment, it is desirable to be able to start up the necessary resources, perform the testing and then remove all of the resources used for the test. Using docker for this works well because the infrastructure services, such as a database, do not already have to be in place. Instead, every component necessary is created as a container and removed when complete.

Why use Docker Compose?

Though we can run each Docker container individually, that quickly becomes tedious and difficult to manage. Docker Compose allows us to create a YAML (.yml) file that defines all of the containers we need to run and configure any environment variables we need to use to run the application. Docker Compose also allows us to communicate easily between multiple containers. Source: Docker documentation.

Application and its services in Docker Compose

Let's consider the following Application Under Test (AUT). There are three services that make up this application: api, ui and db. These services have been defined in the docker-compose.yml file:

version: '3.9'

networks:
 app_network:
   name: fastnetwork
   driver: bridge
   
services:
 api:
   image: api
   build:
     context: ./FastRest.Api
     dockerfile: Dockerfile
   environment:
     DB_CONN_STR: "host=db;port=5432;database=fsr-db;userid=postgres;password=postgr@12"
   ports:
     - "5001:80"
   networks:
     - app_network
   depends_on:
     - "db"

 ui:
   image: ui
   build:
     context: ./FastRest.Ui
     dockerfile: Dockerfile
   environment:
     DB_CONN_STR: "host=db;port=5432;database=fsr-db;userid=postgres;password=postgr@12"
   ports:
     - "5002:80"
   networks:
     - app_network
   depends_on:
     - "db"

 db:
   image: postgres
   environment:
     POSTGRES_DB: 'fsr-db'
     POSTGRES_HOST_AUTH_METHOD: 'trust'
   volumes:
     - fast_db_volume:/var/lib/postgresql/fastdb
     - ./FastRest.Db:/docker-entrypoint-initdb.d/
   ports:
     - 5432:5432
   networks:
     - app_network
   volumes:
     local_admindb_volume: {}
     local_coredb_volume: {}

The services in the file spin up a docker container each; a docker image is pulled from the docker hub and then assembled as per the instructions. api and ui services use the image based on http://mcr.microsoft.com/dotnet/sdk:5.0 while the db uses a Postgres image.

On running the command docker-compose, the following happens:

1. A network called app_network is created.
2. A container is created using api's configuration. It joins the network app_network under the name api.
3. A container is created using ui's configuration. It joins the network app_network under the name ui.
4. A container is created using db's configuration. It joins the network app_network under the name db.

These three dockerized services run together as an application in an isolated environment.

Introducing a Postman Docker Container to the mix

To run our Postman tests on this application, we simply need to add another Docker container to this mix. We do that by defining a service in the docker-compose.yml file; a service that can run the newman command against our Postman Collection.

On running the command docker-compose now, the following additional step will happen:

1. A container is created using Postman's configuration. It joins the network app_network under the name Postman.

Building the Postman Docker Container

We would need to assemble a Docker Container that has Newman setup to run our tests. We have two options at this point:

1. Pull a Docker image with all the pre-requisites installed (Node.js + Newman)
2. Pull a Docker image with Node.js installed and do the rest of the setup manually.

Let us look at these options one by one.

Option 1: A Pre-assembled Docker image

The easiest solution is to pull the postman/newman docker image from Docker Hub. This image has both Node.js and Newman installed and running. We just need to provide our Postman collection .json file as the argument.

We will specify a run command against the Postman tests. We can add the following to the docker-compose.yml file.

postman:
  image: postman/newman:alpine
  container_name: postmanTests
  command:
    run FastRestAPITests.postman_collection.json -k
    -r cli,json
    --reporter-json-export="reports/fastRestAPITests.json"
  volumes:
    - ./fastRest.QA.Tests/postman:/etc/newman
  depends_on:
    - "api"
    - "db"
  networks:
    - app_network

Let's look at the lines of code here:

image: postman/newman:alpine

An Alpine Linux image that has postman/newman installed needs to be pulled from the Docker Hub. Alpine images are recommended because of their smaller size.

container_name: postmanTests

The name specified helps to identify the container on doing a docker ps:

gandhish@PS % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a86d09bb81f6 postman/newman "newman run FastRest…" 22 hours ago Up
Less than a second postmanTests

command:
run FastRestAPITests.postman_collection.json -k

Since Newman is running, all we need to do is issue a run. A Postman collection (.json) has been specified to be run with Newman.

-r cli,json

Desired report formats: command line and JSON.

--reporter-json-export="reports/fastRestAPITests.json"

Location of the JSON report

volumes:
./fastRest.QA.Tests/postman:/etc/newman

Configuration map to host volumes to be used inside the containers. This setting allows the container to use the source code implemented on the host machine. ./fastRest.QA.Tests/postman specifies a local path in our code repository. This path needs to have the .json file

depends_on:
- "api"
- "db"

Wait for api, db services to start before starting this service. The API should be up to respond to the Postman requests.

networks:
- app_network

Use the network: app_network. This network has been defined at the top of the .yml file. All the services are in this network and can be accessed using localhost.

Option 2: Control the image assembly

Pull a base Docker image with Node.js. Install Newman tools manually.

In order to get more control over what gets installed, use a barebones Docker image and install the tools on top. Here, I am installing Newman manually because I would like to use a reporting tool add-on (newman-reporter-htmlextra) which is not available in the postman/newman Docker Image.

We would need a Dockerfile to setup this image. While the docker-compose.yml file defines all the services that make up the application, the Dockerfile is specific to a service and it contains the commands a user could call to assemble an image. Here's what it looks like:

FROM node:alpine
RUN npm install -g newman newman-reporter-htmlextra
WORKDIR /etc/newman

# Set newman as the default container command
ENTRYPOINT ["newman"]

A quick look at the lines of code:

FROM node:alpine 

Pull a docker image with the Alpine Linux distribution and Node.js installed.

RUN npm install -g newman newman-reporter-htmlextra  

Execute this command to globally install Newman and the add-on newman-reporter-htmlextra in the same directory.

WORKDIR /etc/newman 

The WORKDIR command is used to define the working directory of a Docker container at any given time. Here the working directory is being set to /etc/newman. Any RUN, CMD, ADD, COPY, or ENTRYPOINT command will be executed in the specified working directory.

ENTRYPOINT ["newman"] 

The ENTRYPOINT specifies a command that will always be executed when the container starts. Here the default container command is set to newman. This makes our image dedicated to Newman.

Our docker-compose.yml service definition will change slightly:

postman:
  container_name: postmanTests
  build:
    context: ./fastRest.QA.Tests/postman
    dockerfile: Dockerfile
  image: fastRest_postman_tests
  command:
    run FastRestAPITests.postman_collection.json -k
    -r htmlextra,cli,json
    --reporter-htmlextra-export="reports/fastRestAPITests.html"
    --reporter-json-export="reports/fastRestAPITests.json"
  volumes:
    - ./fastRest.QA.Tests/postman:/etc/newman
  depends_on:
    - "api"
    - "db"
  networks:
    - app_network

A quick explanation of the changes:

build:
 context: ./fastRest.QA.Tests/postman
 dockerfile: Dockerfile

Instructions to build the docker image. Path to the Dockerfile is to be mentioned here

-r htmlextra,cli,json

Desired report formats: HTML, command line, and JSON.

--reporter-htmlextra-export="reports/fastRestAPITests.html"

Location of the HTML report. The HTMLExtra reporting add-on creates a dashboard-style summary of the run. For more information, read: HTMLExtra.

HOST_PORT vs. CONTAINER_PORT

It is important to note the distinction between HOST_PORT and CONTAINER_PORT. In the above example, for api service, the HOST_PORT is 5001 and the CONTAINER_PORT is 80. Networked service-to-service communication uses the CONTAINER_PORT. When HOST_PORT is defined, the service is accessible outside the swarm as well.

So, the Postman Tests defined with the URL localhost:5001/…. will change to api:80/.. This change will have to be applied in the JSON collection file in the variable section:

"variable": [{
"key": "apiURL",
"value": "coreapi:80"
}]

Execute your Postman collection with Docker Compose

On running docker-compose, the Postman Docker Image will be built as per the instructions. The command will be executed on the container, and the results will be published in the specified formats. The JSON option creates a .json report file. The CLI option prints the results on the console:

If the Postman container returns 0, it means that all of our tests are successful.

The HTMLExtra report

If Option 2 was chosen, a Dashboard Summary style HTML report is generated using the add-on.

And that's it! Our tests will be run as part of the application start-up without needing to install any local dependencies!

Let's see the Postman on Docker in action:

Summary

A Postman Test Collection can be executed with Docker Compose using the Newman CLI utility.