Testing APIs manually works fine when you're starting out. But as your API grows and your team ships faster, manual testing becomes a bottleneck. You need automated tests running in your CI/CD pipeline to catch bugs before they reach production.
This guide walks through building a complete API testing automation strategy, from unit tests to load tests, all integrated into your deployment pipeline.
Every code change carries risk. A new feature might break existing endpoints. A database migration could slow down response times. A dependency update might introduce security vulnerabilities.
Automated testing in CI/CD catches these issues early:
The goal isn't perfect test coverage. It's catching the bugs that matter before users do.
Not all tests are created equal. Some run in milliseconds, others take minutes. Some catch logic bugs, others find performance issues.
The test pyramid helps you balance speed and coverage:
Unit Tests (70%): Fast, focused tests of individual functions Integration Tests (20%): Tests of API endpoints and database interactions End-to-End Tests (10%): Full workflow tests including external dependencies
For APIs, this translates to:
Let's build each layer.
Newman runs Postman collections from the command line, making it perfect for CI/CD.
First, export your Postman collection. In Postman, click the three dots next to your collection and choose "Export". Save it as petstore-api.json.
Install Newman:
npm install -g newman
npm install -g newman-reporter-htmlextra
Run your collection locally:
newman run petstore-api.json \
--environment petstore-env.json \
--reporters cli,htmlextra \
--reporter-htmlextra-export test-report.html
This runs all requests in your collection and generates a detailed HTML report.
Good Postman tests check more than status codes. Here's a complete test for a GET endpoint:
// Test: Get Pet by ID
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
pm.test("Response time is less than 500ms", function () {
pm.expect(pm.response.responseTime).to.be.below(500);
});
pm.test("Response has correct structure", function () {
const jsonData = pm.response.json();
pm.expect(jsonData).to.have.property('id');
pm.expect(jsonData).to.have.property('name');
pm.expect(jsonData).to.have.property('status');
pm.expect(jsonData.status).to.be.oneOf(['available', 'pending', 'sold']);
});
pm.test("Pet ID matches request", function () {
const jsonData = pm.response.json();
const requestedId = pm.variables.get("petId");
pm.expect(jsonData.id.toString()).to.equal(requestedId);
});
// Save data for next request
pm.environment.set("lastPetName", pm.response.json().name);
This test validates: - HTTP status - Response time - Data structure - Business logic - Sets up data for dependent tests
Here's a GitHub Actions workflow that runs Newman tests:
name: API Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
api-tests:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: testpass
POSTGRES_DB: petstore_test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install Newman
run: |
npm install -g newman
npm install -g newman-reporter-htmlextra
- name: Start API server
run: |
npm install
npm run migrate
npm start &
sleep 10
env:
DATABASE_URL: postgresql://postgres:testpass@localhost:5432/petstore_test
NODE_ENV: test
- name: Run API tests
run: |
newman run tests/petstore-api.json \
--environment tests/ci-environment.json \
--reporters cli,htmlextra,json \
--reporter-htmlextra-export test-report.html \
--reporter-json-export test-results.json \
--bail
- name: Upload test report
if: always()
uses: actions/upload-artifact@v3
with:
name: newman-report
path: test-report.html
- name: Comment PR with results
if: github.event_name == 'pull_request'
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
const results = JSON.parse(fs.readFileSync('test-results.json'));
const summary = `
## API Test Results
- Total Tests: ${results.run.stats.tests.total}
- Passed: ${results.run.stats.tests.passed}
- Failed: ${results.run.stats.tests.failed}
- Duration: ${results.run.timings.completed - results.run.timings.started}ms
`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: summary
});
The --bail flag stops tests on first failure, saving CI minutes when something's broken.
Newman tests functionality. k6 tests performance under load.
Install k6:
# macOS
brew install k6
# Linux
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6
Here's a load test for the PetStore API:
// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';
const errorRate = new Rate('errors');
export const options = {
stages: [
{ duration: '2m', target: 100 }, // Ramp up to 100 users
{ duration: '5m', target: 100 }, // Stay at 100 users
{ duration: '2m', target: 200 }, // Spike to 200 users
{ duration: '5m', target: 200 }, // Stay at 200 users
{ duration: '2m', target: 0 }, // Ramp down
],
thresholds: {
http_req_duration: ['p(95)<500', 'p(99)<1000'], // 95% under 500ms
http_req_failed: ['rate<0.01'], // Less than 1% errors
errors: ['rate<0.05'], // Less than 5% business logic errors
},
};
const BASE_URL = __ENV.API_URL || 'http://localhost:3000';
export default function () {
// Test 1: List pets
let listResponse = http.get(`${BASE_URL}/api/v1/pets?limit=20`);
check(listResponse, {
'list pets status 200': (r) => r.status === 200,
'list pets has data': (r) => JSON.parse(r.body).length > 0,
}) || errorRate.add(1);
sleep(1);
// Test 2: Get specific pet
const pets = JSON.parse(listResponse.body);
if (pets.length > 0) {
const petId = pets[0].id;
let getResponse = http.get(`${BASE_URL}/api/v1/pets/${petId}`);
check(getResponse, {
'get pet status 200': (r) => r.status === 200,
'get pet has name': (r) => JSON.parse(r.body).name !== undefined,
}) || errorRate.add(1);
}
sleep(1);
// Test 3: Create pet
const payload = JSON.stringify({
name: `TestPet-${Date.now()}`,
status: 'available',
category: 'dog',
});
const params = {
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${__ENV.API_TOKEN}`,
},
};
let createResponse = http.post(`${BASE_URL}/api/v1/pets`, payload, params);
check(createResponse, {
'create pet status 201': (r) => r.status === 201,
'create pet returns id': (r) => JSON.parse(r.body).id !== undefined,
}) || errorRate.add(1);
sleep(2);
}
Run it locally:
k6 run load-test.js
Add it to CI/CD:
- name: Run load tests
run: |
k6 run --out json=load-test-results.json load-test.js
env:
API_URL: http://localhost:3000
API_TOKEN: ${{ secrets.TEST_API_TOKEN }}
- name: Check performance thresholds
run: |
if grep -q '"failed":true' load-test-results.json; then
echo "Load test thresholds failed"
exit 1
fi
k6 fails the build if thresholds aren't met, preventing performance regressions.
Contract testing ensures your API matches its OpenAPI specification. When the spec says an endpoint returns a Pet object, contract tests verify it actually does.
Use Schemathesis for contract testing:
pip install schemathesis
Test against your OpenAPI spec:
# test_contract.py
import schemathesis
schema = schemathesis.from_uri("http://localhost:3000/api/v1/openapi.json")
@schema.parametrize()
def test_api_contract(case):
response = case.call()
case.validate_response(response)
Run with pytest:
pytest test_contract.py --hypothesis-max-examples=50
Schemathesis generates test cases from your OpenAPI spec, hitting endpoints with various inputs to find edge cases.
Add to CI/CD:
- name: Run contract tests
run: |
pip install schemathesis pytest
pytest test_contract.py \
--hypothesis-max-examples=100 \
--junit-xml=contract-test-results.xml
- name: Publish test results
if: always()
uses: EnricoMi/publish-unit-test-result-action@v2
with:
files: contract-test-results.xml
Contract tests catch breaking changes before they reach consumers.
Good tests need good data. But test data is tricky:
Here's a test data strategy that works:
Create a seed script that runs before tests:
// seed-test-data.js
const { PrismaClient } = require('@prisma/client');
const prisma = new PrismaClient();
async function seed() {
// Clear existing data
await prisma.pet.deleteMany();
await prisma.category.deleteMany();
// Create categories
const categories = await Promise.all([
prisma.category.create({ data: { name: 'dog' } }),
prisma.category.create({ data: { name: 'cat' } }),
prisma.category.create({ data: { name: 'bird' } }),
]);
// Create test pets
const pets = [
{ name: 'Buddy', status: 'available', categoryId: categories[0].id },
{ name: 'Whiskers', status: 'pending', categoryId: categories[1].id },
{ name: 'Tweety', status: 'sold', categoryId: categories[2].id },
];
for (const pet of pets) {
await prisma.pet.create({ data: pet });
}
console.log('Test data seeded successfully');
}
seed()
.catch(console.error)
.finally(() => prisma.$disconnect());
Run it before tests:
- name: Seed test data
run: node seed-test-data.js
env:
DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}
For dynamic test data, use factories:
// test-factories.js
let petIdCounter = 1000;
function createPet(overrides = {}) {
return {
id: petIdCounter++,
name: `TestPet-${Date.now()}`,
status: 'available',
category: 'dog',
...overrides,
};
}
function createOrder(overrides = {}) {
return {
id: petIdCounter++,
petId: createPet().id,
quantity: 1,
status: 'placed',
...overrides,
};
}
module.exports = { createPet, createOrder };
Use in tests:
const { createPet } = require('./test-factories');
pm.test("Create pet with custom data", function () {
const pet = createPet({ name: 'CustomDog', status: 'pending' });
pm.sendRequest({
url: 'http://localhost:3000/api/v1/pets',
method: 'POST',
header: { 'Content-Type': 'application/json' },
body: { mode: 'raw', raw: JSON.stringify(pet) }
}, function (err, response) {
pm.expect(response.code).to.equal(201);
});
});
Keep environment configs separate:
// ci-environment.json
{
"id": "ci-environment",
"name": "CI Environment",
"values": [
{
"key": "baseUrl",
"value": "http://localhost:3000",
"enabled": true
},
{
"key": "apiToken",
"value": "{{CI_API_TOKEN}}",
"enabled": true
},
{
"key": "testTimeout",
"value": "5000",
"enabled": true
}
]
}
Pass secrets from CI:
- name: Run tests
run: |
newman run tests/petstore-api.json \
--environment tests/ci-environment.json \
--env-var "CI_API_TOKEN=${{ secrets.TEST_API_TOKEN }}"
As your test suite grows, test data becomes harder to manage. Here are patterns that scale:
Capture known-good responses and compare future runs:
pm.test("Response matches snapshot", function () {
const response = pm.response.json();
const snapshot = pm.environment.get("petListSnapshot");
// Compare structure, not exact values
pm.expect(Object.keys(response)).to.deep.equal(Object.keys(JSON.parse(snapshot)));
});
Delete test data after each run:
// In Postman collection's "Tests" tab (runs after all requests)
pm.test("Cleanup test data", function () {
const createdIds = pm.environment.get("createdPetIds") || [];
createdIds.forEach(id => {
pm.sendRequest({
url: `${pm.environment.get("baseUrl")}/api/v1/pets/${id}`,
method: 'DELETE',
header: { 'Authorization': `Bearer ${pm.environment.get("apiToken")}` }
});
});
pm.environment.unset("createdPetIds");
});
Run each test suite against a fresh database:
- name: Create test database
run: |
docker run -d \
--name test-db-${{ github.run_id }} \
-e POSTGRES_PASSWORD=testpass \
-e POSTGRES_DB=petstore_test \
-p 5432:5432 \
postgres:15
- name: Run migrations
run: npm run migrate
env:
DATABASE_URL: postgresql://postgres:testpass@localhost:5432/petstore_test
- name: Run tests
run: newman run tests/petstore-api.json
- name: Cleanup
if: always()
run: docker rm -f test-db-${{ github.run_id }}
Each CI run gets a clean database that's destroyed after tests complete.
Here's a complete CI/CD pipeline with all test types:
name: Complete API Testing Pipeline
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: testpass
POSTGRES_DB: petstore_test
options: >-
--health-cmd pg_isready
--health-interval 10s
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: |
npm install
npm install -g newman newman-reporter-htmlextra
pip install schemathesis pytest
- name: Run database migrations
run: npm run migrate
env:
DATABASE_URL: postgresql://postgres:testpass@localhost:5432/petstore_test
- name: Seed test data
run: node scripts/seed-test-data.js
env:
DATABASE_URL: postgresql://postgres:testpass@localhost:5432/petstore_test
- name: Start API server
run: |
npm start &
sleep 10
env:
DATABASE_URL: postgresql://postgres:testpass@localhost:5432/petstore_test
NODE_ENV: test
- name: Run contract tests
run: pytest tests/test_contract.py --hypothesis-max-examples=100
- name: Run functional tests
run: |
newman run tests/petstore-api.json \
--environment tests/ci-environment.json \
--env-var "API_TOKEN=${{ secrets.TEST_API_TOKEN }}" \
--reporters cli,htmlextra \
--reporter-htmlextra-export newman-report.html
- name: Run load tests
run: k6 run --out json=load-results.json tests/load-test.js
env:
API_URL: http://localhost:3000
API_TOKEN: ${{ secrets.TEST_API_TOKEN }}
- name: Upload test reports
if: always()
uses: actions/upload-artifact@v3
with:
name: test-reports
path: |
newman-report.html
load-results.json
- name: Check test results
run: |
if grep -q '"failed":true' load-results.json; then
echo "Load tests failed"
exit 1
fi
This pipeline runs in about 5-10 minutes and catches most issues before deployment.
After building dozens of API test suites, here's what works:
1. Test the happy path first: Get basic functionality working before edge cases
2. Make tests independent: Each test should run in any order
3. Use meaningful test names: "Get pet by ID returns 200" beats "Test 1"
4. Keep tests fast: Slow tests don't get run. Aim for under 10 minutes total.
5. Test one thing per test: Easier to debug when tests fail
6. Use realistic test data: Production bugs hide in edge cases
7. Monitor test flakiness: Flaky tests erode confidence. Fix or delete them.
8. Version your test collections: Test code is code. Treat it like production code.
9. Run tests locally: Don't wait for CI to catch obvious bugs
10. Review test failures quickly: A failing test that nobody fixes trains the team to ignore failures
Automated API testing in CI/CD isn't optional anymore. It's how modern teams ship fast without breaking things.
Start simple: Add Newman tests for your critical endpoints. Then layer in contract tests and load tests as your API matures.
The goal isn't 100% test coverage. It's catching the bugs that matter before users do. Build that foundation, and you'll ship with confidence.
Your tests are only as good as your commitment to fixing them when they fail. Make that commitment, and automated testing becomes your team's superpower.
