Week 22: Contract Testing & Advanced Test Strategy

1. Consumer-Driven Contract Testing with Pact

Week 18's saga assumed inventory_service would keep responding to reservation requests in the shape order_service expects. Nothing in either service's own test suite verifies that assumption — each service's tests only prove it's internally correct, in isolation. A full end-to-end environment with every service running would catch a mismatch, but it's slow, flaky, and requires every team to coordinate a deploy just to run a test. Contract testing verifies the same boundary without either extreme.

In consumer-driven contract testing, the consumer writes a test expressing exactly what it expects from the provider — this generates a machine-readable "pact" file — and the provider later replays that exact expectation against its real implementation to confirm it still holds.

import pytest
from pact import Consumer, Provider

pact = Consumer("order-service").has_pact_with(Provider("inventory-service"))

def test_reserve_inventory_contract():
    expected = {
        "reservation_id": pact.matchers.Like("res_abc123"),
        "success": True,
    }
    (
        pact
        .given("SKU ABC123 has stock available")
        .upon_receiving("a request to reserve inventory")
        .with_request("post", "/inventory/reserve", body={"sku": "ABC123", "quantity": 2})
        .will_respond_with(200, body=expected)
    )

    with pact:
        response = inventory_client(pact.uri).reserve("ABC123", 2)
        assert response["success"] is True

Running this test produces a pact file — a JSON document literally describing "when I send this request, I expect this response shape back." That file is published to a shared Pact Broker, which inventory_service's own build then downloads and replays:

from pact import Verifier

def test_verify_pacts():
    verifier = Verifier(provider="inventory-service", provider_base_url="http://localhost:8001")
    verifier.set_state_handler_url("http://localhost:8001/_pact/provider_states")

    success, _ = verifier.verify_with_broker(
        broker_url="https://pact-broker.acme.internal",
        publish_verification_results=True,
        provider_version="1.4.2",
    )
    assert success == 0

If inventory_service ever changes its response shape in a way that breaks what order_service actually depends on, this provider-side verification fails in inventory_service's own CI pipeline — before it's ever deployed, and without order_service needing to be running anywhere nearby.

2. Mutation Testing with mutmut

Code coverage answers "did my tests execute this line?" It says nothing about whether those tests would actually notice if that line's logic were wrong — a test that calls a function and asserts nothing meaningful about its result achieves 100% coverage of that function while catching zero bugs in it. Mutation testing answers the sharper question directly: it automatically introduces small, deliberate bugs ("mutants") into your code — flipping a > to >=, changing a + to -, negating a boolean — and reruns your test suite against each mutated version.

pip install mutmut
mutmut run --paths-to-mutate src/billing/discounts.py
mutmut results

If a test suite catches the mutant — some test fails because the mutated behavior differs from what was asserted — the mutant is killed, a good outcome. If every test still passes despite the introduced bug, the mutant survives — a concrete, specific sign that no test actually verifies that piece of logic correctly, no matter what the coverage report claims.

# original
def is_eligible_for_discount(order_total: float) -> bool:
    return order_total > 100

# mutmut's mutant: flips > to >=
def is_eligible_for_discount(order_total: float) -> bool:
    return order_total >= 100

# If this test suite's only test is is_eligible_for_discount(150) -> True,
# BOTH the original and the mutant pass it. The mutant SURVIVES --
# revealing that the boundary condition (order_total == 100) is
# completely untested, something 100% line coverage would never show.

That surviving mutant points to exactly the missing test: an assertion at order_total = 100 itself, the boundary the two operators actually disagree on. Mutation testing doesn't replace writing good tests — it's a tool for finding the specific gaps a coverage report can't see.

3. Testing Resilience Under Fault Injection

Week 9's retry and circuit-breaker configuration has never actually been tested against a real failure — it's been trusted to work because the configuration looks correct. respx (a mocking library for httpx, already familiar from mocking HTTP dependencies) can simulate the specific failure modes those resilience patterns exist to handle, proving the behavior rather than assuming it.

import respx
import httpx
import pytest

@respx.mock
async def test_falls_back_when_inventory_service_times_out():
    respx.post("http://inventory-service/reserve").mock(
        side_effect=httpx.TimeoutException("simulated timeout")
    )

    result = await inventory_client.reserve("ABC123", 2)

    assert result.status == ReservationStatus.FALLBACK_UNAVAILABLE

@respx.mock
async def test_circuit_opens_after_consecutive_failures():
    respx.post("http://inventory-service/reserve").mock(
        return_value=httpx.Response(500)
    )

    for _ in range(10):
        await inventory_client.reserve("ABC123", 1)   # trip the circuit breaker

    # the circuit should now be OPEN -- calls fail fast without even
    # hitting the mocked endpoint, which we can verify directly:
    assert circuit_breaker.current_state == "open"

These tests aren't checking that your business logic is correct — they're checking that the resilience configuration itself behaves as designed: that a slow dependency actually triggers the fallback instead of hanging the caller, and that enough consecutive failures actually trip the circuit breaker rather than the threshold being silently misconfigured.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.