Mocking external HTTP requests in Python using unittest and pytest

There are many ways to mock up HTTP requests in python and to be honest with everything I found on the web, there just seemed to be so many different solutions. After several days of researching and trying solutions I found online to get working in my current project, I had to combine several techniques to match my needs. First to start out with there are python packages that allow you to mock up your HTTP requests, two in particular I tried out are the requests-mock and the responses package. Both these packages work with the requests http package for handling HTTP requests. This article uses the requests package, because that is what I am most familiar with.

Working with those packages was fine, however there was a bunch of repetitive code that had to be setup for each HTTP request under test. I could have probably refactored the repetitive code into pytest fixtures after all, since I am using pytest in the project. However, I am not sure that would have removed some duplicate code either. After further googling I came across this article online, https://rednafi.github.io/reflections/patching-test-dependencies-via-pytest-fixture-unittest-mock.html. This article explains how to use the built-in unittest mock package in Python to mock up the HTTP requests while using pytest fixtures. This article was a good starting point however it uses the httpx http package, so it was not entirely what I needed.

I eventually came up with a working method of using unittest mocks and pytest combined in order to mock up the requests session objects as intended. Let's get started with actual code examples to explain how to do all of this.

Okay so first we need to setup the file that will make the HTTP requests to the external API of our choice. Create a file name anything you want, I am going to choose my_external_api.py. In this file we need to import the `requests` package, so if you have not installed that package run the following:

pip install requests

Now we need to import the requests package and setup our class to access our external api of our choice.

my_external_api.py

import requests

class ExternalApi():
    def __init__(self, base_url = None, sess = None):
         url = 'https://example.com/api/v1/'
         base_url = url if base_url is None else base_url
         self.base_url = base_url
         sess = requests.Session() if sess is None else sess
         self.sess = sess

So we now have our class setup with our constructor that sets up a base_url and a requests.Session() object if one is not provided. Both these attributes are needed so that we can mock up the url and the requests.Session object that will be used in the code that follows.

Now we need to create a method to POST data to a the API and get the returned response from the API.

def reports_per_year(self, params = None):
    if params is None:
        params = ''
    with self.sess as client:
        try:
            resp = client.post(
                self.base_url+'report',
                # These are query params that are passed into the url
                params=params,
                # This is the json data that we are posting to the API
                json={
                    "categories": [
                            'Tests',
                        ],
                    },
                headers={'Content-Type': 'application/json'}
            )
            # This is to check to see if the API is responding with a code
            # other than 2xx
            resp.raise_for_status()
        except requests.exceptions.HTTPError as err:
            print("Http Error:", err)
            raise err
        except requests.exceptions.RequestException as err:
            print("Error:", err)
            raise err
    # If there is a successful response from the API, we return the JSON
    # from the response object
    return resp.json()

Now we need to setup our test file to test this method and see how to mock up our HTTP request. Create a file called test_my_external_api.py.

test_my_external_api.py

from my_external_api import ExternalApi
# Import the patch method from the unittest.mock package in order to patch # the particular methods of the external API.
from unittest.mock import patch
import pytest
# Import the respective requests error types for mocking purposes
from requests.exceptions import HTTPError, RequestException

def base_url():
    return 'https://test.external-api.com/api/v1/'

def example_data_report_json():
    return [
        {
            "id": "1",
            "status": "Closed",
            "category": "Tests",
            "summary": "test power",
        },
        {
            "id": "2",
            "status": "Pending",
            "category": "Tests",
            "summary": "test maintenance",
        }
        
def example_data_report_json_matchers():
    return {
        "categories": [
            "Tests"
        ],
    }
    
class TestExternalApi():
# This is the actual decorator declared to mock up the requests session 
# post method.  Since we are using a session instead of a regular requests 
# object we must mock up the session object as well as the method used on 
# the session which is post in our case
@patch('requests.Session.post', autospec=True)
def test_reports_per_year(self, mock_post):
    url = base_url()
    external_api = ExternalApi(url)
    # Here we mock up the status_code on the mock_post object that
    # is passed into the test function
    mock_post.return_value.status_code.return_value = 200
    # Here we tell the mock_post object to return a method named json and 
    # then tell the json method to return our example json data
    mock_post.return_value.json.return_value = example_data_report_json()
    
    # This is the actual call to the external API that is made.
    # This is mocked up and no external call will actually be made
    resp = external_api.reports_per_year()
    
    # Here we are asserting that the mock_post object is called only once
    # and that it is called with certain arguments
    mock_post.assert_called_once_with(
         external_api.sess,
         url+'report',
         json=example_data_report_json_matchers(),
         headers={"Content-Type": "application/json"}
    )
    
    # Here we assert that the response is a list and that there are
    # exactly 2 items in that list
    assert isinstance(resp, list)
    assert len(resp) == 2

Hopefully the comments in the file made sense as to what is going on and why. Basically a high level overview is that we need to mock up all the requests.Session.post calls that are made in the my_external_api.py file. We are not mocking up the ExternalApi class methods in this case. We mock up the ExternalApi class methods only when calling those from another part of the application or from another file in the application.

Thank you for reading and stayed tuned for more content.