Mocking socket calls in C++

Question

Explanation

I'm trying to get better unit test coverage of my server.

I needed to mock out some system libraries (I could not find anything that worked easily for C++).

Because I did not want to do this ad-hoc, I added some automated tools to my build scripts that do all the work.

In the test directory I have a Mock.def that defines the system calls I want to mock. It supports two types of line: #include and MOCK_SYSTEM_FUNC.

Example:

#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <netdb.h>
#include "Socket.h"




MOCK_SYSTEM_FUNC(fcntlWrapper)
MOCK_SYSTEM_FUNC(closeWrapper)
MOCK_SYSTEM_FUNC(socketWrapper)
MOCK_SYSTEM_FUNC(connect)
MOCK_SYSTEM_FUNC(bind)
MOCK_SYSTEM_FUNC(listen)
MOCK_SYSTEM_FUNC(acceptWrapper)
MOCK_SYSTEM_FUNC(readWrapper)
MOCK_SYSTEM_FUNC(writeWrapper)
MOCK_SYSTEM_FUNC(shutdown)
MOCK_SYSTEM_FUNC(gethostbyname)

With this, my build scripts build three files:

MockHeaders.h
MockHeaders.cpp
ThorMock.h

When building the unit tests, the MockHeaders.cpp is compiled and linked with the unit tests; MockHeaders.h is forced on all the source files with the -include directive from gcc.

The ThorMock.h can be included by unit tests that want to mock out a system file.

The result of the generation above is:

Mockheaders.h

/*
 * This is a generated file do not edit.
 * See ThorBuilder for details
 *
 */


#ifndef THORSANVIL_BUILD_TOOLS_MOCK_H
#define THORSANVIL_BUILD_TOOLS_MOCK_H

#include <functional>

#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <netdb.h>
#include "Socket.h"

namespace ThorsAnvil
{
    namespace BuildTools
    {
        namespace Mock
        {


extern std::function<decltype(::fcntlWrapper)> mockfcntlWrapper;
extern std::function<decltype(::closeWrapper)> mockcloseWrapper;
extern std::function<decltype(::socketWrapper)> mocksocketWrapper;
extern std::function<decltype(::connect)> mockconnect;
extern std::function<decltype(::bind)> mockbind;
extern std::function<decltype(::listen)> mocklisten;
extern std::function<decltype(::acceptWrapper)> mockacceptWrapper;
extern std::function<decltype(::readWrapper)> mockreadWrapper;
extern std::function<decltype(::writeWrapper)> mockwriteWrapper;
extern std::function<decltype(::shutdown)> mockshutdown;
extern std::function<decltype(::gethostbyname)> mockgethostbyname;

        }
    }
}

#define fcntlWrapper ThorsAnvil::BuildTools::Mock::mockfcntlWrapper
#define closeWrapper ThorsAnvil::BuildTools::Mock::mockcloseWrapper
#define socketWrapper ThorsAnvil::BuildTools::Mock::mocksocketWrapper
#define connect ThorsAnvil::BuildTools::Mock::mockconnect
#define bind ThorsAnvil::BuildTools::Mock::mockbind
#define listen ThorsAnvil::BuildTools::Mock::mocklisten
#define acceptWrapper ThorsAnvil::BuildTools::Mock::mockacceptWrapper
#define readWrapper ThorsAnvil::BuildTools::Mock::mockreadWrapper
#define writeWrapper ThorsAnvil::BuildTools::Mock::mockwriteWrapper
#define shutdown ThorsAnvil::BuildTools::Mock::mockshutdown
#define gethostbyname ThorsAnvil::BuildTools::Mock::mockgethostbyname

#endif

MockHeaders.cpp

/*
 * This is a generated file do not edit.
 * See ThorBuilder for details
 *
 */

#include "MockHeaders.h"


#undef fcntlWrapper
#undef closeWrapper
#undef socketWrapper
#undef connect
#undef bind
#undef listen
#undef acceptWrapper
#undef readWrapper
#undef writeWrapper
#undef shutdown
#undef gethostbyname
namespace ThorsAnvil
{
    namespace BuildTools
    {
        namespace Mock
        {

std::function<decltype(::fcntlWrapper)> mockfcntlWrapper = ::fcntlWrapper;
std::function<decltype(::closeWrapper)> mockcloseWrapper = ::closeWrapper;
std::function<decltype(::socketWrapper)> mocksocketWrapper = ::socketWrapper;
std::function<decltype(::connect)> mockconnect = ::connect;
std::function<decltype(::bind)> mockbind = ::bind;
std::function<decltype(::listen)> mocklisten = ::listen;
std::function<decltype(::acceptWrapper)> mockacceptWrapper = ::acceptWrapper;
std::function<decltype(::readWrapper)> mockreadWrapper = ::readWrapper;
std::function<decltype(::writeWrapper)> mockwriteWrapper = ::writeWrapper;
std::function<decltype(::shutdown)> mockshutdown = ::shutdown;
std::function<decltype(::gethostbyname)> mockgethostbyname = ::gethostbyname;


        }
    }
}

ThorMock.h

#ifndef THORSANVIL_BUILDTOOLS_MOCK_MOCK_FUNCTION_H
#define THORSANVIL_BUILDTOOLS_MOCK_MOCK_FUNCTION_H

template<typename Func>
struct MockOutFunction
{
    std::function<Func>     old;
    std::function<Func>&    orig;
    MockOutFunction(std::function<Func>& orig, std::function<Func>&& mock)
        : old(std::move(mock))
        , orig(orig)
    {
        swap(old, orig);
    }
    ~MockOutFunction()
    {
        swap(old, orig);
    }
};
#define MOCK_SYS(func, lambda)      MockOutFunction<decltype(func)>  MockOutFunction ## func(ThorsAnvil::BuildTools::Mock::mock ## func, lambda)



#undef fcntlWrapper
#undef closeWrapper
#undef socketWrapper
#undef connect
#undef bind
#undef listen
#undef acceptWrapper
#undef readWrapper
#undef writeWrapper
#undef shutdown
#undef gethostbyname

namespace ThorsAnvil
{
    namespace BuildTools
    {
        namespace Mock
        {



        }
    }
}

#endif

Usage

In the test file I can temporarily mock out a system call using MOCK_SYS and supplying the function name and a lambda to replace it.

TEST(SocketExceptionTest, ServerSocketAcceptFailsAcceptCall)
{
    MOCK_SYS(socketWrapper, [](int, int, int){return 5;});
    MOCK_SYS(bind,          [](int, SocketAddr const*, std::size_t){return 0;});
    MOCK_SYS(listen,        [](int, int){return 0;});
    MOCK_SYS(closeWrapper,  [](int socketId){return 0;});
    MOCK_SYS(acceptWrapper, [](int, SocketAddr*, socklen_t*){return -1;});

    ServerSocket    server(8080, true);
    ASSERT_THROW(
        server.accept(),
        std::runtime_error
    );
}

Building

The code that generates the three files is part of the generalized Makefile I have, so all my projects can now use the mocking technique. If anybody can help make this neater I would appreciate the input.

coverage/MockHeaders.h: test/Mock.def coverage.Dir
    @cp $(THORSANVIL_ROOT)/build/mock/MockHeaders.h.prefix coverage/MockHeaders.h
    @if [[ -e test/Mock.def ]]; then        \
        perl -ne '/(#include .*)/ and print "$$1\n"' test/Mock.def >> coverage/MockHeaders.h; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.h.preamb >> coverage/MockHeaders.h
    @if [[ -e test/Mock.def ]]; then        \
        perl -ne '/MOCK_SYSTEM_FUNC\(([^)]*)\)/ and print "extern std::function<decltype(::$$1)> mock$$1;\n"' test/Mock.def >> coverage/MockHeaders.h; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.h.median >> coverage/MockHeaders.h
    @if [[ -e test/Mock.def ]]; then        \
        perl -ne '/MOCK_SYSTEM_FUNC\(([^)]*)\)/ and print "#define $$1 ThorsAnvil::BuildTools::Mock::mock$$1\n"' test/Mock.def >> coverage/MockHeaders.h; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.h.suffix >> coverage/MockHeaders.h

coverage/ThorMock.h: test/Mock.def coverage.Dir
    @cp $(THORSANVIL_ROOT)/build/mock/ThorMock.h.prefix coverage/ThorMock.h
    @cat $(THORSANVIL_ROOT)/build/mock/ThorMock.h.preamb >> coverage/ThorMock.h
    @if [[ -e Mock.def ]]; then     \
        perl -ne '/MOCK_SYSTEM_FUNC\(([^)]*)\)/ and print "#undef $$1\n"' Mock.def >> coverage/ThorMock.h; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/ThorMock.h.median >> coverage/ThorMock.h
    @cat $(THORSANVIL_ROOT)/build/mock/ThorMock.h.suffix >> coverage/ThorMock.h

coverage/MockHeaders.cpp:: test/Mock.def coverage/MockHeaders.h coverage.Dir
    @cp $(THORSANVIL_ROOT)/build/mock/MockHeaders.cpp.prefix coverage/MockHeaders.cpp
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.cpp.preamb >> coverage/MockHeaders.cpp
    @if [[ -e test/Mock.def ]]; then        \
        perl -ne '/MOCK_SYSTEM_FUNC\(([^)]*)\)/ and print "#undef $$1\n"' test/Mock.def >> coverage/MockHeaders.cpp; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.cpp.median >> coverage/MockHeaders.cpp
    @if [[ -e test/Mock.def ]]; then        \
        perl -ne '/MOCK_SYSTEM_FUNC\(([^)]*)\)/ and print "std::function<decltype(::$$1)> mock$$1 = $$1;\n"' test/Mock.def >> coverage/MockHeaders.cpp; \
    fi
    @cat $(THORSANVIL_ROOT)/build/mock/MockHeaders.cpp.suffix >> coverage/MockHeaders.cpp

Notes

In unit test builds (or code coverage builds as I call them) there is an indirection. The system call is actually held in a variable and must be looked up. So there is a slight penalty when running the tests.

This indirection is not compiled into the debug or release versions. So these builds do not see any indirection.

I use macros to replace the actual function call. This works most of the time as you can #include the header that defines the overridable function before that macro is defined (hence the includes in the Mock.def file).

Sometimes the system names are just too common. And defining a macro to override them will hit more of your function names than you actually want (examples are read() and write()). In these cases I have had to define a wrapper function in the header file:

inline int closeWrapper(int fd)                             {return ::close(fd);}
inline int socketWrapper(int family, int type, int protocol){return ::socket(family, type, protocol);}
inline ssize_t readWrapper(int fd, void* buf, size_t count) {return ::read(fd, buf, count);}
inline ssize_t writeWrapper(int fd, void const* buf, size_t count){return ::write(fd, buf, count);}
inline int fcntlWrapper(int fd, int cmd, int value)         {return ::fcntl(fd, cmd, value);}

Note these calls will not be affected as I include the header file that defines them in the Mock.def file.

Also because the functions are inline I am hoping that it will not cost me an extra function call in reality as these will be inlined. I then use only the wrapper functions in my code.

Answer 1

This is a worthwhile task - I'm a firm believer in maximising my opportunities for automated testing. It tends to increase my peers' respect for my code, as they know they are less likely to find trivial bugs (and we can focus on reproducing and fixing more challenging stuff).

I do worry that the macro substitution has its problems. As you observe, many of the functions to be mocked share names with other identifiers, such as member functions. You can use different names such as closeWrapper() and friends as shown, but then you¹ need the discipline to always use the wrapper - you might need to use a static analysis tool to catch inadvertent uses of the real function.

¹ "You" includes others who may edit this code in future!

Alternative mechanisms

My two suggestions are:

1. Implement a higher-level abstraction of the sockets library, and consistently use that abstraction, or
2. link with your mocking library ahead of the standard library (either at compile time, or, if you have a dynamic linker with LD_PRELOAD support², at runtime).

² Linux ld.so has such support.

Obviously your tests have more validity if performed in an environment as close as possible to the target platform, but here we're verifying correctness of logic that shouldn't be platform-dependent, so if the two are different for these tests, it may still be valuable on balance.

Consider verifying arguments

At present, the mocks do not verify that they have been called with the expected arguments. Good mocking frameworks allow us to test that the methods are called the right number of times, with the right number of arguments. Consider an interface that can be called something like

MockedMethod m{::bind};
in_addr_t localhost{inet_addr("127.0.0.1")};
m.expectCall(4, localhost, sizeof localhost).return(5);

Makefile simplification

The targets all depend on test/Mock.def, yet they test whether it exists, using [[ -e test/Mock.def ]]. That test must always be true, as Make will have made it as a dependency of the rule.

Those long command sequences are candidates for short Perl scripts (or even a single script with a couple of options to enable the filtering of test/Mock.def). The rules also appear to use additional files not mentioned as dependencies.

My suggestion here is to make some intermediate files, then simply concatenate:

MOCK_DIR=$(THORSANVIL_ROOT)/build/mock

coverage/MockHeaders.h: \
    $(MOCK_DIR)/MockHeaders.h.prefix \
    mock.includes \
    $(MOCK_DIR)/MockHeaders.h.preamb \
    mock.externs \
    $(MOCK_DIR)/MockHeaders.h.median \
    mock.defines \
    $(MOCK_DIR)/MockHeaders.h.suffix
    cat $^ >$@

coverage/MockHeaders.cpp: \
    $(MOCK_DIR)/MockHeaders.h.prefix \
    $(MOCK_DIR)/MockHeaders.h.preamb \
    mock.undefs \
    $(MOCK_DIR)/MockHeaders.h.median \
    mock.functions \
    $(MOCK_DIR)/MockHeaders.h.suffix
    cat $^ >$@

coverage/ThorMock.h: \
    $(MOCK_DIR)/ThorMock.h.prefix \
    $(MOCK_DIR)/ThorMock.h.preamb \
    mock.undefs \
    $(MOCK_DIR)/ThorMock.h.median \
    $(MOCK_DIR)/ThorMock.h.suffix
    cat $^ >$@

mock.includes: test/Mock.def
    grep <$< >$@ 's/^\s*#include\b.*'

mock.externs: test/Mock.def
    sed <$< >$@ -ne 's/MOCK_SYSTEM_FUNC(\([^)]*\))/extern std::function<decltype(::\1)> mock\1;/gp'

mock.functions: mock.externs
    sed <$< >$@ -e 's/extern //g'

mock.defines: test/Mock.def
    sed <$< >$@ -ne 's/MOCK_SYSTEM_FUNC(\([^)]*\))/#define \1 ThorsAnvil::BuildTools::Mock::mock\1;/gp'

mock.undefs: test/Mock.def
    sed <$< >$@ -ne 's/MOCK_SYSTEM_FUNC(\([^)]*\))/#undef \1/gp'

Improve the generated file header

It frustrates me when I see this:

/*
 * This is a generated file; do not edit.
 */

There's some help following, which is good:

/*
 * See ThorBuilder for details.
 */

A really good header for generated files adds some more information:

/*
 * Edits to this file will be overwritten!
 *
 * Generated by build/mock/make_header.pl at 2017-11-01 08:58:25
 * from
 *   build/mock/MockHeaders.h.prefix
 *   build/mock/MockHeaders.h.preamb
 *   build/mock/MockHeaders.h.suffix
 *   test/Mock.def
 */

So as well as a warning, we also learn the source files we might actually want to edit instead, and the program in which we might need to change the processing.

Answer 2

This is pretty hard to pass through. I'd have to say if it works for you... . Two points, i have had good results when doing code generation rather than using my own regex expressions and string substitution to make the step up to template engines. I was using cheetah (which is for python). This would probably save you from having to do the manual assembly through the median, preamble and other parts that you are assembling (e.g. http://search.cpan.org/~mschout/Text-Template-1.47/lib/Text/Template.pm) and would move some of the parsing complexity into the template files rather than having it in the build files.

Have you tried to specify the exact function prototype in your manually rather than using decltype, this might get you a better match and enable you to get rid of the workaround. I don't understand the remark about being "too common".

On a more general note, this is a lot of effort it would seem that, either just mocking the class that is using these systems calls would be a good enough test in most cases, or creating an abstraction and then mocking that would be less effort.