Sending a C++ array to Python/NumPy and back

Question

I am going to send a C++ array to a Python function as NumPy array and get back another NumPy array. After consulting with NumPy documentation and some other threads and tweaking the code, the code is finally working but I would like to know if this code is written optimally considering the:

• Unnecessary copying of the array between C++ and Numpy (Python).
• Correct dereferencing of the variables.
• Easy straight-forward approach.

C++ code:

// python_embed.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "Python.h"
#include "numpy/arrayobject.h"
#include<iostream>

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    import_array()

    // Build the 2D array
    PyObject *pArgs, *pReturn, *pModule, *pFunc;
    PyArrayObject *np_ret, *np_arg;
    const int SIZE{ 10 };
    npy_intp dims[2]{SIZE, SIZE};
    const int ND{ 2 };
    long double(*c_arr)[SIZE]{ new long double[SIZE][SIZE] };
    long double* c_out;
    for (int i{}; i < SIZE; i++)
        for (int j{}; j < SIZE; j++)
            c_arr[i][j] = i * SIZE + j;

    np_arg = reinterpret_cast<PyArrayObject*>(PyArray_SimpleNewFromData(ND, dims, NPY_LONGDOUBLE, 
        reinterpret_cast<void*>(c_arr)));

    // Calling array_tutorial from mymodule
    PyObject *pName = PyUnicode_FromString("mymodule");
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (!pModule){
        cout << "mymodule can not be imported" << endl;
        Py_DECREF(np_arg);
        delete[] c_arr;
        return 1;
    }
    pFunc = PyObject_GetAttrString(pModule, "array_tutorial");
    if (!pFunc || !PyCallable_Check(pFunc)){
        Py_DECREF(pModule);
        Py_XDECREF(pFunc);
        Py_DECREF(np_arg);
        delete[] c_arr;
        cout << "array_tutorial is null or not callable" << endl;
        return 1;
    }
    pArgs = PyTuple_New(1);
    PyTuple_SetItem(pArgs, 0, reinterpret_cast<PyObject*>(np_arg));
    pReturn = PyObject_CallObject(pFunc, pArgs);
    np_ret = reinterpret_cast<PyArrayObject*>(pReturn);
    if (PyArray_NDIM(np_ret) != ND - 1){ // row[0] is returned
        cout << "Function returned with wrong dimension" << endl;
        Py_DECREF(pFunc);
        Py_DECREF(pModule);
        Py_DECREF(np_arg);
        Py_DECREF(np_ret);
        delete[] c_arr;
        return 1;
    }
    int len{ PyArray_SHAPE(np_ret)[0] };
    c_out = reinterpret_cast<long double*>(PyArray_DATA(np_ret));
    cout << "Printing output array" << endl;
    for (int i{}; i < len; i++)
        cout << c_out[i] << ' ';
    cout << endl;

    // Finalizing
    Py_DECREF(pFunc);
    Py_DECREF(pModule);
    Py_DECREF(np_arg);
    Py_DECREF(np_ret);
    delete[] c_arr;
    Py_Finalize();
    return 0;
}

Python Code:

def array_tutorial(a):
    print("a.shape={}, a.dtype={}".format(a.shape, a.dtype))
    print(a)
    a *= 2
    return a[-1]

Answer 1

1. Review

1. It's not recommended to use using namespace std; — the problem is that this imports all of the identifiers from std, and some of these may shadow names from other modules that you need to use. See this question on Stack Overflow.

2. The code doesn't check for success/failure of many of the functions it calls. These can all fail:

   • new long double[SIZE][SIZE]
   • PyArray_SimpleNewFromData
   • PyUnicode_FromString
   • PyTuple_New
   • PyTuple_SetItem
   • PyObject_CallObject

   (and maybe others that I didn't spot).

3. There's no Py_DECREF for pArgs.

4. It would be possible to avoid constructing the argument tuple, by using PyObject_CallFunctionObjArgs instead of PyObject_CallObject.

5. Error messages should be written to standard error (cerr), not standard output.

6. It's clearer to use the macros EXIT_SUCCESS and EXIT_FAILURE from <cstdlib> instead of 0 and 1.

7. The code calls PyArray_NDIM and PyArray_SHAPE on np_ret without checking to see if this object is in fact an array. Call PyArray_Check first.

8. Using value initialization for int variables seems perverse to me. With int i{} you have to remember that the default constructor gives the variable the value 0. With int i = 0 there's no need to remember (and the code is no less efficient: the compiled code doesn't actually create a temporary 0 object and then call the int assignment constructor).

9. This code doesn't provide as much information about errors as it could. In particular, errors inside Python cause Python to create an exception object containing information about the error (see "Exception Handling"). It would be a good idea to print this object if it exists, by checking PyErr_Occurred and then calling PyErr_Print.

10. Each block of error handling code has to undo the effect of all the previous successful blocks of code. This makes the length of the function quadratic in the number of error cases! This is risky, because every time you change something, you have to adjust all the error cases accordingly, and it is very easy to forget (as in §1.3 above). Also, the pain of writing out all the error-handling code makes it tempting to skip error handling for functions that you believe are likely to succeed (as in §1.2 above).

    See the revised code below for one way to arrange that each "undo" operation appears just once.

2. Revised code

This is untested, so probably has some errors.

int _tmain(int argc, _TCHAR* argv[])
{
    int result = EXIT_FAILURE;

    Py_SetProgramName(argv[0]);
    Py_Initialize();
    import_array();

    // Build the 2D array in C++
    const int SIZE = 10;
    npy_intp dims[2]{SIZE, SIZE};
    const int ND = 2;
    long double(*c_arr)[SIZE]{ new long double[SIZE][SIZE] };
    if (!c_arr) {
        std::cerr << "Out of memory." << std::endl;
        goto fail_c_array;
    }
    for (int i = 0; i < SIZE; i++)
        for (int j = 0; j < SIZE; j++)
            c_arr[i][j] = i * SIZE + j;

    // Convert it to a NumPy array.
    PyObject *pArray = PyArray_SimpleNewFromData(
        ND, dims, NPY_LONGDOUBLE, reinterpret_cast<void*>(c_arr));
    if (!pArray)
        goto fail_np_array;
    PyArrayObject *np_arr = reinterpret_cast<PyArrayObject*>(pArray);

    // import mymodule.array_tutorial
    const char *module_name = "mymodule";
    PyObject *pName = PyUnicode_FromString(module_name);
    if (!pName)
        goto fail_name;
    PyObject *pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (!pModule)
        goto fail_import;
    const char *func_name = "array_tutorial";
    PyObject *pFunc = PyObject_GetAttrString(pModule, func_name);
    if (!pFunc)
        goto fail_getattr;
    if (!PyCallable_Check(pFunc)){
        std::cerr << module_name << "." << func_name
                  << " is not callable." << std::endl;
        goto fail_callable;
    }

    // np_ret = mymodule.array_tutorial(np_arr)
    PyObject *pReturn = PyObject_CallFunctionObjArgs(pFunc, pArray, NULL);
    if (!pReturn)
        goto fail_call;
    if (!PyArray_Check(pReturn)) {
        std::cerr << module_name << "." << func_name
                  << " did not return an array." << std::endl;
        goto fail_array_check;
    }
    PyArrayObject *np_ret = reinterpret_cast<PyArrayObject*>(pReturn);
    if (PyArray_NDIM(np_ret) != ND - 1) {
        std::cerr << module_name << "." << func_name
                  << " returned array with wrong dimension." << std::endl;
        goto fail_ndim;
    }

    // Convert back to C++ array and print.
    int len = PyArray_SHAPE(np_ret)[0];
    c_out = reinterpret_cast<long double*>(PyArray_DATA(np_ret));
    std::cout << "Printing output array" << std::endl;
    for (int i = 0; i < len; i++)
        std::cout << c_out[i] << ' ';
    std::cout << std::endl;
    result = EXIT_SUCCESS;

fail_ndim:
fail_array_check:
    Py_DECREF(pReturn);
fail_call:
fail_callable:
    Py_DECREF(pFunc);
fail_getattr:
    Py_DECREF(pModule);
fail_import:
fail_name:
    Py_DECREF(pArray);
fail_np_array:
    delete[] c_arr;
fail_c_array:
    if (PyErr_Check())
        PyErr_Print();  
    PyFinalize();
    return result;
}

Answer 2

I find this simple code work well though it sends array to a Python function as Tuple but not numpy array, but it is easier to change Tuple into numpy array in Python:

c++
...
PyObject* pArgs = PyTuple_New(2);
PyTuple_SetItem(pArgs, 0, Py_BuildValue("i", 1));
PyTuple_SetItem(pArgs, 1, Py_BuildValue("i", 2));
PyObject_CallObject(pFunc, pArgs);
...

&

python
def func(*list):
  print list[0] + list[1]