When Python Weds C

This article assumes that:

• You have a basic understanding of C and Python.
• The required development environment is already in place.

Python version x.y (I used 2.6 for our discussion) is installed under the PYTHON_LIB_PATH directory (e.g., /usr/lib/), and Python headers are under PYTHON_INCLUDE_PATH (e.g., /usr/include).

Sample module source

Let’s begin by writing a C module with the functions to print “Hello world”, to add integers and to decode an example structure. The module is later used from a Python program. The following is the source code — python_interface.c:

#include <Python.h>

static  PyObject* InterfaceMethod_hello_world(PyObject *self)
{
  printf("Hello World!\n");
  Py_INCREF(Py_None);
  return Py_None;
 }
 
 static PyObject* InterfaceMethod_add_integers(PyObject *self, PyObject *args)
 {
 int a=0, b=0, c=0, sum=0;
 PyObject *retsum;
 
 if(PyArg_ParseTuple(args,"ii|i",&a,&b, &c)==0) {
     printf("Error in parsing \n");
     PyErr_SetString(PyExc_StandardError, "Error in parsing");
     return NULL;
 }
 sum=a+b+c;
 printf("sum = %d \n", sum);
 retsum=Py_BuildValue("i",sum);
 return retsum;
 }
 
 #pragma pack(1)
 typedef union {
 unsigned int val;
 struct header {
         unsigned int f1:4;
         unsigned int f2:4;
         unsigned int f3:8;
         unsigned int f4:16;
   } h;
 }header_t;
 #pragma pack()
 static PyObject* InterfaceMethod_decode_header(PyObject *self, PyObject *args)
 {
  char *buff=NULL;
  int len=0;
 
  header_t header;
  PyObject *retval, *entry;
 
 if(PyArg_ParseTuple(args,"s#",&buff, &len)==0) {
     printf("Error in parsing header data \n");
     PyErr_SetString(PyExc_StandardError, "Error in parsing header data");
     retval = Py_BuildValue("s", "FALSE");
     return retval;
 }
 if(buff) {
  header.val = (unsigned int)(strtoul(buff, NULL, 16));
 }
 
  printf(" f1=%x, f2=%x f3=%x f4=%x \n",
  (unsigned int)header.h.f1,
  (unsigned int)header.h.f2,
  (unsigned int)header.h.f3,
  (unsigned int)header.h.f4);
  retval = Py_BuildValue("siiii", "TRUE", (int)header.h.f1, (int)header.h.f2,
                                          (int)header.h.f3, (int)header.h.f4);
  return retval;
 }
 
 static PyMethodDef InterfaceMethod[] = {
 {"hello_world", (PyCFunction)InterfaceMethod_hello_world, METH_NOARGS, "Prints Hello World!"},
 {"add_integers", InterfaceMethod_add_integers, METH_VARARGS, "Adds integers"},
 {"decode_header", InterfaceMethod_decode_header, METH_VARARGS, "Decodes header. TRUE on success FALSE on failure"},
 {NULL, NULL, 0, NULL}
 };
 
 PyMODINIT_FUNC
 initInterfaceMethod(void)
 {
   (void) Py_InitModule3("InterfaceMethod", InterfaceMethod, "Interface Method's docstring");
   /* COMMENT2 : Py_InitModule function without docstring */
   /*(void) Py_InitModule("InterfaceMethod", InterfaceMethod); */
 }

Explanation

You’ve got to include the key header file, Python.h. A peek at this file in PYTHON_INCLUDE_PATH/python2.6/ shows that it, in turn, includes many main C header files such as <limits.h>, <stdio.h>, <string.h>, <errno.h>, <stdlib.h>, <unistd.h>, < stddef.h> and so on.

Let’s take a look at the functions. The interface functions take a pointer to the invoked object, and arguments (optional). They return a pointer to a PyObject class instance. InterfaceMethod_hello_world() just prints “Hello World!” and returns Py_None (equivalent to None in Python). Note that the reference to the Python “None” object is freed if we simply return Py_None. Py_INCREF [3] ensures that the None object reference is not freed.

The PyArg_ParseTuple() [2] API function parses arguments passed to InterfaceMethod_add_integers() to get addends. PyArg_ParseTuple() returns True on success, while on failure it returns False and raises an exception. The second argument in the example, "ii|i" specifies that two mandatory integers, followed by an optional integer, have to be parsed. The failure case sets the error message “Error in parsing”, and raises an exception of type PyExc_StandardError using the PyErr_SetString API function [5]. The NULL value is returned in case of failure. The integer sum is returned using Py_BuildValue [2].

The InterfaceMethod_decode_header() is used to decode the header structure header_t, which has four fields: f1, f2, f3 and f4, of 4, 4, 8 and 16 bits respectively. PyArg_ParseTuple() parses the argument into a string. The string is converted to an unsigned long int using strtoull(). Py_BuildValue builds Python tuples with decoded bit-field values of the header. Note that the first return value indicates TRUE (success) or FALSE (failure).

The table InterfaceMethod of type PyMethodDef lists all the public functions. The self-explanatory PyMethodDef structure (shown below) is taken from the methodobject.h header file in PYTHON_INCLUDE_PATH/python2.6. Note that the hello_world() function needs no arguments (METH_NOARGS), while add_integers() and decode() take arguments (METH_VARARGS).

struct PyMethodDef {
    const char  *ml_name;       /* The name of the built-in function/method */
    PyCFunction  ml_meth;       /* The C function that implements it */
    int          ml_flags;      /* Combination of METH_xxx flags, which mostly
                                   describe the args expected by the C func */
    const char  *ml_doc;        /* The __doc__ attribute, or NULL */
};
typedef struct PyMethodDef PyMethodDef;

Finally, the PyMODINIT_FUNC directive has the framework needed for the dynamic loading of the extension to work. It is mandatory to have the initialisation function name in the form init<module_name>, where module_name is the name of the shared object created. Here, the initInterfaceMethods() function is invoked whenever the InterfaceMethods module is loaded. As a convention, public method names are started with <module_name>, and declared static.

Py_InitModule() exposes the functions described in table InterfaceMethods. Py_InitModule3() [4] takes an additional doc-string as the third argument. The doc-string is used to summarise the function’s behaviour.

Building the module

Let’s review the steps required to build the module. There are two methods, each of which we will cover in turn.

Using a Makefile

Here is the Makefile to build the module. Running make builds the InterfaceMethod.so shared object. The flag -shared -fPIC produces a shared object; -soname is used to specify the shared object name; and -Wl,-Bdynamic informs the linker that the shared object is dynamic.

bash>cat Makefile
VERSION=2.6
PYTHON_INCLUDE_PATH=/usr/include/python${VERSION}
PYTHON_LIB_PATH=/usr/lib/python${VERSION}
all:
        gcc -g -fPIC  python_interface.c -o InterfaceMethod.so -shared -Wl,-soname=InterfaceMethod.so -Wl,-Bdynamic -I${PYTHON_INCLUDE_PATH} -L${PYTHON_LIB_PATH} -lpython${VERSION}

clean:
        @rm InterfaceMethod.so 2>/dev/null
bash>make

Python distutils

Use the distribution utility [6] from Python, and create setup.py to produce the shared object file.

bash>cat setup.py
from distutils.core import setup, Extension

interface_module = Extension('InterfaceMethod', sources = ['python_interface.c'])
setup (name = 'InterfaceMethod',
        version = '1.0',
        description = 'InterfaceMethod demo',
        ext_modules = [interface_module])

bash>python setup.py build
bash>find build -name "*.so"

The distutils package is the core for compilation, distribution, and installation of modules. The build command creates the shared object in the build/ subdirectory.

Testing the module from Python

Once the module is built, the next job is to import the module and test it. Make the module visible during Python execution in one of the following ways:

• Install the module into PYTHON_LIB_PATH/python2.6/site-packages, as indicated in the note above.
• Export the PYTHONPATH variable to which you’ve added the path to the directory in which your module shared object is, before running Python from the same shell, as follows:

  bash>export PYTHONPATH=$PYTHONPATH:/path/to/module/
  bash>python

• Append the path to your module shared object in Python code, or the Python interpreter, using the following command:

  bash>python
  >>>import sys
  >>>sys.path.append('/path/to/module')

Let’s test the hello_world() function with the Python interpreter:

bash>python
>>>help(im)
Help on module InterfaceMethod:
NAME
    InterfaceMethod - Interface Method's docstring
FILE
    ...
FUNCTIONS
    ... 
>>>im.hello_world.__doc__
'Prints Hello World!'
>>>im.hello_world()
Hello World!

You can see the public methods exposed by the module, and the documentation (doc-strings) using help and __doc__. Let’s run add_integers() from a simple Python script given below:

bash>cat interface_add.py
import sys
import InterfaceMethod as im 
def addIntegers(args=[]):
    a=0;b=0
    if len(args)>1 and args[1]:
        a=int(args[1])
    if len(args)>2 and  args[2]:
        b=int(args[2])
    if len(args)>3 and  args[3]:
        c=int(args[3])
        return im.add_integers(a,b,c)
    return im.add_integers(a,b)
def main(argv=None):
    response = addIntegers(sys.argv)
    print response
if __name__ == "__main__":
    sys.exit(main(sys.argv))
bash>python interface_add.py 2 3 5
sum = 10 
10

Let’s examine header decoding:

bash>cat interface_decode_hdr.py
#!/usr/bin/env python
import sys
import InterfaceMethod as im 
def decodeHeaders(args=[]):
    if len(args) > 1 and args[1]:
        packet=args[1]
    else:
       packet="0x10234"
    return im.decode_header(packet)
def main(argv=None):
    res = decodeHeaders(sys.argv)
    if res[0]=="TRUE":
       print "Python:f1=%d f2=%d f3=%d f4=%d" %(res[1],res[2],res[3],res[4])
    else:
       print "Failed to Decode" 
if __name__ == "__main__":
    sys.exit(main(sys.argv))
bash>python interface_decode_hdr.py
 f1=4, f2=3 f3=2 f4=1 
Python:f1=4 f2=3 f3=2 f4=1

We have now finished coding and testing our first simple Python extension module! I hope you have enjoyed this article.

Reference links

1. http://docs.python.org/c-api/intro.html
2. http://docs.python.org/c-api/arg.html
3. http://docs.python.org/dev/c-api/refcounting.html
4. http://docs.python.org/c-api/allocation.html
5. http://docs.python.org/c-api/exceptions.html
6. http://docs.python.org/distutils/introduction.html
7. http://docs.python.org/install/index.html
8. http://docs.python.org/c-api/list.html

Acknowledgement

I acknowledge everybody who contributed to Python, and the Python/C APIs, for inspiring this article. I also thank Sreekrishnan V for his support and tips.