The previous chapters discussed how to extend Python, that is, how to extend thefunctionality of Python by attaching a library of C functions to it. It is alsopossible to do it the other way around: enrich your C/C++ application byembedding Python in it. Embedding provides your application with the ability toimplement some of the functionality of your application in Python rather than Cor C++. This can be used for many purposes; one example would be to allow usersto tailor the application to their needs by writing some scripts in Python. Youcan also use it yourself if some of the functionality can be written in Pythonmore easily.
Embedding Python is similar to extending it, but not quite. The difference isthat when you extend Python, the main program of the application is still thePython interpreter, while if you embed Python, the main program may have nothingto do with Python — instead, some parts of the application occasionally callthe Python interpreter to run some Python code.
So if you are embedding Python, you are providing your own main program. One ofthe things this main program has to do is initialize the Python interpreter. Atthe very least, you have to call the function Py_Initialize(). There areoptional calls to pass command line arguments to Python. Then later you cancall the interpreter from any part of the application.
There are several different ways to call the interpreter: you can pass a stringcontaining Python statements to PyRun_SimpleString(), or you can pass astdio file pointer and a file name (for identification in error messages only)to PyRun_SimpleFile(). You can also call the lower-level operationsdescribed in the previous chapters to construct and use Python objects.
See also
- Python/C API Reference Manual
The details of Python’s C interface are given in this manual. A great deal ofnecessary information can be found here.
1.1. Very High Level Embedding¶
The simplest form of embedding Python is the use of the very high levelinterface. This interface is intended to execute a Python script without needingto interact with the application directly. This can for example be used toperform some operation on a file.
#define PY_SSIZE_T_CLEAN#include <Python.h>intmain(int argc, char *argv[]){ wchar_t *program = Py_DecodeLocale(argv[0], NULL); if (program == NULL) { fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); exit(1); } Py_SetProgramName(program); /* optional but recommended */ Py_Initialize(); PyRun_SimpleString("from time import time,ctime\n" "print('Today is', ctime(time()))\n"); if (Py_FinalizeEx() < 0) { exit(120); } PyMem_RawFree(program); return 0;}
The Py_SetProgramName() function should be called beforePy_Initialize() to inform the interpreter about paths to Python run-timelibraries. Next, the Python interpreter is initialized withPy_Initialize(), followed by the execution of a hard-coded Python scriptthat prints the date and time. Afterwards, the Py_FinalizeEx() call shutsthe interpreter down, followed by the end of the program. In a real program,you may want to get the Python script from another source, perhaps a text-editorroutine, a file, or a database. Getting the Python code from a file can betterbe done by using the PyRun_SimpleFile() function, which saves you thetrouble of allocating memory space and loading the file contents.
1.2. Beyond Very High Level Embedding: An overview¶
The high level interface gives you the ability to execute arbitrary pieces ofPython code from your application, but exchanging data values is quitecumbersome to say the least. If you want that, you should use lower level calls.At the cost of having to write more C code, you can achieve almost anything.
It should be noted that extending Python and embedding Python is quite the sameactivity, despite the different intent. Most topics discussed in the previouschapters are still valid. To show this, consider what the extension code fromPython to C really does:
Convert data values from Python to C,
Perform a function call to a C routine using the converted values, and
Convert the data values from the call from C to Python.
When embedding Python, the interface code does:
Convert data values from C to Python,
Perform a function call to a Python interface routine using the convertedvalues, and
Convert the data values from the call from Python to C.
As you can see, the data conversion steps are simply swapped to accommodate thedifferent direction of the cross-language transfer. The only difference is theroutine that you call between both data conversions. When extending, you call aC routine, when embedding, you call a Python routine.
This chapter will not discuss how to convert data from Python to C and viceversa. Also, proper use of references and dealing with errors is assumed to beunderstood. Since these aspects do not differ from extending the interpreter,you can refer to earlier chapters for the required information.
1.3. Pure Embedding¶
The first program aims to execute a function in a Python script. Like in thesection about the very high level interface, the Python interpreter does notdirectly interact with the application (but that will change in the nextsection).
The code to run a function defined in a Python script is:
#define PY_SSIZE_T_CLEAN#include <Python.h>intmain(int argc, char *argv[]){ PyObject *pName, *pModule, *pFunc; PyObject *pArgs, *pValue; int i; if (argc < 3) { fprintf(stderr,"Usage: call pythonfile funcname [args]\n"); return 1; } Py_Initialize(); pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); Py_DECREF(pName); if (pModule != NULL) { pFunc = PyObject_GetAttrString(pModule, argv[2]); /* pFunc is a new reference */ if (pFunc && PyCallable_Check(pFunc)) { pArgs = PyTuple_New(argc - 3); for (i = 0; i < argc - 3; ++i) { pValue = PyLong_FromLong(atoi(argv[i + 3])); if (!pValue) { Py_DECREF(pArgs); Py_DECREF(pModule); fprintf(stderr, "Cannot convert argument\n"); return 1; } /* pValue reference stolen here: */ PyTuple_SetItem(pArgs, i, pValue); } pValue = PyObject_CallObject(pFunc, pArgs); Py_DECREF(pArgs); if (pValue != NULL) { printf("Result of call: %ld\n", PyLong_AsLong(pValue)); Py_DECREF(pValue); } else { Py_DECREF(pFunc); Py_DECREF(pModule); PyErr_Print(); fprintf(stderr,"Call failed\n"); return 1; } } else { if (PyErr_Occurred()) PyErr_Print(); fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]); } Py_XDECREF(pFunc); Py_DECREF(pModule); } else { PyErr_Print(); fprintf(stderr, "Failed to load \"%s\"\n", argv[1]); return 1; } if (Py_FinalizeEx() < 0) { return 120; } return 0;}
This code loads a Python script using argv[1], and calls the function namedin argv[2]. Its integer arguments are the other values of the argvarray. If you compile and link this program (let’s callthe finished executable call), and use it to execute a Pythonscript, such as:
def multiply(a,b): print("Will compute", a, "times", b) c = 0 for i in range(0, a): c = c + b return c
then the result should be:
$ call multiply multiply 3 2Will compute 3 times 2Result of call: 6
Although the program is quite large for its functionality, most of the code isfor data conversion between Python and C, and for error reporting. Theinteresting part with respect to embedding Python starts with
Py_Initialize();pName = PyUnicode_DecodeFSDefault(argv[1]);/* Error checking of pName left out */pModule = PyImport_Import(pName);
After initializing the interpreter, the script is loaded usingPyImport_Import(). This routine needs a Python string as its argument,which is constructed using the PyUnicode_FromString() data conversionroutine.
pFunc = PyObject_GetAttrString(pModule, argv[2]);/* pFunc is a new reference */if (pFunc && PyCallable_Check(pFunc)) { ...}Py_XDECREF(pFunc);
Once the script is loaded, the name we’re looking for is retrieved usingPyObject_GetAttrString(). If the name exists, and the object returned iscallable, you can safely assume that it is a function. The program thenproceeds by constructing a tuple of arguments as normal. The call to the Pythonfunction is then made with:
pValue = PyObject_CallObject(pFunc, pArgs);
Upon return of the function, pValue is either NULL or it contains areference to the return value of the function. Be sure to release the referenceafter examining the value.
1.4. Extending Embedded Python¶
Until now, the embedded Python interpreter had no access to functionality fromthe application itself. The Python API allows this by extending the embeddedinterpreter. That is, the embedded interpreter gets extended with routinesprovided by the application. While it sounds complex, it is not so bad. Simplyforget for a while that the application starts the Python interpreter. Instead,consider the application to be a set of subroutines, and write some glue codethat gives Python access to those routines, just like you would write a normalPython extension. For example:
static int numargs=0;/* Return the number of arguments of the application command line */static PyObject*emb_numargs(PyObject *self, PyObject *args){ if(!PyArg_ParseTuple(args, ":numargs")) return NULL; return PyLong_FromLong(numargs);}static PyMethodDef EmbMethods[] = { {"numargs", emb_numargs, METH_VARARGS, "Return the number of arguments received by the process."}, {NULL, NULL, 0, NULL}};static PyModuleDef EmbModule = { PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods, NULL, NULL, NULL, NULL};static PyObject*PyInit_emb(void){ return PyModule_Create(&EmbModule);}
Insert the above code just above the main() function. Also, insert thefollowing two statements before the call to Py_Initialize():
numargs = argc;PyImport_AppendInittab("emb", &PyInit_emb);
These two lines initialize the numargs variable, and make theemb.numargs() function accessible to the embedded Python interpreter.With these extensions, the Python script can do things like
import embprint("Number of arguments", emb.numargs())
In a real application, the methods will expose an API of the application toPython.
1.5. Embedding Python in C++¶
It is also possible to embed Python in a C++ program; precisely how this is donewill depend on the details of the C++ system used; in general you will need towrite the main program in C++, and use the C++ compiler to compile and link yourprogram. There is no need to recompile Python itself using C++.
1.6. Compiling and Linking under Unix-like systems¶
It is not necessarily trivial to find the right flags to pass to yourcompiler (and linker) in order to embed the Python interpreter into yourapplication, particularly because Python needs to load library modulesimplemented as C dynamic extensions (.so files) linked againstit.
To find out the required compiler and linker flags, you can execute thepythonX.Y-config script which is generated as part of theinstallation process (a python3-config script may also beavailable). This script has several options, of which the following willbe directly useful to you:
pythonX.Y-config --cflagswill give you the recommended flags whencompiling:$ /opt/bin/python3.11-config --cflags-I/opt/include/python3.11 -I/opt/include/python3.11 -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
pythonX.Y-config --ldflags --embedwill give you the recommended flagswhen linking:$ /opt/bin/python3.11-config --ldflags --embed-L/opt/lib/python3.11/config-3.11-x86_64-linux-gnu -L/opt/lib -lpython3.11 -lpthread -ldl -lutil -lm
Note
To avoid confusion between several Python installations (and especiallybetween the system Python and your own compiled Python), it is recommendedthat you use the absolute path to pythonX.Y-config, as in the aboveexample.
If this procedure doesn’t work for you (it is not guaranteed to work forall Unix-like platforms; however, we welcome bug reports)you will have to read your system’s documentation about dynamic linking and/orexamine Python’s Makefile (use sysconfig.get_makefile_filename()to find its location) and compilationoptions. In this case, the sysconfig module is a useful tool toprogrammatically extract the configuration values that you will want tocombine together. For example:
>>> import sysconfig>>> sysconfig.get_config_var('LIBS')'-lpthread -ldl -lutil'>>> sysconfig.get_config_var('LINKFORSHARED')'-Xlinker -export-dynamic'
