Understanding why we need metaprogramming

Considering what we’ve learned about metaprogramming, we may be wondering the following:

“Is it always mandatory to apply metaprogramming techniques or to manipulate the metadata of the code while developing applications using Python 3 or above?”

This is a common question that can be asked not only while developing applications using Python 3 or above, but also when using any programming language that supports the techniques of metaprogramming and gives developers the option to apply them in the application development process.

To answer this question, it is important to understand the flexibility of metaprogramming and the techniques that are supported by Python to handle code manipulation, which will be covered throughout this book.

One of the reasons to apply metaprogramming is to avoid repetition in various aspects of the Python-based application development process. We will look at an example of this in the Don’t Repeat Yourself section.

In other words, introducing concepts such as code generators at the meta level can save development and execution time in functional- or domain-level programming. Domain-level programming corresponds to writing code for a particular domain, such as finance, networking, social media, and so on.

The other need is to increase the abstraction of your code at the program metadata level rather than at the functional level. Abstraction is the concept of information hiding in the literal sense or in terms of object-oriented programming. Implementing abstraction at the meta-program level would help us decide what information to provide to the next level of coding and what not to provide.

For example, developing a function template at the meta-program level would hide the function definition at the domain or functional level, as well as limit the amount of information that goes to the functional-level code.

Metaprogramming allows us to manipulate programs using metadata at the meta level, which helps define how the grammar and semantics of your program should be. For example, in the Resolving type erors using metaprogramming section, we looked at controlling the outcome of the data types of a function by manipulating the function’s variables.

Don’t Repeat Yourself

In any application development process, thousands of lines of code are written. Don’t Repeat Yourself is a principle defined by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer. The principle states that “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.”

While writing code, there are very high chances of writing multiple functions or methods that perform similar kinds of repetitive tasks, and the functions or methods, in turn, might be repetitive. This leads to redundancy in application development. The greatest disadvantage of redundancy is that when you make any modifications at one location, the implementation, modification, or code fixing needs to be repeated at multiple locations.

Libraries are developed with classes and methods, including object-oriented programming techniques such as abstraction, inheritance, encapsulation, and so on, to avoid redundancy and maintain coding standards as much as possible. Even then, there are chances of repetitive methods being within a class that can still be simplified.

Metaprogramming can help in handling such instances by implementing approaches such as dynamic code generation, dynamic function creation, and more. Throughout this book, we will be looking at various approaches that help you not to repeat yourself while developing applications.

To get a taste of how we can dynamically generate code and avoid repetitions, let’s look at a simple example where arithmetic operations are implemented as repetitive functions.

The following code consists of four basic arithmetic operations that can be performed on two numeric variables. We will be declaring and defining four functions that add, subtract, multiply, and divide two variables, a and b, store the result in a variable, c, and return it while the function is executed:

def add(a,b):  

    c = a + b  

    return c  

def sub(a,b):  

    c = a - b  

    return c  

def multiply(a,b):  

    c = a * b  

    return c  

def divide(a,b):  

    c = a / b  

    return c  

Each of the preceding functions needs to be called separately and variables need to be provided as input to execute them individually, as follows:

add(2,5)  

7

sub(2,5)  

-3

multiply(2,5)  

10

divide(2,5)  

0.4

In this example, there is only one difference – the arithmetic operator that’s used in the function definition. This code can be simplified without implementing metaprogramming, just by declaring a new function that takes in an additional input variable operator.

Let’s learn how to avoid this repetitive function definition and simplify the logic. The following code block defines one common function that can be reused to perform all four arithmetic operations. Let’s start by importing Python’s inbuilt module operator, which contains methods that support multiple arithmetic operations:

import operator as op

def arithmetic(a, b, operation):

    result = operation(a, b)

    return result

In this code snippet, we have declared three variables, including the operation in the function arithmetic. Let’s see this in action:

arithmetic('2', '5', op.add) '25'

Executing this function using input variables would return a concatenated string, 25, that will serve the purpose of creating the common arithmetic function to perform multiple operations. We can look at providing various operations as input to see how this one common function serves multiple purposes.

Calling this function with different arithmetic operators would resolve the need for repetitive function definitions:

arithmetic(2, 5, op.add)

7

arithmetic(2 , 5, op.sub)

-3

arithmetic(2, 5, op.mul)

10

arithmetic(2 , 5, op.truediv)

0.4

This is one approach to resolving code redundancy and avoiding multiple function definitions. But what if we do not want to define the function itself until and unless it is required?

To answer this question, we can implement dynamic function creation using metaprogramming. Dynamic functions are created during the code’s runtime as and when they are required.

Although we are still in the introductory chapter, we will discuss an example of dynamic function creation next to get a view of what kind of programming will be covered throughout this book.

Creating dynamic functions

In this section, we’ll look at an example of how dynamic functions can be created for the same set of arithmetic operations we discussed earlier in this section.

To create an arithmetic function dynamically, we need to import the library types and the FunctionType type. FunctionType is the type of all user-defined functions created by users during the Python-based application development process:

from types import FunctionType  

To begin this process, we will create a string variable that is a function definition of the arithmetic function:

functionstring = '''

def arithmetic(a, b):

    op = __import__('operator')

    result = op.add(a, b)

    return result

    '''  

print(functionstring)

We’ll get the following output:

 def arithmetic(a, b):

    op = __import__('operator')

    result = op.add(a, b)

    return result 

Now, we will create another variable, functiontemplate, and compile 'functionstring' into a code object. We will also set the code object to be executed using 'exec'. The compile method is used to convert the string in Python into a code object that can be further executed using the exec method:

functiontemplate = compile(functionstring, 'functionstring', 'exec')  

functiontemplate 

<code object <module> at 0x000001E20D498660, file "functionstring", line 1>

The code object of the function definition arithmetic will be stored in a tuple in functiontemplate and can be accessed as follows:

functiontemplate.co_consts[0]  

<code object arithmetic at 0x000001E20D4985B0, file "functionstring", line 1>

The next step involves creating a function object using the functiontemplate code object. This can be done using the FunctionType method, which accepts the code object and global variables as input parameters:

dynamicfunction = FunctionType(functiontemplate.co_consts[0], globals(),"add")

dynamicfunction  

<function _main_.arithmetic(a,b)> 

Upon executing, dynamicfunction, it will behave the same way as the add operation works in the operator module’s add method in the arithmetic function:

dynamicfunction(2,5)  

7

Now that we know how to create a function dynamically, we can look at extending it further to create multiple functions, each with a different operation and a different name, dynamically.

To do this, we must create a list of operators and a list of function names:

operator = ['op.add','op.sub','op.mul','op.truediv','op.pow','op.mod', 'op.gt', 'op.lt'] 

functionname = ['add','sub', 'multiply', 'divide', 'power',\

 'modulus', 'greaterthan', 'lesserthan']  

Our earlier list of four functions only contained the add, sub, multiply, and divide operations.

The earlier functionname list contained eight functions. This is the flexibility we get while creating dynamic functions.

For ease of use, let’s also create two input variables, a and b, to be used while executing the function:

a = 2  

b = 5  

In the following code, we will be creating a function called functiongenerator() that implements metaprogramming to dynamically generate as many arithmetic functions as we want. This function will take four input parameters – that is, the list’s functionname, operator, a, and b.

Here is the code:

def functiongenerator(functionname, operator, a,b):    

    from types import FunctionType    

    functionstring = []    

    for i in operator:    

        functionstring.append('''

def arithmetic(a, b):

    op = __import__('operator')

    result = '''+ i + '''(a, b)

    return result

    ''')    

        functiontemplate = []    

    for i in functionstring:    

        functiontemplate.append(compile(i, 'functionstring', 'exec'))    

        dynamicfunction = []    

    for i,j in zip(functiontemplate,functionname):    

        dynamicfunction.append(FunctionType(i.co_consts[0], \

          globals(), j))    

        functiondict = {}    

    for i,j in zip(functionname,dynamicfunction):    

        functiondict[i]=j    

    for i in dynamicfunction:    

        print (i(a,b))    

    return functiondict    

Within functiongenerator(), the following occurs:

• A new functionstring list is created with a function definition for each arithmetic operator provided in the operator list.
• A new functiontemplate list is created with a code object for each function definition.
• A new dynamicfunction list is created with a function object for each code object.
• A new functiondict dictionary is created with a key-value pair of function name-function objects.
• Functiongenerator returns the generated functions as a dictionary.
• Additionally, functiongenerator executes the dynamic functions and prints the results.

Executing this function results in the following output:

funcdict = functiongenerator(functionname, operator, a,b)  

7

-3

10

0.4

32

2

False

True

funcdict  

{'add': <function _main_.arithmetic(a,b)>,

 'sub': <function _main_.arithmetic(a,b)>,

 'multiply': <function _main_.arithmetic(a,b)>,

 'divide': <function _main_.arithmetic(a,b)>,

 'power': <function _main_.arithmetic(a,b)>,

 'modulus': <function _main_.arithmetic(a,b)>,

 'greaterthan': <function _main_.arithmetic(a,b)>,

 'lesserthan': <function _main_.arithmetic(a,b)>,} 

Any specific function from the preceding generated functions can be called individually and used further, as follows:

funcdict['divide'](a,b)  

0.4

The following diagram shows the complete process of metaprogramming to develop these dynamic functions:

Figure 1.8 – Dynamic function generator

Now that we know about dynamic function generators, let’s look at other applications of metaprogramming.