An overview of metaprogramming

Metaprogramming is a concept widely heard of in other programming languages such as C++, Java, .NET, and Ruby but not so widely heard of in Python. Python is a programming language that is easy to learn for beginners to programming and efficient to implement for advanced programmers. Therefore, it has an additional advantage in improving efficiency and optimization while developing high-performance applications when techniques such as metaprogramming are blended with the process of application development.

In this book, we will deep dive into the concepts of metaprogramming using Python 3.

The term meta, as the name suggests, is a process that references itself or its the high-level information. In the context of programming, metaprogramming also describes the similar concept of a program referencing itself or a program object referencing itself. A program referencing itself or its entity gives data on the program or the programming entity that can be used at various levels to perform activities, such as transformations or manipulations, in a programming language.

To understand the term meta, let’s consider the term metadata. As an example, let’s look at a Python DataFrame. For those who are not familiar with the term DataFrame, we can use the term table. The one shown in the following screenshot is called Employee Data:

Figure 1.1 – Employee Data table

This Employee Data table consists of employee information such as the name of the employee, employee ID, qualification, experience, salary, and so on.

All of this information are attributes of single or multiple employees, and it is the data of employees in an organization. So, what will the metadata be? The metadata is the data of how employee data is stored in the Employee Data table.

The metadata for the Employee Data table defines how each column and its values are stored in the table. For example, in the following screenshot, we can see metadata where Name is stored as a string with a length of 64 characters, while Salary is stored as a Float with a length of 12 digits:

Figure 1.2 – Metadata representation for the Employee Data table

Accessing, modifying, transforming, and updating the Employee Data table using information such as the name or ID of an employee is data manipulation, while accessing, modifying, transforming, and updating the data type or size of the column name or employee ID or salary, is metadata manipulation.

With this understanding, let’s look at an example of metaprogramming.

Metaprogramming – a practical introduction

Any programming language that can be used to write code to perform actions consists of a basic unit or piece of code that can be written to perform an action. This is known as a function.

If we have two numbers stored in two variables, a and b, to perform an add action, you can simply add those two numbers by writing a function, as shown in the following code block:

def add(a,b):    

    c = a + b    

    return c  

Now, if we execute this code, it can go through different scenarios, depending on the input data provided to the add function. Let’s take a close look at each of them.

Scenario 1

Running the add function with two integers would result in two numbers being added together, as follows:

add(1,3)  

4

Scenario 2

Running the add function with two strings would result in the concatenation of two words, as follows:

add('meta','program')   

metaprogram

Scenario 3

Let’s take a look at running the add function with one string and one integer:

add('meta',1)  

The preceding code would result in the following error:

Figure 1.3 – TypeError

Let’s examine this error in detail.

The error in the preceding code snippet denotes a TypeError, which was caused by an attempt to add a meta string with an integer of 1. The question that may occur to you is, can we resolve this error using metaprogramming?

The add function in this example denotes a piece of code or program, similar to how the Employee Data table in Figure 1.1 denotes data. In the same line, can we identify the metadata of the add function and use it to resolve the TypeError object returned by the following code:

add('meta',1)  

Next, we will look at a practical example of metaprogramming. We will be making use of the metadata of the add function to understand this concept.

Metadata of the add function

A function in any programming language is written to perform a set of operations on the input variables; it will return the results as a consequence of the operations performed on them. In this section, we will look at a simple example of a function that adds two variables. This will help us understand that metaprogramming can be applied to functions and manipulate the behavior of the function without modifying the algorithm of the function. We will be adding these two variables by writing an add function. To change the results of the add function, we will be manipulating the metadata of its two input variables, thus getting different results each time a different type of input variable is provided to execute the function. Just like we can manipulate what a function should do by writing lines of code to perform various operations, we can also manipulate the function itself by programming its metadata and setting restrictions on what it should and shouldn’t do. Just like a dataset, DataFrame, or table has data and metadata, a program or a function in Python 3 also has data and metadata. In this example, we will be manipulating the actions that are performed by the add function by restricting its behavior – not based on the input data provided to the function but on the type of input data provided to the add function instead. Take a look at the following screenshot:

Figure 1.4 – Examining the data and metadata of the add function

The following code helps us identify the metadata for each data item in the add function:

def add(a,b):    

    c = a +  b    

    print ("Metadata of add", type(add))    

    print ("Metadata of a", type(a))    

    print ("Metadata of b", type(b))    

    print ("Metadata of c", type(c))   

A function call to the preceding function will now return the metadata of the add function instead of its result. Now, let’s call the add method with an integer as input:

add(1,3)  

We’ll get the following output:

Metadata of add <class 'function'>

Metadata of a <class 'int'>

Metadata of b <class 'int'>

Metadata of c <class 'int'>

Similarly, we can also check the addition of strings, as follows:

add('test','string')

We’ll get the following output:

Metadata of add <class 'function'>

Metadata of a <class 'str'>

Metadata of b <class 'str'>

Metadata of c <class 'str'>

Python 3 allows us to use the metadata of the code to manipulate it so that it deviates from its actual behavior. This will also provide customized solutions for the problems we are trying to solve.

In the preceding example, we used the type function, a method in Python that returns the class or data type that any object or variable belongs to.

From the preceding output, it is evident that the a and b variables we passed to the add function belong to the integer data type, and its result, c, is an integer too. The add function itself is of the function class/type.

Resolving type errors using metaprogramming

There are many variations on how we can resolve the type error from the add function we saw in the previous section using metaprogramming. We will look at this in this section.

Scenario 1

The following meta-program handles the error and allows the add function to add two strings or two integers. It also suggests that the user enters the input data with the right data types:

def add(a,b):

    if (type(a) is str and type(b) is int) or\

        (type(a) is int and type(b) is str):

        return "Please enter both input values as integers or\

          string"

    else:

        c = a + b

        return c  

In the function definition of add, we have added two conditions – one to check if the type of a is a string and the type of b is an int, or if the type of a is an int and the type of b is a string. We are checking the combination of these input variables to handle the type mismatch error and directing the users to provide the right data type for input variables.

The following table shows the various combinations of input variable data types and their corresponding output or results based on the conditions set on the metadata of the add function, based on Scenario 1:

Figure 1.5 – Scenario 1 metadata combinations

The following code executes the add function to reinforce the input-output combinations explained in Figure 1.5:

add(1,3)  

4

add('meta','program')  

metaprogram

add('meta',1)  

'Please enter both input values as integers or string'

add(1,'meta')  

'Please enter both input values as integers or string'

Scenario 2

The following meta-program resolves the type mismatch error by converting the mismatching data types into string variables and performing a string concatenation. It is only logical to concatenate a string and an integer using a + operator as we cannot perform arithmetic addition on these two different data types. Take a look at the following program:

def add(a,b):

    if type(a) is int and type(b) is int:

        c = a +  b

        return c

    elif type(a) is str and type(b) is int or\

          type(a) is int and type(b) is str or \

          type(a) is str and type(b) is str:

        c = str(a) + str(b)

        return c

    else:

        print("Please enter string or integer")

Here, no matter what input we provide for the a and b variables, they both get converted into string variables and are then concatenated using +, whereas if both the input variables are integers, they get added using arithmetic addition.

The following table shows the various combinations of input variable data types and their corresponding output or results based on the conditions set on the metadata of the add function based on Scenario 2:

Figure 1.6 – Scenario 2 metadata combinations

Executing the following code provides the combinations of output values we saw in the preceding table:

add(1343,35789)  

37132

add('Meta',' Programming')  

'MetaProgramming'

add('meta',157676)  

'meta157676'

add(65081, 'meta')  

'65081meta'

add(True, 'meta')

Please enter string or integer

Scenario 3

Now, let’s go a step further and restrict the nature of the add function itself to ensure it only performs arithmetic addition and doesn’t accept any other data types or combinations of data types.

In the following code block, we have added another condition to perform a data type check on floating-point values, along with data type checks for the string and integer input values.

This function only accepts numeric values as input and will return a message directing users to input numbers so that only arithmetic addition is performed. Let’s look at the code:

def add(a,b):

    if type(a) is int and type(b) is int or\

       type(a) is float and type(b) is float or\

       type(a) is int and type(b) is float or\

       type(a) is float and type(b) is int:

        c = a +  b

        return c

    else:

        return 'Please input numbers'

The following table shows the various combinations of input variable data types and their corresponding output or results based on the conditions set on the metadata of the add function based on Scenario 3:

Figure 1.7 – Scenario 3 metadata combinations

Executing the following code provides the combination of output values shown in Figure 1.7, including the addition of floating-point values:

add(15443,675683)  

691126

add(54381,3.7876)  

54384.7876

add(6.7754,543.76)  

550.5354

add(79894,0.6568)  

79894.6568

add('meta',14684)  

'Please input numbers'

add(6576,'meta')  

'Please input numbers'

add('meta','program')  

'Please input numbers'

These are some of the approaches that can be applied to perform simple metaprogramming on a function. However, these are not the only solutions that solve type errors or manipulate a function. There is more than one way or approach to implementing solutions using metaprogramming.