Metaprogramming with Metaclass in Python

Metaprogramming is an advanced topic in python. To be exact it is actually a topic beyond the advanced level. Very few developers do metaprogramming (with metaclass and other language features) in Python. The gurus of Python say that 99% of the developers never need it. So, why do we bother to learn it? Well, whether you code with metaclass or not as a developer somehow you are always using it if you know it or not. It’s something like this: most of the end users almost never use Linux, they use Windows or Mac—but there is not a single person who is not using Linux unbeknownst to them. For example if you are browsing a website is most likely running on Linux server. Also, your Android smartphone runs on Linux kernel, so you are using Linux there as well. The end user does not need to know how to operate Linux, how to use shell or how to work without a GUI but they are actually using Linux without knowing it and without any problem. In a similar fashion most Python developers uses metaclasses in this or that way but they know every detail of it. Django developers enjoys using Django’s model and form classes but they do not need to know that those are the magic of metaprogramming.  Metaclasses brings some magic, or black magic if used improperly, to your programing. Many libraries and frameworks in Python that make coding easier for us with more DRY (Do Not Repeat Yourself) are actually the blessing metaprogramming and metaclasses most of the times.

So we’ll note that this article is not for the Python beginner but rather for advanced users who have gone through the beginner and intermediate level and now want to learn something a bit more complex. I am assuming that you have had some experience in popular Python libraries and frameworks. Again topics like metaprogramming cannot be covered completely in a short article like this one but I will try my best to lead you through metaprogramming in Python so that you can start coding with metaclass perfectly. I am going to present the examples in Python 3.x syntax. Python 2.x is no longer relevant and 2.x going to be dropped from official support soon. Again for metaclass 3.x has cleaner syntax.

This article will be divided into several parts. First of all I am going to have some general discussion like the meaning of some of the words that start with ‘meta’ to prepare you mentally about metaprogramming. Then I am going to clarify your concept about metaprogramming. After that I will start our main discussion—the discussion on metaclasses and some examples with code.

Before diving into programming let’s discuss a few things that we encounter in our daily life. I think all of us are familiar with the terms ‘meta key’, ‘metadata’, ‘metaphysics’, etc. The term ‘metaphysics’ comes from Aristotle’s books. Metaphysics means ‘the things that comes after physics’. It’s a branch of Philosophy that deals with things like abstract concepts such as being, knowing, substance, cause, identity, time, space, etc. So, metaphysics does not deal with anything natural or related to daily life physics. It does not describe electricity, heat, mechanics, or anything we know related to physics. So, we may safely say that metaphysics is something that is beyond physics. Now, what is ‘metadata’? Metadata is not the actual data but it is data about the actual data. Imagine a situation when you are browsing some website with your favorite web browser. When you request a page with the browser a bunch of extra information is sent to the server like your user-agent, the encodings your browser supports, the compression algorithm it supports, etc. These things are called ‘headers’ in HTTP terminology. You are sending a request to the server with a URL so that the server can send you the appropriate page, image, video, or another resource. That means the URL is the data you intend to send but in the background the browser is sending some extra information along with it so that you get things working in the proper way. This extra information or extra data is the metadata. When the server sends you a response back it also sends more extra data to your browser that you do not need to see but your browser needs to know to present you the right content in the right way. Now what is a ‘meta key’? Meta key is the key that modifies the behavior of certain ordinary keys. So, it is a helper key.

Enough talk about non-programming things. Now, what is metaprogramming? In simple words metaprogramming is a style or method of programming that helps manipulating other programs or manipulating itself. That is a simple definition but not simple to understand. A metaprogram acts on other program or on itself. It treats the other program or itself as data.  The definition is getting more complicated. To make things easier let’s forget that a metaprogram can manipulate other programs and just remember that it can manipulate itself and treat itself as data (actually only that is needed for this article). Not quite clear yet? Well, in your program a string is data—you can manipulate it in a way of making it all-uppercase, replace some words, insert a new string inside that string, etc. A metaprogram does similar things with itself or its own code.

Still not clear? Well, do you remember any movie in which scientists create such an intelligent robot that at some point in time starts to create more robots like itself and take over human civilization? Well it’s a bit like that. Metaprogramming is such a thing that can create and manipulate itself. In most cases these manipulations happen at runtime. It is something like those viruses from Si-Fi movies that change their own DNA and spread over the world to destroy civilization. There are a lot of techniques in metaprogramming and Python itself has several of them. But here in this article I am going to discuss only one technique. Metaprogramming is a controversial topic with many forms in many languages and most of the time developers are discouraged to use this powerful language feature. But in many cases when we see some magic way of doing things from various libraries and frameworks we see that metaprogramming is behind it. We are not going to discuss the good or bad sides of metaprogramming but we are going to learn it so that in our future development process we can have a second thought about whether we should choose metaprogramming to make things more DRY or not.

In python metaprogramming exists in various forms. I am going to discuss metaclasses only. A metaclass is the most sophisticated way of metaprogramming in Python that makes your code more idiomatic and more DRY. Wrong use of it may result in less DRY-ness and difficulty in debugging programs.

What is a metaclass?

From our above discussion we can say that a metaclass is class about class or is beyond the class or perhaps something that creates classes. To be accurate, a metaclass in Python is a class that can control and manipulate the creation of a class. It is something like the ‘birth controller’ of a class in Python. With the help of metaclasses you can control every aspect of class creation in Python.

Everything in Python is an object. Integers are objects, floats are objects, strings are objects, functions are objects, lists are objects, dictionaries are objects, files are objects and everything else that represents data or a data structure are all objects. Even functions and methods are objects. A class in Python represents a blueprint for objects. We create objects from classes. A class is also an object itself. Classes create instances objects but classes are also objects. So, who or what creates classes? A metaclass creates a class. You can control the creation of instance objects through classes and in a similar fashion you can control the creation of classes with metaclass. During the creation of an instance object you can halt the creation, raise exception or do some extra preparatory work with the help of a class and in a similar fashion you can do the same with metaclass.

Before going into details let’s look at some familiar code in Python.

Outputs:

<class 'type'>

So, the type of the class is ‘type’. In other words a class is an instance of ‘type’. As proof look at the following line of code.

>>> isinstance(UselessClass, type)

Outputs:

True

Now let’s try this:

 >>>type(type)

Outputs:

<class 'type'>

This evidently shows that ‘type’ is an object in Python and its type is itself. So, we can safely conclude that ‘type’ is builtin master type in Python. Actually ‘type’ is at the top of hierarchies in Python.

Here is the proof:

>>> isinstance(type, type)

True

Again ‘type’ is itself an object. But type is an object of itself, it’s a built-in and just like every other built-in it comes with some magic. It is at the top of the chain in Python. Historically ‘type’ also acts like a function in Python but you should not be intimidated by that fact. Depending on parameters ‘type’ acts differently. If three arguments, class name, class bases and attribute-value pairs are provided, ‘type’ acts like a class factory and returns a newly created class. If a single argument is provided it returns the type of the passed object.

Example:

>>> AnotherClass = type("AnotherClass", (object,), {"my_name":"Sabuj"})

>>> AnotherClass

<class '__main__.AnotherClass'>

But creating a class in that way is not useful in most of the cases. When methods and attributes start to grow in numbers we start to lose control. The best way is defining a class that inherits from ‘type’ and overrides appropriate methods to customize the creation of class on the fly. Later we use that class as a metaclass of the class we want to customize creation of. We will see an example soon. Before going further let’s discuss on various methods that we can override to customize class creation.

The important methods we can override are:

• __prepare__
• __new__
• __init__

None of the above are instance methods. They are the class methods of the metaclass. __prepare__ needs @classmethod decorator but the rest of the two do not need that. Be aware of __init__ in a metaclass, it may confuse you. You may end up thinking that __init__ in metaclass is an instance method that is used to call when the object is created. This __init__ is different than what we use in our daily coding. What we always use are instance methods but this one is class method of the metaclass.

__prepare__(metacls, name, bases, **kwds):

__prepare__ is the first method that is called during the creation of a class. It is actually called before the class comes into existence. It is supposed to return a dictionary like object that will be used as the namespace for the class. Whatever method, attributes you define at the class level are stored in this dictionary. If you want to keep trace of order in which those methods, attributes, etc. were defined you can return an OrderedDict. From Python 3.6 by default OrderedDict is returned by the default implementation of __init__.  It accepts four parameters: metacls, name, bases, **kwds. As mentioned before __prepare__ must be decorated with @classmethod decorator.

__new__(cls, name, bases, namespace, **kwds):

__new__ is called after __prepare__ and must return the class. The class body is already executed by this point and the methods and other attributes are stored in the dictionary (or dictionary like object) returned from the __prepare__ method. You can do anything at this point with the class. You can stop the creation of the class, remove methods and attributes from the class, add new methods and attributes to the class, transform methods and attributes, etc. Your imagination is the limit here. Remember __new__ from the metaclass is called only once. Because in the lifetime of your program a class is created once. Always remember that __new__ from the metaclass is never called during instance object creation.

__init__(metacls, name, bases, namespace):

__init__ is used for initialization purpose after the class has been created. This method is not used much but you can use it if you need to carry out initialization task. Always remember that __init__ of a metaclass is not an instance method but it is a class method and it is called only once during the lifetime of the class. This method must not return anything.

To create a metaclass inherit your class from ‘type’ and override all or any of the magic methods mentioned above according to your needs. To use this new class as the metaclass of another class put a keyword argument ‘metaclass’ with the class as the value in the class declaration. For demonstration purpose I am using parameter name ‘cls’ instead of ‘metacls’, ‘parents’ instead of ‘bases’ and ‘dct’ instead of ‘namespace’:

We haven’t run any method and we haven’t instantiated the class yet by this point—we have just created the class but we see the above output. This happens because __new__ is called when the class is being created.

Let’s try it further with __init__:

We haven’t tried __prepare__ yet. Let’s do it and try every method together.

We could have returned OrderedDict from __prepare__ instead of vanilla Python dictionary to keep the order of attributes in the class body. Keeping the order will help us dramatically if we want to make our own ORM for a database dependent app or something similar where order of declaration matters.

There is another thing that I haven’t mentioned yet. Three of these functions are called by the __call__ method of the metaclass in the order __prepare__, __new__, __init__ and most probably you never need to mess with __call__ from the metaclass.

We haven’t done anything fun or useful with metaclasses yet, so let’s get to that now. Think of a situation where you have some API implemented through Python classes at your company to be used by the developers and it has certain rules like the following:

• Every developer must extend the API by creating class with metaclass CompanyMETA.
• The CompanyMETA restricts the use of any class level method that do not start with “c_”
• All other attributes’ default value must be either ‘int’ or ‘str’

Notice carefully that we have avoided dealing with __<method name>__ magic methods.

Now let’s declare two classes.

No error!

This time no mercy. The program will stop and go no further. It will stop even before the object is created, even before the class is created. So, you have protected your company from being flooded by unexpected types of data. I am leaving the exercise of methods starting with “c_” on you for practice.

The most common example of the metaprogramming magic is the most Python web framework Django. It provides a lot of magic through metaclass so that the developer needs to think less about underlining details and business logic more. For example:

• You define classes inheriting django.forms’ Form and set attributes like something = forms.CharField(…) etc. Try putting another type of attribute at the class level. Django won’t complain but those attributes will not be inserted into the html as input fields. Django filters fields during class creation and keeps those in _fields attribute so that it only outputs form fields into the template.
• You create classes by inheriting Django model and you just specify some class level attributes. Magically these fields become the database columns. How do they do that? Simply with metaclass.

There are lots of other frameworks that uses and/or abuses the power of metaprogramming to create some magic so that it becomes easy and fun to use that library or framework. Another framework that uses metaclass to make cleaner syntax is the GUI framework called Kivy.

Metaclasses are a very powerful tool in Python. Use them with caution though, otherwise you may be stuck trying to solve some inevitable bug in your program. Before using them think twice about whether you can solve the problem efficiently and in a DRY way without them or not. If metaclasses gives you a better and cleaner way than anything else, then certainly give it a shot.