Closures in Python 3

In this article I will introduce you to closures in Python 3.

Actually, a closure is a function in Python which allows you to do things that aren’t possible in another language (like C/C++ or Java) with functions.

In Python everything is treated like a first-class citizen, so they can be passed around just as normal variables. So are functions too. You probably have seen code like this already:

>>> def adder(a, b):
...     return a + b
...
>>> def caller(fun):
...     print(fun(2, 4))
...     print(fun(3, 5))
...
>>> caller(adder)
6
8

In this example above we created a function caller which calls another function which is passed to it as an argument.

Because we can pass around functions as arguments we can return functions too from function calls. Let's consider the following example:

>>> def contains_factory(x):
...     def contains(lst):
...         return x in lst
...     return contains
...
>>> contains_15 = contains_factory(15)

After running this example, contains_15 has the type of a function:

<function contains_factory.<locals>.contains at 0x101d78b70>

And these functions are closures.

What makes closures special?

If you think that closures are ordinary functions you might have missed the key difference from the previous example.

Call the contains_15 with some lists / iterables to test the functionality:

>>> contains_15([1,2,3,4,5])
False
>>> contains_15([13, 14, 15, 16, 17])
True
>>> contains_15(range(1, 20))
True

The key point of closures are that they remember their context in which they were created. In the example above contains_15 remembers, that the contains_factory function was called with the value 15. And this 15 is used for later in the contains function as the variable x when you provide multiple iterables to look-up x in them.

Another interesting fact is, that the closure remains existing even if the original creator function (in the example case it is contains_factory) is deleted:

>>> del contains_factory
>>> contains_factory(42)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'contains_factory' is not defined
>>> contains_15(range(14, 20))
True

How to create a closure?

After this introduction on closures you should know how to create closures in Python. But for the sake of brevity let's sum things up:

1. We have to create a nested function (a function inside another function).
2. This nested function has to refer to a variable defined inside the enclosing function.
3. The enclosing function has to return the nested function

Pretty simple, isn't it?

The second point is simple in reality, optional -- but if you do not reference a variable from the enclosing function there is not much sense to create a nested function and return it -- you simply define a function. You could do this in the normal scope too.

Let's create another interesting closure: a counter. The idea behind the counter is that in some cases you just want to count interactions. In those cases you define a global variable (most of the time called counter) and increment it at the right place when an interaction occurs. We can replace this global variable and the incrementation by defining a closure and call this closure every time we want to count something.

>>> def counter_factory():
...     count = 0 # here I create the variable to increment
...     def counter():
...         nonlocal count
...         count += 1
...         return count
...     return counter
...

The example above creates a function which will generate a counter closure every time it is invoked -- and the counter starts from 0 and increases every time you call the closure.

>>> counter1 = counter_factory()
>>> counter1()
1
>>> counter1()
2
>>> counter2 = counter_factory()
>>> counter2()
1
>>> counter1()
3
>>> counter2()
2

Well, this solution is not quite what we want because we return the value of count every time with the function invocation. This means that if we want to verify that there were no interactions then we have to do something like this:

counter1() == 1

And this requires thinking and remembering. And we are humans that make errors.

Fortunately there is a solution for this problem. An enclosing function has no restriction on the number of returned closures. So to fix this issue we can create two closures: one will increment the count variable, the other will return the current value of the count variable:

>>> def counter_factory():
...     count = 0 # here I create the variable to increment
...     def counter():
...         nonlocal count
...         count += 1
...         return count
...     def current():
...         nonlocal count
...         return count
...     return (counter, current)

Now we have access to two closures and we can use them as we want:

>>> incrementer, getter = counter_factory()
>>> getter
<function counter_factory.<locals>.current at 0x102478bf8>
>>> incrementer
<function counter_factory.<locals>.counter at 0x1024789d8>
>>> getter()
0
>>> getter()
0
>>> incrementer()
>>> incrementer()
>>> incrementer()
>>> getter()
3
>>> getter()
3
>>> incrementer()
>>> getter()
4
>>> incrementer, getter = counter_factory()
>>> getter()
0
>>> incrementer()
>>> getter()
1

As you can see, the incrementer increments silently the value of count inside the closure. getter returns the actual value of count without incrementing it -- and this makes the feature usable. Resetting the counter is done by simply re-assigning the closures.

What are closures good for?

Even if closures seem pretty interesting (a function returning another function which knows its creation context!) there is another question: where can we utilize closures to make the best of them?

Here are a few uses for closures:

• Eliminating global variables
• Replacing hard-coded constants
• Providing consistent function signatures

Late binding of closures

A source of problems is the way Python binds the variables in closures. For example you write:

>>> def create_adders():
...     adders = []
...     for i in range(10):
...         def adder(x):
...             return i + x
...         adders.append(adder)
...     return adders
...
>>> for adder in create_adders():
...     print(adder(1))
...

You may expect the following output:

2
3
4
5
6
7
8
9
10
11

In reality you get:

10
10
10
10
10
10
10
10
10
10

What happens? You create 10 functions which add 1 to the number 9. This is because of the late-binding of closures: the variables used inside the closures are looked up at the time the inner function is called. Whenever the adder function is called the inner loop over the range(10) is finished and the variable i has the value 9 -- so you end up with 10 for each 10 functions as result.

To solve this problem you could add an additional parameter to your closures which will maintain the right state:

>>> def create_adders():
...     adders = []
...     for i in range(10):
...         def adder(x, i=i):
...             return i + x
...         adders.append(adder)
...     return adders

Here we added i as a local variable to the closure with a default value of i. This later i comes from the range and sets the value of the closure i to 0 to 9 respectively. This solves the problem if we run the example:

>>> for adder in create_adders():
...     print(adder(1))
...
1
2
3
4
5
6
7
8
9
10

Conclusion

We have seen that closures are functions which remember the context (lexical scope) in which they were created and they can use it even if the program flow is no longer in the enclosing scope.

To create a closure you need to write a function which returns another function. The returned function is the closure itself.

Closure variables are late binding, so take care when you use them: you may end up with unwanted results!