Migrating from Python 2 to Python 3

In this article I will show you how you can migrate your application from an older Python 2.7 version to the new Python 3.5 one.

You may think this is really seldom but sometimes you encounter the task to move on and migrate an application to the newest Python version -- although 2.7 is continued and released often. There can be a lot of different reasons behind this: for example a Python 3 evangelist brain-washed your CTO.

Let's see what you can do to fulfill this requirement.

Differences

Python 3 introduced some core differences to Python 2. However basic scripts can be used without much ado in both versions.

The core differences you may encounter are:

• print is now a function
• input does not evaluate the provided expression
• division operator returns now a float not an int
• modules are re-structured

The first two can be fixed easily, the last one is not so trivial because some changes are nested and you have to know where to search for the new location.

This is a real small set of differences. This article would be too big if I would include every difference you may encounter between the two version.

Tests

I have written an introduction article on testing so I won't tell you how to test. However make sure you have tests because this will make it easy to identify errors in your migrated code.

Migration

We will migrate the following example code from Python 2.7 to Python 3.5, let's save the contents into a file called welcome.py:

def welcome(name=None):
    if not name:
        name = raw_input("What's your name? >>> ")
    print "Welcome to Python, {}!".format(name)
    print '2/3 is {}'.format(2/3)

if __name__ == '__main__':
    welcome('GHajba')
    welcome()

As you can see this script is a very basic one. It shows some above mentioned differences to be prepared and to show how you can migrate to the new version.

Running the script results in the following:

Welcome to Python, GHajba!
2/3 is 0
What's your name? >>> Gabor
Welcome to Python, Gabor!
2/3 is 0

Now one way to migrate the application would be to start by hand and look for prints or raw_inputs and division. For this simple codebase this is OK however for larger scripts it would be overwhelming. For this let's use a tool which was introduced with Python to make migration of larger codebases easier.

The tool provided with Python 3 installations is called 2to3. You can use it with the following command:

2to3 welcome.py

If 2to3 is not on your path, you can find it in the Python installation folder under Tools/Scripts.

Sometimes it can happen that these scripts are not installed on your environment. In this case you can either download the script / install the tools package if it is available, or you write your own script because it is a simple script with 4 lines of code:

#!/usr/bin/env python
import sys
from lib2to3.main import main

sys.exit(main("lib2to3.fixes"))

I will assume through this article that you can run the migration script with the command 2to3. If you can not then add the required extensions / calls to make it work.

The result is only a difference between the two versions written to the console:

RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored welcome.py
--- welcome.py  (original)
+++ welcome.py  (refactored)
@@ -1,8 +1,8 @@
 def welcome(name=None):
     if not name:
-        name = raw_input("What's your name? >>> ")
-    print "Welcome to Python, {}!".format(name)
-    print '2/3 is {}'.format(2/3)
+        name = input("What's your name? >>> ")
+    print("Welcome to Python, {}!".format(name))
+    print('2/3 is {}'.format(2/3))

 if __name__ == '__main__':
     welcome('GHajba')
RefactoringTool: Files that need to be modified:
RefactoringTool: welcome.py

Let's analyze this result a bit. As you can see, the print statements were converted to functions and the raw_input call was converted to an input function call. The division however stays the same. This means that you will get a different result when you run the application with Python 3.

At the beginning you can see that some fixers are skipped. This is because they have to be explicitly called if you want to run them. For example if you want to run the idioms fixer beside all other fixers you would have to execute following code:

2to3 -f all -f idioms welcome.py

Naturally we don't want a difference displayed, we want the code converted into a file. For this let's dig a bit deeper into how 2to3 works.

You can add a -w flag when running the script. This will write the conversion results into the same file you converted -- and a backup file is written with the original contents of the converted script:

2to3 -w welcome.py

The converted code looks like this:

def welcome(name=None):
    if not name:
        print("What's your name?")
        name = input()
    print("Welcome to Python, {}!".format(name))
    print('2/3 is {}'.format(2/3))

if __name__ == '__main__':
    welcome('GHajba')
    welcome()

And this is the result if we run the welcome.py file with Python 3:

Welcome to Python, GHajba!
2/3 is 0.6666666666666666
What's your name?
Gabor
Welcome to Python, Gabor!
2/3 is 0.6666666666666666

Now if you want to keep the original file and have the result in a new one you can use the --add-suffix parameter when calling 2to3. When you use the suffix you have to add the -n option too which disables backup creation of the original file because you save the results into a separate one.

2to3 -n -w --add-suffix=3 ./welcome.py

A strange error message

We have seen, that 2to3 works when you provide a single script without mentioning the location of that script however in the previous example where we appended a suffix we had to include the location of the file ./.

This is because when you run the migration with the options like above you will get following error:

OSError: [Errno 2] No such file or directory: ''

This is because somewhere in the migration os.path.dirname (or something like this) is uses on the file and because we did not provide a directory location it returns an empty string and this gives this error.

This directory path is required because the migration saves the new file into the same directory as your current file is.

Restructured Python modukes

One point you may encounter is migration of Python libraries which have been restructured between versions 2 and 3 like urllib and urllib2, their structure has been changed between Python releases. However this is included in the 2to3 migration too. The following example demonstrates it quickly:

import urllib2

req = urllib2.Request('https://www.python.org/')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page

This simple script uses the urllib2.Request to create a request-response website reading. This is really basic but it shows you how these libraries changed between versions. I added the contents of this example into the urllib_example.py file.

import urllib.request, urllib.error, urllib.parse

req = urllib.request.Request('https://www.python.org/')
response = urllib.request.urlopen(req)
the_page = response.read()
print(the_page)

The code above is the migrated version. As you can see, the migration adds some more imports than required like urllib.error and urllib.parse but this should not bother you -they are included because they would be used in a better script however mine is just a simple example.

Conclusion

Migration from Python 2.7 to Python 3.5 is not a big thing. Most worrying changes are handled with the 2to3 migration tool delivered by Python. Some cases however have to be taken care of like division which changed between these two version. And that's why you should have tests to see if your migration result really works as you expect it.