Packaging Python code

In this article I will continue a previous article on packaging Python code. I will introduce you to setuptools and flit as alternatives to the built-in solution of packaging. This article aims to extend your knowledge on packaging Python applications and distribute them to PyPI, the Python Package Index. With this you can share your code and be part of the big community of Python developers who share what they have.

Thoughts on packaging

A discussion started on the reddit thread for the previous article about modules and packages in Python. The idea is that a module is a single Python file you can import and use. If your project consists of multiple modules and folder-hierarchies you have a package. Libraries found in PyPI are packages because they have at least two .py files.

Sometimes developers use the two terms interchangeably which leads to confusion and discussions. I suggest you use package and library for bigger projects and module for a single file of Python code.

Therefore in this article I will write about packages which include modules of Python code.

The example

The example in this article will be very basic: I will continue to use the adder from the previous article written by Liran. Let's see a brief recap on the topic: we will distribute a simple mathematics module which does basic mathematical calculations: addition, subtraction, multiplication and division of two numbers in the mymath_ghajba.py module:

# -*- coding: utf-8 -*-

__author__ = 'GHajba'


def add(a, b):
    return a + b


def sub(a, b):
    return a - b


def mult(a, b):
    return a * b


def div(a, b):
    return a / b

As you can see, this module is very simple but it is perfect to get you started on distributing.

For the tools I will show how you can add package and Python version dependencies but I won't add them to the distributed packages. One reason for this is that when installing it, it would download and install some packages (like numpy). My second reason is that extra packages are not required in the example code so no dependency should be uploaded.

setuptools

The first library we will look at for distribution is setuptools. You can download and install it with the following command:

GHajba$ pip install --upgrade setuptools

The current version is 31.0.0 so we will look at this distribution. Other versions may differ in functionality.

To have the tool working, the thing we need is to create a module called setup.py. Minimalistic it should look like this:

from setuptools import setup

__author__ = 'GHajba'

setup (
    name="mymath",
    version='0.0.1',
)

This is really the basic content what we have to include. If we run the setup we get an output similar to this:

GHajba$ python3 setup.py sdist
running sdist
running egg_info
creating mymath_ghajba.egg-info
writing mymath_ghajba.egg-info/PKG-INFO
writing dependency_links to mymath_ghajba.egg-info/dependency_links.txt
writing top-level names to mymath_ghajba.egg-info/top_level.txt
writing manifest file 'mymath_ghajba.egg-info/SOURCES.txt'
reading manifest file 'mymath_ghajba.egg-info/SOURCES.txt'
writing manifest file 'mymath_ghajba.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating mymath_ghajba-0.0.1
creating mymath_ghajba-0.0.1/mymath_ghajba.egg-info
copying files to mymath_ghajba-0.0.1...
copying setup.py -> mymath_ghajba-0.0.1
copying mymath_ghajba.egg-info/PKG-INFO -> mymath_ghajba-0.0.1/mymath_ghajba.egg-info
copying mymath_ghajba.egg-info/SOURCES.txt -> mymath_ghajba-0.0.1/mymath_ghajba.egg-info
copying mymath_ghajba.egg-info/dependency_links.txt -> mymath_ghajba-0.0.1/mymath_ghajba.egg-info
copying mymath_ghajba.egg-info/top_level.txt -> mymath_ghajba-0.0.1/mymath_ghajba.egg-info
Writing mymath_ghajba-0.0.1/setup.cfg
creating dist
Creating tar archive
removing 'mymath_ghajba-0.0.1' (and everything under it)

As you can see, there are some fields missing which generate warnings. Let's add them too to have a PyPI conform setup script:

from setuptools import setup

__author__ = 'GHajba'

setup(
    name="mymath_ghajba",
    version='0.0.1',
    url='http://www.discoversdk.com/',
    author='GHajba',
    author_email='<omitted from the article>',
)

Now we have our basic script. Naturally bigger projects require dependencies and a minimal Python version. Let's add some imaginary dependencies and versions to see how to configure it for future projects:

# -*- coding: utf-8 -*-
from setuptools import setup

__author__ = 'GHajba'

setup(
    name="mymath_ghajba",
    version='0.0.1',
    url='http://www.discoversdk.com/',
    author='GHajba',
    author_email='<omitted from the article>',
    install_requires=['matplotlib~=1.5.2'],
    python_requires='>=3.4'
)

After everything is configured, we create a distribution as previously with python3 setup.py sdist. After this it is time to register our package. There are some ways. The most common is to use either python3 setup.py register or twine. If you encounter an error (for example SSL certificate verification failed) it can happen that the cause is not the certificate but something with your package. In this case try to use the PyPI web form and upload your PKG-INFO file. In the example project it is located in the mymath_ghajba.egg-info/ folder.

Uploading the package goes as follows:

GHajba$ python3 setup.py sdist upload

If you encounter an error try uploading with twine:

GHajba$ pip install twine
... some installation output omitted
GHajba$ twine upload dist/mymath_ghajba-0.0.1.tar.gz
Uploading distributions to https://pypi.python.org/pypi
Uploading mymath_ghajba-0.0.1.tar.gz
[================================] 3176/3176 - 00:00:04

Now the example application is available at PyPI: mymath_ghajba

pip-init

If you find writing setup.py files bothersome you can have a look at pip-init. It is a simple tool you can use to generate your setup.py files from the command line with various features and make your life easier. This package is available through PyPI and you can install it with the following command:

GHajba$ pip install pip-init

If it is installed let's try to create a setup.py file for the previous example created and distributed with setuptools:

GHajba$ pip-init
name (setuptools): mymath_ghajba
version (0.1.0): 0.0.2
description (A pip package): A simple mathematics package to demonstrate the usage of pip-init and setuptools
license (MIT):
author (Gábor László Hajba):         
Generate .gitignore file [Y/n]?: n

And this results in the following file:

GHajba$ cat setup.py
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages

try:
    long_description = open("README.rst").read()
except IOError:
    long_description = ""

setup(
    name="mymath_ghajba",
    version="0.0.2",
    description="A simple mathematics package to demonstrate the usage of pip-init and setuptools",
    license="MIT",
    author="Gábor László Hajba",
    packages=find_packages(),
    install_requires=[],
    long_description=long_description,
    classifiers=[
        "Programming Language :: Python",
        "Programming Language :: Python :: 3.6",
    ]
)

It has a bit more content but you have had to type less. Now we can distribute this project the same way as described in the previous section with setuptools.

The new version is available at PyPI: mymath_ghajba

flit

flit is a newer packaging tool which aims to be lightweight and help you to create packages more easily. You can install it with pip:

GHajba$ pip install flit

To use flit we have to create a flit.ini file which contains all the information we want to publish. The easiest way is to use the tool itself to provide the basic structure required so we do not miss any fields, and you get a walk-through to fill-in on the way:

GHajba$ flit init
Module name [mymath]: mymath_ghajba
Author: GHajba
Author email: <omitted from the article>
Home page: http://www.discoversdk.com/
Choose a license (see http://choosealicense.com/ for more info)
1. MIT - simple and permissive
2. Apache - explicitly grants patent rights
3. GPL - ensures that code based on this is shared with the same terms
4. Skip - choose a license later
Enter 1-4: 1

Written flit.ini; edit that file to add optional extra info.

Note that a homepage and email address have to be provided, I omitted them from this article.

The result of my inputs looks like this:

[metadata]
module = mymath_ghajba
author = GHajba
author-email = <omitted from the article>
home-page = http://www.discoversdk.com/
classifiers = License :: OSI Approved :: MIT License

As you can see, we have a valid file we can use to upload our code to PyPI with all the required fields.

Naturally this is not everything you can configure. Bigger projects require dependencies or a specific Python version. To demonstrate this we will add a simple requirement on the matplotlib package and a minimum Python requirement of 3.4:

[metadata]
module = mymath_ghajba
author = GHajba
author-email = <omitted from the article>
home-page = http://www.discoversdk.com/
classifiers = License :: OSI Approved :: MIT License
requires = matplotlib (== 1.5.2)
requires-python  = >= 3.4

After the flit.ini file is ready we have to provide a docstring to our module and a version number to be used. We add them simply to the mymath_ghajba.py file:

"""A simple mathematics module to demonstrate the usage of flit"""

__version__ = '0.0.4'

Now that we have everything ready let's publish the package to the package index:

GHajba$ flit wheel --upload
Copying package file(s) from mymath_ghajba.py
Writing metadata files
Writing the record of files
Wheel built: dist/mymath_ghajba-0.0.4-py2.py3-none-any.whl
Using repository at https://pypi.python.org/pypi
Uploading dist/mymath_ghajba-0.0.4-py2.py3-none-any.whl...
Package is at https://pypi.python.org/pypi/mymath_ghajba  

The drawback of using flit is that it distributes packages in the wheel format which cannot be installed by people using older versions of pip or easy_install. However in my eyes this is not a problem because we want to distribute to newer Python versions and through PyPI.

One thing to note your module or package folder has to be named the same as the module name in the flit.ini file. And with this approach you can easily distribute a simple module.

The newly installed package can be found in PyPI: mymath_ghajba

As an example let's install this package:

GHajba$ pip install mymath_ghajba
Collecting mymath_ghajba
  Downloading mymath_ghajba-0.0.4-py2.py3-none-any.whl
Installing collected packages: mymath-ghajba
Successfully installed mymath-ghajba-0.0.4

Now we can access this simple mathematics module:

>>> import mymath_ghajba
>>> mymath_ghajba.add(1,2)
3
>>> mymath_ghajba.div(2,3)
0.6666666666666666

Conclusion

We have seen two versions of packaging and publishing to the global package index. For setuptools we have taken a look at an extra library which can generate a basic setup.py file without much ado for a basic start which you can configure later on. With fit we have seen a simple way to distribute our modules in the wheel format.

Now it is up to you to find the one which you find comfortable and want to use in the future.