Virtual Environments In Python 3

In our previous post we covered the process for creating a new package and deploy it using pip In this article I will give you a brief introduction to virtual environments. Virtual environments help you to reduce the mess you can do when you install software with pip for example from the Python Package Index -- or it gives you a helping hand if you have to work with different software using different versions of the same library.

What are virtual environments?

Virtual environments are, as their name already suggests, environments which exist but are not available as default. The idea behind Python virtual environments are to give you a clean start where you can install libraries through pip without affecting your main installation.

Every beginner (and sometimes intermediate) Python developer installs all the required libraries with pip into the main installation. I did this too for a long time. And then you get a project which requires an older version of a module but you already have the newest installed. What do you do?

One thing would be to see and hope that the project works with the new version of the dependency. If you are lucky it is working. But most of the times you aren't. In such cases you can set-up a virtual environment and install the older version of the library there.

I am really fond of virtual environments because they eliminate the problem mentioned above, and I can set-up a new environment for every article to see if it requires something more to get the examples up-and-running or it is fine with the basic Python installation and the mentioned extra dependencies if any.

Another option for virtual environment usage is when you create a new deliverable with dependencies listed in the requirements.txt. If your project is big enough this file can get big too -- and you can miss a dependency from the list. To verify that you included the right contents simply set-up a new environment, install all requirements with pip

pip install -r requirements.txt

and start your project. The other way around works as well: you create an environment for your project and develop it and install any required software there. After you are finished you can export your dependencies with the following command:

pip freeze > requirements.txt

Creating some environments

Now it is time to create some virtual environments to see their effects and usage. For Python 3.5 there is a tool called pyvenv which is already available for use:

GHajba$ pyvenv basic

The command above created a new virtual environment called "basic". It does not contain any packages you installed previously from the package index. On my computer I have already installed BeautifulSoup4 to be included in the default Python environment. Now it should not be in the "basic" environment.

Technically it is a new folder in the directory where you executed the command:

drwxr-xr-x@   7 GHajba  staff    238 Nov 28 20:37 basic

Let's try it out and switch to the new environment:

GHajba$ source basic/bin/activate
(basic) GHajba$

Now the (basic) prefix indicates that we are in the "basic" virtual environment for Python. We can verify this when we list all the modules already installed with pip:

(basic) GHajba$ pip list
pip (7.1.2)
setuptools (18.2)

As you can see, it is really bare and has only the core interpreter features. Fortunately you can use pip here too to install additional modules / libraries. As mentioned in the previous section this is useful if you want to try out different versions from the same library or want a restricted environment to see if your requirements.txt file has all the required dependencies for your project.

To exit an activated environment simply call the deactivate command:

(basic) GHajba$ deactivate
GHajba$

If you have an environment which you don't want to use anymore you can remove it (with rm -rf <environment name> on UNIX-like systems or through the Windows Explorer on Windows).

Virtual environments from Python 3.6

The new, 3.6 version of Python deprecated the pyvenv utility. This is because if you call only pyvenv you do not know what Python version you create the environment for. And this can lead to confusion or error if your pyvenv points to a 2.7 version of Python for example but you want to set-up an environment for Python 3.

Naturally the developers of the language provided a solution to continue to use your virtual environments: you can use the venv module to create and use a virtual environment. The example above would look like the this with Python 3.6:

GHajba$ python3 -m venv basic

The functionality stays the same however. This can be verified if you call pyvenv with its help flag:

(basic) GHajba$ pyvenv -h
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip]
            ENV_DIR [ENV_DIR ...]

As you can see the pyvevn bundles the call to the built-in venv module call in older versions too. So the only difference is that you will have to call the whole module without this wrapper script.

How does a virtual environment work?

Virtual environments work by creating some important links to use when they are activated. For example the Python version. Let's compare which is used when I am outside an environment and when I am inside in our "basic" one:

GHajba$ which python
/usr/bin/python
GHajba$ source basic/bin/activate
(basic) GHajba$ which python
/Users/GHajba/basic/bin/python

As you can see, the virtual environment references a different Python or at least points to a different location which could in the end point to the same Python.

GHajba$ ll basic/bin/
total 96
lrwxr-xr-x   1 GHajba  staff    63 Nov 28 11:28 python3.5 -> /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
lrwxr-xr-x   1 GHajba  staff     9 Nov 28 11:28 python3 -> python3.5
lrwxr-xr-x   1 GHajba  staff     9 Nov 28 11:28 python -> python3.5

As you can see, if we list out the contents of the environment's bin folder we get only one Python installation where it references to: Python 3.5. And to get this working the PATH environment variable is changed too to list the current environment's folder with priority against the other entries to have the right Python version referenced when you use the environment.

Alternative solutions

Naturally there are other external libraries which you can use to create virtual environments. One for example is virtualenv and is available through pip. Another example is Conda which manages your complete environment inclusive Python version -- not just the packages installed.

If you get the hang of virtual environments you will have a lot of them. And if you have, you will start to see that you need a way to manage them on their own. Some solutions already exist for the virtualenv library that is called virtualenvwrapper and can be installed through pip. It organizes your environments in one folder, provides commands to easily switch between environments and to create, delete copy and manage your environments.

Introducing these solutions deeper does not fit into the bounds of this article but I wanted to mention them. Feel free to take a look at them and decide for your own which one to use.

Conclusion

Virtual environments are good tools to manage your projects when they have different dependencies. They are supported by Python by default but there are other solutions too to get your needs covered -- even if you want to support different Python versions.

Go ahead and from now on develop only in virtual environments!