Understanding the request-response lifecycle of a Django web application

A web application or a website revolves around the request-response cycle. When you are visiting a website with the help of a web browser, the browser is sending a request to the web server. Receiving your request, the server then processes it and sends you a response back. A response might be in different forms. You go to a link and your browser shows you a web page—that is one kind of response from the server. During the display of your web page the browser needs to load images, css files, javascript files etc.— those are another kind of response from the server. You are downloading a file—yet another response from the web server. You are requesting some data with a RESTful api and the server is sending some data in json format—still another kind of response. Responses can come in a myriad of types.

Your Django application is no exception. It also needs to return some data in response to a request. But it is not just two steps to do it. Your Django application needs to go through various stages to return the end user some result. To understand the Django framework better you must understand how the requests are initiated and the end result is served to the end user. In the following sections I am going to explain various stages of requests and the software or code used there.

End user and the browser

An end user may send a request to the web server with the help of a browser or any other HTTP(S) compliant software. We say the end user the client and the browser or similar software the client-software. End user needs to have no knowledge on how the request is sent to the server. All the technical details are abstracted from the user.

Web servers

Requests from the end user reache some kind of webserver. On the web most of the time it is Apache or Nginx for Django apps, but there are other options too. Apache has a module called mod_wsgi that is used for delegating the request to the Django application through WSGI interface. Apart from using Apache with mod_wsgi we can use dedicated WSGI servers like uWSGI, Gunicorn, etc. and in that case Apache can be used as a reverse proxy—though you can use a dedicated WSGI server it’s not recommended in a production environment. If we use Nginx then we use this server as reverse proxy and delegate the request to a WSGI server that is hosting the Django application. If we want we can leave Apache or Nginx and use the WSGI server directly as mentioned before.

WSGI

WSGI stands for Web Server Gateway Interface. It is the specification for the interface between web servers and web applications for Python. Once upon a time Python had no standardized way to interface between production level web servers and web applications. For example, people used mod_python, CGI, FastCGI, etc. to fill the gap between web servers (e.g. Apache, Nginx, etc) and the web applications (e.g. Django web application). For standardizing a low level server-application interface a Python Enhancement Proposal (PEP) was drafted and after some revision it was accepted by the community and gurus of Python. As mod_python, CGI, FastCGI are no standard interface for Python web applications we will not talk about them in this article. WSGI is just a specification or standard of a low level interface. There are many implementations of WSGI. uWSGI is a great production ready Python web server. Gunicorn is another example. Tornado also implements WSGI. Twisted has a built in WSGI server. There are even several implementations that are written in pure Python. Apache has a module called mod_wsgi that implements WSGI.

Web servers like Nginx, Apache, other reverse proxies, and load balancers accept the initial request from the client (the browser the end user is using), it then delegates the request to the WSGI server or WSGI application hosting container being used. When the WSGI server or container accepts the request it calls the appropriate WSGI application handler function or callable. From this point on, the work of pure Python starts. In the following sections I am going to discuss what happens next in the case of Django.

WSGI application callable

The WSGI server or container calls a function/callable to handle the end user request. We call it a WSGI application from the perspective of WSGI. It can be a pure function or a callable object with a __call__() method defined on it. It accepts two parameters. The first parameter is environment or environ that will contain all the information and data regarding the request with some extra metadata. The second parameter is a function start_response that will be used to initiate the response. start_response() accepts two parameters. The first parameter is the http status line and the second one is an iterable of a key value pair as tuple of headers. The WSGI application callable's return value must be an iterable of response data. Let's clarify our concept with an example.

def wsgi_application(environment, start_response):
   start_response('200 OK', [('Content-Type', 'text/plain')])
   return [b'Hello, World\n', b'I am ready to be published']

Remember the the iterable that will be returned must contain bytes on Python 3.

Now, as we have enough foundation to talk about Django we can remove the curtain.

Django and WSGI

Django provides the WSGI application callable that bootstraps Django. The function roughly looks like the following.

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project_name.settings")

application = get_wsgi_application()

Here the application variable contains the WSGI application provided by Django. Any WSGI compliant server or container should call this to process the requests.

Django bootstrapping

At the very start of everything the wsgi.py from Django project is imported and all the bootstraping, initializing things happen here. The first thing Django does is it sets up its environment and then it loads the settings.py along with other things configured inside settings.py. URLConfs are also loaded in this phase. Request processing is not started immediately yet. Usually the WSGI server or container loads the wsgi.py module when the server starts or at least when the first request comes.

When a request comes the function/callable associated with the application variable is called. Here ends the responsibility of the wsgi.py module.

Request and middlewares

Before your views get the chance to process the request and return some response there comes middleware into play. The list of middlewares are listed inside the settings.py file. settings.py contains the list with the variable named MIDDLEWARE. An initial and typical middleware list looks like the following.

MIDDLEWARE = [
   'django.middleware.security.SecurityMiddleware',
   'django.contrib.sessions.middleware.SessionMiddleware',
   'django.middleware.common.CommonMiddleware',
   'django.middleware.csrf.CsrfViewMiddleware',
   'django.contrib.auth.middleware.AuthenticationMiddleware',
   'django.contrib.messages.middleware.MessageMiddleware',
   'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

A Django application can run without zero number of middlewares. When you start a Django project with django-admin startproject command then frequently used built in middlewares are added to the list.

A middleware has different phases and lifecycle methods that are called during the processing of the request. We will discuss them in a separate article in detail. For now just understand and remember that the request must pass through the list of middlewares and will return back through the same path as response. In some phases of middleware the middleware can stop the request, can return response before it gets to the view.

URLConfs

The URLConf is the list of url() instance. In many other frameworks it can be called the router or url dispatcher. Django consults the URLConf to resolve which view it should call according to the pattern defined there as regular expression. It loads the urlconf during the bootstrapping phase. The rood URLConf is defined in the settings.py file as ROOT_URLCONF; you can change it according to your need but the most sensible default that is set there is urls.py inside the directory where settings.py resides. The root URLConf can load other URLConfs from other apps inside the same Django project.

Views

After consulting the the URLConf and if the proper view is found that matches a certain pattern in the list of url() instance then that view is called. A view is set as callable associated with some specific pattern in one or more url() instance. A view can be a pure function or any other callable. A class based view is defined inheriting View class. You cannot directly set the class as a view in the url() but instead you have to call as_view() on the class to get a callable of that class.

A view function (or method of View class) must accept at least one parameter that is request. It is an instance of HttpRequest. The view callable must return a response that must be an instance object of HttpResponse.

The view is the last station from where the request will be served as a response. In this place you can return some response directly or you can delegate html/xml/txt data creation/processing responsibility to template system. Also it can retrieve data from a database (models) or other places or even from external servers. But the view is the thing that returns the data.

We will have another article only on views so that we can cover every aspect of it.

Return to home

After the HttpResponse object is generated by the view it goes back through the same path it travelled. The response object once again goes back through the list of middlewares, wsgi application callable, wsgi server/container and then it goes back to the public facing web server or reverse proxy or load balancer that returns the final response to the user.

Exceptions and errors

In the process of request/response, at any point any exception or error may take place. Django handles those exceptions and errors and returns a response according to the exception. You can set your custom handler to display a custom result to the user. We will have a dedicated article on this in future, so check back soon.

*Stop by the homepage to search and compare SDKs, Dev Tools, and Libraries.