Home >Blog

Introduction to HTTP(S) programming with Python and requests

HTTP is a protocol that powers most of the web infrastructure; it is in the heart of the web. Whenever you click on a link, load an image, watch a video, or download music, it is HTTP that makes it possible. HTTPS is no different than HTTP except that HTTPS is a secure/encrypted version of http.

If we want to do some kind of work on the web with the help of Python we need some code or a library that will help us do that. Python has few built in libraries and functions, but each of them has some kind of issue. There is another library called requests that made the life of Python programmers much easier, better and happier when working with http(s). So, today we are going to use the requests library to talk to the web. Remember that we are going to show the examples here with Python 3—it doesn't matter whether these block of code will be compatible with Python 2 or not.

Installing requests

First install the requests library with the python package manager pip

$ pip install requests

In this way the latest version of the requests library will be installed on your system.

GET requests

Now, let's talk to a website with the help of requests. When we are retrieving data from the web we usually use the GET HTTP verb (GET method in another word). To do the same with requests library we need to use the get() function in the library.

import requests
r = requests.get('http://example.com')
print(r.text)
Output:
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
    background-color: #f0f0f2;
    margin: 0;
    padding: 0;
    font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
div {
    width: 600px;
    margin: 5em auto;
    padding: 50px;
    background-color: #fff;
    border-radius: 1em;
}
a:link, a:visited {
    color: #38488f;
    text-decoration: none;
}
@media (max-width: 700px) {
    body {
        background-color: #fff;
    }
    div {
        width: auto;
        margin: 0 auto;
        border-radius: 0;
        padding: 1em;
    }
}
</style>
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples in documents. You may use this
domain in examples without prior coordination or asking for permission.</p>
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

Now go to http://example.com with your browser, view source code and you will see the same content there. Hurrah! we have successfully talked to the web with the help of the requests library with just few lines of code.

Status codes and exceptions

Life is not a bed of rose, you will have some kind of issues all the time. Maybe your internet connection is gone suddenly when you send a request to a server or website with requests library's get() function. Not only that, the website/server may go down. Your firewall may block your program. You should be prepared for any type of disaster if you do not want to rerun your program through all night manually. So, before letting our program die let's figure out why our program is dying. The most common precaution you should take is to check the status code of the response object.

import requests

resp = requests.get('http://example.com')

print(resp.status_code)

Output:

200

In HTTP protocol the status code 200 means that everything is alright. Now, let's make some deliberate mistakes.

import requests

resp = requests.get('http://example.com/zhdyasdlahsdadsads')

print(resp.status_code)

Output:

404

404 means that the resource we requested was not found by the server.

Let's make another mistake. This time let's use a domain that does not exist.

import requests

resp = requests.get('http://examplexyzabxcss.com')

print(resp.status_code)

Output:

Traceback (most recent call last):

 File "C:\Python35\lib\site-packages\urllib3\connection.py", line 141, in _new_conn

(self.host, self.port), self.timeout, **extra_kw)

 File "C:\Python35\lib\site-packages\urllib3\util\connection.py", line 60, in create_connection

for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):

 File "C:\Python35\lib\socket.py", line 732, in getaddrinfo

for res in _socket.getaddrinfo(host, port, family, type, proto, flags):

socket.gaierror: [Errno 11001] getaddrinfo failed

...

...

...

The program crashed and my screen is flooded with error messages. This time it is not any status code error message. It happened because our program could not get the ip address of the website we provided because no such website exists. We need to handle such exceptions with a try..except.. code block.

Downloading binary files

Let's practice more by downloading an image with Python and requests.

import requests

img_url = "https://www.python.org/static/img/python-logo.png"

response = requests.get(img_url)

with open("python-logo.png", "wb") as f:

f.write(response.content)

Look at the current working directory and you will see that the official Python logo is saved there. The content attribute of the response object contains raw binary data. As an image is not a text content we cannot just use the text attribute of response object for it.

POST requests

When you are submitting some form online we are sending a request to the server with the HTTP POST method (or verb in another way) most of the time. It's very simple to make post requests with the requests library. To test the post request I am going to use httpbin.org's /post endpoint. It just returns the posted data back as json along with some other information. To make a POST request we can use the post() function from the requests module. The data you want to send can be sent with the help of data parameter of post() function.

import requests
post_url = "https://httpbin.org/post"
data = "Test text for post endpoint of httpbin"
resp = requests.post(post_url, data=data)
print(resp.text)
Output:
{
 "args": {},
 "data": "Test text for post endpoint of httpbin",
 "files": {},
 "form": {},
 "headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Connection": "close",
"Content-Length": "38",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.18.1"
 },
 "json": null,
 "origin": "***.***.***.**",
 "url": "https://httpbin.org/post"
}

With the helper post() function you can programmatically login to various websites and make your daily life automated. Take this as your homework.

Working with RESTful APIs

Beginners have confusion about RESTful web services. Some think that it is one kind of special purpose language, some think that it is a library or framework, etc. REST is nothing but a set of rules to make requests to servers that compile with the rules. So, let's work with REST with the help of the requests library. Data from RESTful endpoints are returned as JSON. We can use the JSON module of Python for deserializing the data or we may use the json() method of the response object. I am going to use a free sample REST api website for testing our code. You are free to use anything else.

import requests

endpoint = "https://jsonplaceholder.typicode.com/posts"

resp = requests.get(endpoint)

post_list = resp.json()

few_post = post_list[:2]

for post in few_post:

print("POST ID: ", post['id'])

print("POST TITLE: ", post['title'])

print("POST BODY: ", post['body'])

Output:

POST ID:  1

POST TITLE:  sunt aut facere repellat provident occaecati excepturi optio reprehenderit

POST BODY:  quia et suscipit

suscipit recusandae consequuntur expedita et cum

reprehenderit molestiae ut ut quas totam

nostrum rerum est autem sunt rem eveniet architecto

POST ID:  2

POST TITLE:  qui est esse

POST BODY:  est rerum tempore vitae

sequi sint nihil reprehenderit dolor beatae ea dolores neque

fugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis

qui aperiam non debitis possimus qui neque nisi nulla

Test your code with various REST endpoints. requests is a great library and HTTP is a vast topic. I am going to post some more intermediate and advanced articles in the future. Until then read the official documentation of the requests library and keep practicing.

By Md. Sabuj Sarker | 8/1/2017 | General