Python requests post and get | Complete tutorial [2022]

This blog is all about python requests post and get where we will discuss two popular HTTP requests method in python, which is GET and POST. For this tutorial, we will use the python request library, and for API testing, we will be using httpbin.org. So let’s get started.

What is a python requests library?

Requests is a simple HTTP library for python that allows you to send HTTP requests easily. we do not need to add query strings manually to your URLs or form-encode your POST data.

The general syntax for the request method is:

requests.methodname(params)

And the following requestmethodname is available for use:

Install python requests library

Installation of the Python request library is relatively straightforward. Using the pip command, we can easily install python.

pip install requests

Let’s verify if the requests module gets successfully installed by connecting to ipython.

import requests

If you can import the requests module without any error, which means the requests module is successfully installed.

Let’s proceed further and understand a few important request methods.

Python request get

Get request is the most commonly used method to obtain the requested data from an API endpoint.

The syntax for the get request is

get(url, params=None, **kwargs)
    Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary, list of tuples or bytes to send
        in the query string for the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response

it takes the URL as an input and other optional parameters.

python request GET example

In this session, we will be using the get method to get a response from the open-source httpd.bin API endpoint.

Let’s go to httpd.bin and see the get method API endpoint.

import requests
url = "https://httpbin.org/get"
r = requests.get(url)

python request get Status Codes

The status code informs you of the status of the request. Generally, the 200 status code means the request is successful.

Type the below command to get the status of the above request

 r.status_code

If you are getting 200 as a response code, it means success.

python requests get CONTENT

Give the content of the response in bytes.

response.content
Out[28]: b'{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.26.0", \n    "X-Amzn-Trace-Id": "Root=1-60fc3591-082c48cf2f532be634be1f7c"\n  }, \n  "origin": "103.58.42.116", \n  "url": "https://httpbin.org/get"\n}\n'

python request get Text

response.text shows the content of the response in Unicode.

response.text
Out[30]: '{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.26.0", \n    "X-Amzn-Trace-Id": "Root=1-60fc3591-082c48cf2f532be634be1f7c"\n  }, \n  "origin": "103.58.42.116", \n  "url": "https://httpbin.org/get"\n}\n'

python request get JSON

Response JSON returns the json-encoded content of a response

The format of the response json follows the below pattern:

json(self, **kwargs)
 |      Returns the json-encoded content of a response, if any.
 |
 |      :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
 |      :raises simplejson.JSONDecodeError: If the response body does not
 |          contain valid json and simplejson is installed.
 |      :raises json.JSONDecodeError: If the response body does not contain
 |          valid json and simplejson is not installed on Python 3.
 |      :raises ValueError: If the response body does not contain valid
 |          json and simplejson is not installed on Python 2.

Type the below command to get the json response

response.json()

python request get Request Headers

A response header gives information like date, server, location and content type, etc.

Type the below command to get the response header

response.headers

response.headers
Out[34]: {'Date': 'Sat, 24 Jul 2021 15:45:21 GMT', 'Content-Type': 'application/json', 'Content-Length': '307', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}

Python request post

Python request post command is used to submit data to be processed to the server.

The syntax for the post request is

post(url, data=None, json=None, **kwargs)
    Sends a POST request.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
        object to send in the body of the :class:`Request`.
    :param json: (optional) json data to send in the body of the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response

it takes the URL as input along with other optional parameters.

python request POST example

In this session, we will be using the post method to submit a response to the httpd.bin API endpoint.

Let’s go to httpd.bin and see the put method API endpoint.

Type the below command to post a request

import requests
url ="https://httpbin.org/post"
response= requests.post(url)

python request post Status Codes

Type the below command to get the status code of the post request.

response.status_code

Status code 200 means that the request was successful.

python request post JSON

Returns the json-encoded content of a response

response.json()

Python request proxy

This session will explain how to configure a proxy in python-requests. To test the request with a proxy, we need a working proxy and a URL to send the request.

Type the below command to send a python proxy request

import requests
url= "http://toscrape.com"
proxies = {
      "http": "http://10.10.10.10:8000",
      "https": "http://10.10.10.10:8000"
}
response = requests.get(url, proxies=proxies)

Python requests authorization

Till now, we have seen how to access the API endpoint, which does not require any authentication. However, many REST APIs require you to authenticate before accessing them.

Python request to fetch API with username and password

In this session, we will understand how to use python requests to fetch URLs, which require a username and password for authentication.

To work with username and password authentication, we need to export the HTTPBasicAuth module.

We will be using the below API endpoint to fetch the data.

Type the below command to fetch a response with username and password

import requests
from requests.auth import HTTPBasicAuth
url="https://httpbin.org/basic-auth/naiveskill/admin"
auth = HTTPBasicAuth('naiveskill', 'admin')
response= requests.get(url, auth=auth)
response.status_code
response.json()

Python request to fetch API over Kerberos

Sometimes we need to fetch the request from a URL backed by Kerberos authenticate.

To fetch such data, we need to import theHTTPKerberosAuth library. We have to make sure that you have the required krb ticket available.

If you have the keytab file. you need to do type the below command

kinit -kt example.keytab example

Now, type the below command to fetch a response with the Kerberos token

import requests
from requests_kerberos import HTTPKerberosAuth
url="http://example.org"
auth=HTTPKerberosAuth()
response= requests.get(url, auth=auth)

More information about Kerberos’s request can be found here

Conclusion

I hope you like this tutorial. We have started with the basics of the python request module, and we have seen how to work with GET and POST requests. Please do let me know in the comment box if you are facing any issues while following this tutorial

More to Read?

MongoDB with python

Jenkins job to create python virtual environment