I am writing a python script which will call a REST POST endpoint but in response I am getting 400 Bad Request where as if I do same request with curl, it returns me 200 OK. Code snippet for python script is below
import httplib,urllib
def printText(txt):
lines = txt.split('n')
for line in lines:
print line.strip()
httpServ = httplib.HTTPConnection("127.0.0.1", 9100)
httpServ.connect()
params = urllib.urlencode({"externalId": "801411","name": "RD Core","description": "Tenant create","subscriptionType": "MINIMAL","features": {"capture":False,"correspondence": True,"vault": False}})
headers = {"Content-type": "application/json"}
httpServ.request("POST", "/tenants", params, headers)
response = httpServ.getresponse()
print response.status, response.reason
httpServ.close()
and corresponding curl request is
curl -iX POST
-H 'Content-Type: application/json'
-d '
{
"externalId": "801411",
"name": "RD Core seed data test",
"description": "Tenant for Core team seed data testing",
"subscriptionType": "MINIMAL",
"features": {
"capture": false,
"correspondence": true,
"vault": false
}
}' http://localhost:9100/tenants/
Now I am not able figure out where is the issue in python script.
Andrei1penguin1, по разному настроены серверы, принимают контент разного типа.
json в requests просто форматирует стандартный словарь в валидную JSON строку, добавляет соответствующий заголовк Content-Type и отправляет запрос. Передавая информацию в data — передается стандартный Content-Type. В принципе, можно хоть вручную установить Content-Type на json, сделать dumps и передать все это в data.
import requests
import json
data = {
"phone_number": "+77777777777"
}
headers = {'Content-Type': 'application/json'}
response = requests.post("https://eda.yandex.ru/api/v1/user/request_authentication_code", data=json.dumps(data), headers=headers)
print(response.text)
I’m trying to make a simple post request via the requests library of Python and I get a bad request error (400) while my url is supposedly correct since I can use it to perform a get.
I’m very new in REST requests, I read many tutorials and documentation but I guess there are still things I don’t get so my error could be basic. Maybe a lack of understanding on the type of url I’m supposed to send via POST. Here my code :
import requests
v_username = "username"
v_password = "password"
v_headers = {'content-type':'application/rdf+xml'}
url = 'https://my.url'
params = {'param': 'val_param'}
payload = {'data': 'my_data'}
r = requests.post(url, params = params, auth=(v_username, v_password), data=payload, headers=v_headers, verify=False)
print r
I used the example of the requests documentation.
asked Nov 8, 2013 at 15:50
3
I had a similar problem, i tried changing params to data or with json.dumps():
from json import dumps
r = requests.post(url, params=dumps(params), auth=(v_username, v_password), data=payload, headers=v_headers, verify=False)
or
r = requests.post(url, data=dumps(params), auth=(v_username, v_password), data=payload, headers=v_headers, verify=False)
answered Apr 25, 2016 at 17:11
1
Answer by Hendrix Morris
The HyperText Transfer Protocol (HTTP) 400 Bad Request response status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).,HTTP response status codes,Warning: The client should not repeat this request without modification.,
HTTP guide
Basics of HTTP
Overview of HTTP
Evolution of HTTP
HTTP Messages
A typical HTTP session
Connection management in HTTP/1.x
Protocol upgrade mechanism
400 Bad Request
Answer by Averi Duncan
Also, enclose your data as JSON in the request body, don’t pass them as URL parameters. You are passing JSON data in your curl example as well.,and corresponding curl request is ,Please be sure to answer the question. Provide details and share your research!,Making statements based on opinion; back them up with references or personal experience.
Also, enclose your data as JSON
in the request body, don’t pass them as URL parameters. You are passing JSON
data in your curl
example as well.
import requests
data = {
"externalId": "801411",
"name": "RD Core",
"description": "Tenant create",
"subscriptionType": "MINIMAL",
"features": {
"capture": False,
"correspondence": True,
"vault": False
}
}
response = requests.post(
url="http://localhost:9100/tenants/",
json=data
)
print response.status_code, response.reason
Answer by Melani Patrick
The 400 (Bad Request) status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).,The key concept to understand here is that the 400 Bad Request error is something that has to do with the submitted request from the client before it is even processed by the server.,The 400 Bad Request Error is an HTTP response status code
that indicates the server was unable to process (understand) the request sent by the client due to incorrect syntax, invalid request message framing, or deceptive request routing.,The 4xx family of status codes is the one we’re investigating here as they relate to invalid or corrupt requests from the client. Specifically, we’ll take a closer look at the 400 Bad Request error: what this error means, what causes it as well as some specific steps to fix the issue.
This is surprisingly easy to do by mistake and can happen if a URL has been encoding incorrectly. The following link is an example of a URL containing characters the server won’t be able to process, hence a 400 Bad Request error is triggered.
https://twitter.com/share?lang=en&text=Example%20of%20malformed%%20characters%20in%20URL
https://twitter.com/share?lang=en&text=Example%20of%20malformed%%20characters%20in%20URL
Answer by Dexter English
I received this error and got this error message: The content value must be a string at least one character in length.,Recommend double-checking to make sure your code is not violating any of those requirements and, if not, filing a support ticket to further debug the request.,I am having the same error using flask-sendgrid. I have the latest version of sendgrid installed. When launching my flask server, I can send one email fine then when I request to send another one it fails.,Thanks @linehammer, I figured that out ? As I stated in my other issue (sendgrid/python-http-client#133), it seems that flask-sendgrid send twice the to parametter after the second request.
sudo pip3 install sendgrid
Answer by Beatrice Dunlap
The Box APIs uses HTTP status codes to communicate if a request
has been successfully processed or not.,Please check our Developer Troubleshooting Articles
for solution to common errors encountered when working with the Box APIs.,Please see the Client Error resource for more details.,Code samples provided under Unilicense
{
"type": "error",
"status": 400,
"code": "bad_digest",
"help_url": "http://developers.box.com/docs/#errors",
"message": "The specified content-md5 did not match what we received",
"request_id": "abcdef123456"
}
Answer by Zahra Salgado
The Client application gets the following response code:,400 Bad request — plain HTTP request sent to HTTPS port,400 Bad request — SSL certificate error,The client application receives an HTTP 400 — Bad request response with the
message «The SSL certificate error». This error is typically sent by the Edge Router
in a two way TLS setup enabled for the incoming connection to Apigee Edge.
The Client application gets the following response code:
HTTP/1.1 400 Bad Request
Followed by the below HTML error page:
<html>
<head>
<title>400 The SSL certificate error</title>
</head>
<body bgcolor="white">
<center> <h1>400 Bad Request</h1>
</center>
<center>The SSL certificate error</center>
<hr>
<center>nginx</center>
</body>
</html>
Typically a virtual host for two-way TLS communication looks as follows:
<VirtualHost name="myTLSVHost">
<HostAliases>
<HostAlias>api.myCompany.com</HostAlias>
</HostAliases>
<Port>443</Port>
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://myKeystoreRef</KeyStore>
<KeyAlias>myKeyAlias</KeyAlias>
<TrustStore>ref://myTruststoreRef</TrustStore>
</SSLInfo>
</VirtualHost>
Once you’ve decided where you would like to capture TCP/IP packets, use the following
tcpdump
command to capture TCP/IP packets:
tcpdump -i any -s 0 host <IP address> -w <File name>
Typically a virtual host for two-way TLS communication looks as follows:
<VirtualHost name="myTLSVHost">
<HostAliases>
<HostAlias>api.myCompany.com</HostAlias>
</HostAliases>
<Port>443</Port>
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://myKeystoreRef</KeyStore>
<KeyAlias>myKeyAlias</KeyAlias>
<TrustStore>ref://myCompanyTruststoreRef</TrustStore>
</SSLInfo>
</VirtualHost>
Typically a virtual host for two-way TLS communication looks as follows:
<VirtualHost name="myTLSVHost">
<HostAliases>
<HostAlias>api.myCompany.com</HostAlias>
</HostAliases>
<Port>443</Port>
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://myKeystoreRef</KeyStore>
<KeyAlias>myKeyAlias</KeyAlias>
<TrustStore>ref://myCompanyTruststoreRef</TrustStore>
</SSLInfo>
</VirtualHost>
openssl
openssl -in <OrgName_envName_vhostName-client.pem> -text -noout
Restart the Router to ensure the latest Certificates are loaded using the below step:
apigee-service edge-router restart
Answer by Ember Nava
The following table lists all the codes that can appear as code attribute of an <error> element if an error has occurred. ,If error messages have been translated, they are returned in the language that’s set in the Accept-Language header of the request. For example, if the headers include Accept-Language: de-de, error messages are returned in German. ,Note: You should not rely on specific text appearing in the <detail> element of an error response. Instead, test the value of the error code attribute to determine why an operation failed.,An HTTP status code of 404 for the response tells you that the operation was not successful because a resource could not be found. In that case, the response body might look like the following example:
For error conditions, the response body also includes an XML block that provides details about the error. For example, if the HTTP response was 404, the response body provides details about what resource in particular was not found. Imagine that you send the following PUT request in order to update information for a user:
http://your-server/api/3.13/sites/9a8b7c6d5-e4f3-a2b1-c0d9-e8f7a6b5c4d/users/9f9e9d9c8-b8a8-f8e7-d7c7-b7a6f6d6e6d
http://your-server/api/3.13/sites/9a8b7c6d5-e4f3-a2b1-c0d9-e8f7a6b5c4d/users/9f9e9d9c8-b8a8-f8e7-d7c7-b7a6f6d6e6d
Answer by Ezra Mora
Errors in Microsoft Graph are returned using standard HTTP status codes, as well as a JSON error response object.,The following table lists and describes the HTTP status codes that can be returned.,The error resource is returned whenever an error occurs in the processing of a request.,The code property contains one of the following possible values. Your apps should be
prepared to handle any one of these errors.
The error response is a single JSON object that contains a single property
named error. This object includes all the details of the error. You can use the information returned here instead of or in addition to the HTTP status code. The following is an example of a full JSON error body.
{
"error": {
"code": "invalidRange",
"message": "Uploaded fragment overlaps with existing data.",
"innerError": {
"requestId": "request-id",
"date": "date-time"
}
}
}
The error resource is composed of these resources:
{
"error": { "@odata.type": "odata.error" }
}
Inside the error response is an error resource that includes the following
properties:
{
"code": "string",
"message": "string",
"innererror": { "@odata.type": "odata.error" }
}
To verify that an error object is an error you are expecting, you must loop
over the innererror
objects, looking for the error codes you expect. For example:
public bool IsError(string expectedErrorCode)
{
OneDriveInnerError errorCode = this.Error;
while (null != errorCode)
{
if (errorCode.Code == expectedErrorCode)
return true;
errorCode = errorCode.InnerError;
}
return false;
}
In this Azure tutorial, we will discuss how to fix the error Python 400 client error: Bad request for URL Azure cognitive services. that I got while working with a requirement to Analyze an image using the Azure cognitive services Computer Vision API and Python.
- Build Intelligent C# Apps With Azure Cognitive Services
Table of Contents
- Python 400 client error: Bad request for URL Azure cognitive services
- Python 400 client error: Bad request for URL Azure cognitive services [Solved]
- Wrapping Up
Recently, I was working with a requirement where I had to get the details of an image using the Azure cognitive services Computer Vision API using Python. While running the application I got the error Python 400 client error: Bad request for URL Azure cognitive services.
“Python 400 client error: Bad request for URL Azure cognitive services“
This error Python 400 client error: Bad request for URL Azure cognitive services is mostly due to the below Problems
1- First thing is there might be some problem with the image, you are using here.
2- There might be some problem with the API Endpoint URL.
In my case, there was some issue with my Image so I got this error. There might be a chance you will also get the same error.
Python 400 client error: Bad request for URL Azure cognitive services [Solved]
Now, to fix this error, I have tried many ways, but at the end, I was getting the same error. Then I tried using another image and this time it was working as expected with out any issue. Few key things you need to verify if you are getting this error
- Verify the image properly like image size, format, etc. If you are getting the error with one image, try with another image, It might work for you.
- The second important thing is to check the API endpoint URL if it is properly formatted with the proper region, version, and all the parameters needed.
- Check the Azure cognitive services API key value properly. If needed ReGenerate the key and use the new one.
You can try to use the EndPoint URL like below
key = "d38b4303c2774962a####ed43ff4b76f"
assert apikey
url = "https://southeastasia.api.cognitive.microsoft.com/vision/v3.0/"
analyse_api = url + "analyze"
image_data = img
headers = {"Ocp-Apim-Subscription-Key": key,
'Content-Type': 'application/octet-stream'}
params = {'visualFeatures':'Categories,Description,Color,Objects,Faces'}
response = requests.post(
analyse_api, headers=headers, params=params, data=image_data)
response.raise_for_status()
analysis = response.json()
//Rest of the code based on your functionality
This is how you can able to fix the error ” Python 400 client error: Bad request for URL Azure cognitive services “.
You may also like following the below articles
- CS1061 C# ‘HttpRequest’ does not contain a definition for ‘Content’ and no accessible extension method ‘Content’ accepting a first argument of type ‘HttpRequest’ could be found
- Failed To Validate The Notification URL SharePoint Webhook Azure Function Endpoint
- How To Implement Azure Face API Using Visual Studio 2019
- How To Convert m4a File To Text Using Azure Cognitive Services
- Web deployment task failed – cannot modify the file on the destination because it is locked by an external process
Wrapping Up
Well, in this article, we have discussed how to fix Python 400 client error: Bad request for URL Azure cognitive services, Python-requests exceptions HTTP error 400 client error: bad request for URL. Hope this will help you to fix your issue as well !!!
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
The requests
library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.
Throughout this article, you’ll see some of the most useful features that requests
has to offer as well as how to customize and optimize those features for different situations you may come across. You’ll also learn how to use requests
in an efficient way as well as how to prevent requests to external services from slowing down your application.
In this tutorial, you’ll learn how to:
- Make requests using the most common HTTP methods
- Customize your requests’ headers and data, using the query string and message body
- Inspect data from your requests and responses
- Make authenticated requests
- Configure your requests to help prevent your application from backing up or slowing down
Though I’ve tried to include as much information as you need to understand the features and examples included in this article, I do assume a very basic general knowledge of HTTP. That said, you still may be able to follow along fine anyway.
Now that that is out of the way, let’s dive in and see how you can use requests
in your application!
Getting Started With requests
Let’s begin by installing the requests
library. To do so, run the following command:
If you prefer to use Pipenv for managing Python packages, you can run the following:
$ pipenv install requests
Once requests
is installed, you can use it in your application. Importing requests
looks like this:
Now that you’re all set up, it’s time to begin your journey through requests
. Your first goal will be learning how to make a GET
request.
The GET Request
HTTP methods such as GET
and POST
, determine which action you’re trying to perform when making an HTTP request. Besides GET
and POST
, there are several other common methods that you’ll use later in this tutorial.
One of the most common HTTP methods is GET
. The GET
method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET
request, invoke requests.get()
.
To test this out, you can make a GET
request to GitHub’s Root REST API by calling get()
with the following URL:
>>>
>>> requests.get('https://api.github.com')
<Response [200]>
Congratulations! You’ve made your first request. Let’s dive a little deeper into the response of that request.
The Response
A Response
is a powerful object for inspecting the results of the request. Let’s make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:
>>>
>>> response = requests.get('https://api.github.com')
In this example, you’ve captured the return value of get()
, which is an instance of Response
, and stored it in a variable called response
. You can now use response
to see a lot of information about the results of your GET
request.
Status Codes
The first bit of information that you can gather from Response
is the status code. A status code informs you of the status of the request.
For example, a 200 OK
status means that your request was successful, whereas a 404 NOT FOUND
status means that the resource you were looking for was not found. There are many other possible status codes as well to give you specific insights into what happened with your request.
By accessing .status_code
, you can see the status code that the server returned:
>>>
>>> response.status_code
200
.status_code
returned a 200
, which means your request was successful and the server responded with the data you were requesting.
Sometimes, you might want to use this information to make decisions in your code:
if response.status_code == 200:
print('Success!')
elif response.status_code == 404:
print('Not Found.')
With this logic, if the server returns a 200
status code, your program will print Success!
. If the result is a 404
, your program will print Not Found
.
requests
goes one step further in simplifying this process for you. If you use a Response
instance in a conditional expression, it will evaluate to True
if the status code was between 200
and 400
, and False
otherwise.
Therefore, you can simplify the last example by rewriting the if
statement:
if response:
print('Success!')
else:
print('An error has occurred.')
Keep in mind that this method is not verifying that the status code is equal to 200
. The reason for this is that other status codes within the 200
to 400
range, such as 204 NO CONTENT
and 304 NOT MODIFIED
, are also considered successful in the sense that they provide some workable response.
For example, the 204
tells you that the response was successful, but there’s no content to return in the message body.
So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code.
Let’s say you don’t want to check the response’s status code in an if
statement. Instead, you want to raise an exception if the request was unsuccessful. You can do this using .raise_for_status()
:
import requests
from requests.exceptions import HTTPError
for url in ['https://api.github.com', 'https://api.github.com/invalid']:
try:
response = requests.get(url)
# If the response was successful, no Exception will be raised
response.raise_for_status()
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}') # Python 3.6
except Exception as err:
print(f'Other error occurred: {err}') # Python 3.6
else:
print('Success!')
If you invoke .raise_for_status()
, an HTTPError
will be raised for certain status codes. If the status code indicates a successful request, the program will proceed without that exception being raised.
Now, you know a lot about how to deal with the status code of the response you got back from the server. However, when you make a GET
request, you rarely only care about the status code of the response. Usually, you want to see more. Next, you’ll see how to view the actual data that the server sent back in the body of the response.
Content
The response of a GET
request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response
, you can view the payload in a variety of different formats.
To see the response’s content in bytes
, you use .content
:
>>>
>>> response = requests.get('https://api.github.com')
>>> response.content
b'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
While .content
gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. response
will do that for you when you access .text
:
>>>
>>> response.text
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
Because the decoding of bytes
to a str
requires an encoding scheme, requests
will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding
before accessing .text
:
>>>
>>> response.encoding = 'utf-8' # Optional: requests infers this internally
>>> response.text
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str
you retrieved from .text
and deserialize it using json.loads()
. However, a simpler way to accomplish this task is to use .json()
:
>>>
>>> response.json()
{'current_user_url': 'https://api.github.com/user', 'current_user_authorizations_html_url': 'https://github.com/settings/connections/applications{/client_id}', 'authorizations_url': 'https://api.github.com/authorizations', 'code_search_url': 'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', 'commit_search_url': 'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', 'emails_url': 'https://api.github.com/user/emails', 'emojis_url': 'https://api.github.com/emojis', 'events_url': 'https://api.github.com/events', 'feeds_url': 'https://api.github.com/feeds', 'followers_url': 'https://api.github.com/user/followers', 'following_url': 'https://api.github.com/user/following{/target}', 'gists_url': 'https://api.github.com/gists{/gist_id}', 'hub_url': 'https://api.github.com/hub', 'issue_search_url': 'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', 'issues_url': 'https://api.github.com/issues', 'keys_url': 'https://api.github.com/user/keys', 'notifications_url': 'https://api.github.com/notifications', 'organization_repositories_url': 'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', 'organization_url': 'https://api.github.com/orgs/{org}', 'public_gists_url': 'https://api.github.com/gists/public', 'rate_limit_url': 'https://api.github.com/rate_limit', 'repository_url': 'https://api.github.com/repos/{owner}/{repo}', 'repository_search_url': 'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', 'current_user_repositories_url': 'https://api.github.com/user/repos{?type,page,per_page,sort}', 'starred_url': 'https://api.github.com/user/starred{/owner}{/repo}', 'starred_gists_url': 'https://api.github.com/gists/starred', 'team_url': 'https://api.github.com/teams', 'user_url': 'https://api.github.com/users/{user}', 'user_organizations_url': 'https://api.github.com/user/orgs', 'user_repositories_url': 'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', 'user_search_url': 'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}'}
The type
of the return value of .json()
is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
The response headers can give you useful information, such as the content type of the response payload and a time limit on how long to cache the response. To view these headers, access .headers
:
>>>
>>> response.headers
{'Server': 'GitHub.com', 'Date': 'Mon, 10 Dec 2018 17:49:54 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Status': '200 OK', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '59', 'X-RateLimit-Reset': '1544467794', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Vary': 'Accept', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'X-GitHub-Media-Type': 'github.v3; format=json', 'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'Content-Security-Policy': "default-src 'none'", 'Content-Encoding': 'gzip', 'X-GitHub-Request-Id': 'E439:4581:CF2351:1CA3E06:5C0EA741'}
.headers
returns a dictionary-like object, allowing you to access header values by key. For example, to see the content type of the response payload, you can access Content-Type
:
>>>
>>> response.headers['Content-Type']
'application/json; charset=utf-8'
There is something special about this dictionary-like headers object, though. The HTTP spec defines headers to be case-insensitive, which means we are able to access these headers without worrying about their capitalization:
>>>
>>> response.headers['content-type']
'application/json; charset=utf-8'
Whether you use the key 'content-type'
or 'Content-Type'
, you’ll get the same value.
Now, you’ve learned the basics about Response
. You’ve seen its most useful attributes and methods in action. Let’s take a step back and see how your responses change when you customize your GET
requests.
Query String Parameters
One common way to customize a GET
request is to pass values through query string parameters in the URL. To do this using get()
, you pass data to params
. For example, you can use GitHub’s Search API to look for the requests
library:
import requests
# Search GitHub's repositories for requests
response = requests.get(
'https://api.github.com/search/repositories',
params={'q': 'requests+language:python'},
)
# Inspect some attributes of the `requests` repository
json_response = response.json()
repository = json_response['items'][0]
print(f'Repository name: {repository["name"]}') # Python 3.6+
print(f'Repository description: {repository["description"]}') # Python 3.6+
By passing the dictionary {'q': 'requests+language:python'}
to the params
parameter of .get()
, you are able to modify the results that come back from the Search API.
You can pass params
to get()
in the form of a dictionary, as you have just done, or as a list of tuples:
>>>
>>> requests.get(
... 'https://api.github.com/search/repositories',
... params=[('q', 'requests+language:python')],
... )
<Response [200]>
You can even pass the values as bytes
:
>>>
>>> requests.get(
... 'https://api.github.com/search/repositories',
... params=b'q=requests+language:python',
... )
<Response [200]>
Query strings are useful for parameterizing GET
requests. You can also customize your requests by adding or modifying the headers you send.
To customize headers, you pass a dictionary of HTTP headers to get()
using the headers
parameter. For example, you can change your previous search request to highlight matching search terms in the results by specifying the text-match
media type in the Accept
header:
import requests
response = requests.get(
'https://api.github.com/search/repositories',
params={'q': 'requests+language:python'},
headers={'Accept': 'application/vnd.github.v3.text-match+json'},
)
# View the new `text-matches` array which provides information
# about your search term within the results
json_response = response.json()
repository = json_response['items'][0]
print(f'Text matches: {repository["text_matches"]}')
The Accept
header tells the server what content types your application can handle. In this case, since you’re expecting the matching search terms to be highlighted, you’re using the header value application/vnd.github.v3.text-match+json
, which is a proprietary GitHub Accept
header where the content is a special JSON format.
Before you learn more ways to customize requests, let’s broaden the horizon by exploring other HTTP methods.
Other HTTP Methods
Aside from GET
, other popular HTTP methods include POST
, PUT
, DELETE
, HEAD
, PATCH
, and OPTIONS
. requests
provides a method, with a similar signature to get()
, for each of these HTTP methods:
>>>
>>> requests.post('https://httpbin.org/post', data={'key':'value'})
>>> requests.put('https://httpbin.org/put', data={'key':'value'})
>>> requests.delete('https://httpbin.org/delete')
>>> requests.head('https://httpbin.org/get')
>>> requests.patch('https://httpbin.org/patch', data={'key':'value'})
>>> requests.options('https://httpbin.org/get')
Each function call makes a request to the httpbin
service using the corresponding HTTP method. For each method, you can inspect their responses in the same way you did before:
>>>
>>> response = requests.head('https://httpbin.org/get')
>>> response.headers['Content-Type']
'application/json'
>>> response = requests.delete('https://httpbin.org/delete')
>>> json_response = response.json()
>>> json_response['args']
{}
Headers, response bodies, status codes, and more are returned in the Response
for each method. Next you’ll take a closer look at the POST
, PUT
, and PATCH
methods and learn how they differ from the other request types.
The Message Body
According to the HTTP specification, POST
, PUT
, and the less common PATCH
requests pass their data through the message body rather than through parameters in the query string. Using requests
, you’ll pass the payload to the corresponding function’s data
parameter.
data
takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with.
For example, if your request’s content type is application/x-www-form-urlencoded
, you can send the form data as a dictionary:
>>>
>>> requests.post('https://httpbin.org/post', data={'key':'value'})
<Response [200]>
You can also send that same data as a list of tuples:
>>>
>>> requests.post('https://httpbin.org/post', data=[('key', 'value')])
<Response [200]>
If, however, you need to send JSON data, you can use the json
parameter. When you pass JSON data via json
, requests
will serialize your data and add the correct Content-Type
header for you.
httpbin.org is a great resource created by the author of requests
, Kenneth Reitz. It’s a service that accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic POST
request:
>>>
>>> response = requests.post('https://httpbin.org/post', json={'key':'value'})
>>> json_response = response.json()
>>> json_response['data']
'{"key": "value"}'
>>> json_response['headers']['Content-Type']
'application/json'
You can see from the response that the server received your request data and headers as you sent them. requests
also provides this information to you in the form of a PreparedRequest
.
Inspecting Your Request
When you make a request, the requests
library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.
You can view the PreparedRequest
by accessing .request
:
>>>
>>> response = requests.post('https://httpbin.org/post', json={'key':'value'})
>>> response.request.headers['Content-Type']
'application/json'
>>> response.request.url
'https://httpbin.org/post'
>>> response.request.body
b'{"key": "value"}'
Inspecting the PreparedRequest
gives you access to all kinds of information about the request being made such as payload, URL, headers, authentication, and more.
So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they’re unauthenticated requests to public APIs. Many services you may come across will want you to authenticate in some way.
Authentication
Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization
header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called auth
, which allows you to pass your credentials.
One example of an API that requires authentication is GitHub’s Authenticated User API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to get()
:
>>>
>>> from getpass import getpass
>>> requests.get('https://api.github.com/user', auth=('username', getpass()))
<Response [200]>
The request succeeded if the credentials you passed in the tuple to auth
are valid. If you try to make this request with no credentials, you’ll see that the status code is 401 Unauthorized
:
>>>
>>> requests.get('https://api.github.com/user')
<Response [401]>
When you pass your username and password in a tuple to the auth
parameter, requests
is applying the credentials using HTTP’s Basic access authentication scheme under the hood.
Therefore, you could make the same request by passing explicit Basic authentication credentials using HTTPBasicAuth
:
>>>
>>> from requests.auth import HTTPBasicAuth
>>> from getpass import getpass
>>> requests.get(
... 'https://api.github.com/user',
... auth=HTTPBasicAuth('username', getpass())
... )
<Response [200]>
Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. requests
provides other methods of authentication out of the box such as HTTPDigestAuth
and HTTPProxyAuth
.
You can even supply your own authentication mechanism. To do so, you must first create a subclass of AuthBase
. Then, you implement __call__()
:
import requests
from requests.auth import AuthBase
class TokenAuth(AuthBase):
"""Implements a custom authentication scheme."""
def __init__(self, token):
self.token = token
def __call__(self, r):
"""Attach an API token to a custom auth header."""
r.headers['X-TokenAuth'] = f'{self.token}' # Python 3.6+
return r
requests.get('https://httpbin.org/get', auth=TokenAuth('12345abcde-token'))
Here, your custom TokenAuth
mechanism receives a token, then includes that token in the X-TokenAuth
header of your request.
Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth.
While you’re thinking about security, let’s consider dealing with SSL Certificates using requests
.
SSL Certificate Verification
Any time the data you are trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL Certificate is critical.
The good news is that requests
does this for you by default. However, there are some cases where you might want to change this behavior.
If you want to disable SSL Certificate verification, you pass False
to the verify
parameter of the request function:
>>>
>>> requests.get('https://api.github.com', verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
<Response [200]>
requests
even warns you when you’re making an insecure request to help you keep your data safe!
Performance
When using requests
, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.
Timeouts
When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang.
By default, requests
will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the timeout
parameter. timeout
can be an integer or float representing the number of seconds to wait on a response before timing out:
>>>
>>> requests.get('https://api.github.com', timeout=1)
<Response [200]>
>>> requests.get('https://api.github.com', timeout=3.05)
<Response [200]>
In the first request, the request will timeout after 1 second. In the second request, the request will timeout after 3.05 seconds.
You can also pass a tuple to timeout
with the first element being a connect timeout (the time it allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection):
>>>
>>> requests.get('https://api.github.com', timeout=(2, 5))
<Response [200]>
If the request establishes a connection within 2 seconds and receives data within 5 seconds of the connection being established, then the response will be returned as it was before. If the request times out, then the function will raise a Timeout
exception:
import requests
from requests.exceptions import Timeout
try:
response = requests.get('https://api.github.com', timeout=1)
except Timeout:
print('The request timed out')
else:
print('The request did not time out')
Your program can catch the Timeout
exception and respond accordingly.
The Session Object
Until now, you’ve been dealing with high level requests
APIs such as get()
and post()
. These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.
Underneath those abstractions is a class called Session
. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session
instance directly.
Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session:
import requests
from getpass import getpass
# By using a context manager, you can ensure the resources used by
# the session will be released after use
with requests.Session() as session:
session.auth = ('username', getpass())
# Instead of requests.get(), you'll use session.get()
response = session.get('https://api.github.com/user')
# You can inspect the response just like you did before
print(response.headers)
print(response.json())
Each time you make a request with session
, once it has been initialized with authentication credentials, the credentials will be persisted.
The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a Session
, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one.
Max Retries
When a request fails, you may want your application to retry the same request. However, requests
will not do this for you by default. To apply this functionality, you need to implement a custom Transport Adapter.
Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to https://api.github.com
to retry three times before finally raising a ConnectionError
. You would build a Transport Adapter, set its max_retries
parameter, and mount it to an existing Session
:
import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import ConnectionError
github_adapter = HTTPAdapter(max_retries=3)
session = requests.Session()
# Use `github_adapter` for all requests to endpoints that start with this URL
session.mount('https://api.github.com', github_adapter)
try:
session.get('https://api.github.com')
except ConnectionError as ce:
print(ce)
When you mount the HTTPAdapter
, github_adapter
, to session
, session
will adhere to its configuration for each request to https://api.github.com.
Timeouts, Transport Adapters, and sessions are for keeping your code efficient and your application resilient.
Conclusion
You’ve come a long way in learning about Python’s powerful requests
library.
You’re now able to:
- Make requests using a variety of different HTTP methods such as
GET
,POST
, andPUT
- Customize your requests by modifying headers, authentication, query strings, and message bodies
- Inspect the data you send to the server and the data the server sends back to you
- Work with SSL Certificate verification
- Use
requests
effectively usingmax_retries
,timeout
, Sessions, and Transport Adapters
Because you learned how to use requests
, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Making HTTP Requests With Python
The urllib.error.HTTPError
is a class in the Python urllib
library that represents an HTTP error. An HTTPError
is raised when an HTTP request returns a status code that represents an error, such as 4xx (client error) or 5xx (server error).
HTTPError Attributes
The urllib.error.HTTPError
class has the following attributes:
code
: The HTTP status code of the error.reason
: The human-readable reason phrase associated with the status code.headers
: The HTTP response headers for the request that caused theHTTPError
.
What Causes HTTPError
Here are some common reasons why an HTTPError
might be raised:
- Invalid or malformed request URL.
- Invalid or malformed request parameters or body.
- Invalid or missing authentication credentials.
- Server internal error or malfunction.
- Server temporarily unavailable due to maintenance or overload.
Python HTTPError Examples
Here are a few examples of HTTP errors in Python:
404 Not Found
import urllib.request
import urllib.error
try:
response = urllib.request.urlopen('http://httpbin.org/status/404')
except urllib.error.HTTPError as err:
print(f'A HTTPError was thrown: {err.code} {err.reason}')
In the above example, an invalid URL is attempted to be opened using the urllib.request.urlopen()
function. Running the above code raises an HTTPError
with code 404:
A HTTPError was thrown: 404 NOT FOUND
400 Bad Request
import urllib.request
try:
response = urllib.request.urlopen('http://httpbin.org/status/400')
except urllib.error.HTTPError as err:
if err.code == 400:
print('Bad request!')
else:
print(f'An HTTP error occurred: {err}')
In the above example, a bad request is sent to the server. Running the above code raises a HTTPError
with code 400:
Bad request!
401 Unauthorized
import urllib.request
import urllib.error
try:
response = urllib.request.urlopen('http://httpbin.org/status/401')
except urllib.error.HTTPError as err:
if err.code == 401:
print('Unauthorized!')
else:
print(f'An HTTP error occurred: {err}')
In the above example, a request is sent to the server with missing credentials. Running the above code raises a HTTPError
with code 401:
Unauthorized!
500 Internal Server Error
import urllib.request
import urllib.error
try:
response = urllib.request.urlopen('http://httpbin.org/status/500')
except urllib.error.HTTPError as err:
if err.code == 500:
print('Internal server error!')
else:
print(f'An HTTP error occurred: {err}')
In the above example, the server experiences an error internally. Running the above code raises a HTTPError
with code 500:
Internal server error!
How to Fix HTTPError in Python
To fix HTTP errors in Python, the following steps can be taken:
- Check the network connection and ensure it is stable and working.
- Check the URL being accessed and make sure it is correct and properly formatted.
- Check the request parameters and body to ensure they are valid and correct.
- Check whether the request requires authentication credentials and make sure they are included in the request and are correct.
- If the request and URL are correct, check the HTTP status code and reason returned in the error message. This can give more information about the error.
- Try adding error handling code for the specific error. For example, the request can be attempted again or missing parameters can be added to the request.
Track, Analyze and Manage Errors With Rollbar
Managing errors and exceptions in your code is challenging. It can make deploying production code an unnerving experience. Being able to track, analyze, and manage errors in real-time can help you to proceed with more confidence. Rollbar automates error monitoring and triaging, making fixing Python errors easier than ever. Try it today!
sorry, here is the trackback. and im using Mac OSX
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
self._prepare_proxy(conn)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 810, in _prepare_proxy
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 294, in connect
self._tunnel()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 832, in _tunnel
message.strip()))
OSError: Tunnel connection failed: 400 Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/adapters.py", line 438, in send
timeout=timeout
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/edwardbiswas/Desktop/Script/proxy.py", line 107, in <module>
r = requests.get('https://icanhazip.com', proxies={'https': 'http://{}'.format(random.choice(proxies_list))})
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 518, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 639, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/adapters.py", line 500, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request',)))