Python requests ошибка ssl

I’m working on a simple script that involves CAS, jspring security check, redirection, etc. I would like to use Kenneth Reitz’s python requests because it’s a great piece of work! However, CAS requires getting validated via SSL so I have to get past that step first. I don’t know what Python requests is wanting? Where is this SSL certificate supposed to reside?

Traceback (most recent call last):
  File "./test.py", line 24, in <module>
  response = requests.get(url1, headers=headers)
  File "build/bdist.linux-x86_64/egg/requests/api.py", line 52, in get
  File "build/bdist.linux-x86_64/egg/requests/api.py", line 40, in request
  File "build/bdist.linux-x86_64/egg/requests/sessions.py", line 209, in request 
  File "build/bdist.linux-x86_64/egg/requests/models.py", line 624, in send
  File "build/bdist.linux-x86_64/egg/requests/models.py", line 300, in _build_response
  File "build/bdist.linux-x86_64/egg/requests/models.py", line 611, in send
requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Andrew Brēza's user avatar

Andrew Brēza

7,5673 gold badges33 silver badges40 bronze badges

asked May 19, 2012 at 18:45

TedBurrows's user avatar

4

The problem you are having is caused by an untrusted SSL certificate.

Like @dirk mentioned in a previous comment, the quickest fix is setting verify=False:

requests.get('https://example.com', verify=False)

Please note that this will cause the certificate not to be verified. This will expose your application to security risks, such as man-in-the-middle attacks.

Of course, apply judgment. As mentioned in the comments, this may be acceptable for quick/throwaway applications/scripts, but really should not go to production software.

If just skipping the certificate check is not acceptable in your particular context, consider the following options, your best option is to set the verify parameter to a string that is the path of the .pem file of the certificate (which you should obtain by some sort of secure means).

So, as of version 2.0, the verify parameter accepts the following values, with their respective semantics:

  • True: causes the certificate to validated against the library’s own trusted certificate authorities (Note: you can see which Root Certificates Requests uses via the Certifi library, a trust database of RCs extracted from Requests: Certifi — Trust Database for Humans).
  • False: bypasses certificate validation completely.
  • Path to a CA_BUNDLE file for Requests to use to validate the certificates.

Source: Requests — SSL Cert Verification

Also take a look at the cert parameter on the same link.

Martijn Pieters's user avatar

answered Oct 12, 2012 at 18:19

Rafael Almeida's user avatar

Rafael AlmeidaRafael Almeida

10.3k6 gold badges45 silver badges60 bronze badges

15

From requests documentation on SSL verification:

Requests can verify SSL certificates for HTTPS requests, just like a web browser. To check a host’s SSL certificate, you can use the verify argument:

>>> requests.get('https://kennethreitz.com', verify=True)

If you don’t want to verify your SSL certificate, make verify=False

ostergaard's user avatar

ostergaard

3,3672 gold badges30 silver badges40 bronze badges

answered May 19, 2012 at 19:20

Zeugma's user avatar

ZeugmaZeugma

31k8 gold badges67 silver badges80 bronze badges

8

I encountered the same issue and ssl certificate verify failed issue when using aws boto3, by review boto3 code, I found the REQUESTS_CA_BUNDLE is not set, so I fixed the both issue by setting it manually:

from boto3.session import Session
import os

# debian
os.environ['REQUESTS_CA_BUNDLE'] = os.path.join(
    '/etc/ssl/certs/',
    'ca-certificates.crt')
# centos
#   'ca-bundle.crt')

For aws-cli, I guess setting REQUESTS_CA_BUNDLE in ~/.bashrc will fix this issue (not tested because my aws-cli works without it).

REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt # ca-bundle.crt
export REQUESTS_CA_BUNDLE

rodorgas's user avatar

rodorgas

9522 gold badges12 silver badges29 bronze badges

answered Nov 15, 2015 at 7:47

Yong's user avatar

YongYong

6815 silver badges3 bronze badges

4

The name of CA file to use you could pass via verify:

cafile = 'cacert.pem' # http://curl.haxx.se/ca/cacert.pem
r = requests.get(url, verify=cafile)

If you use verify=True then requests uses its own CA set that might not have CA that signed your server certificate.

answered Oct 12, 2012 at 18:38

jfs's user avatar

jfsjfs

396k192 gold badges978 silver badges1667 bronze badges

9

$ pip install -U requests[security]

  • Tested on Python 2.7.6 @ Ubuntu 14.04.4 LTS
  • Tested on Python 2.7.5 @ MacOSX 10.9.5 (Mavericks)

When this question was opened (2012-05) the Requests version was 0.13.1. On version 2.4.1 (2014-09) the «security» extras were introduced, using certifi package if available.

Right now (2016-09) the main version is 2.11.1, that works good without verify=False. No need to use requests.get(url, verify=False), if installed with requests[security] extras.

answered Sep 19, 2016 at 18:57

alanjds's user avatar

alanjdsalanjds

3,9342 gold badges33 silver badges43 bronze badges

6

In case you have a library that relies on requests and you cannot modify the verify path (like with pyvmomi) then you’ll have to find the cacert.pem bundled with requests and append your CA there. Here’s a generic approach to find the cacert.pem location:

windows

C:>python -c "import requests; print requests.certs.where()"
c:Python27libsite-packagesrequests-2.8.1-py2.7.eggrequestscacert.pem

linux

#  (py2.7.5,requests 2.7.0, verify not enforced)
root@host:~/# python -c "import requests; print requests.certs.where()"
/usr/lib/python2.7/dist-packages/certifi/cacert.pem

#  (py2.7.10, verify enforced)
root@host:~/# python -c "import requests; print requests.certs.where()"
/usr/local/lib/python2.7/dist-packages/requests/cacert.pem

btw. @requests-devs, bundling your own cacerts with request is really, really annoying… especially the fact that you do not seem to use the system ca store first and this is not documented anywhere.

update

in situations, where you’re using a library and have no control over the ca-bundle location you could also explicitly set the ca-bundle location to be your host-wide ca-bundle:

REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt python -c "import requests; requests.get('https://somesite.com')";

ibre5041's user avatar

ibre5041

4,8081 gold badge19 silver badges34 bronze badges

answered Mar 4, 2016 at 8:38

tintin's user avatar

tintintintin

3,15630 silver badges34 bronze badges

3

As pointed out by others, this problem «is caused by an untrusted SSL certificate». My answer is based on the top-rated answer and this answer.

You can test the certificate using curl:

curl -vvI https://example.com

If an error returns, you have 3 options:

  1. For a quick fix, you could just not verify the certificate:
requests.get('https://example.com', verify=False)
  1. Pass the path to the CA_BUNDLE file or directory with certificates of trusted CAs:
requests.get('https://example.com', verify='/path/to/certfile')
  1. If you have access to, fix the web server certificate.

My problem was because I was using only my site’s certificate, not the intermediate (a.k.a. chain) certificate.

If you are using Let’s Encrypt, you should use the fullchain.pem file, not cert.pem.

answered Nov 20, 2020 at 15:09

Derik Oliveira's user avatar

1

If you want to remove the warnings, use the code below.

import urllib3

urllib3.disable_warnings()

and verify=False with request.get() or post() method

import requests

requests.get(url, verify=False)

imxitiz's user avatar

imxitiz

3,9102 gold badges9 silver badges33 bronze badges

answered Nov 3, 2015 at 7:42

AniketGole's user avatar

AniketGoleAniketGole

8192 gold badges11 silver badges22 bronze badges

0

I face the same problem using gspread and these commands works for me:

sudo pip uninstall -y certifi
sudo pip install certifi==2015.04.28

answered Feb 16, 2016 at 4:53

user941581's user avatar

user941581user941581

3792 silver badges4 bronze badges

3

I have found an specific approach for solving a similar issue. The idea is pointing the cacert file stored at the system and used by another ssl based applications.

In Debian (I’m not sure if same in other distributions) the certificate files (.pem) are stored at /etc/ssl/certs/ So, this is the code that work for me:

import requests
verify='/etc/ssl/certs/cacert.org.pem'
response = requests.get('https://lists.cacert.org', verify=verify)

For guessing what pem file choose, I have browse to the url and check which Certificate Authority (CA) has generated the certificate.

EDIT: if you cannot edit the code (because you are running a third app) you can try to add the pem certificate directly into /usr/local/lib/python2.7/dist-packages/requests/cacert.pem (e.g. copying it to the end of the file).

NSNoob's user avatar

NSNoob

5,5286 gold badges41 silver badges54 bronze badges

answered Apr 18, 2013 at 14:29

slamora's user avatar

slamoraslamora

69710 silver badges17 bronze badges

2

If you don’t bother about certificate just use verify=False.

import requests

url = "Write your url here"

returnResponse = requests.get(url, verify=False)

BoJack Horseman's user avatar

answered May 21, 2015 at 12:01

yogesh prasad's user avatar

After hours of debugging I could only get this to work using the following packages:

requests[security]==2.7.0  # not 2.18.1
cryptography==1.9  # not 2.0

using OpenSSL 1.0.2g 1 Mar 2016

Without these packages verify=False was not working.

I hope this helps someone.

answered Jul 20, 2017 at 20:10

michael's user avatar

michaelmichael

6529 silver badges12 bronze badges

I ran into the same issue. Turns out I hadn’t installed the intermediate certificate on my server (just append it to the bottom of your certificate as seen below).

https://www.digicert.com/ssl-support/pem-ssl-creation.htm

Make sure you have the ca-certificates package installed:

sudo apt-get install ca-certificates

Updating the time may also resolve this:

sudo apt-get install ntpdate
sudo ntpdate -u ntp.ubuntu.com

If you’re using a self-signed certificate, you’ll probably have to add it to your system manually.

answered Jun 13, 2014 at 19:19

Marius Craciunoiu's user avatar

2

If the request calls are buried somewhere deep in the code and you do not want to install the server certificate, then, just for debug purposes only, it’s possible to monkeypatch requests:

import requests.api
import warnings


def requestspatch(method, url, **kwargs):
    kwargs['verify'] = False
    return _origcall(method, url, **kwargs)

_origcall = requests.api.request
requests.api.request = requestspatch
warnings.warn('Patched requests: SSL verification disabled!')

Never use in production!

answered Aug 29, 2017 at 6:06

xmedeko's user avatar

xmedekoxmedeko

7,2166 gold badges53 silver badges84 bronze badges

Too late to the party I guess but I wanted to paste the fix for fellow wanderers like myself! So the following worked out for me on Python 3.7.x

Type the following in your terminal

pip install --upgrade certifi      # hold your breath..

Try running your script/requests again and see if it works (I’m sure it won’t be fixed yet!). If it didn’t work then try running the following command in the terminal directly

open /Applications/Python 3.6/Install Certificates.command  # please replace 3.6 here with your suitable python version

answered Nov 29, 2018 at 11:32

d-coder's user avatar

d-coderd-coder

12.5k4 gold badges25 silver badges36 bronze badges

This is similar to @rafael-almeida ‘s answer, but I want to point out that as of requests 2.11+, there are not 3 values that verify can take, there are actually 4:

  • True: validates against requests’s internal trusted CAs.
  • False: bypasses certificate validation completely. (Not recommended)
  • Path to a CA_BUNDLE file. requests will use this to validate the server’s certificates.
  • Path to a directory containing public certificate files. requests will use this to validate the server’s certificates.

The rest of my answer is about #4, how to use a directory containing certificates to validate:

Obtain the public certificates needed and place them in a directory.

Strictly speaking, you probably «should» use an out-of-band method of obtaining the certificates, but you could also just download them using any browser.

If the server uses a certificate chain, be sure to obtain every single certificate in the chain.

According to the requests documentation, the directory containing the certificates must first be processed with the «rehash» utility (openssl rehash).

(This requires openssl 1.1.1+, and not all Windows openssl implementations support rehash. If openssl rehash won’t work for you, you could try running the rehash ruby script at https://github.com/ruby/openssl/blob/master/sample/c_rehash.rb , though I haven’t tried this. )

I had some trouble with getting requests to recognize my certificates, but after I used the openssl x509 -outform PEM command to convert the certs to Base64 .pem format, everything worked perfectly.

You can also just do lazy rehashing:

try:
    # As long as the certificates in the certs directory are in the OS's certificate store, `verify=True` is fine.
    return requests.get(url, auth=auth, verify=True)
except requests.exceptions.SSLError:
    subprocess.run(f"openssl rehash -compat -v my_certs_dir", shell=True, check=True)
    return requests.get(url, auth=auth, verify="my_certs_dir")

answered Oct 10, 2019 at 20:44

cowlinator's user avatar

cowlinatorcowlinator

6,9726 gold badges41 silver badges60 bronze badges

I fought this problem for HOURS.

I tried to update requests. Then I updated certifi. I pointed verify to certifi.where() (The code does this by default anyways). Nothing worked.

Finally I updated my version of python to python 2.7.11. I was on Python 2.7.5 which had some incompatibilities with the way that the certificates are verified. Once I updated Python (and a handful of other dependencies) it started working.

answered May 6, 2016 at 21:13

ajon's user avatar

ajonajon

7,79811 gold badges48 silver badges86 bronze badges

2

Some servers do not have the trusted root cert for Letsencrypt.

For example, assume the server pointed by the url below is protected by a Letsencrypt SSL.

requests.post(url, json=data)

This request can fail with [SSL: CERTIFICATE_VERIFY_FAILED] because the requesting server does not have the root cert for Letsencrypt.

When this happens download the active self-signed ‘pem’ cert from the link below.

https://letsencrypt.org/certificates/. (Active ISRG Root X1 as of this writing)

Now, use that in the verify parameter as follows.

requests.post(url, json=data, verify='path-to/isrgrootx1.pem')

answered Oct 20, 2021 at 20:09

Rajitha Wijayaratne's user avatar

There is currently an issue in the requests module causing this error, present in v2.6.2 to v2.12.4 (ATOW): https://github.com/kennethreitz/requests/issues/2573

Workaround for this issue is adding the following line: requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS'

answered Jan 15, 2017 at 20:10

Peter's user avatar

PeterPeter

1,26211 silver badges11 bronze badges

1

I think you have several ways for fix this issue. I mentioned 5 ways below:

  1. You can define context for each request and pass the context on each request for use it like below:
import certifi
import ssl
import urllib
context = ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com', context=context)
  1. OR Set certificate file in environment.
import os
import certifi
import urllib
os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
os.environ["SSL_CERT_FILE"] = certifi.where()
result = urllib.request.urlopen('https://www.example.com')

  1. OR Replace create default https context method:
import certifi
import ssl
ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com')
  1. OR If you use Linux machine, generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it.
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
  1. OR If you use Mac machine, generating fresh certificates
$ cd "/Applications/$(python3 --version | awk '{print $2}'| awk  -F. '{print "Python " $1"."$2}')"
$ sudo "./Install Certificates.command"

answered Aug 7, 2022 at 19:07

Ali ZahediGol's user avatar

Ali ZahediGolAli ZahediGol

8162 gold badges9 silver badges20 bronze badges

1

As mentioned by @Rafael Almeida, the problem you are having is caused by an untrusted SSL certificate. In my case, the SSL certificate was untrusted by my server. To get around this without compromising security, I downloaded the certificate, and installed it on the server (by simply double clicking on the .crt file and then Install Certificate…).

answered Jun 14, 2017 at 11:07

Michael's user avatar

MichaelMichael

1991 silver badge9 bronze badges

In my case the reason was fairly trivial.

I had known that the SSL verification had worked until a few days earlier, and was infact working on a different machine.

My next step was to compare the certificate contents and size between the machine on which verification was working, and the one on which it was not.

This quickly led to me determining that the Certificate on the ‘incorrectly’ working machine was not good, and once I replaced it with the ‘good’ cert, everything was fine.

answered Apr 30, 2019 at 11:59

mastDrinkNimbuPani's user avatar

1

It is not feasible to add options if requests is being called from another package. In that case adding certificates to the cacert bundle is the straight path, e.g. I had to add «StartCom Class 1 Primary Intermediate Server CA», for which I downloaded the root cert into StartComClass1.pem. given my virtualenv is named caldav, I added the certificate with:

cat StartComClass1.pem >> .virtualenvs/caldav/lib/python2.7/site-packages/pip/_vendor/requests/cacert.pem
cat temp/StartComClass1.pem >> .virtualenvs/caldav/lib/python2.7/site-packages/requests/cacert.pem

one of those might be enough, I did not check

answered Aug 3, 2015 at 17:29

rhoerbe's user avatar

rhoerberhoerbe

4631 gold badge4 silver badges17 bronze badges

I was having a similar or the same certification validation problem. I read that OpenSSL versions less than 1.0.2, which requests depends upon sometimes have trouble validating strong certificates (see here). CentOS 7 seems to use 1.0.1e which seems to have the problem.

I wasn’t sure how to get around this problem on CentOS, so I decided to allow weaker 1024bit CA certificates.

import certifi # This should be already installed as a dependency of 'requests'
requests.get("https://example.com", verify=certifi.old_where())

answered Jul 11, 2017 at 13:26

Tim Ludwinski's user avatar

1

I had to upgrade from Python 3.4.0 to 3.4.6

pyenv virtualenv 3.4.6 myvenv
pyenv activate myvenv
pip install -r requirements.txt

answered Mar 7, 2018 at 23:22

Paul's user avatar

PaulPaul

2,3992 gold badges24 silver badges29 bronze badges

I found this answer which fixed it:

import ssl
import certifi
import urllib.request

url = "https://www.google.com/"
html = urllib.request.urlopen(url, context=ssl.create_default_context(cafile=certifi.where()))

I have no idea what it does, though.

answered Oct 2, 2021 at 6:41

Urban P.'s user avatar

Urban P.Urban P.

1191 silver badge9 bronze badges

When it says verify takes ‘path to certificate’, I pointed it to the issuer certificate so that it can use that to verify the url’s certificate. curl and wget were fine with that certificate. But not python requests.

I had to create a certificate chain with all the certificates from end (leaf?) to root for python requests to process it fine. And the chain works fine with cURL and Wget too naturally.

Hope it helps someone and saves few hours.

answered Oct 27, 2022 at 11:54

Neutral's user avatar

Everything here failed for me. My company uses Z-Scaler and recently activated some extra protection that resulted in python packages such as plantweb failing with this error:

SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

The problem was that Z-Scaler is using its own certificate, so I needed to get that file from IT and tell python to use it.

export REQUESTS_CA_BUNDLE="/path/to/zscaler.crt"
plantweb --format png some_file.puml

However, this introduced another problem: Some other code I had started failing with the same SSL error because it needed to use the standard certificates from python’s certifi package.

The final solution that worked for me was to combine the normal certificates and the Z-Scaler one into a single certificate and tell python to use that:

pip3 install certifi
normal_python_cert="$(python3 -m certifi)"
cat $normal_python_cert /path/to/zscaler.crt > combined.crt
export REQUESTS_CA_BUNDLE=/path/to/combined.crt
plantweb --format png some_file.puml

Super useful reference:
https://help.zscaler.com/zia/adding-custom-certificate-application-specific-trust-store#python

answered Mar 1 at 23:09

retsigam's user avatar

retsigamretsigam

5401 gold badge5 silver badges13 bronze badges

This is just another way you can try to solve the issue.

If you put «www.example.com», requests shouts at you. If you put «https://www.example.com», you get this error. So if you DO NOT NEED https, you can avoid the error by changing «https» to «http». eg. «http://www.example.com»

WARNING: Not using HTTPS is generally not a good idea. See Why HTTPS for Everything? Why HTTPS matters

E net4's user avatar

E net4

27.2k13 gold badges96 silver badges135 bronze badges

answered Sep 2, 2020 at 8:09

Default's user avatar

DefaultDefault

6749 silver badges18 bronze badges

SSLError occurs when you request a remote URL that does not provide a trusted SSL certificate. The easiest way to fix this issue is to disable SSL verification for that particular web address by passing in verify=False as an argument to the method calls. Just make sure you are not sending any sensitive data in your request.

Here is some sample code that disables SSL verification:

import requests

response = requests.get("https://example.com/", verify=False)

You can optionally provide a custom certificate for the website to fix this error as well. Here is some sample code for providing a custom .pem certificate file to requests:

import requests

custom_certificate_path = "./certificates/custom-certificate.pem"
response = requests.get("https://example.com/", verify=custom_certificate_path)

Related Requests web scraping questions:

Проверка SSL-сертификатов и SSL на стороне клиента.

Содержание:

  • Проверка SSL сертификата;
  • Сертификаты на стороне клиента;
  • Доверенные сертификаты CA.

Проверка SSL сертификата.

Библиотека requests проверяет SSL-сертификаты для HTTPS-запросов, как это делает веб-браузер. По умолчанию проверка SSL включена, и запросы выдадут SSLError, если она не сможет проверить сертификат:

>>> import requests
>>> requests.get('https://requestb.in')
# requests.exceptions.SSLError: hostname 'requestb.in' doesn't match ...

На этом домене нет установленного SSL, поэтому запрос создает исключение. Запрос к GitHub проходит без каких либо ошибок:

>>> requests.get('https://github.com')
# <Response [200]>

Можно передать аргументу verify путь к файлу CA_BUNDLE или каталогу с доверенными сертификатами CA:

# указание доверенных сертификатов в запросе
>>> requests.get('https://github.com', verify='/path/to/certfile')

# указание доверенных сертификатов для сессии
>>> sess = requests.Session()
>>> sess.verify = '/path/to/certfile'

Примечание. Если для параметра verify задан путь к каталогу, то этот каталог должен быть обработан с помощью утилиты c_rehash, поставляемой с OpenSSL.

Список доверенных CA также можно указать с помощью переменных сред REQUESTS_CA_BUNDLE. Если REQUESTS_CA_BUNDLE не установлена, то CURL_CA_BUNDLE будет использоваться в качестве запасного варианта.

Запросы также могут игнорировать проверку SSL-сертификата, если для параметра verify задано значение False:

>>> requests.get('https://kennethreitz.org', verify=False)
# <Response [200]>

Обратите внимание, что если аргумент verify=False, то запросы будут принимать любой TLS-сертификат, представленный сервером, и будут игнорировать несоответствия имен хостов и/или просроченные сертификаты, что сделает приложение уязвимым для атак man-in-the-middle (MitM). Установка значения verify в False может быть полезна во время локальной разработки или тестирования.

По умолчанию для параметра verify установлено значение True. Опция verify применяется только к сертификатам хоста.

Сертификаты на стороне клиента.

Также можно указать локальный сертификат для использования в качестве сертификата на стороне клиента, как один файл (содержащий закрытый ключ и сертификат) или как кортеж путей к обоим файлам:

>>> requests.get('https://kennethreitz.org', 
...                cert=('/path/client.cert', '/path/client.key'))
# <Response [200]>

# для сессии
s = requests.Session()
s.cert = '/path/client.cert'

Если укать неправильный путь или неверный сертификат, то получим SSLError:

>>> requests.get('https://kennethreitz.org', cert='/wrong_path/client.pem')
# SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL...

Предупреждение. Закрытый ключ к локальному сертификату должен быть незашифрованным. В настоящее время библиотека requests не поддерживают использование зашифрованных ключей.

Доверенные сертификаты CA

Библиотека requests используют сертификаты из пакета certifi. Это позволяет пользователям обновлять доверенные сертификаты без изменения версии запросов.

До версии requests-2.16 модуль объединял набор доверенных корневых центров сертификации, полученных из хранилища доверенных сертификатов Mozilla. Сертификаты обновлялись только один раз для каждой версии запросов. Когда не был установлен certifi , это приводило к чрезвычайно устаревшим пакетам сертификатов при использовании значительно более старых версий запросов.

В целях безопасности команда разработчиков библиотеки requests рекомендует почаще обновлять сертификаты!

This document covers some of Requests more advanced features.

Session Objects¶

The Session object allows you to persist certain parameters across
requests. It also persists cookies across all requests made from the
Session instance, and will use urllib3’s connection pooling. So if
you’re making several requests to the same host, the underlying TCP
connection will be reused, which can result in a significant performance
increase (see HTTP persistent connection).

A Session object has all the methods of the main Requests API.

Let’s persist some cookies across requests:

s = requests.Session()

s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('https://httpbin.org/cookies')

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

Sessions can also be used to provide default data to the request methods. This
is done by providing data to the properties on a Session object:

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

# both 'x-test' and 'x-test2' are sent
s.get('https://httpbin.org/headers', headers={'x-test2': 'true'})

Any dictionaries that you pass to a request method will be merged with the
session-level values that are set. The method-level parameters override session
parameters.

Note, however, that method-level parameters will not be persisted across
requests, even if using a session. This example will only send the cookies
with the first request, but not the second:

s = requests.Session()

r = s.get('https://httpbin.org/cookies', cookies={'from-my': 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'

r = s.get('https://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'

If you want to manually add cookies to your session, use the
Cookie utility functions to manipulate
Session.cookies.

Sessions can also be used as context managers:

with requests.Session() as s:
    s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')

This will make sure the session is closed as soon as the with block is
exited, even if unhandled exceptions occurred.

Remove a Value From a Dict Parameter

Sometimes you’ll want to omit session-level keys from a dict parameter. To
do this, you simply set that key’s value to None in the method-level
parameter. It will automatically be omitted.

All values that are contained within a session are directly available to you.
See the Session API Docs to learn more.

Request and Response Objects¶

Whenever a call is made to requests.get() and friends, you are doing two
major things. First, you are constructing a Request object which will be
sent off to a server to request or query some resource. Second, a Response
object is generated once Requests gets a response back from the server.
The Response object contains all of the information returned by the server and
also contains the Request object you created originally. Here is a simple
request to get some very important information from Wikipedia’s servers:

>>> r = requests.get('https://en.wikipedia.org/wiki/Monty_Python')

If we want to access the headers the server sent back to us, we do this:

>>> r.headers
{'content-length': '56170', 'x-content-type-options': 'nosniff', 'x-cache':
'HIT from cp1006.eqiad.wmnet, MISS from cp1010.eqiad.wmnet', 'content-encoding':
'gzip', 'age': '3080', 'content-language': 'en', 'vary': 'Accept-Encoding,Cookie',
'server': 'Apache', 'last-modified': 'Wed, 13 Jun 2012 01:33:50 GMT',
'connection': 'close', 'cache-control': 'private, s-maxage=0, max-age=0,
must-revalidate', 'date': 'Thu, 14 Jun 2012 12:59:39 GMT', 'content-type':
'text/html; charset=UTF-8', 'x-cache-lookup': 'HIT from cp1006.eqiad.wmnet:3128,
MISS from cp1010.eqiad.wmnet:80'}

However, if we want to get the headers we sent the server, we simply access the
request, and then the request’s headers:

>>> r.request.headers
{'Accept-Encoding': 'identity, deflate, compress, gzip',
'Accept': '*/*', 'User-Agent': 'python-requests/1.2.0'}

Prepared Requests¶

Whenever you receive a Response object
from an API call or a Session call, the request attribute is actually the
PreparedRequest that was used. In some cases you may wish to do some extra
work to the body or headers (or anything else really) before sending a
request. The simple recipe for this is the following:

from requests import Request, Session

s = Session()

req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()

# do something with prepped.body
prepped.body = 'No, I want exactly this as the body.'

# do something with prepped.headers
del prepped.headers['Content-Type']

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

Since you are not doing anything special with the Request object, you
prepare it immediately and modify the PreparedRequest object. You then
send that with the other parameters you would have sent to requests.* or
Session.*.

However, the above code will lose some of the advantages of having a Requests
Session object. In particular,
Session-level state such as cookies will
not get applied to your request. To get a
PreparedRequest with that state
applied, replace the call to Request.prepare() with a call to
Session.prepare_request(), like this:

from requests import Request, Session

s = Session()
req = Request('GET',  url, data=data, headers=headers)

prepped = s.prepare_request(req)

# do something with prepped.body
prepped.body = 'Seriously, send exactly these bytes.'

# do something with prepped.headers
prepped.headers['Keep-Dead'] = 'parrot'

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

When you are using the prepared request flow, keep in mind that it does not take into account the environment.
This can cause problems if you are using environment variables to change the behaviour of requests.
For example: Self-signed SSL certificates specified in REQUESTS_CA_BUNDLE will not be taken into account.
As a result an SSL: CERTIFICATE_VERIFY_FAILED is thrown.
You can get around this behaviour by explicitly merging the environment settings into your session:

from requests import Request, Session

s = Session()
req = Request('GET', url)

prepped = s.prepare_request(req)

# Merge environment settings into session
settings = s.merge_environment_settings(prepped.url, {}, None, None, None)
resp = s.send(prepped, **settings)

print(resp.status_code)

SSL Cert Verification¶

Requests verifies SSL certificates for HTTPS requests, just like a web browser.
By default, SSL verification is enabled, and Requests will throw a SSLError if
it’s unable to verify the certificate:

>>> requests.get('https://requestb.in')
requests.exceptions.SSLError: hostname 'requestb.in' doesn't match either of '*.herokuapp.com', 'herokuapp.com'

I don’t have SSL setup on this domain, so it throws an exception. Excellent. GitHub does though:

>>> requests.get('https://github.com')
<Response [200]>

You can pass verify the path to a CA_BUNDLE file or directory with certificates of trusted CAs:

>>> requests.get('https://github.com', verify='/path/to/certfile')

or persistent:

s = requests.Session()
s.verify = '/path/to/certfile'

Note

If verify is set to a path to a directory, the directory must have been processed using
the c_rehash utility supplied with OpenSSL.

This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.
If REQUESTS_CA_BUNDLE is not set, CURL_CA_BUNDLE will be used as fallback.

Requests can also ignore verifying the SSL certificate if you set verify to False:

>>> requests.get('https://kennethreitz.org', verify=False)
<Response [200]>

Note that when verify is set to False, requests will accept any TLS
certificate presented by the server, and will ignore hostname mismatches
and/or expired certificates, which will make your application vulnerable to
man-in-the-middle (MitM) attacks. Setting verify to False may be useful
during local development or testing.

By default, verify is set to True. Option verify only applies to host certs.

Client Side Certificates¶

You can also specify a local cert to use as client side certificate, as a single
file (containing the private key and the certificate) or as a tuple of both
files’ paths:

>>> requests.get('https://kennethreitz.org', cert=('/path/client.cert', '/path/client.key'))
<Response [200]>

or persistent:

s = requests.Session()
s.cert = '/path/client.cert'

If you specify a wrong path or an invalid cert, you’ll get a SSLError:

>>> requests.get('https://kennethreitz.org', cert='/wrong_path/client.pem')
SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

Warning

The private key to your local certificate must be unencrypted.
Currently, Requests does not support using encrypted keys.

CA Certificates¶

Requests uses certificates from the package certifi. This allows for users
to update their trusted certificates without changing the version of Requests.

Before version 2.16, Requests bundled a set of root CAs that it trusted,
sourced from the Mozilla trust store. The certificates were only updated
once for each Requests version. When certifi was not installed, this led to
extremely out-of-date certificate bundles when using significantly older
versions of Requests.

For the sake of security we recommend upgrading certifi frequently!

Body Content Workflow¶

By default, when you make a request, the body of the response is downloaded
immediately. You can override this behaviour and defer downloading the response
body until you access the Response.content
attribute with the stream parameter:

tarball_url = 'https://github.com/psf/requests/tarball/main'
r = requests.get(tarball_url, stream=True)

At this point only the response headers have been downloaded and the connection
remains open, hence allowing us to make content retrieval conditional:

if int(r.headers['content-length']) < TOO_LONG:
  content = r.content
  ...

You can further control the workflow by use of the Response.iter_content()
and Response.iter_lines() methods.
Alternatively, you can read the undecoded body from the underlying
urllib3 urllib3.HTTPResponse at
Response.raw.

If you set stream to True when making a request, Requests cannot
release the connection back to the pool unless you consume all the data or call
Response.close. This can lead to
inefficiency with connections. If you find yourself partially reading request
bodies (or not reading them at all) while using stream=True, you should
make the request within a with statement to ensure it’s always closed:

with requests.get('https://httpbin.org/get', stream=True) as r:
    # Do things with the response here.

Keep-Alive¶

Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session!
Any requests that you make within a session will automatically reuse the appropriate
connection!

Note that connections are only released back to the pool for reuse once all body
data has been read; be sure to either set stream to False or read the
content property of the Response object.

Streaming Uploads¶

Requests supports streaming uploads, which allow you to send large streams or
files without reading them into memory. To stream and upload, simply provide a
file-like object for your body:

with open('massive-body', 'rb') as f:
    requests.post('http://some.url/streamed', data=f)

Warning

It is strongly recommended that you open files in binary
mode
. This is because Requests may attempt to provide
the Content-Length header for you, and if it does this value
will be set to the number of bytes in the file. Errors may occur
if you open the file in text mode.

Chunk-Encoded Requests¶

Requests also supports Chunked transfer encoding for outgoing and incoming requests.
To send a chunk-encoded request, simply provide a generator (or any iterator without
a length) for your body:

def gen():
    yield 'hi'
    yield 'there'

requests.post('http://some.url/chunked', data=gen())

For chunked encoded responses, it’s best to iterate over the data using
Response.iter_content(). In
an ideal situation you’ll have set stream=True on the request, in which
case you can iterate chunk-by-chunk by calling iter_content with a chunk_size
parameter of None. If you want to set a maximum size of the chunk,
you can set a chunk_size parameter to any integer.

POST Multiple Multipart-Encoded Files¶

You can send multiple files in one request. For example, suppose you want to
upload image files to an HTML form with a multiple file field ‘images’:

<input type="file" name="images" multiple="true" required="true"/>

To do that, just set files to a list of tuples of (form_field_name, file_info):

>>> url = 'https://httpbin.org/post'
>>> multiple_files = [
...     ('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
...     ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]
>>> r = requests.post(url, files=multiple_files)
>>> r.text
{
  ...
  'files': {'images': 'data:image/png;base64,iVBORw ....'}
  'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7aa0b3a',
  ...
}

Warning

It is strongly recommended that you open files in binary
mode
. This is because Requests may attempt to provide
the Content-Length header for you, and if it does this value
will be set to the number of bytes in the file. Errors may occur
if you open the file in text mode.

Event Hooks¶

Requests has a hook system that you can use to manipulate portions of
the request process, or signal event handling.

Available hooks:

response:

The response generated from a Request.

You can assign a hook function on a per-request basis by passing a
{hook_name: callback_function} dictionary to the hooks request
parameter:

hooks={'response': print_url}

That callback_function will receive a chunk of data as its first
argument.

def print_url(r, *args, **kwargs):
    print(r.url)

Your callback function must handle its own exceptions. Any unhandled exception won’t be passed silently and thus should be handled by the code calling Requests.

If the callback function returns a value, it is assumed that it is to
replace the data that was passed in. If the function doesn’t return
anything, nothing else is affected.

def record_hook(r, *args, **kwargs):
    r.hook_called = True
    return r

Let’s print some request method arguments at runtime:

>>> requests.get('https://httpbin.org/', hooks={'response': print_url})
https://httpbin.org/
<Response [200]>

You can add multiple hooks to a single request. Let’s call two hooks at once:

>>> r = requests.get('https://httpbin.org/', hooks={'response': [print_url, record_hook]})
>>> r.hook_called
True

You can also add hooks to a Session instance. Any hooks you add will then
be called on every request made to the session. For example:

>>> s = requests.Session()
>>> s.hooks['response'].append(print_url)
>>> s.get('https://httpbin.org/')
 https://httpbin.org/
 <Response [200]>

A Session can have multiple hooks, which will be called in the order
they are added.

Custom Authentication¶

Requests allows you to specify your own authentication mechanism.

Any callable which is passed as the auth argument to a request method will
have the opportunity to modify the request before it is dispatched.

Authentication implementations are subclasses of AuthBase,
and are easy to define. Requests provides two common authentication scheme
implementations in requests.auth: HTTPBasicAuth and
HTTPDigestAuth.

Let’s pretend that we have a web service that will only respond if the
X-Pizza header is set to a password value. Unlikely, but just go with it.

from requests.auth import AuthBase

class PizzaAuth(AuthBase):
    """Attaches HTTP Pizza Authentication to the given Request object."""
    def __init__(self, username):
        # setup any auth-related data here
        self.username = username

    def __call__(self, r):
        # modify and return the request
        r.headers['X-Pizza'] = self.username
        return r

Then, we can make a request using our Pizza Auth:

>>> requests.get('http://pizzabin.org/admin', auth=PizzaAuth('kenneth'))
<Response [200]>

Streaming Requests¶

With Response.iter_lines() you can easily
iterate over streaming APIs such as the Twitter Streaming
API. Simply
set stream to True and iterate over the response with
iter_lines:

import json
import requests

r = requests.get('https://httpbin.org/stream/20', stream=True)

for line in r.iter_lines():

    # filter out keep-alive new lines
    if line:
        decoded_line = line.decode('utf-8')
        print(json.loads(decoded_line))

When using decode_unicode=True with
Response.iter_lines() or
Response.iter_content(), you’ll want
to provide a fallback encoding in the event the server doesn’t provide one:

r = requests.get('https://httpbin.org/stream/20', stream=True)

if r.encoding is None:
    r.encoding = 'utf-8'

for line in r.iter_lines(decode_unicode=True):
    if line:
        print(json.loads(line))

Warning

iter_lines is not reentrant safe.
Calling this method multiple times causes some of the received data
being lost. In case you need to call it from multiple places, use
the resulting iterator object instead:

lines = r.iter_lines()
# Save the first line for later or just skip it

first_line = next(lines)

for line in lines:
    print(line)

Proxies¶

If you need to use a proxy, you can configure individual requests with the
proxies argument to any request method:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

requests.get('http://example.org', proxies=proxies)

Alternatively you can configure it once for an entire
Session:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.update(proxies)

session.get('http://example.org')

Warning

Setting session.proxies may behave differently than expected.
Values provided will be overwritten by environmental proxies
(those returned by urllib.request.getproxies).
To ensure the use of proxies in the presence of environmental proxies,
explicitly specify the proxies argument on all individual requests as
initially explained above.

See #2018 for details.

When the proxies configuration is not overridden per request as shown above,
Requests relies on the proxy configuration defined by standard
environment variables http_proxy, https_proxy, no_proxy,
and all_proxy. Uppercase variants of these variables are also supported.
You can therefore set them to configure Requests (only set the ones relevant
to your needs):

$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"
$ export ALL_PROXY="socks5://10.10.1.10:3434"

$ python
>>> import requests
>>> requests.get('http://example.org')

To use HTTP Basic Auth with your proxy, use the http://user:password@host/
syntax in any of the above configuration entries:

$ export HTTPS_PROXY="http://user:pass@10.10.1.10:1080"

$ python
>>> proxies = {'http': 'http://user:pass@10.10.1.10:3128/'}

Warning

Storing sensitive username and password information in an
environment variable or a version-controlled file is a security risk and is
highly discouraged.

To give a proxy for a specific scheme and host, use the
scheme://hostname form for the key. This will match for
any request to the given scheme and exact hostname.

proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

Note that proxy URLs must include the scheme.

Finally, note that using a proxy for https connections typically requires your
local machine to trust the proxy’s root certificate. By default the list of
certificates trusted by Requests can be found with:

from requests.utils import DEFAULT_CA_BUNDLE_PATH
print(DEFAULT_CA_BUNDLE_PATH)

You override this default certificate bundle by setting the REQUESTS_CA_BUNDLE
(or CURL_CA_BUNDLE) environment variable to another file path:

$ export REQUESTS_CA_BUNDLE="/usr/local/myproxy_info/cacert.pem"
$ export https_proxy="http://10.10.1.10:1080"

$ python
>>> import requests
>>> requests.get('https://example.org')

SOCKS¶

New in version 2.10.0.

In addition to basic HTTP proxies, Requests also supports proxies using the
SOCKS protocol. This is an optional feature that requires that additional
third-party libraries be installed before use.

You can get the dependencies for this feature from pip:

$ python -m pip install requests[socks]

Once you’ve installed those dependencies, using a SOCKS proxy is just as easy
as using a HTTP one:

proxies = {
    'http': 'socks5://user:pass@host:port',
    'https': 'socks5://user:pass@host:port'
}

Using the scheme socks5 causes the DNS resolution to happen on the client, rather than on the proxy server. This is in line with curl, which uses the scheme to decide whether to do the DNS resolution on the client or proxy. If you want to resolve the domains on the proxy server, use socks5h as the scheme.

Compliance¶

Requests is intended to be compliant with all relevant specifications and
RFCs where that compliance will not cause difficulties for users. This
attention to the specification can lead to some behaviour that may seem
unusual to those not familiar with the relevant specification.

Encodings¶

When you receive a response, Requests makes a guess at the encoding to
use for decoding the response when you access the Response.text attribute. Requests will first check for an
encoding in the HTTP header, and if none is present, will use
charset_normalizer
or chardet to attempt to
guess the encoding.

If chardet is installed, requests uses it, however for python3
chardet is no longer a mandatory dependency. The chardet
library is an LGPL-licenced dependency and some users of requests
cannot depend on mandatory LGPL-licensed dependencies.

When you install requests without specifying [use_chardet_on_py3] extra,
and chardet is not already installed, requests uses charset-normalizer
(MIT-licensed) to guess the encoding.

The only time Requests will not guess the encoding is if no explicit charset
is present in the HTTP headers and the Content-Type
header contains text. In this situation, RFC 2616 specifies
that the default charset must be ISO-8859-1. Requests follows the
specification in this case. If you require a different encoding, you can
manually set the Response.encoding
property, or use the raw Response.content.

HTTP Verbs¶

Requests provides access to almost the full range of HTTP verbs: GET, OPTIONS,
HEAD, POST, PUT, PATCH and DELETE. The following provides detailed examples of
using these various verbs in Requests, using the GitHub API.

We will begin with the verb most commonly used: GET. HTTP GET is an idempotent
method that returns a resource from a given URL. As a result, it is the verb
you ought to use when attempting to retrieve data from a web location. An
example usage would be attempting to get information about a specific commit
from GitHub. Suppose we wanted commit a050faf on Requests. We would get it
like so:

>>> import requests
>>> r = requests.get('https://api.github.com/repos/psf/requests/git/commits/a050faf084662f3a352dd1a941f2c7c9f886d4ad')

We should confirm that GitHub responded correctly. If it has, we want to work
out what type of content it is. Do this like so:

>>> if r.status_code == requests.codes.ok:
...     print(r.headers['content-type'])
...
application/json; charset=utf-8

So, GitHub returns JSON. That’s great, we can use the r.json method to parse it into Python objects.

>>> commit_data = r.json()

>>> print(commit_data.keys())
['committer', 'author', 'url', 'tree', 'sha', 'parents', 'message']

>>> print(commit_data['committer'])
{'date': '2012-05-10T11:10:50-07:00', 'email': 'me@kennethreitz.com', 'name': 'Kenneth Reitz'}

>>> print(commit_data['message'])
makin' history

So far, so simple. Well, let’s investigate the GitHub API a little bit. Now,
we could look at the documentation, but we might have a little more fun if we
use Requests instead. We can take advantage of the Requests OPTIONS verb to
see what kinds of HTTP methods are supported on the url we just used.

>>> verbs = requests.options(r.url)
>>> verbs.status_code
500

Uh, what? That’s unhelpful! Turns out GitHub, like many API providers, don’t
actually implement the OPTIONS method. This is an annoying oversight, but it’s
OK, we can just use the boring documentation. If GitHub had correctly
implemented OPTIONS, however, they should return the allowed methods in the
headers, e.g.

>>> verbs = requests.options('http://a-good-website.com/api/cats')
>>> print(verbs.headers['allow'])
GET,HEAD,POST,OPTIONS

Turning to the documentation, we see that the only other method allowed for
commits is POST, which creates a new commit. As we’re using the Requests repo,
we should probably avoid making ham-handed POSTS to it. Instead, let’s play
with the Issues feature of GitHub.

This documentation was added in response to
Issue #482. Given that
this issue already exists, we will use it as an example. Let’s start by getting it.

>>> r = requests.get('https://api.github.com/repos/psf/requests/issues/482')
>>> r.status_code
200

>>> issue = json.loads(r.text)

>>> print(issue['title'])
Feature any http verb in docs

>>> print(issue['comments'])
3

Cool, we have three comments. Let’s take a look at the last of them.

>>> r = requests.get(r.url + '/comments')
>>> r.status_code
200

>>> comments = r.json()

>>> print(comments[0].keys())
['body', 'url', 'created_at', 'updated_at', 'user', 'id']

>>> print(comments[2]['body'])
Probably in the "advanced" section

Well, that seems like a silly place. Let’s post a comment telling the poster
that he’s silly. Who is the poster, anyway?

>>> print(comments[2]['user']['login'])
kennethreitz

OK, so let’s tell this Kenneth guy that we think this example should go in the
quickstart guide instead. According to the GitHub API doc, the way to do this
is to POST to the thread. Let’s do it.

>>> body = json.dumps({u"body": u"Sounds great! I'll get right on it!"})
>>> url = u"https://api.github.com/repos/psf/requests/issues/482/comments"

>>> r = requests.post(url=url, data=body)
>>> r.status_code
404

Huh, that’s weird. We probably need to authenticate. That’ll be a pain, right?
Wrong. Requests makes it easy to use many forms of authentication, including
the very common Basic Auth.

>>> from requests.auth import HTTPBasicAuth
>>> auth = HTTPBasicAuth('fake@example.com', 'not_a_real_password')

>>> r = requests.post(url=url, data=body, auth=auth)
>>> r.status_code
201

>>> content = r.json()
>>> print(content['body'])
Sounds great! I'll get right on it.

Brilliant. Oh, wait, no! I meant to add that it would take me a while, because
I had to go feed my cat. If only I could edit this comment! Happily, GitHub
allows us to use another HTTP verb, PATCH, to edit this comment. Let’s do
that.

>>> print(content[u"id"])
5804413

>>> body = json.dumps({u"body": u"Sounds great! I'll get right on it once I feed my cat."})
>>> url = u"https://api.github.com/repos/psf/requests/issues/comments/5804413"

>>> r = requests.patch(url=url, data=body, auth=auth)
>>> r.status_code
200

Excellent. Now, just to torture this Kenneth guy, I’ve decided to let him
sweat and not tell him that I’m working on this. That means I want to delete
this comment. GitHub lets us delete comments using the incredibly aptly named
DELETE method. Let’s get rid of it.

>>> r = requests.delete(url=url, auth=auth)
>>> r.status_code
204
>>> r.headers['status']
'204 No Content'

Excellent. All gone. The last thing I want to know is how much of my ratelimit
I’ve used. Let’s find out. GitHub sends that information in the headers, so
rather than download the whole page I’ll send a HEAD request to get the
headers.

>>> r = requests.head(url=url, auth=auth)
>>> print(r.headers)
...
'x-ratelimit-remaining': '4995'
'x-ratelimit-limit': '5000'
...

Excellent. Time to write a Python program that abuses the GitHub API in all
kinds of exciting ways, 4995 more times.

Custom Verbs¶

From time to time you may be working with a server that, for whatever reason,
allows use or even requires use of HTTP verbs not covered above. One example of
this would be the MKCOL method some WEBDAV servers use. Do not fret, these can
still be used with Requests. These make use of the built-in .request
method. For example:

>>> r = requests.request('MKCOL', url, data=data)
>>> r.status_code
200 # Assuming your call was correct

Utilising this, you can make use of any method verb that your server allows.

Transport Adapters¶

As of v1.0.0, Requests has moved to a modular internal design. Part of the
reason this was done was to implement Transport Adapters, originally
described here. Transport Adapters provide a mechanism to define interaction
methods for an HTTP service. In particular, they allow you to apply per-service
configuration.

Requests ships with a single Transport Adapter, the HTTPAdapter. This adapter provides the default Requests
interaction with HTTP and HTTPS using the powerful urllib3 library. Whenever
a Requests Session is initialized, one of these is
attached to the Session object for HTTP, and one
for HTTPS.

Requests enables users to create and use their own Transport Adapters that
provide specific functionality. Once created, a Transport Adapter can be
mounted to a Session object, along with an indication of which web services
it should apply to.

>>> s = requests.Session()
>>> s.mount('https://github.com/', MyAdapter())

The mount call registers a specific instance of a Transport Adapter to a
prefix. Once mounted, any HTTP request made using that session whose URL starts
with the given prefix will use the given Transport Adapter.

Many of the details of implementing a Transport Adapter are beyond the scope of
this documentation, but take a look at the next example for a simple SSL use-
case. For more than that, you might look at subclassing the
BaseAdapter.

Example: Specific SSL Version¶

The Requests team has made a specific choice to use whatever SSL version is
default in the underlying library (urllib3). Normally this is fine, but from
time to time, you might find yourself needing to connect to a service-endpoint
that uses a version that isn’t compatible with the default.

You can use Transport Adapters for this by taking most of the existing
implementation of HTTPAdapter, and adding a parameter ssl_version that gets
passed-through to urllib3. We’ll make a Transport Adapter that instructs the
library to use SSLv3:

import ssl
from urllib3.poolmanager import PoolManager

from requests.adapters import HTTPAdapter


class Ssl3HttpAdapter(HTTPAdapter):
    """"Transport adapter" that allows us to use SSLv3."""

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_version=ssl.PROTOCOL_SSLv3)

Blocking Or Non-Blocking?¶

With the default Transport Adapter in place, Requests does not provide any kind
of non-blocking IO. The Response.content
property will block until the entire response has been downloaded. If
you require more granularity, the streaming features of the library (see
Streaming Requests) allow you to retrieve smaller quantities of the
response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects
out there that combine Requests with one of Python’s asynchronicity frameworks.
Some excellent examples are requests-threads, grequests, requests-futures, and httpx.

Timeouts¶

Most requests to external servers should have a timeout attached, in case the
server is not responding in a timely manner. By default, requests do not time
out unless a timeout value is set explicitly. Without a timeout, your code may
hang for minutes or more.

The connect timeout is the number of seconds Requests will wait for your
client to establish a connection to a remote machine (corresponding to the
connect()) call on the socket. It’s a good practice to set connect timeouts
to slightly larger than a multiple of 3, which is the default TCP packet
retransmission window.

Once your client has connected to the server and sent the HTTP request, the
read timeout is the number of seconds the client will wait for the server
to send a response. (Specifically, it’s the number of seconds that the client
will wait between bytes sent from the server. In 99.9% of cases, this is the
time before the server sends the first byte).

If you specify a single value for the timeout, like this:

r = requests.get('https://github.com', timeout=5)

The timeout value will be applied to both the connect and the read
timeouts. Specify a tuple if you would like to set the values separately:

r = requests.get('https://github.com', timeout=(3.05, 27))

If the remote server is very slow, you can tell Requests to wait forever for
a response, by passing None as a timeout value and then retrieving a cup of
coffee.

r = requests.get('https://github.com', timeout=None)

Requests verifies SSL certificates for HTTPS requests, just like a web browser. SSL Certificates are small data files that digitally bind a cryptographic key to an organization’s details. Often, a website with a SSL certificate is termed as secure website. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate.

Disable SSL certificate verification

Let us try to access a website with an invalid SSL certificate, using Python requests 

Python3

Output :- 
 

ssl-certificate-verification-python-requests

This website doesn’t have SSL setup so it raises this error.
To disable certificate verification, at the client side, one can use verify attribute.  

Python3

Output 

ssl-certificate-verification-python-requests-1

Since output response 200 is printed, we can assume that request was successful.  

Manual SSL Verification

one can also pass the link to the certificate for validation via python requests only.  

Python3

import requests

response = requests.get('https://github.com', verify ='/path / to / certfile')

print(response)

This would work in case the path provided is correct for SSL certificate for github.com. 

Client Side Certificates

You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both files’ paths: 

>>> requests.get('https://kennethreitz.org', cert=('/path/client.cert', '/path/client.key'))

or persistent:  

s = requests.Session()
s.cert = '/path/client.cert'

If you specify a wrong path or an invalid cert, you’ll get a SSLError: 

>>> requests.get('https://kennethreitz.org', cert='/wrong_path/client.pem')
SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

Last Updated :
09 Sep, 2021

Like Article

Save Article

Понравилась статья? Поделить с друзьями:
  • Python requests обработка ошибок
  • Python requests коды ошибок
  • Python pyqt5 сообщение об ошибке
  • Python postgresql обработка ошибок
  • Python manage py createsuperuser ошибка