requests adapters this module contains the transport adapters that requests uses to define and maintain connections the base transport adapter def initself super init def send self request streamfalse timeoutnone verifytrue certnone proxiesnone raise notimplementederror def closeself the builtin http adapter for urllib3 provides a generalcase interface for requests sessions to contact http and https urls by implementing the transport adapter interface this class will usually be the class session session class under the covers param poolconnections the number of urllib3 connection pools to cache param poolmaxsize the maximum number of connections to save in the pool param maxretries the maximum number of retries each connection should attempt note this applies only to failed dns lookups socket connections and connection timeouts never to requests where data has made it to the server by default requests does not retry failed connections if you need granular control over the conditions under which we retry a request import urllib3 s retry class and pass that instead param poolblock whether the connection pool should block for connections usage import requests s requests session a requests adapters httpadaptermaxretries3 s mount http a can t handle by adding proxymanager to self attrs because self poolmanager uses a lambda function which isn t pickleable initializes a urllib3 poolmanager this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param connections the number of urllib3 connection pools to cache param maxsize the maximum number of connections to save in the pool param block block when no free connections are available param poolkwargs extra keyword arguments used to initialize the pool manager save these values for pickling return urllib3 proxymanager for the given proxy this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param proxy the proxy to return a urllib3 proxymanager for param proxykwargs extra keyword arguments used to configure the proxy manager returns proxymanager rtype urllib3 proxymanager verify a ssl certificate this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param conn the urllib3 connection object associated with the cert param url the requested url param verify either a boolean in which case it controls whether we verify the server s tls certificate or a string in which case it must be a path to a ca bundle to use param cert the ssl certificate to verify allow selfspecified cert location builds a class response requests response object from a urllib3 response this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param req the class preparedrequest preparedrequest used to generate the response param resp the urllib3 response object rtype requests response fallback to none if there s no statuscode for whatever reason make headers caseinsensitive set encoding add new cookies from the server give the response some context returns a urllib3 connection for the given url this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param url the url to connect to param proxies optional a requestsstyle dictionary of proxies used on this request rtype urllib3 connectionpool only scheme should be lower case disposes of any internal state currently this closes the poolmanager and any active proxymanager which closes any pooled connections obtain the url to use when making the final request if the message is being sent through a http proxy the full url has to be used otherwise we should only use the path portion of the url this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param request the class preparedrequest preparedrequest being sent param proxies a dictionary of schemes or schemes and hosts to proxy urls rtype str add any headers needed by the connection as of v2 0 this does nothing by default but is left for overriding by users that subclass the class httpadapter requests adapters httpadapter this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param request the class preparedrequest preparedrequest to add headers to param kwargs the keyword arguments from the call to send returns a dictionary of the headers to add to any request sent through a proxy this works with urllib3 magic to ensure that they are correctly sent to the proxy rather than in a tunnelled request if connect is being used this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param proxy the url of the proxy being used for this request rtype dict sends preparedrequest object returns response object param request the class preparedrequest preparedrequest being sent param stream optional whether to stream the request content param timeout optional how long to wait for the server to send data before giving up as a float or a ref connect timeout read timeout timeouts tuple type timeout float or tuple or urllib3 timeout object param verify optional either a boolean in which case it controls whether we verify the server s tls certificate or a string in which case it must be a path to a ca bundle to use param cert optional any userprovided ssl certificate to be trusted param proxies optional the proxies dictionary to apply to the request rtype requests response todo remove this in 3 0 0 see 2811 this branch is for urllib3 v1 22 and later this branch is for urllib3 versions earlier than v1 22 noqa f401 the base transport adapter sends preparedrequest object returns response object param request the class preparedrequest preparedrequest being sent param stream optional whether to stream the request content param timeout optional how long to wait for the server to send data before giving up as a float or a ref connect timeout read timeout timeouts tuple type timeout float or tuple param verify optional either a boolean in which case it controls whether we verify the server s tls certificate or a string in which case it must be a path to a ca bundle to use param cert optional any user provided ssl certificate to be trusted param proxies optional the proxies dictionary to apply to the request cleans up adapter specific items the built in http adapter for urllib3 provides a general case interface for requests sessions to contact http and https urls by implementing the transport adapter interface this class will usually be the class session session class under the covers param pool_connections the number of urllib3 connection pools to cache param pool_maxsize the maximum number of connections to save in the pool param max_retries the maximum number of retries each connection should attempt note this applies only to failed dns lookups socket connections and connection timeouts never to requests where data has made it to the server by default requests does not retry failed connections if you need granular control over the conditions under which we retry a request import urllib3 s retry class and pass that instead param pool_block whether the connection pool should block for connections usage import requests s requests session a requests adapters httpadapter max_retries 3 s mount http a can t handle by adding proxy_manager to self __attrs__ because self poolmanager uses a lambda function which isn t pickleable initializes a urllib3 poolmanager this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param connections the number of urllib3 connection pools to cache param maxsize the maximum number of connections to save in the pool param block block when no free connections are available param pool_kwargs extra keyword arguments used to initialize the pool manager save these values for pickling return urllib3 proxymanager for the given proxy this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param proxy the proxy to return a urllib3 proxymanager for param proxy_kwargs extra keyword arguments used to configure the proxy manager returns proxymanager rtype urllib3 proxymanager verify a ssl certificate this method should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param conn the urllib3 connection object associated with the cert param url the requested url param verify either a boolean in which case it controls whether we verify the server s tls certificate or a string in which case it must be a path to a ca bundle to use param cert the ssl certificate to verify allow self specified cert location builds a class response requests response object from a urllib3 response this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param req the class preparedrequest preparedrequest used to generate the response param resp the urllib3 response object rtype requests response fallback to none if there s no status_code for whatever reason make headers case insensitive set encoding add new cookies from the server give the response some context returns a urllib3 connection for the given url this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param url the url to connect to param proxies optional a requests style dictionary of proxies used on this request rtype urllib3 connectionpool only scheme should be lower case disposes of any internal state currently this closes the poolmanager and any active proxymanager which closes any pooled connections obtain the url to use when making the final request if the message is being sent through a http proxy the full url has to be used otherwise we should only use the path portion of the url this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param request the class preparedrequest preparedrequest being sent param proxies a dictionary of schemes or schemes and hosts to proxy urls rtype str add any headers needed by the connection as of v2 0 this does nothing by default but is left for overriding by users that subclass the class httpadapter requests adapters httpadapter this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param request the class preparedrequest preparedrequest to add headers to param kwargs the keyword arguments from the call to send returns a dictionary of the headers to add to any request sent through a proxy this works with urllib3 magic to ensure that they are correctly sent to the proxy rather than in a tunnelled request if connect is being used this should not be called from user code and is only exposed for use when subclassing the class httpadapter requests adapters httpadapter param proxy the url of the proxy being used for this request rtype dict sends preparedrequest object returns response object param request the class preparedrequest preparedrequest being sent param stream optional whether to stream the request content param timeout optional how long to wait for the server to send data before giving up as a float or a ref connect timeout read timeout timeouts tuple type timeout float or tuple or urllib3 timeout object param verify optional either a boolean in which case it controls whether we verify the server s tls certificate or a string in which case it must be a path to a ca bundle to use param cert optional any user provided ssl certificate to be trusted param proxies optional the proxies dictionary to apply to the request rtype requests response todo remove this in 3 0 0 see 2811 this branch is for urllib3 v1 22 and later this branch is for urllib3 versions earlier than v1 22
import os.path
import socket  

from urllib3.exceptions import ClosedPoolError, ConnectTimeoutError
from urllib3.exceptions import HTTPError as _HTTPError
from urllib3.exceptions import InvalidHeader as _InvalidHeader
from urllib3.exceptions import (
    LocationValueError,
    MaxRetryError,
    NewConnectionError,
    ProtocolError,
)
from urllib3.exceptions import ProxyError as _ProxyError
from urllib3.exceptions import ReadTimeoutError, ResponseError
from urllib3.exceptions import SSLError as _SSLError
from urllib3.poolmanager import PoolManager, proxy_from_url
from urllib3.util import Timeout as TimeoutSauce
from urllib3.util import parse_url
from urllib3.util.retry import Retry

from .auth import _basic_auth_str
from .compat import basestring, urlparse
from .cookies import extract_cookies_to_jar
from .exceptions import (
    ConnectionError,
    ConnectTimeout,
    InvalidHeader,
    InvalidProxyURL,
    InvalidSchema,
    InvalidURL,
    ProxyError,
    ReadTimeout,
    RetryError,
    SSLError,
)
from .models import Response
from .structures import CaseInsensitiveDict
from .utils import (
    DEFAULT_CA_BUNDLE_PATH,
    extract_zipped_paths,
    get_auth_from_url,
    get_encoding_from_headers,
    prepend_scheme_if_needed,
    select_proxy,
    urldefragauth,
)

try:
    from urllib3.contrib.socks import SOCKSProxyManager
except ImportError:

    def SOCKSProxyManager(*args, **kwargs):
        raise InvalidSchema("Missing dependencies for SOCKS support.")


DEFAULT_POOLBLOCK = False
DEFAULT_POOLSIZE = 10
DEFAULT_RETRIES = 0
DEFAULT_POOL_TIMEOUT = None


class BaseAdapter:
    

    def __init__(self):
        super().__init__()

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        
        raise NotImplementedError

    def close(self):
        
        raise NotImplementedError


class HTTPAdapter(BaseAdapter):
    

    __attrs__ = [
        "max_retries",
        "config",
        "_pool_connections",
        "_pool_maxsize",
        "_pool_block",
    ]

    def __init__(
        self,
        pool_connections=DEFAULT_POOLSIZE,
        pool_maxsize=DEFAULT_POOLSIZE,
        max_retries=DEFAULT_RETRIES,
        pool_block=DEFAULT_POOLBLOCK,
    ):
        if max_retries == DEFAULT_RETRIES:
            self.max_retries = Retry(0, read=False)
        else:
            self.max_retries = Retry.from_int(max_retries)
        self.config = {}
        self.proxy_manager = {}

        super().__init__()

        self._pool_connections = pool_connections
        self._pool_maxsize = pool_maxsize
        self._pool_block = pool_block

        self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

    def __getstate__(self):
        return {attr: getattr(self, attr, None) for attr in self.__attrs__}

    def __setstate__(self, state):
        
        
        self.proxy_manager = {}
        self.config = {}

        for attr, value in state.items():
            setattr(self, attr, value)

        self.init_poolmanager(
            self._pool_connections, self._pool_maxsize, block=self._pool_block
        )

    def init_poolmanager(
        self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs
    ):
        
        
        self._pool_connections = connections
        self._pool_maxsize = maxsize
        self._pool_block = block

        self.poolmanager = PoolManager(
            num_pools=connections,
            maxsize=maxsize,
            block=block,
            **pool_kwargs,
        )

    def proxy_manager_for(self, proxy, **proxy_kwargs):
        
        if proxy in self.proxy_manager:
            manager = self.proxy_manager[proxy]
        elif proxy.lower().startswith("socks"):
            username, password = get_auth_from_url(proxy)
            manager = self.proxy_manager[proxy] = SOCKSProxyManager(
                proxy,
                username=username,
                password=password,
                num_pools=self._pool_connections,
                maxsize=self._pool_maxsize,
                block=self._pool_block,
                **proxy_kwargs,
            )
        else:
            proxy_headers = self.proxy_headers(proxy)
            manager = self.proxy_manager[proxy] = proxy_from_url(
                proxy,
                proxy_headers=proxy_headers,
                num_pools=self._pool_connections,
                maxsize=self._pool_maxsize,
                block=self._pool_block,
                **proxy_kwargs,
            )

        return manager

    def cert_verify(self, conn, url, verify, cert):
        
        if url.lower().startswith("https") and verify:
            cert_loc = None

            
            if verify is not True:
                cert_loc = verify

            if not cert_loc:
                cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH)

            if not cert_loc or not os.path.exists(cert_loc):
                raise OSError(
                    f"Could not find a suitable TLS CA certificate bundle, "
                    f"invalid path: {cert_loc}"
                )

            conn.cert_reqs = "CERT_REQUIRED"

            if not os.path.isdir(cert_loc):
                conn.ca_certs = cert_loc
            else:
                conn.ca_cert_dir = cert_loc
        else:
            conn.cert_reqs = "CERT_NONE"
            conn.ca_certs = None
            conn.ca_cert_dir = None

        if cert:
            if not isinstance(cert, basestring):
                conn.cert_file = cert[0]
                conn.key_file = cert[1]
            else:
                conn.cert_file = cert
                conn.key_file = None
            if conn.cert_file and not os.path.exists(conn.cert_file):
                raise OSError(
                    f"Could not find the TLS certificate file, "
                    f"invalid path: {conn.cert_file}"
                )
            if conn.key_file and not os.path.exists(conn.key_file):
                raise OSError(
                    f"Could not find the TLS key file, invalid path: {conn.key_file}"
                )

    def build_response(self, req, resp):
        
        response = Response()

        
        response.status_code = getattr(resp, "status", None)

        
        response.headers = CaseInsensitiveDict(getattr(resp, "headers", {}))

        
        response.encoding = get_encoding_from_headers(response.headers)
        response.raw = resp
        response.reason = response.raw.reason

        if isinstance(req.url, bytes):
            response.url = req.url.decode("utf-8")
        else:
            response.url = req.url

        
        extract_cookies_to_jar(response.cookies, req, resp)

        
        response.request = req
        response.connection = self

        return response

    def get_connection(self, url, proxies=None):
        
        proxy = select_proxy(url, proxies)

        if proxy:
            proxy = prepend_scheme_if_needed(proxy, "http")
            proxy_url = parse_url(proxy)
            if not proxy_url.host:
                raise InvalidProxyURL(
                    "Please check proxy URL. It is malformed "
                    "and could be missing the host."
                )
            proxy_manager = self.proxy_manager_for(proxy)
            conn = proxy_manager.connection_from_url(url)
        else:
            
            parsed = urlparse(url)
            url = parsed.geturl()
            conn = self.poolmanager.connection_from_url(url)

        return conn

    def close(self):
        
        self.poolmanager.clear()
        for proxy in self.proxy_manager.values():
            proxy.clear()

    def request_url(self, request, proxies):
        
        proxy = select_proxy(request.url, proxies)
        scheme = urlparse(request.url).scheme

        is_proxied_http_request = proxy and scheme != "https"
        using_socks_proxy = False
        if proxy:
            proxy_scheme = urlparse(proxy).scheme.lower()
            using_socks_proxy = proxy_scheme.startswith("socks")

        url = request.path_url
        if is_proxied_http_request and not using_socks_proxy:
            url = urldefragauth(request.url)

        return url

    def add_headers(self, request, **kwargs):
        
        pass

    def proxy_headers(self, proxy):
        
        headers = {}
        username, password = get_auth_from_url(proxy)

        if username:
            headers["Proxy-Authorization"] = _basic_auth_str(username, password)

        return headers

    def send(
        self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
    ):
        

        try:
            conn = self.get_connection(request.url, proxies)
        except LocationValueError as e:
            raise InvalidURL(e, request=request)

        self.cert_verify(conn, request.url, verify, cert)
        url = self.request_url(request, proxies)
        self.add_headers(
            request,
            stream=stream,
            timeout=timeout,
            verify=verify,
            cert=cert,
            proxies=proxies,
        )

        chunked = not (request.body is None or "Content-Length" in request.headers)

        if isinstance(timeout, tuple):
            try:
                connect, read = timeout
                timeout = TimeoutSauce(connect=connect, read=read)
            except ValueError:
                raise ValueError(
                    f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                    f"or a single float to set both timeouts to the same value."
                )
        elif isinstance(timeout, TimeoutSauce):
            pass
        else:
            timeout = TimeoutSauce(connect=timeout, read=timeout)

        try:
            resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,
                headers=request.headers,
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=timeout,
                chunked=chunked,
            )

        except (ProtocolError, OSError) as err:
            raise ConnectionError(err, request=request)

        except MaxRetryError as e:
            if isinstance(e.reason, ConnectTimeoutError):
                
                if not isinstance(e.reason, NewConnectionError):
                    raise ConnectTimeout(e, request=request)

            if isinstance(e.reason, ResponseError):
                raise RetryError(e, request=request)

            if isinstance(e.reason, _ProxyError):
                raise ProxyError(e, request=request)

            if isinstance(e.reason, _SSLError):
                
                raise SSLError(e, request=request)

            raise ConnectionError(e, request=request)

        except ClosedPoolError as e:
            raise ConnectionError(e, request=request)

        except _ProxyError as e:
            raise ProxyError(e)

        except (_SSLError, _HTTPError) as e:
            if isinstance(e, _SSLError):
                
                raise SSLError(e, request=request)
            elif isinstance(e, ReadTimeoutError):
                raise ReadTimeout(e, request=request)
            elif isinstance(e, _InvalidHeader):
                raise InvalidHeader(e, request=request)
            else:
                raise

        return self.build_response(request, resp)