<jupyter_start><jupyter_text>Python - Requests LibraryRequests is an elegant and simple HTTP library for Python built for human beings.  Installation Using pipenv```shpipenv install requests``` Using Source Code```sh$ git clone git://github.com/requests/requests.git$ curl -OL https://github.com/requests/requests/tarball/master$ cd requests$ pip install .```  Make a request<jupyter_code>import requests
# get request
r = requests.get('https://api.github.com/events')
r.url
# post request
r = requests.post('http://httpbin.org/post', data={'name':'john'})
r.status_code
# put request
r = requests.put('http://httpbin.org/put',data={'name':'<NAME>'})
r.status_code
# delete request
r = requests.delete('http://httpbin.org/delete')
r.status_code
# header request
r = requests.head('http://httpbin.org/get')
r.content<jupyter_output><empty_output><jupyter_text>Passing Parameters in Urls<jupyter_code># passing params
payload={'name':'john','age':12}
r = requests.get('http://httpbin.org/get',params=payload)
r.url<jupyter_output><empty_output><jupyter_text>Whenever a call is made to requests.get() first a Request object is constructed and a response object is generated once Requests gets a response back from the server. The Response object contains all of the information returned by the server and also contains the Request object you created originally.<jupyter_code>r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
# access the headers the server sent back to us
r.headers
# headers we sent to the server
r.request.headers<jupyter_output><empty_output><jupyter_text>Prepared RequestsWhenever we receive a Response object from an API call or a Session call, the request attribute is actually the PreparedRequest that was used. In some cases you may wish to do some extra work to the body or headers before sending a request.<jupyter_code>s = requests.Session()

req = requests.request('POST',url,data=data,headers=headers)
prepped = req.prepare()

# do something with prepped body
prepped.body = 'No I want exactly this as the body'

# do something with prepped.headers
del prepped.headers['Content-Type']

resp = s.send(prepped,stream=stream,verify=verify,proxies=proxies,cert=cert,timeout=timeout)

print(resp.status_code)<jupyter_output><empty_output><jupyter_text>However the above code will lose some of the advantages of having a Requests Session object. In particular, Session-level state such as cookies will not get applied to your request. To get a PreparedRequest with that state applied, replace the call to Request.prepare() with a call to Session.prepare_request() like this:<jupyter_code>s = requests.Session()

req = requests.request('GET',url,data=data,headers=headers)
prepped = s.prepare_request(req)

# do something with prepped body
prepped.body = 'No I want exactly this as the body'

# do something with prepped.headers
del prepped.headers['Keep-Deatd'] = 'parrot'

resp = s.send(prepped,stream=stream,verify=verify,proxies=proxies,cert=cert,timeout=timeout)

print(resp.status_code)<jupyter_output><empty_output><jupyter_text>When you are using the prepared request flow, keep in mind that it does not take into account the environment. This can cause problems if you are using environment variables to change the behavior of requests. For example Self-signed SSL certificates specified in REQUESTS_CA_BUNDLE will not taken into account. You can get around this behavior by explicitly merging the environment settings into the session:<jupyter_code>s = requests.Session()

req = requests.request('POST',url,data=data,headers=headers)
prepped = s.prepare_request(req)

# do something with prepped body
prepped.body = 'No I want exactly this as the body'

# Merger environment settings into session
settings = s.merge_environment_settings(prepped.url,None,None,None,None)

resp = s.send(prepped, **settings)

print(resp.status_code)<jupyter_output><empty_output><jupyter_text>Response Content Text Response Content<jupyter_code># get request
r = requests.get('https://api.github.com/events')
print(r.text)
# get the encoding
r.encoding<jupyter_output><empty_output><jupyter_text>Binary Response ContentYou can also access the response body as bytes for non text-requests. The gzip and deflate transfer-encodings are automatically decoded for you. For example to create an image from binary data returned by a request you can use the following code:```pythonfrom PIL import Imagefrom io import BytesIOi = Image.open(BytesIO.(r.content))```  JSON Response Content<jupyter_code># get request
r = requests.get('https://api.github.com/events')
r.json()<jupyter_output><empty_output><jupyter_text>In case the JSON decoding fails, r.json() raises an exception. For example, if the response gets a 204(No Content) or the response contains invalid JSON, attempting r.json() raises```pythonValueError: No JSON object could be decoded```It should also be noted that the success of the call to r.json() does not indicate the success of the response. Some servers may return a JSON object in a failed response. Such JSON will be decoded and returned. To check that a request is successful use r.raise_for_status() or check r.status_code is what you expect.  Raw response content<jupyter_code># get request
r = requests.get('https://api.github.com/events',stream=True)
r.raw.read(10)<jupyter_output><empty_output><jupyter_text>In general however you should use a pattern like this to save what is being streamed to file.```pythonwith open(filename,'wb') as fd:    for chunk in r.iter_content(chunk_size=128):        fd.write(chunk)```Using Response.iter_content will handle a lot of what you would otherwise have to handle when using Response.raw directly. When streaming a download, the above is the preferred and recommended way to retrieve the content. Note that chunk_size can be freely adjusted to a number that may better fit your use cases.  Custom HeadersIf you'd like to add HTTP headers to a request, simply pass in a dict to the headers parameter.For example we did n't specify the user-agent in the previous example<jupyter_code># get request
url = 'https://api.github.com/events'
headers = {'user-agent':'app-request'}
r = requests.get(url,headers)
r.headers<jupyter_output><empty_output><jupyter_text>Note that the custom headers are given less precedence than more specific sources of information.* Authorization headers set with headers= will be overridden if credentials are specified in .netrc which will be overridden by the auth= parameter.* Authorization headers will be removed if you get redirected off-host.* Proxy-Authorization headers will be overridden by proxy credentials provided in the URL* Content-Length headers will be overridden when we can determine the length of the content.  Advanced POST requestsTypically, you want to send some form-encoded data much like an HTML form. To do this, simply pass a dictionary to the data argument. You dictionary of data will automatically be form-encoded when the request is made.<jupyter_code># using form-encoded data with a dictionary
payload = {'name':'john','age':12}
r = requests.post('https://httpbin.org/post',data=payload)
print(r.text)<jupyter_output>{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "age": "12", 
    "name": "john"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "16", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.14.2"
  }, 
  "json": null, 
  "origin": "192.168.127.12", 
  "url": "https://httpbin.org/post"
}<jupyter_text>You can also pass a list of tuples to the data argument. This is particularly useful when the form has multiple elements that use the same key:<jupyter_code># using form-encoded data with a tuple
payload = (('name','john'),('age',12))
r = requests.post('https://httpbin.org/post',data=payload)
print(r.text)
# using json
import jsonpayload = {'name':'john','age':12}
r = requests.post('https://httpbin.org/post',data=payload)
print(r.text)
payload = {'name':'john','age':12}
r = requests.post('https://httpbin.org/post',data=json.dumps(payload))
print(r.text)
payload = {'name':'john','age':12}
r = requests.post('https://httpbin.org/post',json=payload)
print(r.text)<jupyter_output>{
  "args": {}, 
  "data": "{\"name\": \"john\", \"age\": 12}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "27", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.14.2"
  }, 
  "json": {
    "age": 12, 
    "name": "john"
  }, 
  "origin": "192.168.127.12", 
  "url": "https://httpbin.org/post"
}<jupyter_text>Post a Multipart - Encoded File<jupyter_code>url = 'http://httpbin.org/post'
files = {'file':open('test.xls','rb')}

r = requests.post(url,files=files)
r.text
# set filename, content-type and headers explicitly
url = 'http://httpbin.org/post'
files = {'file':('report.xls',open(),'application/vnd.ms-excel',{'Expires':'0'})}
r = requests.post(url,files=files)<jupyter_output><empty_output><jupyter_text>In the event you are posting a very large file as multipart/form-data request you may want to stream the request. By default requests does not support this but there is a separate package which does -requests-toolbelt.  Post Multiple Multipart-Encoded FilesYou can send multiple files in one request. For example, suppose you want to upload image files to an HTML form with a multiple file field 'images':<jupyter_code>url = 'http://httpbin.org/post'
multiple_files = [
    ('images',('foo.png',open('foo.png','rb'),'image/png')),
    ('images',('bar.png',open('foo.png','rb'),'image/png'))
]
r = requests.post(url,file=multiple_files)
r.text<jupyter_output><empty_output><jupyter_text>Response Status CodesWe can check the response status code:<jupyter_code>r = requests.get('http://httpbin.org/get')
r.status_code
# built in status code objects
r.status_code == requests.codes.ok
# bad request, raise exception
bad_r = requests.get('http://httpbin.org/status/404')
bad_r.status_code
bad_r.raise_for_status()
r.raise_for_status()<jupyter_output><empty_output><jupyter_text>Response Headers<jupyter_code># get the headers
r.headers
# access headers
r.headers['Content-Type']
# access headers using get
r.headers.get('content-type')<jupyter_output><empty_output><jupyter_text>Cookies<jupyter_code>url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies['example-cookie-name']
# send cookies
url = 'http://httpbin.org/cookies'
cookies = dict(cookies_are='working')
r = requests.get(url,cookies=cookies)
r.text<jupyter_output><empty_output><jupyter_text>Cookies are returned in a RequestsCookieJar, which acts like a dict but also offers a more complete interface, suitable for use over multiple domains or paths.<jupyter_code>jar = requests.cookies.RequestsCookieJar()
jar.set('tasty-cookie','yum',domain='httpbin.org',path='/cookies')
url = 'http://httpbin.org/cookies'
r = requests.get(url,cookies=jar)
r.text
r.cookies<jupyter_output><empty_output><jupyter_text>Redirection and HistoryBy default Requests will perform location redirection for all verbs except HEAD.We can use the history property of the Response object to track redirection.The Response.history list contains the Response objects that were created in order to complete the request. The list is sorted from the oldest to the most recent response.<jupyter_code>r = requests.get('http://github.com')
r.url
r.status_code
r.history<jupyter_output><empty_output><jupyter_text>If you are using GET,OPTIONS,POST,PUT,PATCH or DELETE you can disable redirection handling with the allow_redirects parameter.<jupyter_code>r = requests.get('http://github.com',allow_redirects=False)
r.status_code
r.history<jupyter_output><empty_output><jupyter_text>TimeoutsMost requests to externals servers should have a timeout attached, in case the server is not responding in a timely manner. By default, requests do not time out unless a timeout value is set explicitly. Without a timeout, your code may hang for minutes or more.The connect timeout is the number of seconds Requests will wait for your client to establish a connection to a remote machine call on the socket. It is a good practice to set connect timeouts to slightly larger than a multiple of 3, which is the default TCP packet retransmission window.Once your client has connected to the server and sent the HTTP request, the read timeout is the number of the seconds the client will wait for the server to send a response. Specifically its the number of seconds that the client will wait between bytes sent from a server. In 99.9% of cases, this is the time before the server sends the first byte).<jupyter_code>requests.get('http://github.com',timeout=0.001)<jupyter_output><empty_output><jupyter_text>Note: timeout is not a time limit on the entire response download, rather an exception is raised when server has not issued a response for timeout seconds.If no timeout is specified explicitly requests do not timeout.<jupyter_code># single value timeout
# good for both 'connect' and 'read' timeouts
r = requests.get('https://github.com',timeout=5)
# separate value for 'connect' and 'readout' timeouts
r = requests.get('https://github.com',timeout=(3.05,27))<jupyter_output><empty_output><jupyter_text>If the remote server is very slow, you can tell Requests to wait forever for a response, by passing None as a timeout value and then retrieving a cup of coffee.<jupyter_code>r = requests.get('https://github.com',timeout=None)<jupyter_output><empty_output><jupyter_text>Errors and ExceptionIn the event of a network problem, Requests will raise a ConnectionError exceptionResponse.raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.If a request times out, a Timeout exception is raises.If a request exceeds the configured the number of maximum redirections a TooManyRedirects exception is raised.All exceptions that Requests explicitly raises inherit fromrequests.exceptions.RequestException  Event HooksRequests has a hook system that you can use to manipulate portions of the request process or signal event handling.Available Hooks:* response - response generated from a requestYou can assign a hook function on a per-request basis by passing a {hook_name:callback_function} dictionary to the hooks request parameter.<jupyter_code># call back function - print url function
def print_url(r, *args, **kwargs):
    print(r.url)
    
requests.get('http://httpbin.org',hooks={'response':print_url})
# second callback function
def record_hook(r,*args,**kwargs):
    r.hook_called = True
    return r
# add multiple hooks to a single request
requests.get('http://httpbin.org',hooks={'response':[print_url,record_hook]})
r = requests.get('http://httpbin.org',hooks={'response':[print_url,record_hook]})
r.hook_called<jupyter_output>http://httpbin.org/<jupyter_text>ProxiesIf you need to use a proxy, you can configure individual requests with the proxies argument to any request method:<jupyter_code># proxies = {'http':'http://10.10.1.10:3128','https':'http://10.10.1.10:1080'}
# requests.get('http://example.org',proxies = proxies)<jupyter_output><empty_output><jupyter_text>To use HTTP Basic Auth with your proxy use the http://user:password@host/syntaxTo give a proxy for a specific scheme and host use the scheme://hostname form for the key. This will match for any request to the given scheme and hostname.proxies={'http://10.20.1.18':'http://10.20.1.292:5342'}  Chunk Encoded RequestsRequests also supports Chunked transfer encoding for outgoing and incoming requests. To send a chunk-encoded request, simply provide a generator (or any iterator without a length) for your body.```pythondef gen():    yield 'hi'    yield 'there'requests.post('http://some.url/chunked',data=gen())```For chunked encoded repsonses its best to iterate over the data using Response.iter_content(). In an ideal situation you'll have set stream=True on the request, in which case you can iterate chunk-by-chunk by calling iter_content with a chunk_size parameter of None. If you want to set a maximum size of the chunk, you can set a chunk_size parameter to any integer.  Streaming RequestsWith Response.iter_lines() you can easily iterate over streaming APIs such as Twitter Streaming API. Simply set to Stream to True and iterate over the response with iter_lines:<jupyter_code>import json
import requests

r = requests.get('http://httpbin.org/stream/5',stream=True)

for line in r.iter_lines():
    # filter out keep-alive new lines
    if line:
        decoded_line = line.decode('utf-8')
        print(json.loads(decoded_line))<jupyter_output>{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 0, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 1, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 2, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 3, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzi[...]<jupyter_text>When using decode_unicode=True with Response.iter_lines() or Response.iter_content(), provide a fallback encoding in the event the server does n't provide one:<jupyter_code>r = requests.get('http://httpbin.org/stream/5', stream=True)

if r.encoding is None:
    r.encoding = 'utf-8'
    
for line in r.iter_lines(decode_unicode=True):
    if line:
        print(json.loads(line))<jupyter_output>{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 0, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 1, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 2, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'python-requests/2.14.2', 'Host': 'httpbin.org'}, 'id': 3, 'origin': '192.168.127.12', 'args': {}, 'url': 'http://httpbin.org/stream/5'}
{'headers': {'Accept-Encoding': 'gzi[...]<jupyter_text>Session ObjectsThe Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance and will use urllib3's connection pooling. So if you are making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase.<jupyter_code>s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('http://httpbin.org/cookies')

print(r.text)<jupyter_output>{
  "cookies": {
    "sessioncookie": "123456789"
  }
}<jupyter_text>Sessions can also be used to provide default data to the request methods. This is done by providing data to the properties on a Session object:<jupyter_code>s = requests.Session()
s.auth = ('user','pass')
s.headers.update({'x-test':'true'})

# both x-test and x-test are send
s.get('http://httpbin.org/headers', headers={'x-test-2':'true'})<jupyter_output><empty_output><jupyter_text>Any dictionaries you pass to a request method will be merged with the session-level values that are set. The method level parameters override the session parameters.Note however that method level parameteres will not be persisted across requests even if using a session. The following example will only send the cooking with the first request but not the second:<jupyter_code>s = requests.Session()

r = s.get('http://httpbin.org/cookies',cookies={'from-my':'browser'})
print(r.text)
r = s.get('http://httpbin.org/cookies')
print(r.text)<jupyter_output>{
  "cookies": {}
}
{'fraction_non_alphanumeric': 0.1294988227379751, 'fraction_numerical': 0.013406371630387775, 'mean_word_length': 5.515967438948028, 'pattern_counts': {'":': 39, '<': 115, '<?xml version=': 0, '>': 115, 'https://': 16, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 14, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '550162', 'license': 'MIT', 'n_tokens_mistral': 6462, 'n_tokens_neox': 6187, 'n_words': 2089, 'path': 'libraries/Python Libraries - Requests/Requests Library.ipynb', 'repo_name': 'vravishankar/Jupyter-Books', 'size': '20823'}