<jupyter_start><jupyter_text>HTTP and RequestsEstimated time needed: **15** minutes ObjectivesAfter completing this lab you will be able to:*   Understand HTTP*   Handle HTTP Requests Table of Contents                        Overview of HTTP                             Uniform Resource Locator:URL                 Request                Response                                        Requests in Python                              Get Request with URL Parameters                Post Requests  Overview of HTTP  When you, the **client**, use a web page your browser sends an **HTTP** request to the **server** where the page is hosted. The server tries to find the desired **resource** by default "index.html". If your request is successful, the server will send the object to the client in an **HTTP response**. This includes information like the type of the **resource**, the length of the **resource**, and other information.The figure below represents the process. The circle on the left represents the client, the circle on the right represents the Web server. The table under the Web server represents a list of resources stored in the web server. In  this case an HTML file, png image, and txt file .The HTTP protocol allows you to send and receive information through the web including webpages, images, and other web resources. In this lab, we will provide an overview of the Requests library for interacting with the HTTP protocol. </p           Uniform Resource Locator: URL Uniform resource locator (URL) is the most popular way to find resources on the web.  We can break the URL into three parts.    scheme this is this protocol, for this lab it will always be http://       Internet address or  Base URL  this will be used to find the location here are some examples: www.ibm.com and   www.gitlab.com      route location on the web server for example: /images/IDSNlogo.png  You may also hear the term Uniform Resource Identifier (URI), URL are actually a subset of URIs. Another popular term is endpoint, this is the URL of an operation provided by a Web server. Request  The process can be broken into the request and response  process.  The request using the get method is partially illustrated below. In the start line we have the GET method, this is an HTTP method. Also the location of the resource  /index.html and the HTTP version. The Request header passes additional information with an HTTP request:           When an HTTP request is made, an HTTP method is sent, this tells the server what action to perform.  A list of several HTTP methods is shown below. We will go over more examples later.           Response The figure below represents the response; the response start line contains the version number HTTP/1.0, a status code (200) meaning success, followed by a descriptive phrase (OK). The response header contains useful information. Finally, we have the response body containing the requested file, an  HTML  document.  It should be noted that some requests have headers.           Some status code examples are shown in the table below, the prefix indicates the class. These are shown in yellow, with actual status codes shown in  white. Check out the following link  for more descriptions.           Requests in Python Requests is a Python Library that allows you to send HTTP/1.1 requests easily. We can import the library as follows:<jupyter_code>import requests<jupyter_output><empty_output><jupyter_text>We will also use the following libraries:<jupyter_code>import os 
from PIL import Image
from IPython.display import IFrame<jupyter_output><empty_output><jupyter_text>You can make a GET request via the method get to [www.ibm.com](http://www.ibm.com/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01):<jupyter_code>url='https://www.ibm.com/'
r=requests.get(url)<jupyter_output><empty_output><jupyter_text>We have the response object r, this has information about the request, like the status of the request. We can view the status code using the attribute status_code.<jupyter_code>r.status_code<jupyter_output><empty_output><jupyter_text>You can view the request headers:<jupyter_code>print(r.request.headers)<jupyter_output><empty_output><jupyter_text>You can view the request body, in the following line, as there is no body for a get request we get a None:<jupyter_code>print("request body:", r.request.body)<jupyter_output><empty_output><jupyter_text>You can view the HTTP response header using the attribute headers. This returns a python dictionary of HTTP response headers.<jupyter_code>header=r.headers
print(r.headers)<jupyter_output><empty_output><jupyter_text>We can obtain the date the request was sent using the key Date<jupyter_code>header['date']<jupyter_output><empty_output><jupyter_text>Content-Type indicates the type of data:<jupyter_code>header['Content-Type']<jupyter_output><empty_output><jupyter_text>You can also check the encoding:<jupyter_code>r.encoding<jupyter_output><empty_output><jupyter_text>As the Content-Type is text/html we can use the attribute text to display the HTML in the body. We can review the first 100 characters:<jupyter_code>r.text[0:100]<jupyter_output><empty_output><jupyter_text>You can load other types of data for non-text requests, like images. Consider the URL of the following image:<jupyter_code># Use single quotation marks for defining string
url='https://gitlab.com/ibm/skills-network/courses/placeholder101/-/raw/master/labs/module%201/images/IDSNlogo.png'<jupyter_output><empty_output><jupyter_text>We can make a get request:<jupyter_code>r=requests.get(url)<jupyter_output><empty_output><jupyter_text>We can look at the response header:<jupyter_code>print(r.headers)<jupyter_output><empty_output><jupyter_text>We can see the 'Content-Type'<jupyter_code>r.headers['Content-Type']<jupyter_output><empty_output><jupyter_text>An image is a response object that contains the image as a bytes-like object. As a result, we must save it using a file object. First, we specify the file path andname<jupyter_code>path=os.path.join(os.getcwd(),'image.png')
path<jupyter_output><empty_output><jupyter_text>We save the file, in order to access the body of the response we use the attribute content then save it using the open function and write method:<jupyter_code>with open(path,'wb') as f:
    f.write(r.content)<jupyter_output><empty_output><jupyter_text>We can view the image:<jupyter_code>Image.open(path)<jupyter_output><empty_output><jupyter_text>Question 1: write  wget  In the previous section, we used the wget function to retrieve content from the web server as shown below.  Write the python code to perform the same task. The code should be the same as the one used to download the image, but the file name should be 'Example1.txt'. !wget -O /resources/data/Example1.txt<jupyter_code>url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt'
path=os.path.join(os.getcwd(),'example1.txt')
r=requests.get(url)
with open(path,'wb') as f:
    f.write(r.content)<jupyter_output><empty_output><jupyter_text>Click here for the solution```pythonurl='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/labs/Module%205/data/Example1.txt'path=os.path.join(os.getcwd(),'example1.txt')r=requests.get(url)with open(path,'wb') as f:    f.write(r.content)``` Get Request with URL Parameters  You can use the GET method to modify the results of your query, for example retrieving data from an API. We send a GET request to the  server. Like before we have the Base URL, in the Route we append /get, this indicates we would like to preform a GET request. This is demonstrated in the following table:           The Base URL is for [http://httpbin.org/](http://httpbin.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01) is a simple HTTP Request & Response Service. The URL in Python is given by:<jupyter_code>url_get='http://httpbin.org/get'<jupyter_output><empty_output><jupyter_text>A query string is a part of a uniform resource locator (URL), this sends other information to the web server. The start of the query is a ?, followed by a series of parameter and value pairs, as shown in the table below. The first parameter name is name and the value is Joseph. The second parameter name is ID and the Value is 123. Each pair, parameter, and value is separated by an equals sign, =.The series of pairs is separated by the ampersand &.           To create a Query string, add a dictionary. The keys are the parameter names and the values are the value of the Query string.<jupyter_code>payload={"name":"Joseph","ID":"123"}<jupyter_output><empty_output><jupyter_text>Then passing the dictionary payload to the params parameter of the  get() function:<jupyter_code>r=requests.get(url_get,params=payload)<jupyter_output><empty_output><jupyter_text>We can print out the URL and see the name and values<jupyter_code>r.url<jupyter_output><empty_output><jupyter_text>There is no request body<jupyter_code>print("request body:", r.request.body)<jupyter_output><empty_output><jupyter_text>We can print out the status code<jupyter_code>print(r.status_code)<jupyter_output><empty_output><jupyter_text>We can view the response as text:<jupyter_code>print(r.text)<jupyter_output><empty_output><jupyter_text>We can look at the 'Content-Type'.<jupyter_code>r.headers['Content-Type']<jupyter_output><empty_output><jupyter_text>As the content 'Content-Type' is in the JSON format we can use the method json(), it returns a Python dict:<jupyter_code>r.json()<jupyter_output><empty_output><jupyter_text>The key args has the name and values:<jupyter_code>r.json()['args']<jupyter_output><empty_output><jupyter_text>Post Requests   Like a GET request, a POST is used to send data to a server, but the POST request sends the data in a request body. In order to send the Post Request in Python, in the URL we change the route to POST:<jupyter_code>url_post='http://httpbin.org/post'<jupyter_output><empty_output><jupyter_text>This endpoint will expect data as a file or as a form. A form is convenient way to configure an HTTP request to send data to a server. To make a POST request we use the post() function, the variable payload is passed to the parameter  data :<jupyter_code>r_post=requests.post(url_post,data=payload)<jupyter_output><empty_output><jupyter_text>Comparing the URL from the response object of the GET and POST request we see the POST request has no name or value pairs.<jupyter_code>print("POST request URL:",r_post.url )
print("GET request URL:",r.url)<jupyter_output><empty_output><jupyter_text>We can compare the POST and GET request body, we see only the POST request has a body:<jupyter_code>print("POST request body:",r_post.request.body)
print("GET request body:",r.request.body)<jupyter_output><empty_output><jupyter_text>We can view the form as well:<jupyter_code>r_post.json()['form']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07898698308686797, 'fraction_numerical': 0.010537501106880369, 'mean_word_length': 5.3342680874929895, 'pattern_counts': {'":': 2, '<': 138, '<?xml version=': 0, '>': 137, 'https://': 4, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '61951', 'license': 'MIT', 'n_tokens_mistral': 3363, 'n_tokens_neox': 3234, 'n_words': 1247, 'path': 'PY0101EN-5.3_Requests_HTTP.ipynb', 'repo_name': 'amanpreet-5045/Python-Basics-For-Data-Science', 'size': '11293'}