<jupyter_start><jupyter_text>Importing water flow data and saving results to a CSV file. Mimicking the Excel exercise, we begin our analysis by fetching the data we need into our analytical environment. In that previous Excel exercise, we used the NWIS web form to generate a web page containing the data we wanted. Here, however, we use Python's `requests` and `io` libraries to send the same request to the NWIS servers and pull the data directly into a `Pandas` data frame, which we save as a local CSV file. Furthermore, we write our code using strategic variables that enable us to easily modify the gage site and/or the start/end dates of the data to retreive.   How it's done: REST APIsWithout going into detail, this approach leverages the NWIS's *REST API* to fetch the data. Documentation for this API is provided here: https://waterservices.usgs.gov/rest/DV-Test-Tool.html. REST APIs work simply by sending a command, or **request**, to a server via the URL, which then spurs the server to generate a **response** which is passed back to the machine generating the request. This is exactly how web browsers work, but here we are using Python to send the request (for data) and handle the response!The URL for the gage site we want is here:http://waterservices.usgs.gov/nwis/dv/?format=rdb&sites=02087500&startDT=2010-10-01&endDT=2017-09-30&statCd=00003&parameterCd=00060&siteStatus=allYou'll see in the code below that we'll parse this into its components for easier manipulation.   The code... Import the Python libraries required to run the scriptLike *R*, Python has a huge developer environment, and these developers are constantly creating new libraries that run specific tasks. Here we load these into our current scripting environment.*Note: While many 3rd party libraries may be included in your default Python installation, some are not an need to be **installed** prior importing them in a script. We examined how the `requests` library was manually installed in the setup for this exercise.*<jupyter_code>#Import libraries
import requests
import io
import pandas as pd<jupyter_output><empty_output><jupyter_text>Assemble the parameters that will be used in the data requestThe **URL request** is made of 3 components: a web address (of the server providing the service), the service, and a number of parameters passed to the service. Decomposing it into specific elements gives us more control over its execution. So here, to make our script more dynamic, we store parameter values such as the site number and the specific data values we want to fetch. That way we can easily tweak these values and somewhat easily generate results for a different site or parameter.<jupyter_code>#Set site, parameter, and stat codes
siteNo = '02087500' #Neuse R. Near Clayton 
pcode = '00060'     #Discharge (cfs)
scode = '00003'     #Daily mean
#Set start and end dates
startDate = '1930-10-01'
endDate = '2017-09-30'<jupyter_output><empty_output><jupyter_text>Format the request, then send it and store the response as a variableHere we construct the URL used to retrieve the data. It's standard format to include the web address and service as one *string* variable, and the service parameters as a *dictionary* of paramter `name:value` pairs. Compare this structure to the URL mentioned above to see how the two relate.<jupyter_code>#Construct the service URL and parameters
#https://waterdata.usgs.gov/nwis/dv?
url =  'https://waterservices.usgs.gov/nwis/dv'
params = {'sites':siteNo,
          'parameterCd':pcode,
          'statCd':scode,
          'startDT':startDate,
          'endDT':endDate,
          'format':'rdb',
          'siteStatus':'all'
         }<jupyter_output><empty_output><jupyter_text>Send the request to the server, saving the response as a variable. The `requests.get` statment engages the server and ideally sends back the response we want. If all goes well, the servers response, which will be in text format, will be stored in the `response_raw` variable. We then clean it up, in case it has funny characters that might cause odd things to happen later.<jupyter_code>#Send the requests and translate the response
response_raw = requests.get(url,params)
response_clean = response_raw.content.decode('utf-8')<jupyter_output><empty_output><jupyter_text>Clean up the response and read it into a data frameRecall from the Excel exercise that the NWIS response had a number of metadata lines (28, to be precise), then a header line, then a data format line, and finally the raw data separated by tabs. Here we add a bit of Python code to skip the first 28 and also the 30th lines when importing the data into a dataframe. If you're wondering why we use `29` to add the 30th line, it's because *Python lists start at 0, not 1...*The second box below reads the raw text response into a dataframe. This is a fairly complex statement with a lot going on. First the `io.StringIO(response_clean)` bit converts our text object (stored in memory) into something that resembles a file stored on disk. We do this because this `read_csv` function is designed to read files, not memory. The remaining three lines are specific to the file we are reading: we want to skip the lines listed in the object created above (1:28 & 30), indicate that the values are separated by tabs, and lastly indicate that the `site_no` column is actually text, not a number.<jupyter_code>#Create a list of metadata rows to skip; rows 1-29 and 31 
rowsToSkip = list(range(28))
#Append '30' to the list
rowsToSkip.append(29)
#Convert the data into a data frame
df = pd.read_csv(io.StringIO(response_clean),
                 skiprows=rowsToSkip,     #Skip metadta and data spec lines
                 delimiter='\t',          #Set to tab delimited
                 dtype={'site_no':'str'}) #Set site_no to a string datatype<jupyter_output><empty_output><jupyter_text>Examine the resulting data frameWe now have a Pandas dataframe object stored as the variable `df`. We'll spend some time examining all the fantastic stuff we can do to a dataframe, but for now, we can just view the 1st 5 rows using the `head()` function.<jupyter_code>#Display the first 5 rows
df.head()<jupyter_output><empty_output><jupyter_text>Rename the columnsBefore we save this to a CSV file, we should rename the columns something more meaningful.<jupyter_code>#Rename the last two fields
df.rename(columns={'85235_00060_00003':'MeanFlow_cfs','85235_00060_00003_cd':'Confidence'},inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Examine the data types importedThe `dtypes` command returns a listing of the data types of each column. Looks good other than that the datetime is not a `datetime` object, which we'll return to later.<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Save the dataframe to a csv fileFinally, we'll write the file to the local drive so that we can more easily import it later!<jupyter_code>#Save to a csv file
df.to_csv('GageData.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06645750142612664, 'fraction_numerical': 0.018824871648602397, 'mean_word_length': 4.7625308134757605, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '135001', 'license': 'MIT', 'n_tokens_mistral': 2013, 'n_tokens_neox': 1858, 'n_words': 959, 'path': '03-ImportData.ipynb', 'repo_name': 'DataDevils/WaterFlow-Python', 'size': '7012'}