<jupyter_start><jupyter_text>Analysis WorkflowThis lecture takes some of the ideas we developed last week, and organizes them into a workflow. We'll do this by example, analyzing the pronto data. Topics will include:- Retrieving data- Encapsulating repeated code into functions- Creating python modules- Organizing the analysis directory with multiple directoriesThese topics provide a rough outline for an analysis workflow.  Retrieving Data<jupyter_code># Packages
from urllib import request
import os
import pandas as pd
# Constants used in analysis
TRIP_DATA = "https://data.seattle.gov/api/views/tw7j-dfaw/rows.csv?accessType=DOWNLOAD"
TRIP_FILE = "pronto_trips.csv"

WEATHER_DATA = "http://uwseds.github.io/data/pronto_weather.csv"
WEATHER_FILE = "pronto_weather.csv"
# Get the URL data
#request.urlretrieve(TRIP_DATA, TRIP_FILE)
!ls -lh<jupyter_output>total 44M
-rw-rw-r-- 1 ubuntu ubuntu 4.7K Oct  9 14:53 analysis_workflow.ipynb
-rw-rw-r-- 1 ubuntu ubuntu 556K Oct  4 14:15 Project-overview.pptx
-rw-rw-r-- 1 ubuntu ubuntu  43M Oct  4 14:43 pronto_trips.csv<jupyter_text>Two challenges1. The file is big. We don't want to download it if it's already present.1. We're going to repeatedly download files. We don't want to just copy and paste the same code.  Encapsulating Repeated Code In FunctionsA function is code that can be invoked by many callers. A function may have arguments that are specified by the caller, and returns values created by the function.<jupyter_code># Example function
def xyz(input):     # The function's name is "func". It has one argument "input".
    return int(input) + 1 # The function returns one value, input + 1

print (xyz("3"))
#a = xyz(3)
#print (xyz(a))
def addTwo(input1, input2):
    return input1 + input2
#
addTwo(1, 2)<jupyter_output><empty_output><jupyter_text>Colin will provide more details about function, such as variable scope, and multiple return values.<jupyter_code># Function to download from a URL
def download(url, filename):
    print("Downloading", filename)
    #request.urlretrieve(url, filename)
download(TRIP_DATA, TRIP_FILE)
# Enhancing function to detect file already present
import os.path
def download(url, filename):
    if os.path.isfile(filename):
        print("Already present %s." % filename)
    else:
        print("Downloading %s" % filename)
        #request.urlretrieve(url, filename)
        
download(TRIP_DATA, "none.csv")
import download
download.download_file(TRIP_DATA, "none.csv")<jupyter_output>Downloading none.csv
{'fraction_non_alphanumeric': 0.08808290155440414, 'fraction_numerical': 0.015942606616181746, 'mean_word_length': 4.74370709382151, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '609148', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 829, 'n_tokens_neox': 770, 'n_words': 280, 'path': '04_ProjectOverview_AnalysisWorkflow/analysis_workflow.ipynb', 'repo_name': 'ecl95/LectureNotes', 'size': '2509'}