<jupyter_start><jupyter_text>HW3 Q4 [10 pts] Important Notices    WARNING: Do NOT add any cells to this Jupyter Notebook, because that will crash the autograder. Additionally, Make sure to delete or comment out any code you add to this notebook for testing purposes prior to submitting the notebook to Gradescope. Failure to do so may crash the autograder. All instructions, code comments, etc. in this notebook **are part of the assignment instructions**. That is, if there is instructions about completing a task in this notebook, that task is not optional.      You must implement the following functions in this notebook to receive credit.`user()``load_data()``exclude_no_pickuplocations()``exclude_no_tripdistance()``include_fare_range()``get_highest_tip()``get_total_toll()`Each method will be auto-graded using different sets of parameters or data, to ensure that values are not hard-coded.  You may assume we will only use your code to work with data from NYC Taxi Trips during auto-grading. You do not need to write code for unreasonable scenarios.  Since the overall correctness of your code will require multiple function to work together correctly (i.e., all methods are interdepedent), implementing only a subset of the functions likely will lead to a low score. Helper functionsYou should not use - and do not need - any helper functions; implement the required functions by completing the function skeletons below. All of the fuctions you implement should be self-contained; each function should carry all of the code it needs to execute on its expected input(s) within its body.  Kernel SelectionBe sure that you select the PySpark kernel from the kernel options above, if it is not already selected. Do not use the Python3 kernel. The notebook should default to the PySpark kernel.  GCP and GradescopeWe have defined a sqlContext for you, in case you want to experiment with or explore using SparkSQL to accomplish the below tasks while working with the jupyter notebook on GCP. You should _not_ use the SQL context provided for your final submission to Gradescope. For submitting to Gradescope, you should use spark dataframes and dataframe operations _only_. While it is not strictly required that you investigate different approaches to solving this problem using both a sqlContext and dataframe operations, there are subtle differences in the way that seemingly identical actions (e.g., reading in a csv file) sometimes work depending upon which of these you elect to go with. They can be surprising and are worth being aware of. So feel free to explore in GCP, but note that you need only complete the functions below using dataframe operations.  SparkSession Objects and SparkContextsNote that you have access to a SparkSession object in the notebook environment in virtue of using the PySpark kernel. It can be referenced using the variable name _spark_ , and does not need to be explicitly created or defined. This beavior can be leveraged to read in the initial dataset. You also have access to a SparkContext _sc_ which is used to provide you with a sqlContext to execute SparkSQL statements on GCP. You should not need to invoke _spark_ or the provided sqlContext in any of your functions but the load_data() function. Do not hardcode references to _spark_ , _sc_ , or _sqlContext_ into any of your functions except for the load_data() function.  Assumptions Regarding Function InputsThis question assumes that you're carrying the results of your prior work foward at each stage of data preparation and analysis. Thus, the _input_ to each of the _filtering_ functions below should be the _output_ of the _prior_ function. The inputs to the two analytical functions should be the output of the last filtering function.   Pyspark Imports*Please don't modify the below cell*<jupyter_code>import pyspark
from pyspark.sql import SQLContext
from pyspark.sql import *<jupyter_output><empty_output><jupyter_text>Define Spark Context*Please don't modify the below cell*<jupyter_code>sc
sqlContext = SQLContext(sc)
### Student Section - Please compete all the functions below<jupyter_output><empty_output><jupyter_text>Function to return GT Username<jupyter_code>#export
def user():
        """
        :return: string
        your GTUsername, NOT your 9-Digit GTId  
        """         
        return 'gpburdell3'<jupyter_output><empty_output><jupyter_text>Function to load data<jupyter_code>#export
def load_data(gcp_storage_path):
    """
        :param gcp_storage_path: string (full gs path including file name e.g gs://bucket_name/data.csv) 
        :return: spark dataframe  
    """
    ################################################################
    # code to load yellow_tripdata_2019-01.csv data from your GCP  #
    # storage bucket                                               #      
    ################################################################<jupyter_output><empty_output><jupyter_text>Function to exclude trips that don't have a pickup location<jupyter_code>#export
def exclude_no_pickuplocations(df):
    """
        :param nyc tax trips dataframe: spark dataframe 
        :return: spark dataframe  
    """
    ################################################################
    # code to exclude trips with no pickup locations               #
    # Note: Exclude nulls and zeros                                #        
    ################################################################<jupyter_output><empty_output><jupyter_text>Function to exclude trips with no distance<jupyter_code>#export
def exclude_no_tripdistance(df):
    """
        :param nyc tax trips dataframe: spark dataframe 
        :return: spark dataframe  
    """
    ################################################################
    # code to exclude trips with no trip distances                 #
    # Note: Exclude nulls and zeros                                #        
    ################################################################<jupyter_output><empty_output><jupyter_text>Function to include fare amount between the range of 20 to 60 Dollars<jupyter_code>#export
def include_fare_range(df):
    
    """
        :param nyc tax trips dataframe: spark dataframe 
        :return: spark dataframe  
    """
    ################################################################
    # code to include trips with only within the fare range of     #
    # 20 to 60 dollars (including 20 and 60 dollars)               #        
    ################################################################<jupyter_output><empty_output><jupyter_text>Function to get the highest tip amount<jupyter_code>#export
def get_highest_tip(df):
    """
        :param nyc tax trips dataframe: spark dataframe 
        :return: decimal (rounded to 2 digits)  (NOTE: DON'T USE FLOAT)
    """
    
    ################################################################
    # code to get the highest tip amount                           #
    #                                                              #        
    ################################################################<jupyter_output><empty_output><jupyter_text>Function to get total toll amount<jupyter_code>#export
def get_total_toll(df):
    """
        :param nyc tax trips dataframe: spark dataframe 
        :return: decimal (rounded to 2 digits)  (NOTE: DON'T USE FLOAT)
    """
    
    ################################################################
    # code to get total toll amount                                #
    #                                                              #        
    ################################################################<jupyter_output><empty_output><jupyter_text>Run above functions and print Uncomment the cells below and test your implemented functions  Load data from yellow_tripdata09-08-2021.csv<jupyter_code>#gcp_storage_path = "gs://<replace_with_your_storage_bucket>/yellow_tripdata09-08-2021.csv"
#df = load_data(gcp_storage_path)
#df.printSchema()<jupyter_output><empty_output><jupyter_text>Print total numbers of rows in the dataframe<jupyter_code>#df.count()<jupyter_output><empty_output><jupyter_text>Print total number of rows in the dataframe after excluding trips with no pickup location<jupyter_code>#df_no_pickup_locations = exclude_no_pickuplocations(df)
#df_no_pickup_locations.count()<jupyter_output><empty_output><jupyter_text>Print total number of rows in the dataframe after exclude trips with no distance<jupyter_code>#df_no_trip_distance = exclude_no_tripdistance(df_no_pickup_locations)
#df_no_trip_distance.count()<jupyter_output><empty_output><jupyter_text>Print total number of rows in the dataframe after including trips with fair amount between the range of 20 to 60 Dollars<jupyter_code>#df_include_fare_range = include_fare_range(df_no_trip_distance)
#df_include_fare_range.count()<jupyter_output><empty_output><jupyter_text>Print the highest tip amount<jupyter_code>#max_tip = get_highest_tip(df_include_fare_range)
#print(max_tip)<jupyter_output><empty_output><jupyter_text>Print the total toll amount<jupyter_code>#total_toll = get_total_toll(df_include_fare_range)
#print(total_toll)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13489384866630375, 'fraction_numerical': 0.005117038649972781, 'mean_word_length': 4.181049069373943, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '359649', 'license': 'MIT', 'n_tokens_mistral': 2447, 'n_tokens_neox': 2272, 'n_words': 927, 'path': 'Q4/q4.ipynb', 'repo_name': 'garbamoussa/Pyspark', 'size': '9185'}