<jupyter_start><jupyter_text>Company XYZ is an Online Travel Agent site, such as Expedia, Booking.com, etc. They haven't invested in data science yet and all the data they have about user searches are simply stored in the URLs generated when users search for a hotel. If you are not familiar with URLs, you can run a search on any OTA site and see how all search parameters are present in the URL. You are asked to: (1) Create a clean data set where each column is a field in the URL, each row is a given search and the cells are the corresponding URL values. (2) For each search query, how many amenities were selected? (3) Often, to measure the quality of a search algorithm, data scientists use some metric based on how often users click on the second page, third page, and so on. The idea here is that a great search algorithm should return all interesting results on the first page and never force users to visit the other pages (how often do you click on the second page results when you search on Google? Almost never, right?). Create a metric based on the above idea and find the city with the worst search algorithm.<jupyter_code>import numpy as np
import pandas as pd
pd.set_option("display.max_columns", 10)
pd.set_option("display.width", 350)

from urllib.parse import urlparse

import seaborn as sns
import matplotlib.pyplot as plt

import h2o
from h2o.frame import H2OFrame
from h2o.estimators.random_forest import H2ORandomForestEstimator
from h2o.grid.grid_search import H2OGridSearch<jupyter_output><empty_output><jupyter_text>Read in the data set<jupyter_code>url = pd.read_csv("../Datasets/url_list.txt", sep = " ", header = None)
print(url.shape)
url.head()<jupyter_output>(77677, 1)<jupyter_text>Data processing<jupyter_code>def search_parse(dat):
    """ function to parse users" search history """
    n = len(dat)
    col_names = ["checkin", "checkout", "customMinimumPriceFilter", "customMaximumPriceFilter", "freeCancellation",
                 "stars_5", "stars_4", "stars_3", "stars_2", "stars_1", "max_score", "min_score", "couponCode",
                 "adults", "city", "children", "amenities", "search_page"]
    
    search_maps = {}
    for name in col_names:
        search_maps[name] = [np.nan] * n
          
    for i in range(n):
        search_conditions = urlparse(dat.iloc[i][0])[4]
        items = search_conditions.split("&")
        conditions = set()
        
        for item in items:
            key, value = item.split("=")
            key = key.split(".")[1]
            if key == "city":
                value = value.strip().replace("+", " ")
            if key not in conditions:
                search_maps[key][i] = value
                conditions.add(key)
            else:
                search_maps[key][i] = search_maps[key][i] + ", " + value
            
    search_dat = pd.DataFrame(search_maps, columns = col_names)
    search_dat["checkin"] = pd.to_datetime(search_dat["checkin"])
    search_dat["checkout"] = pd.to_datetime(search_dat["checkout"])
    search_dat = search_dat.rename(columns = {"customMinimumPriceFilter": "minPrice",
                                              "customMaximumPriceFilter": "maxPrice"})
    return search_dat
search_dat = search_parse(dat = url)<jupyter_output><empty_output><jupyter_text>(1) Create a clean data set where each column is a field in the URL, each row is a given search and the cells are the corresponding URL values.<jupyter_code>search_dat.head()<jupyter_output><empty_output><jupyter_text>(2) For each search query, how many amenities were selected?<jupyter_code>search_dat["amenities"].value_counts()
search_dat["amenities_count"] = search_dat["amenities"].apply(lambda x: 0 if pd.isnull(x) else len(x.split(", ")))
search_dat["amenities_count"].value_counts()<jupyter_output><empty_output><jupyter_text>(3) Often, to measure the quality of a search algorithm, data scientists use some metric based on how often users click on the second page, third page, and so on. The idea here is that a great search algorithm should return all interesting results on the first page and never force users to visit the other pages (how often do you click on the second page results when you search on Google? Almost never, right?). Create a metric based on the above idea and find the city with the worst search algorithm.<jupyter_code>search_dat["first_page_flag"] = search_dat["search_page"].apply(lambda x: 1.0 if x == "1" else 0.0)
search_dat.head()
list(search_dat.columns)
grp_city = search_dat[["city", "first_page_flag"]].groupby("city").mean().reset_index().sort_values(by = "first_page_flag")
print(grp_city)
plt.figure(figsize = [12, 6])
sns.barplot(x = "city", y = "first_page_flag", data = grp_city, palette = "PuBuGn")
plt.title("First Page Rate by City", fontsize = 16)
plt.xlabel("City", fontsize = 12)
plt.ylabel("First Page Rate ", fontsize = 12)<jupyter_output>city  first_page_flag
1                    London, United Kingdom         0.526588
2               New York, NY, United States         0.557616
0                      Hong Kong, Hong Kong         0.910826
3  San Francisco, California, United States         0.959285
{'fraction_non_alphanumeric': 0.09918478260869565, 'fraction_numerical': 0.015333850931677018, 'mean_word_length': 3.753690036900369, 'pattern_counts': {'":': 3, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '444833', 'license': 'MIT', 'n_tokens_mistral': 1567, 'n_tokens_neox': 1502, 'n_words': 548, 'path': '16.URL Parsing Challenge.ipynb', 'repo_name': 'Veronica0206/DS_Project_Sample', 'size': '5152'}