<jupyter_start><jupyter_text>Scraping Sneaker Data from the Stock X website  There are 3 components to this notebook:1. Using the individual shoe keys to scrape all the shoe info 2. Cleaning the scraped data.3. Using the shoe name search/random shoe generator to get the price history for a specific shoe<jupyter_code>import requests
from datetime import datetime
from datetime import timedelta
import pandas as pd
import numpy as np
import glob
from urllib.request import Request, urlopen
import time
import random
import io
import dropbox
# open a csv with all the sneakers (used for continuation after error)
# second_path = r'/Users/gabbyvinco/Desktop/sneakers_df.csv'
# round2 = pd.read_csv(second_path, index_col=None, header=0)
# import the query key csv from dropbox
frame = pd.read_csv('https://www.dropbox.com/s/90lnoot6o9usjtl/cumulative_list?dl=1')
# rename column to 'urlKeys' and rename dataframe
dataframe = frame.rename(columns={'0': 'urlKeys'})
# taking the urlKeys and inserting them in to th complete url with parameters
full_urls = []

for value in dataframe['urlKeys']:
    full = 'https://stockx.com/api/products/'+value+'?includes=market,360&currency=EUR&country=IT'
    #'https://stockx.com/api/products/'+value+
        # this part is determining which file/shoe you want to work with 
    #'?includes=market,360&currency=EUR&country=IT' 
        # this part is basically a string that is being passed down to your web browser to the web server
    # with this url we can send a GET request and fetch the information that is listed in the Inspect>Network>Preview
    full_urls.append(full)
    
# create a new column with the full urls    
dataframe['urlFull'] = full_urls
len(full_urls)
# create a new list to hold the sneakers and all their information
info = []<jupyter_output><empty_output><jupyter_text>1. Shoe Info (getting the basic information regarding the shoe)Here we use the keys that we grabbed in the previous script. These keys when added to the query take you to the page for that specific shoe. From there we were able to make a request and gather information like the Stock X shoe identification number, brand, colorway, release date, retail price,official shoe name, volatility, change percentage, and marketed gender of the shoe.<jupyter_code># create a function that loops through the full url list, extracts the variables that we want
# and prints them out in a neat format

def get_shoe_info(url_list):
    for url in url_list:
        headers = {
            "accept-encoding": "gzip, deflate, br",
            "sec-fetch-mode": "cors",
            "sec=fetch-site": "same-origin",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
            "x-requested-with": "XMLHttpRequest"
        }
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        product = response.json()["Product"]
        try:
            id_num = product["id"]
        except:
            print("no id")
            id_num = 0
        try:
            brand = product["brand"]
        except:
            print("no brand")
            brand = 0
        try:
            colorway = product["colorway"]
        except:
            print("no colorway")
            colorway = 0
        try:    
            release_date = product["releaseDate"]
        except:
            print("no release date")
            release_date = 0
        try:
            retail_price = product["retailPrice"]
        except:
            print("no retail price")
            retail_price = 0
        try:
            shoe_name = product["shoe"]
        except:
            print("no shoe name")
            shoe_name = 0
        try:
            volatility = product["market"]["volatility"]
        except:
            print("no volatility")
            volatility = 0
        try:
            change_percentage = product["market"]["changePercentage"]
        except:
            print("no change percentage")
            change_percentage = 0
        try:
            gender = product["gender"]
        except:
            print("no gender")
            gender = 0

        info.append([id_num,
                     brand,
                     colorway,
                     release_date,
                     retail_price,
                     shoe_name,
                     volatility,
                     change_percentage,
                     gender])

        print("shoe info added")
        time.sleep(5)

    return 0
    
# (for personal use) brief explanation of __name__ == "__main__"
# the global variable = __name__ and the entry point = __main__ (or the name that you import the module by)
# so the code below this if statement will only run if the module == entry point to your program
# it allows the code in the module to be importable by other modules without executing the code beneath the block on import

if __name__ == "__main__":
    import sys
    sys.exit(get_shoe_info(full_urls))
# add info into sneakers_df
# sneaker_df = pd.DataFrame(info, columns = ["ID","Brand", "Colorway","ReleaseDate","RetailPrice","Name","Volatility","ChangePercentage","Gender"])

# re-ran the script because there was a timeout error so here we just added to the point at which it was left off
# round2 = round2.append(info)
# adding to dataset after timeout error
# round2.to_csv (r'/Users/gabbyvinco/Desktop/sneakers_df.csv', index = False, header=True)<jupyter_output><empty_output><jupyter_text>2. Cleaning the scraped dataHere we are taking the data gathered about each shoe and cleaning it. There were some issues in the scrape with errors stopping the script and as a result of that there were some issues in offsets of the columns. This is all fixed in this section and prepares the data for futher analysis.<jupyter_code># import sneakers_df from dropbox
sneakers_df = pd.read_csv('https://www.dropbox.com/s/fridkrkb5ifydt0/sneakers_df.csv?dl=1')
sneakers_df.head()
# align and rearrange the data by the offseted groups of rows

set0 = sneakers_df.loc[0:211,:]
set1 = sneakers_df.loc[212:366,:]
set2 = sneakers_df.loc[367:522,:]
set3 = sneakers_df.loc[523:677,:]
set4 = sneakers_df.loc[678:832,:]
set5 = sneakers_df.loc[833:987,:]
set6 = sneakers_df.loc[988:1141,:]
set7 = sneakers_df.loc[1142:1291,:]
set8 = sneakers_df.loc[1292:1446,:]
set9 = sneakers_df.loc[1447:1601,:]
set10 = sneakers_df.loc[1602:1756,:]
set11 = sneakers_df.loc[1757:1911,:]
set12 = sneakers_df.loc[1912:2066,:]
set13 = sneakers_df.loc[2067:2221,:]
set14 = sneakers_df.loc[2222:2376,:]
set15 = sneakers_df.loc[2377:2532,:]
set16 = sneakers_df.loc[2533:,:]
# create a list of all the dataframes to then loop through 
df_list = [set1,set2,set3,set4,set5,set6,set7,set8,set9,set10,set11,set12,set13,set14,set15,set16]
# create a list to add to after removing all NaN values
dropped_list = []
# get rid of all NaN rows that were created by some issue in the scraping code
for s in df_list:
    s = s.dropna(axis='columns', how ='all')
    dropped_list.append(s)
dropped_list[0].head()
# create a new cumulative df
sneaker_info_df = pd.DataFrame()
# create an empty list to append to 
sets_list = []
# change the column names for all of these randomly named columns
for df in dropped_list:
    df = df.rename(columns={df.columns[0]: 'ID',
                            df.columns[1]: 'Brand',
                            df.columns[2]: 'Colorway',
                            df.columns[3]: 'ReleaseDate',
                            df.columns[4]: 'RetailPrice',
                            df.columns[5]: 'Name',
                            df.columns[6]: 'Volatility',
                            df.columns[7]: 'ChangePercent',
                            df.columns[8]: 'Gender'})
    sets_list.append(df)
    
# take the first set which had the correct column names and reorder to fit the rest of the sets
cols = list(set0.columns.values)
col_correct = list(sneaker_info_df.columns.values)
set0 = set0[['ID','Brand','Colorway','ReleaseDate','RetailPrice','Name','Volatility','ChangePercent','Gender']]
# add in set 0 to complete the dataframe
sets_list.append(set0)
sneaker_info_df = pd.concat(sets_list)
# sort by ascending index
sneaker_info_df = sneaker_info_df.sort_index(ascending=True)
sneaker_info_df.head()
# drop any duplicates 
sneaker_info_df.drop_duplicates(subset ="ID",keep = False, inplace = True)
# remove brand from the sneaker name so that it is only the official style name
sneaker_info_df['Name'] = sneaker_info_df['Name'].str.replace(r'adidas ', '')
sneaker_info_df['Name'] = sneaker_info_df['Name'].str.replace(r'Nike ', '')

sneaker_info_df.head()
# value counts to see which are the most popular style of shoe
sneaker_info_df["Name"].value_counts().head(630)
# remove duplicates and only have unique values
unique_sneaker_models = sneaker_info_df["Name"].unique()
# unique_sneaker_models
# test the random sample of 30 sneakers
sneakers_to_search_30 = np.random.choice(unique_sneaker_models,30)
print(sneakers_to_search_30)
# save the csv to the project's dropbox folder

# running the notebook that contains the twitter api keys to pass the variables in
%run ./dropbox_key.ipynb
# connect to the group project dropbox
DBX = dropbox.Dropbox(dropbox_token)
# save df as csv file
data = sneaker_info_df.to_csv(index=False) # The index parameter is optional
# upload to the project's dropbox folder
with io.BytesIO(data.encode()) as stream:
    stream.seek(0)
    DBX.files_upload(stream.read(), "/sneaker_info_data", mode=dropbox.files.WriteMode.overwrite)<jupyter_output><empty_output><jupyter_text>3. Shoe Name Seach to generate a url for price historyThis portion was created with the intention to use with our time-series analysis. Here we have two methods as to which we can gather our price history info. 1. By using the shoe search function where we can enter in the shoe name we are searching for, then select the colorway we want, and finally it will provide us with the "sku_id" which is the Stock X shoe identification number. Using this number we can continue and request the information from the price history plot.2. We use the random shoe generator which will provide us with "sku_id" for a random shoe on the Stock X website.<jupyter_code># import the cleaned sneaker dataset from dropbox
sneaker_info_df = pd.read_csv('https://www.dropbox.com/s/6mj2ntm6meddnqh/sneaker_info_data?dl=1')
# select shoe by name
# make an input where the shoe name can be entered
print("What is the name of the shoe you want to search? ")
shoe_search = input()

# make input lowercase to eliminate any variation in sizing
shoe_search = shoe_search.lower()
# create a list with all the sneaker names in the dataset ensure that they are all 
# lowercased so we can compare it with the input
shoe_names = sneaker_info_df["Name"].to_list()
lower_names = []

for i in shoe_names:
    i = i.lower()
    lower_names.append(i)

# unique brands    
unique_names = set(shoe_names)
# print(unique_names)

#unique brands by lowercase
unique_lower_names = set(lower_names)
# print(unique_lower_names)

# capitalize the first letter of the searched term from the input
shoe_search_capitalized = shoe_search.title()
if shoe_search in unique_lower_names:
    print("We have found that shoe in our data.")
    
    # count of how many types of that name shoe there are
    count_of_name = sum(shoe_search in s for s in lower_names)
    print(f'There are {count_of_name} different colorways of that shoe.')

    # take the shoes of that name and display the colorways to choose from
    sneaker_search_responses = sneaker_info_df.set_index(["Name"])
    print("     ")
    colorway_options = sneaker_info_df[sneaker_info_df["Name"] == shoe_search_capitalized]
    pd.set_option('display.max_rows', colorway_options.shape[0]+1)
    print(colorway_options[["Colorway","Gender"]])
    print("     ")
    
    # choose the colorway
    print("Please specify which colorway you would like: ")
    color_search = input()
    print("     ")
    
    # take the selected colorway and output the ID number
    selected_shoe = colorway_options[colorway_options["Colorway"] == color_search]
    sku_id = selected_shoe["ID"]
    print("ID has been grabbed, proceed to make the request")
#     print(id_num)

else:
    # error message if the shoe isnt in the dataset
    print("We're sorry, the brand you entered wasn't found in our data.")
# to check if the id was grabbed
print(sku_id)
# take the ID number from previous cell and insert it in the url
search_url = "https://stockx.com/api/products/" + sku_id + "/chart"
search_url = search_url.values[0]
# display the entire id not just the first few characters
pd.options.display.max_colwidth = 150
# check the link generated
print(search_url)<jupyter_output>https://stockx.com/api/products/c24cd262-21b0-447a-bbd6-6f1ce15be54c/chart<jupyter_text>Random Shoe Generator to create url for price historyThis part uses a random number generator to then select a shoe out of the dataset and then provide a url for which we can then access the price history.<jupyter_code># check the length of the dataset
len(sneaker_info_df)
# create a random number generator to then select a shoe out of the dataset
n = random.randint(0,2205)
random_sneaker = sneaker_info_df.loc[n,:]
print(random_sneaker[["Name","Colorway"]])
random_id = random_sneaker["ID"]
# create the url using the id from the random generator
random_url = "https://stockx.com/api/products/" + random_id + "/chart"

# check the link
print(random_url)<jupyter_output>https://stockx.com/api/products/87fff3cf-69d5-46de-8432-a062be9d5547/chart<jupyter_text>Price History Function (Getting the data from the interactive plots)This portion now uses the url from either the shoe search fucntion or the random shoe generator to access the price history plot on the Stock X website. This information is visible on the website in an interactive plot, however we were able to make a request and gather all of the price history information.<jupyter_code># allow us to control the days for which we want the price history information
day_ago = datetime.today() - timedelta(days=89)
day_ago = day_ago.strftime('%Y-%m-%d')
day_choose = datetime.today().strftime('%Y-%m-%d')
y = []
def get_price_history(url_with_id, day_get, day_ini):

    params = {
        "start_date": day_ini,
        "end_date": day_get,
        "intervals": "100",
        "format": "highstock",
        "currency": "EUR",
        "country": "IT"
    }

    headers = {
        "accept-encoding": "gzip, deflate",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "same-origin",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
        "x-requested-with": "XMLHttpRequest"
    }

    response = requests.get(url_with_id, params=params, headers=headers)
    response.raise_for_status()
    price_history = response.json()["series"][0]["data"]
    
    for timestamp, price in response.json()["series"][0]["data"]:
        date = datetime.utcfromtimestamp(int(timestamp) // 1000)
        print(f"[{date}]: €{price}")
        # append to list so data frame can be created
        y.append([date, price])
    
    return 0<jupyter_output><empty_output><jupyter_text>Don't forget to pass in the url variable (either search_url OR random_url)You must use one or the other, but not both.<jupyter_code>if __name__ == "__main__":
    import sys
    sys.exit(get_price_history(search_url, day_choose, day_ago))
# create a dataframe from the y list
price_history_df = pd.DataFrame(y)
price_history_df
# save the dataframe to the dropbox file

# running the notebook that contains the twitter api keys to pass the variables in
%run ./dropbox_key.ipynb
# connect to the group project dropbox
DBX = dropbox.Dropbox(dropbox_token)
# save df as csv file
data = price_history_df.to_csv(index=False) # The index parameter is optional
# upload to the project's dropbox folder
with io.BytesIO(data.encode()) as stream:
    stream.seek(0)
    DBX.files_upload(stream.read(), "/price_history_df.csv", mode=dropbox.files.WriteMode.overwrite)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09534368070953436, 'fraction_numerical': 0.021864991377186498, 'mean_word_length': 3.5635188308038224, 'pattern_counts': {'":': 18, '<': 27, '<?xml version=': 0, '>': 29, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '121697', 'license': 'MIT', 'n_tokens_mistral': 5182, 'n_tokens_neox': 4917, 'n_words': 1644, 'path': 'Sneakers_Notebooks/StockX Scraping/sneaker_scraping.ipynb', 'repo_name': 'gabriellavinco/Sneaker_Data', 'size': '16236'}