<jupyter_start><jupyter_text>Sourcing the DataThe product data I've chosen doesn't come directly from Amazon, but rather from three AI researchers, <NAME>, <NAME>, and <NAME>, who gathered it for their paper "Justifying Recommendations using Distantly-Labeled Reviews and Fine-grained Aspects." The review data extends from May 1996 to October 2018, which is about when they released the update. Their focus was primarily on reviews, but the dataset also has metadata for ~15-million products. The purpose of this notebook is to obtain the data, clean it, and reduce it to a more manageable size.  Downloading the DataFor an overview of the dataset, see `main_notebook.ipynb`. This notebook developed out of practical necessity, due to the unwieldy size of the dataset (~15M samples). I needed a space to reduce the dataset to a more manageable ~4M.I begin by downloading the data from some kindly academic researchers. Either run the following cell or use the [link](http://deepyeti.ucsd.edu/jianmo/amazon/metaFiles/All_Amazon_Meta.json.gz) to download the file to `data/amazon_raw.json.gz`. The researchers request that you fill out a [short form](https://forms.gle/A8hBfPxKkKGFCP238) if you download the data.This single compressed JSON file is not ideal for such a large, 15-million product dataset. After the download completes, I'll export the it to Apache Parquet. Parquet is a (much faster) binary format which is designed for fast IO on large datasets.<jupyter_code>src = "http://deepyeti.ucsd.edu/jianmo/amazon/metaFiles/All_Amazon_Meta.json.gz"
dst = "data/amazon_raw.json.gz"

if DOWNLOAD:
    # Adjust `chunk_size` as necessary
    utils.download(src, dst, chunk_size=10 ** 7)<jupyter_output><empty_output><jupyter_text>Now I'm ready for the export, assuming `INIT_EXPORT` is set. This may take a while. It would be faster to use Dask here than Pandas, but Dask doesn't support chunked reading of compressed JSONs.<jupyter_code>if INIT_EXPORT:
    cols = [
        "asin",
        "description",
        "title",
        "feature",
        "brand",
        "main_cat",
    ]
    json_path = os.path.normpath("data/amazon_raw.json.gz")
    out_dir = os.path.normpath("data/amazon_raw")

    print(f"Exporting to {out_dir}...")
    # Make sure empty directory exists
    if os.path.isdir(out_dir):
        shutil.rmtree(out_dir)
    os.makedirs(out_dir)
    reader = pd.read_json(json_path, lines=True, chunksize=10 ** 5)

    with tqdm(total=14_874_972) as pbar:
        for i, chunk in enumerate(reader):
            chunk = chunk.loc[:, cols]
            out_path = os.path.join(out_dir, f"part.{i}.parquet")
            chunk.to_parquet(
                out_path, engine="pyarrow", index=True,
            )
            pbar.update(len(chunk))<jupyter_output>Exporting to data\amazon_raw...<jupyter_text>Initial Pruning<jupyter_code>cols = ["description", "main_cat", "asin"]
df = dd.read_parquet("data/amazon_raw", engine="pyarrow", columns=cols)
print(f"n_samples: {len(df):,.0f}")
df
df = df.dropna(subset=["description", "main_cat"])
df.head()
df = df.compute()
df
df["description"] = df["description"].str.join(" ")
df.head()<jupyter_output><empty_output><jupyter_text>I drop duplicates with respect to the ID feature 'asin', and with respect to 'description'. Weeding out duplicate descriptions reduces the influence of boilerplate copy-paste style product descriptions.<jupyter_code>df = df.drop_duplicates(subset=["asin"])
df = df.drop_duplicates(subset=["description"])
df.shape
df = df.set_index("asin")
df.head()<jupyter_output><empty_output><jupyter_text>Pruning Length OutliersI want to reduce the dataset to a more manageable size by selecting only the most valuable parts. Document vectors can be normalized to adjust for varying document length, but there are still reasons to pay attention to it. Short documents contain very few terms, and thus very little information. The vectors representing short documents are more coarse-grained and crude than those representing long ones.On the other hand, some documents may contain too much information. Extremely long documents may contain pages upon pages of irrelevant details. For example, the product description for a book might be the entire first chapter. They also may contain artifacts which inflate their length, such lists of codes or blocks of artificial language (e.g. HTML, JavaScript).<jupyter_code>fig = lang.length_dist(
    data=df.sample(10 ** 6), subset="description", log_scale=True, tick_prec=1
)<jupyter_output><empty_output><jupyter_text>If it looks normally distributed, that's because it's plotted on a logarithmic scale. The description lengths vary in orders of magnitude.<jupyter_code># Print stats
lang.length_info(df["description"])<jupyter_output>len_description
count        8,288,595
mean               665
std              1,120
min                  0
25%                174
50%                386
75%                802
max            278,061<jupyter_text>The longest description is ~280K characters and the shortest is 0 characters. Let's take a look at the longest.<jupyter_code>df.loc[df.description.str.len().idxmax(), "description"]<jupyter_output><empty_output><jupyter_text>It's not weird, but it's filled with HTML formatting. Reminds me a philosophy of mathematics course I once took. It'd make me a little sad to drop this one, though it does contain a lot of irrelevant detail.Next, I'll strip out HTML tags to get more useful length measurements.<jupyter_code>df["description"] = lang.strip_html_tags(df.loc[:, "description"], n_jobs=-1)
df.head()
df.sort_values(
    "description", key=lambda x: x.str.len(), ascending=False, inplace=True,
)
df.head(20)<jupyter_output><empty_output><jupyter_text>All the longest descriptions are for books. That's hardly surprising. Let's look at the one that's not a book, B00LQDEF5Q.<jupyter_code>df.at["B00LQDEF5Q", "description"]<jupyter_output><empty_output><jupyter_text>And there you have it, an obnoxiously long list of SKUs. I'll drop the longest 1%.Since I want to create a relatively fine-grained vector space, I'll also drop the shortest 50%.<jupyter_code>fig = lang.length_dist(
    data=df, subset="description", log_scale=True, tick_prec=0, height=8
)
ax = fig.get_axes()[0]
lower = df.description.str.len().quantile(0.5)
upper = df.description.str.len().quantile(0.99)
plotting.annotate.draw_line(ax, x=lower)
plotting.annotate.draw_line(ax, x=upper)
fig.savefig("figures/length_outliers.svg", bbox_inches="tight")
df = lang.trim_length_outliers(df, subset="description", q_lower=0.5, q_upper=0.99)
df.head()<jupyter_output>C:\Users\ndgig\anaconda3\envs\capstone\lib\site-packages\ndg_tools\outliers.py:358: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated infavour of `both` or `neither`.
  inliers = data.between(lower, upper, inclusive=False) | data.isna()<jupyter_text>The following is now the shortest product description. It's a mere 379 characters, but seems long enough to be useful. I can't imagine getting much use out of anything shorter. That said, I do plan on concatenating the description with a couple other text features.<jupyter_code>df.at[df.description.str.len().idxmin(), "description"]
fig = lang.length_dist(df.sample(10 ** 6), subset="description")<jupyter_output><empty_output><jupyter_text>The document lengths are now more or less on the same order of magnitude. I hope I've gotten rid of the anomalies.  Cleaning 'main_cat'The next step is to clean up the target variable: 'main_cat'.<jupyter_code># Get mask of samples with HTML
is_html = df["main_cat"].str.match(r"^<\w+")

# Extract `alt` property
df.loc[is_html, "main_cat"] = (
    df.loc[is_html, "main_cat"].str.extract(r"alt=\"(.+)\"").squeeze()
)
df["main_cat"].unique()<jupyter_output><empty_output><jupyter_text>Next, I decode HTML entities and convert to lowercase. By "HTML entities," I mean those codes like `&amp;` you see above.<jupyter_code>steps = [lang.decode_html_entities, lang.lowercase]

df["main_cat"] = lang.chain_processors(df.loc[:, "main_cat"], steps, n_jobs=-1)
df["main_cat"].unique()<jupyter_output><empty_output><jupyter_text>Next, I examine the class balance.<jupyter_code>df["main_cat"].value_counts()<jupyter_output><empty_output><jupyter_text>Looks pretty imbalanced, but serviceable. I'll drop the extremely small classes with less than 100 samples.<jupyter_code>df = utils.prune_categories(df, column="main_cat", cut=100)<jupyter_output>Status  Support
books                         retained   985915
amazon home                   retained   501315
amazon fashion                retained   471062
sports & outdoors             retained   250249
tools & home improvement      retained   208432
toys & games                  retained   206211
automotive                    retained   196327
grocery                       retained   148844
cell phones & accessories     retained   147845
digital music                 retained   117223
movies & tv                   retained   102003
computers                     retained    98518
office products               retained    73173
all electronics               retained    70385
pet supplies                  retained    65062
industrial & scientific       retained    61787
camera & photo                retained    60324
arts, crafts & sewing         retained    60002
home audio & theater          retained    45335
musical instruments           retained  [...]<jupyter_text>I define a function below for saving the data throughout this notebook. By saving periodically, I can avoid having to rerun the whole notebook.<jupyter_code>def data_checkpoint(data, path):
    """Save data to disk."""
    path = os.path.normpath(path)

    if os.path.isdir(path):
        shutil.rmtree(path)
        print("Overwriting...")

    data.to_parquet(
        path, engine="pyarrow", index=True, partition_cols=["main_cat"],
    )

    n_files = len(glob.glob(f"{path}/*/*.parquet"))
    print(f"{n_files} files created.")
data_checkpoint(df, "data/skeleton_checkpoint")<jupyter_output>38 files created.<jupyter_text>Adding 'title' and 'brand'The next step is to load the 'title' and 'brand' features.<jupyter_code>titles = pd.read_parquet(
    "data/amazon_raw", engine="pyarrow", columns=["title", "brand", "asin"]
)
titles<jupyter_output><empty_output><jupyter_text>Then I drop duplicates from the ID column 'asin' and set it as the index.<jupyter_code>titles = titles.drop_duplicates(subset=["asin"])
titles = titles.set_index("asin")
titles.head()<jupyter_output><empty_output><jupyter_text>I slice out the intersection between `titles` and `df`, discarding any samples which I previously discarded.<jupyter_code>titles = titles.loc[titles.index.intersection(df.index)]
titles.shape<jupyter_output><empty_output><jupyter_text>Then I drop duplicates with respect to 'title' and 'brand'. Anything with the same title and brand is too close for comfort.<jupyter_code>titles = titles.drop_duplicates(subset=["title", "brand"])
titles.shape<jupyter_output><empty_output><jupyter_text>Looks like that shaved off about 20K samples. Not bad. Next, I'll survey the length distributions.<jupyter_code>fig = lang.length_dist(
    titles.sample(10 ** 6), subset="title", log_scale=True, tick_prec=1
)<jupyter_output><empty_output><jupyter_text>Short titles don't bother me, but I'm worried about the extremely long ones. Let's look at the longest.<jupyter_code>titles.at[titles.title.str.len().idxmax(), "title"]<jupyter_output><empty_output><jupyter_text>It's a long list of alphanumeric codes. They look to be SKUs of compatible products. Since I'm building a natural language model and not a repository of codes, I'll drop the ultra-long titles. I'll also go ahead and drop the ultra-short titles by shaving off 1% from each tail of the distribution.<jupyter_code>titles = lang.trim_length_outliers(titles, subset="title", q_inner=0.98)
titles<jupyter_output>C:\Users\ndgig\anaconda3\envs\capstone\lib\site-packages\ndg_tools\outliers.py:358: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated infavour of `both` or `neither`.
  inliers = data.between(lower, upper, inclusive=False) | data.isna()<jupyter_text>Much better. The longest title is now ~200 characters, and the shortest is ~10. Time to merge `titles` with `df`.<jupyter_code>df = pd.merge(df, titles, left_index=True, right_index=True)
df
del titles<jupyter_output><empty_output><jupyter_text>Filtering TextNext, I apply some basic preprocessing filters to the text. I begin by decoding HTML entities and lowercasing. If you recall, I defined a list of functions earlier called `steps`.<jupyter_code>steps
df.update(
    # Apply `steps` as a pipeline, elementwise
    lang.chain_processors(
        df.loc[:, ["description", "title", "brand"]], steps, n_jobs=-1,
    )
)
df.head()<jupyter_output><empty_output><jupyter_text>Next I'll remove HTML tags from 'title' and 'brand'. I know there are an abundance of them in this dataset.<jupyter_code>df.update(lang.strip_html_tags(df.loc[:, ["title", "brand"]], n_jobs=-1))
df.head()<jupyter_output><empty_output><jupyter_text>Dropping NullsNext, I'll drop 0-length documents after clearing any extra whitespace that may be around.<jupyter_code>df.update(lang.strip_extra_space(df, n_jobs=-1))
df.head()
# Convert 0-length strings to null
df = df.mask(df.applymap(len, "ignore") == 0)
df.isna().sum()<jupyter_output><empty_output><jupyter_text>I leave the 'brand' nulls intact for now. I will revisit them later.<jupyter_code>df = df.dropna(subset=["description", "title", "main_cat"])
df.shape<jupyter_output><empty_output><jupyter_text>Dropping DuplicatesNext, I drop duplicates again after filtering. I look for duplicates with respect to both 'title' and 'brand', and also with respect to 'description'.<jupyter_code>df = df.drop_duplicates(subset=["title", "brand"])
df = df.drop_duplicates(subset=["description"])
df.shape<jupyter_output><empty_output><jupyter_text>That shaved off tens of thousands of samples. Pretty good.Now to save the filtered text.<jupyter_code>data_checkpoint(df, "data/filter_checkpoint")<jupyter_output>37 files created.<jupyter_text>Revisiting Length OutliersI plot length distributions on a linear scale to see if anything has changed.<jupyter_code>df = pd.read_parquet("data/filter_checkpoint", engine="pyarrow")
df.head()
fig = lang.length_dist(
    df.sample(10 ** 6), subset=["description", "title", "brand"], tick_prec=1,
)
lang.other.length_info(df)<jupyter_output>len_description  len_title  len_brand
count        3,886,482  3,886,482  3,722,635
mean               971         68         15
std                615         38         10
min                 88          2          1
25%                528         41          7
50%                761         61         12
75%              1,216         85         18
max              3,878        203      1,986<jupyter_text>Looks like my filtering has resulted in a few very short titles and descriptions. I'll ignore brands for the moment, but trim the shortest 1% of titles and descriptions.<jupyter_code>df = lang.trim_length_outliers(df, subset=["title", "description"], q_lower=0.01,)
df.head()<jupyter_output>C:\Users\ndgig\anaconda3\envs\capstone\lib\site-packages\ndg_tools\outliers.py:358: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated infavour of `both` or `neither`.
  inliers = data.between(lower, upper, inclusive=False) | data.isna()<jupyter_text>I haven't trimmed the brands yet, but clearly the massive positive skew is a problem. Apparently the longest brand name is ~2K characters. I can tell just by the character count that I need to drop the longest 1%.<jupyter_code>df = lang.trim_length_outliers(df, subset="brand", q_upper=0.99,)
df.head()
fig = lang.length_dist(df.sample(10 ** 6), subset=["description", "title", "brand"])<jupyter_output><empty_output><jupyter_text>The lengths are looking pretty good, considering how skewed they were to begin with. These are plotted on a linear scale.<jupyter_code>data_checkpoint(df, "data/length_checkpoint")<jupyter_output>37 files created.<jupyter_text>Clean BrandNow I'll do my best to clean up the 'brand' feature. I expect it to have a strong relationship with category.<jupyter_code>df = pd.read_parquet("data/length_checkpoint", engine="pyarrow")
df.head()<jupyter_output><empty_output><jupyter_text>I examine the top brands.<jupyter_code>brand_counts = df["brand"].value_counts()
brand_counts.head(15)<jupyter_output><empty_output><jupyter_text>Next I put together a list of null placeholders. I'll have to add to the list whenever I spot a new one.<jupyter_code>null_brands = [
    "generic",
    "n/a",
    "no brand",
    "none",
    "not available",
    "null",
    "unavailable",
    "unbranded",
    "unknown",
    "learn more",
    "other",
    "anonymous",
]

brand_counts.loc[null_brands].sort_values()
df.loc[df.brand == "generic"]
if "generic" in null_brands:
    null_brands.remove("generic")
df.loc[df.brand.isin(null_brands), "brand"] = np.nan
df.isna().sum()
by_brands = df.brand.str.startswith("by ").dropna()
df.dropna().loc[by_brands, ["brand"]]
df["brand"] = df.loc[:, "brand"].str.replace("^(by\s+)", "", regex=True)
df.dropna().loc[by_brands, ["brand"]]
amazon_pages = df.brand.str.extract(r"visit amazon's (.+) page")
amazon_pages = amazon_pages.dropna().squeeze()
df.loc[amazon_pages.index, "brand"] = amazon_pages
df.loc[amazon_pages.index]
regex = df.brand.value_counts().reset_index()
regex.columns = ["brand", "support"]
regex["brand"] = lang.strip_non_word(regex.loc[:, "brand"])
regex = regex.loc[
    (~regex.brand.str.contains("\s", regex=True))
    & (regex.brand != "various")
    & (regex.brand.str.len() > 3)
]
regex["brand"] = r"^" + regex.brand.map(re.escape) + r"\b"
regex.head(20)
no_brand = df.loc[df.brand.isna()]
fillers = lang.locate_patterns(regex.brand.head(100), no_brand.title)
fillers.sort_values()
df.loc[fillers.index, "brand"] = fillers
df.loc[fillers.index]
df.isna().sum()
no_brand = df.isna().groupby(df.main_cat).sum()[["brand"]]
no_brand["support"] = df.main_cat.value_counts()
no_brand = no_brand.rename(columns={"brand": "null_brand"})
no_brand["frac_null"] = no_brand["null_brand"] / no_brand["support"]
no_brand.sort_values("frac_null", ascending=False).head(10).style.bar("frac_null")
df.loc[df.brand.isna(), "brand"] = ""
df.brand.value_counts().head(10)<jupyter_output><empty_output><jupyter_text>Revisiting Class Balance<jupyter_code>df = utils.prune_categories(df, "main_cat", cut=100)
data_checkpoint(df, "data/brand_checkpoint")<jupyter_output>36 files created.<jupyter_text>Adding 'feature' text<jupyter_code>df = pd.read_parquet("data/brand_checkpoint", engine="pyarrow")
df.head()
feat = dd.read_parquet(
    "data/amazon_raw", columns="feature", index="asin", engine="pyarrow"
)
feat = feat.loc[feat.index.isin(df.index)].str.join(" ")
feat.head()
df = pd.merge(df, feat.compute(), left_index=True, right_index=True, copy=False)
df.head()
del feat
display(df.index.duplicated().sum())
df = df.loc[~df.index.duplicated()]
df.index.duplicated().sum()
steps = [lang.decode_html_entities, lang.lowercase, lang.strip_html_tags]
df["feature"] = lang.chain_processors(df.loc[:, "feature"], steps, n_jobs=-1)
df["feature"].head()
lang.length_info(df["feature"])
fig = lang.length_dist(df.sample(10 ** 4, random_state=39), subset=["feature"],)
df.at[df.feature.str.len().idxmax(), "feature"]
df = lang.trim_length_outliers(df, subset="feature", method="iqr")<jupyter_output>len_description  len_title  len_brand  len_main_cat  len_feature
count        3,299,764  3,299,764  3,299,764     3,299,764    3,299,764
mean               979         68         11            12          198
std                618         38          6             6          253
min                384         13          0             4            0
25%                533         41          7             5            0
50%                767         61         11            11          114
75%              1,221         86         15            15          316
max              3,877        202         39            28        1,100


             n_trimmed  pct_trimmed
description          0            0
title                0            0
brand                0            0
main_cat             0            0
feature        450,573           12
total_obs      450,573           12<jupyter_text>Next, I delete everything between brackets or parentheses.<jupyter_code>longest_feat = df.feature.str.len().idxmax()
df.at[longest_feat, "feature"]
df["feature"] = df.loc[:, "feature"].str.replace(r"([{\(].+[\)}])", "", regex=True)
df.at[longest_feat, "feature"]
df["feature"].fillna("", inplace=True)
df.head()
data_checkpoint(df, "data/feature_checkpoint")<jupyter_output>36 files created.<jupyter_text>Constructing Model Text<jupyter_code>df = dd.read_parquet("data/feature_checkpoint", engine="pyarrow", index="asin")
df.head()
text_schema = "{title}. {description}. {feature}. {brand}."


def construct_text(item, schema=text_schema):
    text = schema.format(**item)
    return lang.strip_extra_periods(text)


construct_text(df.head(1).squeeze())
# Apply function row-wise
df["text"] = df.apply(construct_text, meta=str, axis=1)
df["text"].head()<jupyter_output>[########################################] | 100% Completed |  3.4s
[########################################] | 100% Completed |  3.5s
[########################################] | 100% Completed |  3.6s<jupyter_text>Export<jupyter_code>df.to_parquet(
    "data/amazon_clean",
    engine="pyarrow",
    schema="infer",
    write_index=True,
    overwrite=True,
)
conf = input("Delete checkpoints? (y/n): ")
if strtobool(conf):
    for path in glob.glob("data/*_checkpoint"):
        shutil.rmtree(path)
        print(f"'{path}' deleted.")
else:
    print(f"Checkpoints not deleted.")<jupyter_output>Delete checkpoints? (y/n): y
'data\brand_checkpoint' deleted.
'data\feature_checkpoint' deleted.
'data\filter_checkpoint' deleted.
'data\length_checkpoint' deleted.
'data\skeleton_checkpoint' deleted.
{'fraction_non_alphanumeric': 0.1238677536231884, 'fraction_numerical': 0.02441123188405797, 'mean_word_length': 4.201648998822144, 'pattern_counts': {'":': 1, '<': 158, '<?xml version=': 0, '>': 158, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '398912', 'license': 'ADSL', 'n_tokens_mistral': 7402, 'n_tokens_neox': 6950, 'n_words': 1910, 'path': 'big_clean.ipynb', 'repo_name': 'ndgigliotti/dsc-capstone', 'size': '22096'}