<jupyter_start><jupyter_text>Data analisysSome preliminary analisys of the dataset<jupyter_code>print("First 5 rows")
print("------------")
df.head()
print("No. of rows and columns")
print("-----------------------")
df.shape
print("Check null values")
print("-----------------")
df.isnull().any().any()
print("Check duplicate values")
print("----------------------")
len(df['url'].unique()) != df.shape[0]
print("DataFrame column types")
print("----------------------")
df.info()
print("Some stats")
print("----------------")
df.describe()<jupyter_output>Some stats
----------------<jupyter_text>Page labellingLabel (possibly) each page with its own template. Let's start with product (books) pages. The following URL links to a page which shows info regarding a book:```https://blackwells.co.uk/bookshop/product/Players-Handbook-by-Jeremy-Crawford-author-James-Wyatt-author-Robert-J-Schwalb-author-Bruce-R-Cordell-author/9780786965601```So we could infer that each product page has the following url schema:```https://blackwells.co.uk/bookshop/product/```<jupyter_code>import re

#Urls associated with product pages
testStrings1= ['https://blackwells.co.uk/bookshop/product/Dungeons--Dragons-Starter-Set-Six-Dice-Five-Ready-to-Play-DD-Characters-With-Character-Sheets-a-Rulebook-and-One-Adventure-by-Wizards-RPG-Team-author/9780786965595',
              'https://blackwells.co.uk/bookshop/product/Education-by-Gary-Thomas-author/9780199643264',
              'https://blackwells.co.uk/bookshop/product/Game-of-Thrones-House-Stark-Ruled-Notebook-Reformat-by-Insight-Editions-author/9781683836650',
              'https://blackwells.co.uk/bookshop/product/The-World-of-Peter-Rabbit-by-Beatrix-Potter-author/9780723275923',
              'https://blackwells.co.uk/bookshop/product/Happy-Old-Me-by-DAVIES-HUNTER/9781472626028',
              'https://blackwells.co.uk/bookshop/product/Hey-Warrior-by-Karen-Young-Norvile-Dovidonyte-artist/9781912678037']

#Various urls which aren't product pages
testStrings2 = ['https://blackwells.co.uk/bookshop/bestsellers',
               'https://blackwells.co.uk/rarebooks/book.php?id=55981',
               'https://blackwells.co.uk/bookshop/category/_computing/',
               'https://blackwells.co.uk/bookshop/search/author/Subrata%20Dasgupta%20(author)',
               'https://blackwells.co.uk/bookshop/search/?productTitle=game+of+thrones&quantity=6&author=&quantity=6&publisher=&quantity=6&keyword=&quantity=6&pubDateFrom=&pubDateTo=&isbn=&priceFrom=&priceTo=&academicLevel=E',
               'https://blackwells.co.uk/bookshop/category/WN/',
               'https://blackwells.co.uk/bookshop/category/WQ/',
               'https://blackwells.co.uk/bookshop/search/?keyword=',
               'https://blackwells.co.uk/bookshop/basket',
               'https://blackwells.co.uk/bookshop/events/',
               'https://blackwells.co.uk/bookshop/collection/Gift-and-Stationery-bestsellers/-186139',
               'https://blackwells.co.uk/bookshop/search/?keyword=game+of+thrones',
               'https://blackwells.co.uk/bookshop/category/_travel/']

testStrings = testStrings1 + testStrings2

productRegex = r'^https\:\/\/blackwells\.co\.uk\/bookshop\/product\/.+$'

# simple tests
print("Testing article regex:")
print("Only the first 6 urls should be matching with the article regex")
print("---------------------------------------------------------------")

for uri in testStrings:
    print(re.match(productRegex, uri))
    
print('---------------------------------------------------------------')<jupyter_output>Testing article regex:
Only the first 6 urls should be matching with the article regex
---------------------------------------------------------------
<re.Match object; span=(0, 205), match='https://blackwells.co.uk/bookshop/product/Dungeon>
<re.Match object; span=(0, 87), match='https://blackwells.co.uk/bookshop/product/Educati>
<re.Match object; span=(0, 134), match='https://blackwells.co.uk/bookshop/product/Game-of>
<re.Match object; span=(0, 106), match='https://blackwells.co.uk/bookshop/product/The-Wor>
<re.Match object; span=(0, 85), match='https://blackwells.co.uk/bookshop/product/Happy-O>
<re.Match object; span=(0, 108), match='https://blackwells.co.uk/bookshop/product/Hey-War>
None
None
None
None
None
None
None
None
None
None
None
None
None
---------------------------------------------------------------<jupyter_text>Next we have to label pages which show list of books. We noted that books in blackwells.co.uk are grouped by category, so that books which share the same category are listed in the same page in a paginated way. ```https://blackwells.co.uk/bookshop/category/_travel/```So urls referring to categories (seems to) have the following schema:```https://blackwells.co.uk/bookshop/category/```Note that also search results (except `https://blackwells.co.uk/bookshop/search/?keyword=` and `https://blackwells.co.uk/bookshop/search/`) are paginated:```https://blackwells.co.uk/bookshop/search/?keyword=game+of+thrones```In that case the url schema follows this pattern:```https://blackwells.co.uk/bookshop/search/```<jupyter_code>catalogRegex = r'^https\:\/\/blackwells\.co\.uk\/bookshop\/(category\/.+|(search\?.+|search\/\?keyword\=.+|search\/(?!(\?keyword\=))))'

print("Testing catalog regex:")
print("Only 7 urls should be matching with the bestsellers regex")
print("-----------------------------------------------------------")

for uri in testStrings:
    print(re.match(catalogRegex, uri))
print('---------------------------------------------------------------')<jupyter_output>Testing catalog regex:
Only 7 urls should be matching with the bestsellers regex
-----------------------------------------------------------
None
None
None
None
None
None
None
None
<re.Match object; span=(0, 54), match='https://blackwells.co.uk/bookshop/category/_compu>
<re.Match object; span=(0, 41), match='https://blackwells.co.uk/bookshop/search/'>
<re.Match object; span=(0, 41), match='https://blackwells.co.uk/bookshop/search/'>
<re.Match object; span=(0, 46), match='https://blackwells.co.uk/bookshop/category/WN/'>
<re.Match object; span=(0, 46), match='https://blackwells.co.uk/bookshop/category/WQ/'>
None
None
None
None
<re.Match object; span=(0, 65), match='https://blackwells.co.uk/bookshop/search/?keyword>
<re.Match object; span=(0, 51), match='https://blackwells.co.uk/bookshop/category/_trave>
---------------------------------------------------------------<jupyter_text>Add `label` columnLabelling each possible row of the dataframe<jupyter_code>df['label'] = None
df.head()
def set_label(url, regex, old_label, new_label):
    if re.match(regex, url):
        if old_label is None:
            return new_label
        else:
            return 'collision'
    else:
        return old_label<jupyter_output><empty_output><jupyter_text>Add product label<jupyter_code>df['label'] = df.apply(lambda x: set_label(x['url'], productRegex, x['label'], 'product'), axis=1)
print("There should be no row with 'collision' label")
print("---------------------------------------------")
df[df['label']=='collision'].head()<jupyter_output>There should be no row with 'collision' label
---------------------------------------------<jupyter_text>Add list label for paginated pages<jupyter_code>df['label'] = df.apply(lambda x: set_label(x['url'], catalogRegex, x['label'], 'list'), axis=1)
print("There should be no row with 'collision' label")
print("---------------------------------------------")
df[df['label']=='collision'].head()<jupyter_output>There should be no row with 'collision' label
---------------------------------------------<jupyter_text>Summary:<jupyter_code>fmt_string = 'There are {} row with {} label'
print(fmt_string.format(len(df[df['label'].isnull()]),'no'))
print(fmt_string.format(len(df[df['label']=='product']), 'product'))
print(fmt_string.format(len(df[df['label']=='list']), 'list'))
df[df['label'].isnull()][['url', 'referer_url']].head(20)<jupyter_output><empty_output><jupyter_text>Note that there are some rows which aren't labeled. That is because these pages aren't generated neither by the 'product' template nor the 'list' template.  Write dataset to a file<jupyter_code>df.to_csv(FILEPATH, encoding='utf-8', index=False)
test = pd.read_csv(FILEPATH)
test.head()
test.shape
print(fmt_string.format(len(test[test['label'].isnull()]),'no'))
print(fmt_string.format(len(test[test['label']=='product']), 'product'))
print(fmt_string.format(len(df[df['label']=='list']), 'list'))<jupyter_output>There are 20 row with no label
There are 10405 row with product label
There are 494 row with list label
{'fraction_non_alphanumeric': 0.24460681976339596, 'fraction_numerical': 0.020644861980978892, 'mean_word_length': 3.046457062412013, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 40, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '124642', 'license': 'MIT', 'n_tokens_mistral': 2845, 'n_tokens_neox': 2659, 'n_words': 508, 'path': 'notebooks/datasetLabeling/02-setTemplateLabel/blackwells.ipynb', 'repo_name': 'a-wars/AGIW_First_Project', 'size': '8622'}