<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Setting up Environment2&nbsp;&nbsp;Initializing Storage Variables3&nbsp;&nbsp;Wikipedia Scraping3.1&nbsp;&nbsp;Extrating Chinese food names3.2&nbsp;&nbsp;Extracting all other cuisines3.3&nbsp;&nbsp;Data Cleaning4&nbsp;&nbsp;html Scraping4.1&nbsp;&nbsp;Adding Japanese food from html manually4.2&nbsp;&nbsp;Adding Korean food from html with beautiful soup4.3&nbsp;&nbsp;Exporting Corpus5&nbsp;&nbsp;References  Setting up Environment<jupyter_code>import pandas as pd
import numpy as np
# import matplotlib.pyplot as plt
import wikipedia<jupyter_output><empty_output><jupyter_text>Initializing Storage Variables<jupyter_code>df_corpus = pd.DataFrame(columns=["Food", "Cuisine"])
df_corpus
cuisines = ["Chinese", "Malay", "Indian", "Cross-cultural",
            "Seafood", "Fruit", "Desserts", "Drinks and beverages"]<jupyter_output><empty_output><jupyter_text>Wikipedia ScrapingDocumentation at https://wikipedia.readthedocs.io/en/latest/code.html<jupyter_code>wiki = wikipedia.page("Singaporean cuisine")
links = wiki.links
wiki.sections<jupyter_output><empty_output><jupyter_text>Extrating Chinese food names<jupyter_code>txt_chinese = wiki.section("Chinese") #section(section_title)
txt_chinese
len(txt_chinese)
def get_corpus(txt, start_chars, end_chars):
    start, end = 0, 0
    corpus = []
    new = False
    for i in range(len(txt)):
        if txt[i] in start_chars:
            start = i+1 # start of food name just after start_char
            new = True # found start of new word flag
        if txt[i] in end_chars:
            end = i # end of food name, non inclusive of txt[i]
        if new and end > start:
            while txt[start] == " ":
                start += 1 # remove space at start
            while txt[end-1] == " ":
                end -= 1 # remove space at end
            corpus.append(txt[start: end].lower()) # change to lower case
            new = False # word copied out, prevents duplicates
    return corpus<jupyter_output><empty_output><jupyter_text>We notice that Chinese cuisine is the only cuisine where all food names are bounded by ```'\n'``` and ```'('``` characters, (eg. \nSliced fish soup (鱼片汤; yú piàn tāng)) we shall extract it step by step as an example.<jupyter_code>corpus_chinese = get_corpus(txt_chinese, ['\n'], ['(', '/'])
corpus_chinese
corpus_chinese = corpus_chinese[1:]
corpus_chinese_size = len(corpus_chinese)
print(corpus_chinese_size)
df_corpus = df_corpus.append(pd.DataFrame({"Food":corpus_chinese,
                                          "Cuisine":["Chinese"]*len(corpus_chinese)}))
df_corpus<jupyter_output><empty_output><jupyter_text>Extracting all other cuisinesBy viewing the wikipedia page, it is observed that all other cuisines have food names are bounded by ```'\n'```, ```','``` and ```'/'``` characters.Similarly, we will expect the section summary to have a substring bounded by the same characters and will need to remove those.<jupyter_code>for cuisine in cuisines[1:]: # exclude "Chinese"
    txt = wiki.section(cuisine) #section(section_title)
    corpus = get_corpus(txt, ['\n'], [',', '(', '/'])
    print("\n", cuisine, ", Corpus size =", len(corpus))
    print(corpus)<jupyter_output>Malay , Corpus size = 27
['acar', 'assam pedas', 'ayam penyet', 'bakso', 'begedil', 'curry puff', 'dendeng paru', 'goreng pisang', 'gulai daun ubi', 'keropok', 'ketupat', 'lemak siput', 'lontong', 'nagasari', 'nasi goreng', 'nasi padang', 'otak-otak', 'pecel lele', 'rawon', 'rojak bandung', 'roti john', 'sambal', 'satay', 'sayur lodeh', 'soto', 'soto ayam', 'tumpeng']

 Indian , Corpus size = 10
['appam', 'dosa', 'murtabak', 'naan', 'roti prata', 'soup kambing', 'soup tulang', 'soup tulang merah', 'tandoori chicken', 'vadai']

 Cross-cultural , Corpus size = 20
['ayam buah keluak', 'biryani', 'cereal prawns', 'chili crab pasta', 'curry laksa', 'fish head curry', 'kari debal', 'kari lemak ayam', 'katong laksa', 'kueh pie tee', 'kway teow goreng', 'mee rebus', 'mee siam', 'mee goreng', 'mee soto', 'rojak', 'sambal kangkong', 'satay bee hoon', 'tauhu goreng', '"western food" in hawker centres where "singapore-style" chicken chop']

 Seafood , Corpus size = 5
['black pepper crab', 'chill[...]<jupyter_text>From the results above, we realised that we will only need to remove the summary text (the first element of the corpus) for "Fruit", and the last element for "Cross-cultural".However, to make our code reusable, we shall fulter the food names by length instead. Let's use 40 characters as a filter.Running the above loop again, we have:<jupyter_code>corpus_size = corpus_chinese_size
for cuisine in cuisines[1:]: # exclude "Chinese"
    txt = wiki.section(cuisine) #section(section_title)
    corpus = get_corpus(txt, ['\n'], [',', '(', '/'])
    corpus = list(filter(lambda x: len(x) <= 40, corpus))
    print("\n", cuisine, ", Corpus size =", len(corpus))
    print(corpus)
    
    corpus_size += len(corpus)
    df_corpus = df_corpus.append(pd.DataFrame({"Food":corpus,
                                          "Cuisine":[cuisine]*len(corpus)}))<jupyter_output>Malay , Corpus size = 27
['acar', 'assam pedas', 'ayam penyet', 'bakso', 'begedil', 'curry puff', 'dendeng paru', 'goreng pisang', 'gulai daun ubi', 'keropok', 'ketupat', 'lemak siput', 'lontong', 'nagasari', 'nasi goreng', 'nasi padang', 'otak-otak', 'pecel lele', 'rawon', 'rojak bandung', 'roti john', 'sambal', 'satay', 'sayur lodeh', 'soto', 'soto ayam', 'tumpeng']

 Indian , Corpus size = 10
['appam', 'dosa', 'murtabak', 'naan', 'roti prata', 'soup kambing', 'soup tulang', 'soup tulang merah', 'tandoori chicken', 'vadai']

 Cross-cultural , Corpus size = 19
['ayam buah keluak', 'biryani', 'cereal prawns', 'chili crab pasta', 'curry laksa', 'fish head curry', 'kari debal', 'kari lemak ayam', 'katong laksa', 'kueh pie tee', 'kway teow goreng', 'mee rebus', 'mee siam', 'mee goreng', 'mee soto', 'rojak', 'sambal kangkong', 'satay bee hoon', 'tauhu goreng']

 Seafood , Corpus size = 5
['black pepper crab', 'chilli crab', 'oyster omelette', 'sambal lala', 'sambal stingray']

 Fruit , C[...]<jupyter_text>Verifying that our corpus has been created correctly, we check that the size of the created dataframe has the same size as ```corpus_size```<jupyter_code>print("Dataframe size =", df_corpus.shape)
print("Expected corpus size =", corpus_size)<jupyter_output>Dataframe size = (112, 2)
Expected corpus size = 112<jupyter_text>Data Cleaning From a visual inspection of the corpus, we found that the following food names can be modified:**Chinese**- 'beef kway teow', 'hainanese chicken rice', 'shredded chicken noodles' *- add alternative name***Malay**- 'nasi goreng', 'nasi padang', 'roti john' *- add alternative name***Indian**- 'roti prata' *- add alternative name***Desserts:**- 'kuih or kueh' *- change to alternative name*- 'kueh lapis is a rich'- 'lapis sagu is also a popular kueh with layers of alternating colour and a sweet'**Drinks and beverages:**- 'chin chow drink', 'yuenyeung coffee' *- add alternative name*<jupyter_code>df_corpus["alternative names"] = df_corpus["Food"]
wrong_names = ['kueh lapis is a rich',
               'lapis sagu is also a popular kueh with layers of alternating colour and a sweet',
              'kuih or kueh']

# Dropping wrong rows
df_corpus = df_corpus[~df_corpus["Food"].isin(wrong_names)]
df_corpus.shape
# Adding corrected names
names = ['kueh lapis', 'lapis sagu']
df_corpus = df_corpus.append(pd.DataFrame({"Food": names,
                                           "alternative names": names,
                                           "Cuisine":['Desserts']*2}))

# Adding Alternative names
cus = 'Chinese'
names = ['beef kway teow', 'hainanese chicken rice', 'shredded chicken noodles']
alternative = ['kway teow', 'chicken rice', 'chicken noodles']

df_corpus = df_corpus.append(pd.DataFrame({"Food":names,
                                           "alternative names": alternative,
                                           "Cuisine":[cus]*len(alternative)}))

cus = 'Malay'
names = ['nasi goreng', 'nasi padang', 'nasi goreng', 'nasi padang', 'roti john']
alternative = ['nasigoreng', 'nasipadang', 'nasi-goreng', 'nasi-padang', 'roti-john']

df_corpus = df_corpus.append(pd.DataFrame({"Food":names,
                                           "alternative names": alternative,
                                           "Cuisine":[cus]*len(alternative)}))

cus = 'Indian'
names = ['roti prata', 'roti prata', 'roti prata']
alternative = ['rotiprata', 'roti-prata', 'prata']

df_corpus = df_corpus.append(pd.DataFrame({"Food":names,
                                           "alternative names": alternative,
                                           "Cuisine":[cus]*len(alternative)}))

cus = 'Desserts'
names = ['kueh', 'kueh']
alternative = ['kuih', 'kueh']

df_corpus = df_corpus.append(pd.DataFrame({"Food":names,
                                           "alternative names": alternative,
                                           "Cuisine":[cus]*len(alternative)}))

cus = 'Drinks and beverages'
names = ['chin chow drink', 'chin chow drink', 'chin chow drink', 'yuenyeung coffee']
alternative = ['chin chow', 'chinchow', 'chin-chow', 'yuenyeung']

df_corpus = df_corpus.append(pd.DataFrame({"Food":names,
                                           "alternative names": alternative,
                                           "Cuisine":[cus]*len(alternative)}))

corpus_size = df_corpus.shape[0]
df_corpus.shape<jupyter_output><empty_output><jupyter_text>Finally, we generate some summary statistics of our corpus<jupyter_code>print(df_corpus.shape)
print(df_corpus["Cuisine"].value_counts())<jupyter_output>(129, 3)
Chinese                 36
Malay                   32
Cross-cultural          19
Drinks and beverages    13
Indian                  13
Desserts                11
Seafood                  5
Name: Cuisine, dtype: int64<jupyter_text>html Scraping<jupyter_code>import codecs
def get_corpus(txt, start_str, end_str):
    start, end = 0, 0
    corpus = []
    new = False
    i=0
    
    while len(txt) > len(end_str):
        start = txt.find(start_str) + len(start_str)
        new = True # found start of new word flag
        end = txt.find(end_str)
        
        if new and end > start and end-start < 40:
            while txt[start] == " ":
                start += 1 # remove space at start
            while txt[end-1] == " ":
                end -= 1 # remove space at end
            corpus.append(txt[start: end].lower()) # change to lower case
            new = False # word copied out, prevents duplicates
        
            #update search space when word is found
            txt = txt[end:]
            i+=1
        elif end > start and end-start >= 40:
            txt = txt[start+len(start_str):] # for start to be updated
        else:
            txt = txt[start:] # end < start, for end to be updated
        
        if start == -1 or end == -1:
            break
    print("found", i, "food names")
    return corpus<jupyter_output><empty_output><jupyter_text>Adding Japanese food from html manually<jupyter_code>cuisine = "Japanese"
f = codecs.open("japfood.html", 'r', 'utf-8')
txt = f.read()
txt
corpus = get_corpus(txt, ". ", "</h")
corpus = list(filter(lambda x: len(x) <= 40, corpus))
print("\n", cuisine, ", Corpus size =", len(corpus))
print(corpus)
corpus = corpus[:-1]
corpus
# Check for duplicate names
for x in df_corpus["alternative names"]:
    for food in corpus:
        if food in x or x in food:
            print("'" + x + "',")
corpus_size += len(corpus)
df_corpus = df_corpus.append(pd.DataFrame({"Food":corpus,
                                          "Cuisine":[cuisine]*len(corpus),
                                          "alternative names": corpus}))

print("Dataframe size =", df_corpus.shape)
print("Expected corpus size =", corpus_size)<jupyter_output>Dataframe size = (159, 3)
Expected corpus size = 159<jupyter_text>Adding Korean food from html with beautiful soup<jupyter_code>from urllib.request import urlopen
from bs4 import BeautifulSoup

cuisine = "Korean"

# html = urlopen('koreanfood.html').read().decode("utf-8")
html = f = codecs.open("koreanfood.html", 'r', 'utf-8')

# using beautifulsoup
bs = BeautifulSoup(html, 'html.parser')
corpus = list(map(lambda x: x.get_text(), bs.find_all(["h1"])))
corpus
corpus = corpus[1:] # removing article title
alternative = []

# Removing start and end extra characters
start_str = '. '
end_str = ' ('
for i in range(len(corpus)):
    x = corpus[i]
    corpus[i] = x[x.find(start_str) + len(start_str):]
    alternative.append(x[x.find(start_str) + len(start_str): x.find(end_str)])
alternative<jupyter_output><empty_output><jupyter_text>Further finetuning and adding alternative names<jupyter_code>corpus = ['Chili Pickled Cabbage (Kimchi 김치)',
            'Samgyeopsal (삼겹살)',
            'Pork Bulgogi (Daeji Bulgogi 불고기)',
            'Korean Barbecue (Gogigui 고기구이)',
            'Korean Barbecue (Gogigui 고기구이)',
            'Hangover Stew (Haejangguk 해장국)',
            'Soft Tofu Stew (Sundubu\xa0Jjigae 순두부찌게)',
            'Mixed Seafood Stew',
            'Kimchi Stew (Kimchi Jjigae\xa0김치찌개)',
            'Fish Stew (Saengseon Jjigae 생선찌개)',
            'Spicy Stir Fried Octopus (Nakji Bokkeum\xa0낙지볶음)',
            'Korean Ox Bone Soup (Seolleongtang\xa0설렁탕)',
            'Hotpot Mixed Rice (Dolsot Bibimbap\xa0돌솥 비빔밥)',
            'Korean Mixed Rice (Cold Bibimbap 비빔밥)',
            'Steamed Mandu Dumplings (Jjinmandu 찐만두)',
            'Deep Fried Mandu (Yaki Mandu)',
            'Noodles in Ice Soup (Mul Naengmyeon 물 냉면)',
            'Mixed Cold Noodles (Bibim Naengmyeon 비빔 냉면)',
            'Kimchi Fried Rice (Kimchi Bokkeumbap\xa0김치 볶음밥)',
            'Fried Sweet Potato Noodles (Japchae 잡채)',
            'Mung Bean Pancake (Bindaetteok 빈대떡)',
            'Korean Blood Sausage (Soondae 순대)',
            'Octopus Mixed Plate',
            'Gimbap 김밥',
            'Korean Chicken Skewers (Dakkochi 닭꼬치)',
            'Korean Side Dishes (Banchan반찬)',
            'Tornado Potatoes',
          'Tornado Potatoes',
            'Gooey Deep Fried Snack (Hotteok 호떡)',
            'Korean Tempura (Twigim 튀김)',
            'Red Rice Cakes (Tteokbokki 떡볶이)']
alternative = ['Chili Pickled Cabbage',
                'Samgyeopsal',
                'Bulgogi',
                'Korean Barbecue',
                'Korean BBQ',
                'Hangover Stew',
                'Soft Tofu Stew',
                'Mixed Seafood Stew',
                'Kimchi Stew',
                'Fish Stew',
                'Fried Octopus',
                'Ox Bone Soup',
                'Hotpot Mixed Rice',
                'Korean Mixed Rice',
                'Mandu Dumplings',
                'Deep Fried Mandu',
                'Ice Soup',
                'Cold Noodles',
                'Kimchi Fried Rice',
                'Sweet Potato Noodles',
                'Mung Bean Pancake',
                'Korean Blood Sausage',
                'Octopus Mixed Plat',
                'Gimbap',
                'Korean Chicken Skewers',
                'Korean Side Dishes',
                'Tornado Potatoe',
               'Tornado Potato',
                'Gooey Deep Fried Snack',
                'Korean Tempura',
                'Red Rice Cakes']
# Check for duplicate names
for x in df_corpus["alternative names"]:
    for food in alternative:
        if food in x or x in food:
            print("'" + x + "',")
corpus_size += len(corpus)
df_corpus = df_corpus.append(pd.DataFrame({"Food":corpus,
                                          "Cuisine":[cuisine]*len(corpus),
                                          "alternative names": corpus}))

print("Dataframe size =", df_corpus.shape)
print("Expected corpus size =", corpus_size)<jupyter_output>Dataframe size = (190, 3)
Expected corpus size = 190<jupyter_text>Exporting Corpus<jupyter_code>df_corpus.to_csv("../Instagram/" + "corpus_wikipedia.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.131518428740133, 'fraction_numerical': 0.007085586425508111, 'mean_word_length': 2.9072365225837786, 'pattern_counts': {'":': 32, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '106636', 'license': 'Apache-2.0', 'n_tokens_mistral': 5642, 'n_tokens_neox': 5321, 'n_words': 1449, 'path': 'Code/.ipynb_checkpoints/Corpus_Wikipedia-checkpoint.ipynb', 'repo_name': 'terenceneo/Social-Analytics', 'size': '16089'}