<jupyter_start><jupyter_text>Fuzzy_match_of_public- <NAME> & <NAME> Problem describtionThe aim of this project is to pick the public companies(have listed on American stock exchanges) from the original 222,450 companies(some are repeated). The file **work1.csv** is the work file and the **work2** is the public companies(more than 6,000) in United States' share exchanges. Mainly we apply *fuzzy wuzzy* algorithm and picked the required ones. https://github.com/seatgeek/fuzzywuzzy (developped by SeatGeet)**There are two main difficulties. **- The first names in the file with unchecked companies are not very formal, say **'Apple company '** instead of **'Apple Inc'**. We designed the *fuzzy wuzzy* value by making experiments manually in advance. - The second is massive data leads to much running time (it is estimated more than **12 hours** for the first version programe). We reduced it to **less than five minutes** by devide the matching procedure into two steps after carefully observed the data structure.<jupyter_code>import pandas as pd
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import warnings
warnings.filterwarnings('ignore')
# fuzz.token_set_ratio("Advanced Semiconductor Engineering, Inc.", "Advance Pain Centers"    )
#1.import company data we want search
co = pd.read_csv("work1.csv", sep=',')
# co.head()
co.describe()
#import public companies data
share = pd.read_csv("work2.csv")
share.sort_values('sharename', inplace=True)
share.reset_index(inplace=True)
share.drop('index', axis=1, inplace=True)
share
#2.make company name unique
co_name = co[['coname']]
co_name.drop_duplicates(inplace=True)
co_name.sort_values('coname', inplace=True)
co_name.reset_index(inplace=True)
co_name.drop(['index'], axis=1, inplace=True)
co_name.head()
co_name.describe()
#3. use fuzzy match algorithm 
import time
t0 = time.time()

pre=0
name2sharecode = {}

offset = 0
limit = len(co_name)

for j, co_row in co_name.iloc[offset: offset + limit].iterrows():
    cur = round((j - offset) / limit * 100)
    if cur > pre:
        pre = cur
        print(f'{cur}%...')   #give progress bar
    
    share_pre2 = share[share['sharename'].str.upper().str.startswith(co_row['coname'][0:2])]
    
    #match company names
    for i, share_row in share_pre2.iterrows():
        if fuzz.token_set_ratio(share_row['sharename'], co_row['coname']) > 75:
            name2sharecode[co_row['coname']] = share_row['sharecode']
            print(i, j)
            break
print(time.time() - t0)
import json
json.dump(name2sharecode, open('name2sharecode{}.json'.format(offset), 'w'))
name2sharecode
len(name2sharecode)
co['result'] = [name2sharecode.get(co_row['coname'], 'nopublic') for _, co_row in co.iterrows()]
co['result'].head()

#write back
co['result'].describe()
co[co['result'] != 'nopublic']
co.to_csv("./jy_project.csv")
#write to csv<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12864077669902912, 'fraction_numerical': 0.014216366158113732, 'mean_word_length': 5.035564853556485, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '446742', 'license': 'MIT', 'n_tokens_mistral': 962, 'n_tokens_neox': 919, 'n_words': 284, 'path': 'Fuzzy_match_of_public_companies.ipynb', 'repo_name': 'jinwangjoshua/Fuzzy-match-on-listed-companies', 'size': '2891'}