<jupyter_start><jupyter_text>Names Database This database will serve two purposes:* To enable record linkage in order to standarize authors names which may have mild variations in writing between different years or posters.* See which authors correspond to science and technology personel in order to exclude foreign authors, undergraduate students and others.According to the [data source](https://datos.gob.ar/dataset/mincyt-personal-ciencia-tecnologia), the following description of the dataset is given:> "Conjunto de datos que describe mediante atributos tales como nombre, apellido, sexo, edad, a las personas físicas."<jupyter_code>personas = pd.read_csv('../CyT_Datasets/personas.csv', sep = ';', low_memory = False,
                      encoding='UTF-8')<jupyter_output><empty_output><jupyter_text>We examine the dataset a bit closer<jupyter_code>personas.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 185618 entries, 0 to 185617
Data columns (total 6 columns):
 #   Column              Non-Null Count   Dtype  
---  ------              --------------   -----  
 0   persona_id          185618 non-null  object 
 1   nombre              183411 non-null  object 
 2   apellido            182285 non-null  object 
 3   sexo_id             128420 non-null  float64
 4   edad                128420 non-null  float64
 5   cvar_ultimo_acceso  128330 non-null  object 
dtypes: float64(2), object(4)
memory usage: 8.5+ MB<jupyter_text>There are 185 thousand unique ids in the database which would supposedly translate to 185 thousand people. We want the names and the sex of each individual which means we only have 183 thousand names and 128 thousand non-null sex rows. Since we are limited by the lowest of these values which is the sex column, we only have data for 128 thousand individuals. Thus, we remove all the rows for which the sex columns is null.<jupyter_code>personas_limpia = personas[personas['sexo_id'].notna()]<jupyter_output><empty_output><jupyter_text>Generate a new pandas series from the union of the firstname and lastname of every person<jupyter_code>nombres_personas = personas_limpia['nombre'] + ' ' + personas_limpia['apellido']<jupyter_output><empty_output><jupyter_text>Names Cleaning Transform the names to lowercase<jupyter_code>nombres_personas = nombres_personas.str.lower()<jupyter_output><empty_output><jupyter_text>Recode strange characters<jupyter_code>nombres_personas = nombres_personas.apply(lambda x: unidecode(x))<jupyter_output><empty_output><jupyter_text>Perform a last cleaning step using a regex pattern<jupyter_code>nombres_personas_clean = nombres_personas.apply(lambda x: re.sub(r'[^a-zA-Z]', r'', string = x))<jupyter_output><empty_output><jupyter_text>This last variable contains the names withouth any whitespace and it will be the one we will use in our tf-idf implementation *** We must use the data frame that contains data for all posters combined so we load the corresponding csv file altough we could use the output from the extraction.py file it would take an unnecessary long time.<jupyter_code>all_posters_data = pd.read_csv('../SAN_csv/all_posters.csv')<jupyter_output><empty_output><jupyter_text>Since we only need one sample of each author name we remove duplicated names from the data frame.<jupyter_code>autores_sin_dup = all_posters_data['autor'].drop_duplicates()<jupyter_output><empty_output><jupyter_text>The 'author' column contains the authors names as they were parsed from the posters but these strings must be normalized.First we convert to lowercase.<jupyter_code>name_spam = autores_sin_dup.str.lower()<jupyter_output><empty_output><jupyter_text>Then we strip the trailing whitespaces, newlines, etc.<jupyter_code>name_spam = name_spam.str.strip()<jupyter_output><empty_output><jupyter_text>Remove any numbers and the character '°' which appeared frequently next to the numbers that indicated affiliation.<jupyter_code>name_spam = name_spam.apply(lambda x: re.sub(r'[\d°]', r'', string = str(x)))<jupyter_output><empty_output><jupyter_text>Solve coding issues like accents and other strange characters<jupyter_code>name_spam = name_spam.apply(lambda x: unidecode(str(x)))<jupyter_output><empty_output><jupyter_text>Lastly we include only letters and remove any whitespace to use these strings in our tf-idf implementation.<jupyter_code>name_spam_clean = name_spam.apply(lambda x: re.sub(r'[^a-zA-Z]', r'', string = str(x)))<jupyter_output><empty_output><jupyter_text>***  Record Linkage Implementation  Tf-idf and K-Nearest Neighbors The ngrams function separates the strings into trigrams by default, thus returning a list of ngrams for every string. It will be useful to use trigrams since many names may have spelling errors and other mistakes thus impeding correct matching when separating by entire words.<jupyter_code>def ngrams(string, n = 3):
    ngrams = zip(*[string[i:] for i in range(n)])
    return [''.join(ngram) for ngram in ngrams]<jupyter_output><empty_output><jupyter_text>We initialize the tf-idf class, pass it the ngrams function we just made and create the tf-idf matrix from the clean names.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase= False)
tf_idf_matrix = vectorizer.fit_transform(nombres_personas_clean)<jupyter_output><empty_output><jupyter_text>We initiliaze the NearesNeighbors class using only one neighbor and fitting the tf-idf matrix.<jupyter_code>from sklearn.neighbors import NearestNeighbors
nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tf_idf_matrix)<jupyter_output><empty_output><jupyter_text>The getNearestN function will get a query which will one of the strings we want to match and will return the index of the closest match from the clean names variable and the distance to that point.<jupyter_code>def getNearestN(query):
    queryTFIDF_ = vectorizer.transform(query)
    distances, indices = nbrs.kneighbors(queryTFIDF_)
    return distances, indices<jupyter_output><empty_output><jupyter_text>We apply the function to our messy authors names.<jupyter_code>%time dis, inds = getNearestN(name_spam_clean)<jupyter_output>CPU times: user 42.1 s, sys: 18 s, total: 1min
Wall time: 15.7 s<jupyter_text>*** **Note:** Before procedding we can compare the timings of these method with the timing of more traditional fuzzy string matching by using the fuzzywuzzy library. We calculate the time it would take to search just one name so we can get an idea of how long would it take to calculate the whole dataframe.<jupyter_code>%time process.extractBests(query= name_spam.iloc[0],choices= list(nombres_personas), limit = 1)<jupyter_output>CPU times: user 5.08 s, sys: 6.83 ms, total: 5.09 s
Wall time: 5.09 s<jupyter_text>*** We create a dataframe containing the query and the match as well as the distance calculated from the K-Nearest Neighbor algorithm.<jupyter_code>name_spam = list(name_spam) #need to convert back to a list
matches = []
for i,j in enumerate(inds):
    temp = [round(dis[i][0],2), nombres_personas.values[j][0], name_spam[i], autores_sin_dup.values[i], personas_limpia['persona_id'].values[j][0], j[0]]
    matches.append(temp)
final_res = pd.DataFrame(matches, columns = ['score', 'match','messy', 'original', 'id', 'index'])<jupyter_output><empty_output><jupyter_text>Checking matching validity Although undoubtedly faster, the above algorithm may not be as accurate as others methods. But we can now use more traditional fuzzy string matching by using pairwise comparissons between the matched name from the K-Nearest Neighbors algorithm and the target name to create a criterion composed of multiple scores. For this implementation we employ 4 measures from the fuzzywuzzy library:* Partial Token Set Ratio: Should give 100 when a portion of the string is present in both cases. For example when they share a common lastname, independtely of the position. * Partial Ratio: Attempts to account for partial string matches.* Token Sort Ratio: Should give high scores independently of the words order when they don't differ.* WRatio: A balanced score considering many factorsFurthermore, we take the average of the last 3 measures as a fifth metric.<jupyter_code>def multiple_scorer(col1, col2):
    partial_token_set = fuzz.partial_token_set_ratio(col1,col2)
    partial_ratio = fuzz.partial_ratio(col1, col2)
    token_sort = fuzz.token_sort_ratio(col1, col2)
    wratio = fuzz.WRatio(col1, col2)
    return [partial_token_set, partial_ratio, token_sort, wratio]
resultado = []
for indice, row in final_res.iterrows():
    tmp_resultado = multiple_scorer(row['match'],row['messy'])
    promedio = np.round(np.mean(tmp_resultado[1:]), 3)
    tmp_resultado.append(promedio)
    resultado.append(tmp_resultado)
scores = pd.DataFrame(resultado, columns=['Partial Token Set', 'Partial Ratio', 'Token Sort', 'WRatio', 'Score Average'])
names_matched = pd.merge(final_res, scores, left_index=True, right_index=True)
names_matched.to_csv('matche_nombres.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08572691117984976, 'fraction_numerical': 0.014471939902783915, 'mean_word_length': 5.700962250185048, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 88, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '141346', 'license': 'MIT', 'n_tokens_mistral': 2894, 'n_tokens_neox': 2749, 'n_words': 921, 'path': 'Data_PreProcessing/Fuzzy Name Matching.ipynb', 'repo_name': 'Aler-u/DotSAN', 'size': '9052'}