<jupyter_start><jupyter_text>I realize the following algorithm has great space to be improved and be more elegant by using some consice operations. So please feel free to edit it.   Load Image View Table<jupyter_code>image_view = pd.read_csv("image view.txt", delimiter="\t", names=["identifier", "provider", "title", "meta_data", "tags", "standardized_popularity"])
image_view.head()<jupyter_output><empty_output><jupyter_text>Extract JSON objects First examine how many different keys we have in tags:<jupyter_code>tags = image_view["tags"]
tags
diff_tags_keys = []
def find_diff_json_keys(lst):
    for n in pd.read_json(lst).columns:
        if n not in diff_tags_keys:
            diff_tags_keys.append(n)
tags.apply(find_diff_json_keys)
diff_tags_keys<jupyter_output><empty_output><jupyter_text>Since we have already had a provider column, we will only extract name and accuracy from tags.<jupyter_code>def tags_extract(lst):
    
    json_tbl = pd.read_json(lst)
    if "name" in json_tbl.columns:
        name_lst = list(json_tbl["name"])
        name_col.append(name_lst)
    else:
        name_col.append(None)
        
    if "accuracy" in json_tbl.columns:
        accuracy_lst = list(json_tbl["accuracy"])
        accuracy_col.append(accuracy_lst)
    else:
        accuracy_col.append(None)
    
name_col = []
accuracy_col = []
tags.apply(tags_extract)
image_view["name"] = name_col
image_view["accuracy"] = accuracy_col
image_view<jupyter_output><empty_output><jupyter_text>As you may see in the accuracy column, we have None and nan. Generally, None happens when we have no accuracy data for the image. nan happens when a JSON object in the tags of the image has only name element and no accuracy element. I am not very clear with what accuracy really is. So I decide to leave it as this for now and not to do any imputation or null removal. The following code is try to print the first ith rows in a cleaner format<jupyter_code>rows_to_display = 100
row_counter = 0

for i in image_view["name"]:
    if row_counter >= rows_to_display:
        break
    print("Row" + str(row_counter), i)
    print("\n")
    row_counter += 1
rows_to_display = 100
row_counter = 0

for i in image_view["accuracy"]:
    if row_counter >= rows_to_display:
        break
    print("Row" + str(row_counter), i)
    print("\n")
    row_counter += 1<jupyter_output>Row0 None


Row1 [nan, 0.9973600000000001, 0.9889200000000001, 0.9882000000000001, 0.98567, 0.98377, 0.9815, 0.9813900000000001, 0.9795300000000001, 0.9778600000000001, 0.9692200000000001, 0.96242, 0.95959, 0.9576600000000001, 0.9520700000000001, 0.9483600000000001, 0.9463300000000001, 0.94105, 0.9394, 0.93788, 0.9256900000000001]


Row2 [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.9690500000000001, 0.9572200000000001, 0.95691, 0.9538500000000001, 0.9388000000000001, 0.9316700000000001, 0.9175800000000001, 0.8968300000000001, 0.89495, 0.8901500000000001, 0.87699, 0.8615800000000001, 0.85113, 0.8369200000000001, 0.8263400000000001, 0.8172900000000001, 0.8132900000000001, 0.8095, 0.8061100000000001, 0.8060900000000001]


Row3 [nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.9927900000000001, 0.9899600000000001, 0.98228, 0.97889, 0.9683600000000001, 0.9585, 0.9476100000000001, 0.9464800000000001, 0.9350200000000001, 0.93376, 0.93348[...]
{'fraction_non_alphanumeric': 0.10526315789473684, 'fraction_numerical': 0.20190306274159978, 'mean_word_length': 4.653781512605042, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 33, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '551220', 'license': 'MIT', 'n_tokens_mistral': 1660, 'n_tokens_neox': 1158, 'n_words': 323, 'path': 'Image Popularity and Authority/ImageNet/Image view Table.ipynb', 'repo_name': 'MichaelRenMR/data-science-playground', 'size': '3363'}