<jupyter_start><jupyter_text>Table of Contents<jupyter_code>folder = Path('/Users/klay6683/Dropbox/data/planet4/p4_analysis/P4_catalog_v1.0/')
newfolder = folder.with_name("P4_catalog_v1.1")
newfolder.mkdir()
fanfname = folder / "P4_catalog_v1.0_L1C_cut_0.5_fan_meta_merged.csv"
newfanfname = fanfname.with_name("P4_catalog_v1.1_L1C_cut_0.5_fan_meta_merged.csv")
fans = pd.read_csv(fanfname)
fans.marking_id.count()
fans.marking_id.nunique()
fans.groupby('marking_id').size().sort_values(ascending=False).head()<jupyter_output><empty_output><jupyter_text>From several fnotching events I got several entries with different vote_ratios in the catalog.I think the best way to deal with them is to average the vote_ratio, as a fair judgement on its "fan-ness".<jupyter_code>fans.query("marking_id=='F018f75'").vote_ratio.describe()
fans.query("marking_id=='F00cfa9'").vote_ratio
# this will create mean value for each column
mean_fans = fans.groupby('marking_id').mean()
mean_fans.loc['F00cfa9']
fans.shape
mean_fans = mean_fans.reset_index()
mean_fans.shape
fans.shape
mean_fans.shape
mean_fans.head()
mean_fans = mean_fans.join(fans.drop_duplicates(subset='marking_id').set_index('marking_id')[['tile_id', 'obsid']],
                           how='inner')
mean_fans.shape
mean_fans.reset_index(inplace=True)
(mean_fans.shape[0]-fans.shape[0])/fans.shape[0]
mean_fans[fans.columns].to_csv(newfanfname, index=False)
mean_fans.shape
fans.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13342409802586794, 'fraction_numerical': 0.025867937372362152, 'mean_word_length': 7.305084745762712, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '655122', 'license': '0BSD', 'n_tokens_mistral': 604, 'n_tokens_neox': 587, 'n_words': 80, 'path': 'notebooks/catalog_checks.ipynb', 'repo_name': 'CitizenScienceInAstronomyWorkshop/P4_sandbox', 'size': '1469'}