<jupyter_start><jupyter_text>Import and prepare data<jupyter_code>df = pd.read_csv('/home/lars/app-document-recommender/parliamentary_questions/questions_topics.txt', sep='\t', header=None, index_col=0)<jupyter_output><empty_output><jupyter_text>The dataframe contains documents that have been assigned topics with a Mallet topic modeller<jupyter_code>df
df3 = df
df2 = df.iloc[:, 1:]<jupyter_output><empty_output><jupyter_text>Set inputIn the following cell we select the document we want to compare with<jupyter_code>selected_doc = df.iloc[300, 1:]
df3['c'] = df2.apply(lambda x : 1 - spatial.distance.cosine(selected_doc, x), axis=1)<jupyter_output><empty_output><jupyter_text>The following table shows the selected document (cosine similarity of 1.0) and the four most similar results<jupyter_code>results = df3.sort_values('c', ascending=False).head()
results.loc[:, [1 ,'c']]
results.index
lst = [((x.split('/'))[-1].split('.'))[0] + '.html' for x in df.iloc[results.index, 0]]
lst
ts =pd.read_csv('data/texts.csv', index_col=0)
ts.head(2)<jupyter_output><empty_output><jupyter_text>View textsThe following cell displays the selected text and the four most similar<jupyter_code>for x in ts.iterrows():
    if str(x[0]) in lst:
        print(x[1]['Text'], '\n')<jupyter_output>Regulation (EU, Euratom) 2020/2092 on a general regime of conditionality for the protection of the Union budget was published in the Official Journal of the European Union on 22 December 2020.The conclusions of the European Council meeting of 10 and 11 December 2020 state that ‘the Commission intends to develop and adopt guidelines on the way it will apply the Regulation, including a methodology for carrying out its assessment.’ Furthermore, ‘should an action for annulment be introduced with regard to the Regulation, the guidelines will be finalised after the judgment of the Court of Justice so as to incorporate any relevant elements stemming from such judgment’. Moreover, ‘until such guidelines are finalised, the Commission will not propose measures under the Regulation’.Which of the above guidelines has the Commission developed and adopted so far?What steps has the Commission taken so far concerning which Member States to analyse and decide whether it needs to propose measures under [...]
{'fraction_non_alphanumeric': 0.09226060341058155, 'fraction_numerical': 0.019676432006996064, 'mean_word_length': 6.061728395061729, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '420126', 'license': 'MIT', 'n_tokens_mistral': 687, 'n_tokens_neox': 644, 'n_words': 258, 'path': 'pq_recommendations.ipynb', 'repo_name': 'tungland/document-recommender', 'size': '2287'}