<jupyter_start><jupyter_text>Navigating a Knowledge BaseUsing Learned Representations for Encyclopedia Navigation<NAME>- und Content Management WiSe 20/21 Overview    Context: Navigating an encyclopedic body of work         Data: Stanford Encyclopedia of Philosophy and Wikipedia Philosophers        Features: Node2vec, Specter and USE        Retrieval: Elasticsearch and k-nearest neighbor  Context: How *not* to consume an encyclopediaIn 1797, <NAME> was given a complete set of the Britannica's 3rd edition, which he read completely; after this feat, he extended his royal title to include "Most Formidable Lord and Master of the Encyclopædia Britannica."(Source)The Know-It-All: One Man's Humble Quest to Become the Smartest Person in the World is a book by Esquire editor <NAME>, published in 2004. It recounts his experience of reading the entire Encyclopædia Britannica; all 32 volumes of the 2002 edition, extending to over 33,000 pages with some 44 million words. He set out on this endeavour to become the "smartest person in the world". The book is organized alphabetically in encyclopedia format and recounts both interesting facts from the encyclopedia and the author's experiences. (Source)In 2008, <NAME> published his account of reading the complete Oxford English Dictionary. (Source) Context: How to consume an encyclopedia    Encyclopedias: Distillation of human knowledge         Goal of this project:  Facilitate the consumption of a medium that is only meant for direct and targeted consumption        Problems: Huge scale of encyclopedic knowledge bases        Solution: Reduce the problem to one topical corner; philosophy Context: High level goal    Writing Assistant: Based on semantic content, retrieve a relevant subset of nodes in the graph            Traverse the knowledge base in a locally relevant way: Once in the graph, find a relevant subset of nodes Data: In vivo    The Stanford Encyclopedia of Philosophy: All articles            All Wikipedia Philosophers: All philosophers on Wikipedia     Data: High level view    Graph structure: Use that to discover            Text: Use that to anchor into the graph and also to discover     Data: Scraping    The Stanford Encyclopedia of Philosophy: Scrapy            All Wikipedia Philosophers: Local DBPedia instance[Scrapy notebook](scraping_plato.ipynb) / [DBPedia notebook](scraping_wiki_dbpedia.ipynb)  Data: *In vitro*<jupyter_code>!ls ../data/<jupyter_output>dbpedia.pkl			plato_undirected.graphml
dbpedia_with_articles.pkl	spectre_embeddings.npy
homophily_plato_embeddings.npy	structural_plato_embeddings.npy
homophily_wiki_embeddings.npy	structural_wiki_embeddings.npy
node2vec_v1			to_index.csv
old_node2vec_embeddings		to_index_p4.pkl
plato_directed.gephi		to_index.pkl
plato_directed.graphml		use_abstract_embeddings.npy
plato.pkl			use_article_embeddings.npy
{'fraction_non_alphanumeric': 0.039524309199020634, 'fraction_numerical': 0.01224204267226303, 'mean_word_length': 4.607843137254902, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '234811', 'license': 'Apache-2.0', 'n_tokens_mistral': 824, 'n_tokens_neox': 735, 'n_words': 341, 'path': 'notebooks/presentation.ipynb', 'repo_name': 'cakiki/philosophy-graph', 'size': '2889'}