<jupyter_start><jupyter_text>Build the qnode to index dictionary<jupyter_code>qnode_to_index = {qnode: i for i, qnode in enumerate(np.loadtxt(embedding_file, skiprows=1, max_rows=max_rows, \
                                                                usecols=range(1), dtype='str'))}
with open(qnode_to_index_file, 'w') as fd:
    json.dump(qnode_to_index, fd)<jupyter_output><empty_output><jupyter_text>Build the FAISS index<jupyter_code>%time xb = np.loadtxt(embedding_file, skiprows=1, max_rows=max_rows, delimiter=' ', \
                      usecols=range(1,dim+1)).astype(np.float32)
# Use METRIC_L2 in case you don't want projections
index = faiss.index_factory(dim, 'IVF65536_HNSW32,Flat', faiss.METRIC_L2)
index.set_direct_map_type(faiss.DirectMap.Array)
index.is_trained
%time index.train(xb)
index.is_trained
%time index.add(xb)
%time faiss.write_index(index, output_index)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11655773420479303, 'fraction_numerical': 0.017429193899782137, 'mean_word_length': 4.437869822485207, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '275410', 'license': 'MIT', 'n_tokens_mistral': 346, 'n_tokens_neox': 336, 'n_words': 55, 'path': 'complEx/create_index_from_embedding.ipynb', 'repo_name': 'Otamio/GraphEmbedEval', 'size': '918'}