<jupyter_start><jupyter_text>Hemimatch ExampleBefore starting the library, you will want to unzip the contents of https://open.quiltdata.com/b/janelia-flylight-color-depth/tree/Color_Depth_MIPs_For_Download/EM_Hemibrain11_0630_2020_radi2_PackBits_noNeuronName.zip to a local images/ folder. These images will be used for the matching. Precomputed binary match data is available at https://ffbodata.neuronlp.fruitflybrain.org/match_data/ as well.Then, start by importing the library:<jupyter_code>from hemimatch_lib import *<jupyter_output><empty_output><jupyter_text>Binary matches are fastest:<jupyter_code>X = load_binary_dataset()
score_order, scores, onlyfiles = match_cpu_inmemory('eb_ex.png', X)<jupyter_output><empty_output><jupyter_text>You can use the matches as follows to generate NLP queries; here, we used local EB neurons as our input patterns, and we find such neurons. You can use hemibrain_template.png as a template and eb_ex.png as an example of a image to use:<jupyter_code>get_top_k(onlyfiles, score_order)<jupyter_output>show /:referenceId:[1261082560, 1198330641, 1261086755, 1261091157, 1100792636, 5813057274, 1228692168, 5813047446, 1198330641, 1261091174, 1261423480, 1261086823, 1228692168, 1261423444, 1198680426, 1261436305, 5812983887, 5812979082, 5813020587, 5813080272, 1261432052, 1261091124, 1261086756, 1198680441, 1261427765, 1261436508, 1231070863, 5812979686, 1230738163, 1351874277, 1261427892, 1198680122, 1261419142, 1261760187, 1261427857, 1292458366, 1100792636, 1261432076, 1230121634, 1292113218, 1261427999, 1261432159, 1292445517, 1261086734, 1261423454, 1261773006, 1261768832, 1261756007, 1198684596, 1260422110]<jupyter_text>Sequential matches are slower:<jupyter_code>score_order, scores, onlyfiles = match_cpu_sequential('eb_ex.png')<jupyter_output><empty_output><jupyter_text>You can also use tensorflow for matching with GPU support:<jupyter_code>score_order, scores, onlyfiles = match_gpu('eb_ex.png', batch_size = 1, workers = 1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08382936507936507, 'fraction_numerical': 0.2544642857142857, 'mean_word_length': 8.79126213592233, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '162038', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1087, 'n_tokens_neox': 721, 'n_words': 169, 'path': 'HemiMatchExample.ipynb', 'repo_name': 'mkturkcan/hemimatch', 'size': '2016'}