<jupyter_start><jupyter_text>Putting together the PIGEON databaseAnneliek ter Horst1. Download all available viral contigs from datasets specified in the paper2. Retain only contigs >= 10 kb, unless it is RefSeq data3. Cluster viral contigs into vOTUs using cd-hit  Retain only contigs => 10kb- For each fasta file of viral contigs that is downloaded, retain only the contigs => 10 kb- Except for the RefSeq (Pruitt et al., 2009) file and the Gregory et al., 2019 file (bc circular or >10kb)- Using the bbtools reformat tool- https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/reformat-guide/<jupyter_code>reformat.sh in=in_file.fa out=out_file_10k.fa minlength=10000<jupyter_output><empty_output><jupyter_text>Concatenate all fasta files to one big fasta file- Use the Linux cat command<jupyter_code>cat fasta1.fa fasta2.fa fasta3.fa fastax.fa >> all_fastas.fa<jupyter_output><empty_output><jupyter_text>Cluster all viral contigs into vOTUs using cd-hit- Clustering sequences at 95% ANI over 85% of the sequence, relative to the shorter sequence- https://github.com/weizhongli/cdhit/wiki/2.-Installation- Using cd-hit-est<jupyter_code>cd-hit-est-2d \
-i all_fastas.fa \
-o all_fastas_clustered.fa \
-c 0.95 -aS 0.85<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09062003179650238, 'fraction_numerical': 0.03259141494435612, 'mean_word_length': 5.111650485436893, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '638705', 'license': 'MIT', 'n_tokens_mistral': 473, 'n_tokens_neox': 432, 'n_words': 135, 'path': 'PIGEON_database.ipynb', 'repo_name': 'AnneliektH/SPRUCE', 'size': '1258'}