<commit_before>py-metagenomics: databases
==========================

Instructions for downloading and building local copies of RefSeq, Silva, or KEGG sequence databases and metadata

Overview
--------
In this folder is a makefile for each database. You should be able to simply `make` each database with the command:

    make -f download_XXX.make

If you are impatient and are working on a multi-core computer, you can add "-j N" to the command to use N threads when possible.

# LASTDBCHUNK #
Unless you have a lot of RAM available (>200GB), you should limit the size of the generated last databases so they fit into memory. To do so, set the make variable: LASTDBCHUNK=100G (or whatever is appropriate for your system).

KEGG
----
To download the KEGG database, you'll need a paid account. Sorry. To tell the makefile what your account is, you'll need to set the KEGG_USER and KEGG_PASSWORD variables. You can set these in the file, in your environment, or in the make call:

    make -f download_kegg.make KEGG_USER=<EMAIL> KEGG_PASSWORD=<PASSWORD>

RefSeq and KEGG
---------------
You can build a mapping file from RefSeq accessions to KEGG ko ids if you have download the KEGG database link files. These are automatically downloaded with the KEGG db. If you don't want the whold KEGG db, but do want to build the RefSeq to KO map, you can download just the link files with:

	make -f download_kegg.make links

Remember to define KEGG_USER and KEGG_PASSWORD either in the command or in the makefile!
<commit_msg>Adding some requirements for getting make to work<commit_after>py-metagenomics: databases
==========================

Instructions for downloading and building local copies of RefSeq, Silva, or KEGG sequence databases and metadata

Overview
--------
In this folder is a makefile for each database. You should be able to simply `make` each database with the command:

    make -f download_XXX.make

If you are impatient and are working on a multi-core computer, you can add "-j N" to the command to use N threads when possible.

You need to have a number of things installed for this to work properly, including:
 
 * gnu make
 * perl
 * lastal/lastdb
 * tantan
 * python with modules: biopython, numpy

# LASTDBCHUNK #
Unless you have a lot of RAM available (>200GB), you should limit the size of the generated last databases so they fit into memory. To do so, set the make variable: LASTDBCHUNK=100G (or whatever is appropriate for your system).

KEGG
----
To download the KEGG database, you'll need a paid account. Sorry. To tell the makefile what your account is, you'll need to set the KEGG_USER and KEGG_PASSWORD variables. You can set these in the file, in your environment, or in the make call:

    make -f download_kegg.make KEGG_USER=<EMAIL> KEGG_PASSWORD=<PASSWORD>

RefSeq and KEGG
---------------
You can build a mapping file from RefSeq accessions to KEGG ko ids if you have download the KEGG database link files. These are automatically downloaded with the KEGG db. If you don't want the whold KEGG db, but do want to build the RefSeq to KO map, you can download just the link files with:

	make -f download_kegg.make links

Remember to define KEGG_USER and KEGG_PASSWORD either in the command or in the makefile!

{'fraction_non_alphanumeric': 0.07206652294425624, 'fraction_numerical': 0.0036957191253464737, 'mean_word_length': 3.9587786259541984, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4358860', 'n_tokens_mistral': 929, 'n_tokens_neox': 853, 'n_words': 500}