<commit_before>  COMPARA PRODUCTION CYCLE
============================

1. Once the release coordinator has sent the mail for declaration of
intention, ask the other members of compara group (or related) what
they intend to produce.  Sometimes, you have to wait a bit that gene
builders do their declaration first to know what compara will need to
produce and also potential schema changes.
Compara has one extra days to send the DOI, because we have to know which new 
species are added. 

For homologues and family, ask Albert (<EMAIL>) or Abel
(<EMAIL>). For whole genome alignments and synteny ask Javier
(<EMAIL>) or Abel (<EMAIL>). For ComparaMart ask
Benoit (<EMAIL>)

2. Mail the compara declaration of intentions to <EMAIL>
send the DOI at after the Genebuilders

3. Define priorities and assign jobs to the team members.

4. Add in the master compara database
(compara1:3306/jh7_ensembl_compara_master) the new entries in the
genome_db and dnafrag tables. You can set up your registry and use the
~ensembl-compara/scripts/pipeline/update_genome.pl script. This script
sets the new genome_dbs as the default assemblies.

eg. 
perl update_genome.pl --reg_conf reg_conf_v44.pl --compara compara_master --species bushbaby


5. Wait for the handover before starting to build the new database in case
any of the new species cannot make it. Don't forget to switch the
assembly_default values of genome_db in this case.
 [1] for species making it / used in the pipeline
 [0] for species not making it / or old assemblies

6. Update the ncbi_taxa_nodes and ncbi_taxa_names in the master DB using the
ncbi_taxonomy database located in mysql::/ecs2:3365/ncbi_taxonomy

7. New method_link_species_set entries might be added using the
~ensembl-compara/scripts/pipeline/create_mlss.pl script. The release
coordinator (or any team member) should create a new
method_link_species_set in the master database before starting a new
pipeline in order to get a unique method_link_species_set_id. Ideally
they can be created before starting to build the new database although
new method_link_species_sets can be added later on.

8. Create the new database for the new release and add it to your
registry configuration file. Use the ~ensembl-compara/sql/table.sql
file to create the tables and populate the database with the relevant
primary data and genomic alignments that can be reused from the
previous release. This can be done with the
~ensembl-compara/scripts/pipeline/populate_new_database.pl script.  It
requires the master database, the previous released database and the
fresh new database with the tables already created. The script will
copy relevant data from the master and the old database into the new
one.

  mysql -u USER -pPASSWD -h compara1 -P 3306 \
      -e "CREATE DATABASE jh7_ensembl_compara_39"
  mysql -u USER -pPASSWD -h compara1 -P 3306 \
      jh7_ensembl_compara_39 < ~/src/ensembl-compara/sql/table.sql<commit_msg>added example for mlss<commit_after>  COMPARA PRODUCTION CYCLE
============================

1. Once the release coordinator has sent the mail for declaration of
intention, ask the other members of compara group (or related) what
they intend to produce.  Sometimes, you have to wait a bit that gene
builders do their declaration first to know what compara will need to
produce and also potential schema changes.
Compara has one extra days to send the DOI, because we have to know which new 
species are added. 

For homologues and family, ask Albert (<EMAIL>) or Abel
(<EMAIL>). For whole genome alignments and synteny ask Javier
(<EMAIL>) or Abel (<EMAIL>). For ComparaMart ask
Benoit (<EMAIL>)

2. Mail the compara declaration of intentions to <EMAIL>
send the DOI at after the Genebuilders

3. Define priorities and assign jobs to the team members.

4. Add in the master compara database
(compara1:3306/jh7_ensembl_compara_master) the new entries in the
genome_db and dnafrag tables. You can set up your registry and use the
~ensembl-compara/scripts/pipeline/update_genome.pl script. This script
sets the new genome_dbs as the default assemblies.

eg. 
perl update_genome.pl --reg_conf reg_conf_v44.pl --compara compara_master --species bushbaby


5. Wait for the handover before starting to build the new database in case
any of the new species cannot make it. Don't forget to switch the
assembly_default values of genome_db in this case.
 [1] for species making it / used in the pipeline
 [0] for species not making it / or old assemblies

6. Update the ncbi_taxa_nodes and ncbi_taxa_names in the master DB using the
ncbi_taxonomy database located in mysql::/ecs2:3365/ncbi_taxonomy

7. New method_link_species_set entries might be added using the
~ensembl-compara/scripts/pipeline/create_mlss.pl script. The release
coordinator (or any team member) should create a new
method_link_species_set in the master database before starting a new
pipeline in order to get a unique method_link_species_set_id. Ideally
they can be created before starting to build the new database although
new method_link_species_sets can be added later on.

eg.
 perl create_mlss.pl --method_link_type  BLASTZ_NET --genome_db_id 22,51 --source "ensembl"  --reg_conf reg_conf_v44.pl --compara compara_master

8. Create the new database for the new release and add it to your
registry configuration file. Use the ~ensembl-compara/sql/table.sql
file to create the tables and populate the database with the relevant
primary data and genomic alignments that can be reused from the
previous release. This can be done with the
~ensembl-compara/scripts/pipeline/populate_new_database.pl script.  It
requires the master database, the previous released database and the
fresh new database with the tables already created. The script will
copy relevant data from the master and the old database into the new
one.

  mysql -u USER -pPASSWD -h compara1 -P 3306 \
      -e "CREATE DATABASE jh7_ensembl_compara_39"
  mysql -u USER -pPASSWD -h compara1 -P 3306 \
      jh7_ensembl_compara_39 < ~/src/ensembl-compara/sql/table.sql
{'fraction_non_alphanumeric': 0.05508544881367181, 'fraction_numerical': 0.013937282229965157, 'mean_word_length': 4.47005444646098, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2461088', 'n_tokens_mistral': 1827, 'n_tokens_neox': 1703, 'n_words': 849}