<commit_before>Time-stamp: <2019-04-06 08:30:45 eu>
* Gisaid [[file:~/AD/sources/acmacs-whocc/doc/gisaid.org][instructions for downloading sequences from gisaid]] and making seqdb
* Check config
~/AD/sources/acmacs-whocc/conf/make-signature-pages.config.json
* On albertine
** Generate pdfs for individual flu subtypes
#+BEGIN_SRC sh
~/AD/sources/acmacs-whocc/bin/make-signature-pages -t $(date '+%Y-%m%d') -f <h1 h3 bv by> -s <week day>
#+END_SRC
Command above creates directory, submits to condor and then runs in background.
If processing directory exists, program starts again from the last saved state.
Program sends email on completion or failure.
See ../acmacs-whocc/conf/make-signature-pages.config.json
** Generate pdf with all sig pages
* Runtime
** Custom version raxml with --stop-after-seconds [2019-04-05 Fri]
*** H3 3997 sequences A%28H3N2%29/HAWAII/22/2012__MDCK1, A%28H3N2%29/ONTARIO/RV2886/2018__P1 ... A%28H3N2%29/ANCONA/3/2019__
**** full time --stop-after-seconds 0 no threads
i22: 11652s (03:14:12)
/syn/eu/raxml-acorg/raxmlHPC-oriane -p 13579 -N 1 -s ../source-4k.fas -n 4k-0s-$(date +%m%d-%H%M%S) -m GTRGAMMAI -e 0.001 -o A%28H3N2%29/HAWAII/22/2012__MDCK1 -c 4 -f d --silent --no-seq-check -D

**** --stop-after-seconds 1 no threads
i22: 4227s (01:10:27)
/syn/eu/raxml-acorg/raxmlHPC-oriane --stop-after-seconds 1 -p 13579 -N 1 -s ../source-4k.fas -n 4k-1s-$(date +%m%d-%H%M%S) -m GTRGAMMAI -e 0.001 -o A%28H3N2%29/HAWAII/22/2012__MDCK1 -c 4 -f d --silent --no-seq-check -D

* OLD (before <2018-12-28 Fri>)
Run on albertine ~/AD/sources/acmacs-whocc/bin/make-signature-pages.before-20181228 -t $(date '+%Y-%m%d')
* OLD (before <2018-07-16 Mon>)
** Making tree for the signature page from the WHO CC sequence and HI tables
*** download sequences from gisaid (see GISAID.md) in put it into ~/ac/tables-store/sequences/gisaid-all-20160101-$(date +%Y%m%d).fas.bz2
*** make ~/AD/data/seqdb.json.xz (make sure hidb in ~/AD/data/ are up to date), approximate running time: 5 minutes
         ~/AD/bin/seqdb-update-whocc
*** mkdir $(date +%Y-%m%d-%H%M), copy seqdb
         D=$(date +%Y-%m%d-%H%M) && for V in bvic byam h1 h3; do mkdir /syn/eu/ac/results/whocc-tree/$V/$D; cp ~/AD/data/seqdb.json.xz /syn/eu/ac/results/whocc-tree/$V/$D; done
*** export sequences from ./seqdb.json.xz
**** base-seq. Use ~/AD/bin/seqdb-list --db ./seqdb.json.xz --re <name> to check, if the name below returns just one seq_id<commit_msg>docs updated<commit_after>Time-stamp: <2019-04-06 22:12:19 eu>
* Gisaid [[file:~/AD/sources/acmacs-whocc/doc/gisaid.org][instructions for downloading sequences from gisaid]] and making seqdb
* Check config
~/AD/sources/acmacs-whocc/conf/make-signature-pages.config.json
* On albertine
** Generate pdfs for individual flu subtypes
#+BEGIN_SRC sh
~/AD/sources/acmacs-whocc/bin/make-signature-pages -t $(date '+%Y-%m%d') -f <h1 h3 bv by> -s <week day>
#+END_SRC
Command above creates directory, submits to condor and then runs in background.
If processing directory exists, program starts again from the last saved state.
Program sends email on completion or failure.
See ../acmacs-whocc/conf/make-signature-pages.config.json
** Generate pdf with all sig pages
* Runtime
T1 - 1 thread
S0 - full time, S1 - --stop-after-seconds 1

|       | 4k           | 5k                 | 6k             | 7k                | 8k | 9k | 10k | 11k |
|-------+--------------+--------------------+----------------+-------------------+----+----+-----+-----|
| T1 S0 | 11652s 3.24h | 15106s 4.20h 48704 |                |                   |    |    |     |     |
|-------+--------------+--------------------+----------------+-------------------+----+----+-----+-----|
| T1 S1 |  4227s 1.17h |  5222s 1.45h 48774 | 7176s 2h 55717 | 9116s 2.53h 61643 |    |    |     |     |
|-------+--------------+--------------------+----------------+-------------------+----+----+-----+-----|
| T  S0 |              |                    |                |                   |    |    |     |     |
|-------+--------------+--------------------+----------------+-------------------+----+----+-----+-----|
| T  S1 |              |                    |                |                   |    |    |     |     |
|-------+--------------+--------------------+----------------+-------------------+----+----+-----+-----|

** Custom version raxml with --stop-after-seconds [2019-04-05 Fri]
*** H3 3997 sequences A%28H3N2%29/HAWAII/22/2012__MDCK1, A%28H3N2%29/ONTARIO/RV2886/2018__P1 ... A%28H3N2%29/ANCONA/3/2019__
**** full time --stop-after-seconds 0 no threads
i22: 11652s (03:14:12)
/syn/eu/raxml-acorg/raxmlHPC-oriane -p 13579 -N 1 -s ../source-4k.fas -n 4k-0s-$(date +%m%d-%H%M%S) -m GTRGAMMAI -e 0.001 -o A%28H3N2%29/HAWAII/22/2012__MDCK1 -c 4 -f d --silent --no-seq-check -D
**** --stop-after-seconds 1 no threads
i22: 4227s (01:10:27)
s=8k; /syn/eu/raxml-acorg/raxml-i22-avx --stop-after-seconds 1 -p 13579 -N 1 -s ../source-$s.fas -n $s-1s-$(date +%m%d-%H%M%S) -m GTRGAMMAI -e 0.001 -o A%28H3N2%29/HAWAII/22/2012__MDCK1 -c 4 -f d --silent --no-seq-check -D
*** H3 5k sequences A%28H3N2%29/HAWAII/22/2012__MDCK1, 
**** full time --stop-after-seconds 0 no threads
i22: 15106s 4.2h 48704
**** --stop-after-seconds 1 no threads
i22: 5222s 1.45h 48774
*** H3 6k sequences A%28H3N2%29/HAWAII/22/2012__MDCK1, 
**** full time --stop-after-seconds 0 no threads
i22: 
**** --stop-after-seconds 1 no threads
i22: 7176s 2h 55717
*** H3 7k sequences A%28H3N2%29/HAWAII/22/2012__MDCK1, 
**** full time --stop-after-seconds 0 no threads
i22: 
**** --stop-after-seconds 1 no threads
i22: 9116s 2.53h 61643
* OLD (before <2018-12-28 Fri>)
Run on albertine ~/AD/sources/acmacs-whocc/bin/make-signature-pages.before-20181228 -t $(date '+%Y-%m%d')
* OLD (before <2018-07-16 Mon>)
** Making tree for the signature page from the WHO CC sequence and HI tables
*** download sequences from gisaid (see GISAID.md) in put it into ~/ac/tables-store/sequences/gisaid-all-20160101-$(date +%Y%m%d).fas.bz2
*** make ~/AD/data/seqdb.json.xz (make sure hidb in ~/AD/data/ are up to date), approximate running time: 5 minutes
         ~/AD/bin/seqdb-update-whocc
*** mkdir $(date +%Y-%m%d-%H%M), copy seqdb
         D=$(date +%Y-%m%d-%H%M) && for V in bvic byam h1 h3; do mkdir /syn/eu/ac/results/whocc-tree/$V/$D; cp ~/AD/data/seqdb.json.xz /syn/eu/ac/results/whocc-tree/$V/$D; done
*** export sequences from ./seqdb.json.xz
**** base-seq. Use ~/AD/bin/seqdb-list --db ./seqdb.json.xz --re <name> to check, if the name below returns just one seq_id
{'fraction_non_alphanumeric': 0.2302732575990175, 'fraction_numerical': 0.08673626036229659, 'mean_word_length': 2.22365165759525, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 10}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3911503', 'n_tokens_mistral': 2840, 'n_tokens_neox': 2437, 'n_words': 616}