<commit_before>### System-level prerequisites:
 * A Postgres installation
 ```
   sudo apt-get install postgresql
 ```
 * A ZeroMQ installation and bindings
 ```
  sudo apt-get install libzmq3 libzmq3-dev
 ```

 * GnuPlot for classic plotting:
 ```
  sudo apt-get install libgd2-noxpm-dev libcairo2-dev gnuplot
 ```

* libarchive for dealing with complex directory jobs
```
 sudo apt-get install libarchive-dev
```

* A Redis caching server for mitigating DB load and maintaining temporary frontend state
```
  cd /tmp &&
  wget http://download.redis.io/redis-stable.tar.gz &&
  tar xvzf redis-stable.tar.gz &&
  cd redis-stable &&
  make &&
  make install
```

### Setting up postgresql:
 This is not normative, but the simplest (insecure!) approach is just:
 ```
 sudo emacs /etc/postgresql/9.1/main/pg_hba.conf
     change "local all all peer" to "local all all password"

 sudo -u postgres psql
     create database cortex_tester;
     create database cortex;

     create user cortex with password '<PASSWORD>';
     create user cortex_tester with password '<PASSWORD>';

     grant all privileges on database cortex_tester to cortex_tester;
     grant all privileges on database cortex to cortex;
 ```

 This should evolve as we get nearer to production deploys... Also, for now postgresql is expected on the default 5432 port.
 
 One of the problems that is experienced with arXiv, is that as we enter the tens-of-millions of rows for log messages, performance degrades very rapidly. One good solution to avoid that is to use a small threshold for VACUUM ANALYZE, given that the inserts are generally quick at the moment. [source of this trick](https://lob.com/blog/supercharge-your-postgresql-performance/) 
 ```
ALTER TABLE logs  
SET (autovacuum_enabled = true,
autovacuum_vacuum_scale_factor = 0.001,
autovacuum_analyze_scale_factor = 0.0005,
autovacuum_analyze_threshold = 50,
autovacuum_vacuum_threshold = 50);

ALTER TABLE tasks  
SET (autovacuum_enabled = true,
autovacuum_vacuum_scale_factor = 0.001,
autovacuum_analyze_scale_factor = 0.0005,
autovacuum_analyze_threshold = 50,
autovacuum_vacuum_threshold = 50);
```

Also, ensure you have the Postgres data directory on a sufficiently large disk. You may want 250GB available at a minimum for a LaTeXML run over arXiv. (See [here](https://github.com/dginev/CorTeX/issues/10) for details). Another consideration is running a newer version of Postgres, which may not be available in the default source list of some Linux server environments. The mathweb.org deployment currently uses Postgres 9.4 and upgraded from 9.1 roughly following an adaptation of [these upgrade guidelines](https://gist.github.com/tamoyal/2ea1fcdf99c819b4e07d).
<commit_msg>more aggro vacuum, aiming for every 3000 updates<commit_after>### System-level prerequisites:
 * A Postgres installation
 ```
   sudo apt-get install postgresql
 ```
 * A ZeroMQ installation and bindings
 ```
  sudo apt-get install libzmq3 libzmq3-dev
 ```

 * GnuPlot for classic plotting:
 ```
  sudo apt-get install libgd2-noxpm-dev libcairo2-dev gnuplot
 ```

* libarchive for dealing with complex directory jobs
```
 sudo apt-get install libarchive-dev
```

* A Redis caching server for mitigating DB load and maintaining temporary frontend state
```
  cd /tmp &&
  wget http://download.redis.io/redis-stable.tar.gz &&
  tar xvzf redis-stable.tar.gz &&
  cd redis-stable &&
  make &&
  make install
```

### Setting up postgresql:
 This is not normative, but the simplest (insecure!) approach is just:
 ```
 sudo emacs /etc/postgresql/9.1/main/pg_hba.conf
     change "local all all peer" to "local all all password"

 sudo -u postgres psql
     create database cortex_tester;
     create database cortex;

     create user cortex with password '<PASSWORD>';
     create user cortex_tester with password '<PASSWORD>';

     grant all privileges on database cortex_tester to cortex_tester;
     grant all privileges on database cortex to cortex;
 ```

 This should evolve as we get nearer to production deploys... Also, for now postgresql is expected on the default 5432 port.
 
 One of the problems that is experienced with arXiv, is that as we enter the tens-of-millions of rows for log messages, performance degrades very rapidly. One good solution to avoid that is to use a small threshold for VACUUM ANALYZE, given that the inserts are generally quick at the moment. [source of this trick](https://lob.com/blog/supercharge-your-postgresql-performance/) 
 ```
ALTER TABLE logs  
SET (autovacuum_enabled = true,
autovacuum_vacuum_scale_factor = 0.0002,
autovacuum_analyze_scale_factor = 0.0005,
autovacuum_analyze_threshold = 50,
autovacuum_vacuum_threshold = 50);

ALTER TABLE tasks  
SET (autovacuum_enabled = true,
autovacuum_vacuum_scale_factor = 0.0002,
autovacuum_analyze_scale_factor = 0.0005,
autovacuum_analyze_threshold = 50,
autovacuum_vacuum_threshold = 50);
```

Also, ensure you have the Postgres data directory on a sufficiently large disk. You may want 250GB available at a minimum for a LaTeXML run over arXiv. (See [here](https://github.com/dginev/CorTeX/issues/10) for details). Another consideration is running a newer version of Postgres, which may not be available in the default source list of some Linux server environments. The mathweb.org deployment currently uses Postgres 9.4 and upgraded from 9.1 roughly following an adaptation of [these upgrade guidelines](https://gist.github.com/tamoyal/2ea1fcdf99c819b4e07d).

{'fraction_non_alphanumeric': 0.0781621819513088, 'fraction_numerical': 0.021233754347428153, 'mean_word_length': 4.367387033398821, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7163109', 'n_tokens_mistral': 1769, 'n_tokens_neox': 1593, 'n_words': 623}