<commit_before>untap
===

In order to facilitate ease of access, some of the information available through [Harvard Personal Genome Project](http://www.personalgenomes.org/) page and the [GET-Evidence](http://evidence.pgp-hms.org/about) site has been consolidated into a small SQLite database (~120Mb uncompressed).  This project is a collection of scripts to download data, consolidate into a SQLite database, upload to an [Arvados](https://arvados.org/) project and create an HTML visualization front end for easy exploration of the data.

You can explore the most recent snapshot of the Harvard Personal Genome Project database snapshot available through [a Curoverse hosted collection](https://workbench.su92l.arvadosapi.com/collections/2210f7ee07fc1c8b926e5db28eff9635+3284/html/index.html?disposition=inline&size=33766)

Quick start
---

To grab the repository:

```bash
$ git clone https://github.com/abeconnelly/untap
$ cd untap
```

We need to run the application inside a server such as nginx.

```bash
$ cd $HOME
$ sudo apt-get install nginx
$ sudo /etc/init.d/nginx start
$ mkdir /var/www
$ cat > /etc/nginx/sites-enabled/untap <<EOF
	server {
	  root /var/www;

	  location / {
	  }
	}
EOF
$ sudo ln -s $HOME/untap /var/www/untap
$ sudo chmod -R 777 /var/www/untap
$ sudo nginx -s reload
```

Now we need to obtain a dataset. Either 1) download the snapshot provided at [the Untap hosted on Curoverse](https://workbench.su92l.arvadosapi.com/collections/2210f7ee07fc1c8b926e5db28eff9635+3284/html/index.html?disposition=inline&size=33766) or 2) follow the instructions in the following section to scrape [Tapestry](http://my.pgp-hms.org) and build your own snapshot. In both cases, the database should be put in the root directory, i.e. `/untap/hu-pgp.sqlite3.gz`. 
<commit_msg>added python module instead of nginx<commit_after>untap
===

In order to facilitate ease of access, some of the information available through [Harvard Personal Genome Project](http://www.personalgenomes.org/) page and the [GET-Evidence](http://evidence.pgp-hms.org/about) site has been consolidated into a small SQLite database (~120Mb uncompressed).  This project is a collection of scripts to download data, consolidate into a SQLite database, upload to an [Arvados](https://arvados.org/) project and create an HTML visualization front end for easy exploration of the data.

You can explore the most recent snapshot of the Harvard Personal Genome Project database snapshot available through [a Curoverse hosted collection](https://workbench.su92l.arvadosapi.com/collections/2210f7ee07fc1c8b926e5db28eff9635+3284/html/index.html?disposition=inline&size=33766)

Quick start
---

To grab the repository:

```bash
$ git clone https://github.com/abeconnelly/untap
$ cd untap
```

We need to run the application inside a HTTP server,

### such as nginx

```bash
$ cd $HOME
$ sudo apt-get install nginx
$ sudo /etc/init.d/nginx start
$ mkdir /var/www
$ cat > /etc/nginx/sites-enabled/untap <<EOF
	server {
	  root /var/www;

	  location / {
	  }
	}
EOF
$ sudo ln -s $HOME/untap /var/www/untap
$ sudo chmod -R 777 /var/www/untap
$ sudo nginx -s reload
```

### or with a python module

```bash
$ cd html
$ python -m SimpleHTTPServer
```

Now we need to obtain a dataset. Either 1) download the snapshot provided at [the Untap hosted on Curoverse](https://workbench.su92l.arvadosapi.com/collections/2210f7ee07fc1c8b926e5db28eff9635+3284/html/index.html?disposition=inline&size=33766) or 2) follow the instructions in the following section to scrape [Tapestry](http://my.pgp-hms.org) and build your own snapshot. In both cases, the database should be put in the root directory, i.e. `/untap/hu-pgp.sqlite3.gz`. 

{'fraction_non_alphanumeric': 0.10097455332972388, 'fraction_numerical': 0.03735787763941527, 'mean_word_length': 4.482195845697329, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '843398', 'n_tokens_mistral': 1291, 'n_tokens_neox': 1158, 'n_words': 407}