<commit_before># Spatial Metabolomics Engine [![Build Status](https://travis-ci.org/SpatialMetabolomics/SM_distributed.svg?branch=master)](https://travis-ci.org/SpatialMetabolomics/SM_distributed) [![Documentation Status](https://readthedocs.org/projects/sm-distributed/badge/?version=latest)](http://sm-distributed.readthedocs.org/en/latest/?badge=latest)
High throughput molecules annotation for imaging mass spectrometry datasets.

## Main Features
- centroided imzML files as input dataset files
- Apache Spark based implementation for easy scaling from one machine to a cluster
- Can be run in both local and distributed modes
- Comes with unit, regression and scientific tests
- Web application for browsing the search results
- Fabric script for easy deploy and an AWS cluster start

## Installation Manual
### Spark Installation and Setup

Install Oracle Java 7

    sudo apt-add-repository ppa:webupd8team/java
	sudo apt-get update
	sudo apt-get install oracle-java7-installer
	java -version  # should work now

Add an env variable to .bashrc
    
    echo -e '\nexport JAVA_HOME=/usr/lib/jvm/java-7-oracle' >> ~/.bashrc

Download Spark 1.5 with pre-build for Hadoop 2.6 and higher (http://spark.apache.org/downloads.html)
Don't forget to replace **USER:GROUP** with your linux user and group 
    
    sudo mkdir /opt/dev
    id -un  # your user
    id -gn  # your group
	sudo chown USER:GROUP /opt/dev
	cd /opt/dev
	wget -O - http://apache.lauf-forum.at/spark/spark-1.5.2/spark-1.5.2-bin-hadoop2.6.tgz | tar xz -C /opt/dev

Install ssh daemon and create a public/private key pair

    sudo apt-get install ssh
    mkdir -p ~/.ssh  # create if not exists
    chmod 700 ~/.ssh  # set proper permissions
    cd ~/.ssh/
	ssh-keygen -t rsa
	cat id_rsa.pub >> authorized_keys
	ssh localhost  # should work now
	exit
    
### Postgres Installation and Setup
More details can be found here https://help.ubuntu.com/community/PostgreSQL

Server installation from a repository

    sudo apt-get install postgresql postgresql-contrib libpq-dev
	sudo service postgresql status  # if everything is fine should return 9.3/main (port 5432): online
	
Make local ipv4 connections password free. Put **trust** instead of **md5**
    
    sudo nano /etc/postgresql/9.3/main/pg_hba.conf
    ...
    # # IPv4 local connections:
    # host    all             all             127.0.0.1/32            md5

Don't forget to restart the database server

    sudo service postgresql restart
	
### ElasticSearch Installation and Setup

Download ElasticSearch deb package and install it

`wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb`
`dpkg -i /tmp/elasticsearch-2.3.3.deb`

Make ElasticSearch listen to any ip address

`nano /etc/elasticsearch/elasticsearch.yml`
set network.host: 0.0.0.0

Restart ElasticSearch daemon

`sudo service elasticsearch restart`

### SM Engine Installation and Setup

Install git, pip and libffi

	sudo apt-get install git python-pip libffi-dev
	
Clone the latest "stable" **SM_distributed** repository branch into the home directory and rename it to **sm**
	
	git clone -b v0.4 --single-branch https://github.com/SpatialMetabolomics/SM_distributed.git
	mv SM_distributed sm

Install SM engine python dependencies

	sudo pip install numpy==1.10.4
	sudo pip install scipy==0.16.0 Fabric==1.10.2 lxml==3.3.3 pypng==0.0.18 matplotlib==1.5.0 mock==1.3.0 psycopg2==2.6.1 py4j==0.9 pytest==2.8.2 tornado==4.2.1 tornpsql==1.1.0
	sudo pip install --no-deps pyimzML==1.0.1
	sudo pip install pyMSpec pyImagingMSpec cpyMSpec cpyImagingMSpec

Postgres server setup for local use only

	sudo -u postgres psql
	
	CREATE ROLE sm LOGIN CREATEDB NOSUPERUSER PASSWORD '<PASSWORD>';
	CREATE DATABASE sm WITH OWNER sm;
	\q  # exit
	
SM engine schema creation

	cd sm
	psql -h localhost -U sm sm < scripts/create_schema.sql

ElasticSearch index creation

`python scripts/create_es_index.py --drop`

Create data directory for SM engine. Don't forget to replace **USER:GROUP** with your linux user and group (id -un and id -gn)

	sudo mkdir /opt/data && sudo chown USER:GROUP /opt/data

Create config files. For **conf/config.json** put values for needed parameters

	cd sm/conf
	cp sm_log.cfg.template sm_log.cfg
	cp config.json.template config.json
	nano config.json
	# replace fs.data_dir with path to the data directory (/opt/data)

Add environment variables to .bashrc

    echo -e '\nexport SPARK_HOME=/opt/dev/spark-1.5.2-bin-hadoop2.6' >> ~/.bashrc
    echo 'export PYTHONPATH=$HOME/sm:/opt/dev/spark-1.5.2-bin-hadoop2.6/python:/opt/dev/spark-1.5.2-bin-hadoop2.6/python/build:$PYTHONPATH' >> ~/.bashrc
    source ~/.bashrc
    
### Run Tests
Run the engine unit tests

	cd ~/sm
    ./test_runner.py --unit

Run the regression tests

    ./test_runner.py --regr

Download and import molecule database (HMDB) first

    wget https://s3-eu-west-1.amazonaws.com/sm-engine/hmdb.csv
    python scripts/import_molecule_formula_db.py HMDB hmdb.csv<commit_msg>README correction<commit_after># Spatial Metabolomics Engine [![Build Status](https://travis-ci.org/SpatialMetabolomics/SM_distributed.svg?branch=master)](https://travis-ci.org/SpatialMetabolomics/SM_distributed) [![Documentation Status](https://readthedocs.org/projects/sm-distributed/badge/?version=latest)](http://sm-distributed.readthedocs.org/en/latest/?badge=latest)
High throughput molecules annotation for imaging mass spectrometry datasets.

## Main Features
- centroided imzML files as input dataset files
- Apache Spark based implementation for easy scaling from one machine to a cluster
- Can be run in both local and distributed modes
- Comes with unit, regression and scientific tests
- Web application for browsing the search results
- Fabric script for easy deploy and an AWS cluster start

## Installation Manual
### Spark Installation and Setup

Install Oracle Java 7

    sudo apt-add-repository ppa:webupd8team/java
	sudo apt-get update
	sudo apt-get install oracle-java7-installer
	java -version  # should work now

Add an env variable to .bashrc
    
    echo -e '\nexport JAVA_HOME=/usr/lib/jvm/java-7-oracle' >> ~/.bashrc

Download Spark 1.5 with pre-build for Hadoop 2.6 and higher (http://spark.apache.org/downloads.html)
Don't forget to replace **USER:GROUP** with your linux user and group 
    
    sudo mkdir /opt/dev
    id -un  # your user
    id -gn  # your group
	sudo chown USER:GROUP /opt/dev
	cd /opt/dev
	wget -O - http://apache.lauf-forum.at/spark/spark-1.5.2/spark-1.5.2-bin-hadoop2.6.tgz | tar xz -C /opt/dev

Install ssh daemon and create a public/private key pair

    sudo apt-get install ssh
    mkdir -p ~/.ssh  # create if not exists
    chmod 700 ~/.ssh  # set proper permissions
    cd ~/.ssh/
	ssh-keygen -t rsa
	cat id_rsa.pub >> authorized_keys
	ssh localhost  # should work now
	exit
    
### Postgres Installation and Setup
More details can be found here https://help.ubuntu.com/community/PostgreSQL

Server installation from a repository

    sudo apt-get install postgresql postgresql-contrib libpq-dev
	sudo service postgresql status  # if everything is fine should return 9.3/main (port 5432): online
	
Make local ipv4 connections password free. Put **trust** instead of **md5**
    
    sudo nano /etc/postgresql/9.3/main/pg_hba.conf
    ...
    # # IPv4 local connections:
    # host    all             all             127.0.0.1/32            md5

Don't forget to restart the database server

    sudo service postgresql restart
	
### ElasticSearch Installation and Setup

Download ElasticSearch deb package and install it

`wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb`
`dpkg -i /tmp/elasticsearch-2.3.3.deb`

Make ElasticSearch listen to any ip address

`nano /etc/elasticsearch/elasticsearch.yml`
set network.host: 0.0.0.0

Restart ElasticSearch daemon

`sudo service elasticsearch restart`

### SM Engine Installation and Setup

Install git, pip and libffi

	sudo apt-get install git python-pip libffi-dev
	
Clone the latest "stable" **SM_distributed** repository branch into the home directory and rename it to **sm**
	
	git clone -b v0.4 --single-branch https://github.com/SpatialMetabolomics/SM_distributed.git
	mv SM_distributed sm

Install SM engine python dependencies

	sudo pip install numpy==1.10.4
	sudo pip install scipy==0.16.0 Fabric==1.10.2 lxml==3.3.3 pypng==0.0.18 matplotlib==1.5.0 mock==1.3.0 psycopg2==2.6.1 py4j==0.9 pytest==2.8.2 tornado==4.2.1 tornpsql==1.1.0
	sudo pip install --no-deps pyimzML==1.0.1
	sudo pip install pyMSpec pyImagingMSpec cpyMSpec cpyImagingMSpec

Postgres server setup for local use only

	sudo -u postgres psql
	
	CREATE ROLE sm LOGIN CREATEDB NOSUPERUSER PASSWORD '<PASSWORD>';
	CREATE DATABASE sm WITH OWNER sm;
	\q  # exit

Create config files. For **conf/config.json** put values for needed parameters

	cd sm/conf
	cp sm_log.cfg.template sm_log.cfg
	cp config.json.template config.json
	nano config.json
	# replace fs.data_dir with path to the data directory (e.g. /opt/data)

SM engine schema creation

	cd sm
	psql -h localhost -U sm sm < scripts/create_schema.sql

ElasticSearch index creation

`python scripts/create_es_index.py --drop`

Create data directory for SM engine. Don't forget to replace **USER:GROUP** with your linux user and group (id -un and id -gn)

	sudo mkdir /opt/data && sudo chown USER:GROUP /opt/data

Add environment variables to .bashrc

    echo -e '\nexport SPARK_HOME=/opt/dev/spark-1.5.2-bin-hadoop2.6' >> ~/.bashrc
    echo 'export PYTHONPATH=$HOME/sm:/opt/dev/spark-1.5.2-bin-hadoop2.6/python:/opt/dev/spark-1.5.2-bin-hadoop2.6/python/build:$PYTHONPATH' >> ~/.bashrc
    source ~/.bashrc
    
### Run Tests
Run the engine unit tests

	cd ~/sm
    ./test_runner.py --unit

Run the regression tests

    ./test_runner.py --regr

Download and import molecule database (HMDB) first

    wget https://s3-eu-west-1.amazonaws.com/sm-engine/hmdb.csv
    python scripts/import_molecule_formula_db.py HMDB hmdb.csv
{'fraction_non_alphanumeric': 0.1002381897578404, 'fraction_numerical': 0.022826518459706232, 'mean_word_length': 3.551490514905149, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 21, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4680333', 'n_tokens_mistral': 3534, 'n_tokens_neox': 3255, 'n_words': 1049}