<commit_before># 4most-4gp

This is the code for the IWG7 Galactic pipeline for the 4MOST multi-object
spectrograph. It comprises a collection of Python modules which use a common data format to
store and manipulate spectra and their associated metadata. It makes it easy to pass spectra
between a range of spectral synthesis and processing tools including Turbospectrum, the
4MOST Facility Simulator (4FS) and the Cannon, without the need for manual data format
conversion. It includes the ability to store spectra in libraries and search for them by arbitrary
metadata constraints, making it easy to create new tests on subgroups of stars filtered from
larger samples.

In addition, a simple web interface allows the contents of spectrum libraries to be searched
and viewed quickly for diagnostic purposes.

The framework is available in two repositories on GitHub, and includes step-by-step
installation instructions. The first repository contains the Python modules which provide
programmatic interfaces for creating and manipulating libraries of spectra, including wrappers
for passing them to various analysis tools:

[https://github.com/dcf21/4most-4gp]

The second repository contains python scripts which utilise these modules to synthesise the
spectra and perform the tests described in this report:

[https://github.com/dcf21/4most-4gp-scripts]

This code is under active development, and stable releases are periodically
made.

Visiting the GitHub URLs above will present you with the `master` branch of our
code, which should always correspond to the latest stable release. If you click
on the "branches" dropdown menu, you can select a different version of the code
to download.

Stable releases are given date stamps, for example, `release-2018-01-09-1`. The
master branch points to the most recent release. The `dev` branch is not stable
and should be used with extreme caution.

# Code structure

The code is organised into a collection of python modules, most of which do not
depend on each other. These can be found in the directory `src/pythonModules`.

The code is divided up in this way since each module has different dependencies,
and it allows functions to be used without installing every dependency.

It is strongly recommended that they be installed in the python virtual
environment as described below, and that you do not tamper with your
system-wide python installation.

The python modules are as follows:

**fourgp_speclib** - Defines core classes representing spectra and libraries to keep them in. Functionality to search for spectra within libraries is provided using both SQLite and MySQL. All other 4GP modules depend on this core module.

**fourgp_specsynth** -- Provides a wrapper for synthesising spectra with given stellar parameters using Turbospectrum.

**fourgp_fourfs** -- Provides a wrapper for degrading spectra using 4FS; specifically, using the 4MOST Exposure Time Calculator (ETC).

**fourgp_cannon** -- Provides a wrapper for passing arrays of spectra to the Cannon.

**fourgp_rv** -- Provides a simple MCMC code for estimating the radial velocities of spectra.

**fourgp_degrade** -- Provides very simple functions for convolving spectra with Gaussians and for interpolating them onto new wavelength rasters.

**fourgp_telescope_data** -- Provides some basic data about 4MOST, including its wavelength coverage and spectral resolution.

# Getting 4GP to do useful things

All of this code in this repository is simply a collection of python modules which provide a programmatic interface for handling spectra. There is no example code here to show how to use these modules in practice.

Such examples can be found separately in the respository [4most-4gp-scripts](https://github.com/dcf21/4most-4gp-scripts).

# Installing 4GP

For more complete instructions, visit the [Wiki](https://github.com/dcf21/4most-4gp/wiki/dependencies) on our GitHib page.

## Dependencies

Before you start, you need to have all the dependencies you need to build Turbospectrum, 4FS, and the other tools that 4GP wraps.

4GP currently only runs on python 2.7.

These packages and libraries are required:

* *git* (required to check the code out from GitHub)
* *SQLite3* (including the python-sqlite3 binding; you can test for this by typing `import sqlite3` into a python terminal)
* *python-tk* (required to use the 4GP Spectrum Browser and the Cannon; you can test for this by typing `import tkinter` into a python terminal)

The following packages are strongly recommended:

* *python-virtualenv* (required to set up a python virtual environment)

The following packages are needed to run certain parts of 4GP:

* *libchealpix-dev* (required to build and install 4FS)
* *python-healpy* (required to build and install 4FS)
* *libcfitsio3-dev* (required to build and install 4FS)
* *gfortran* (required to build and install Turbospectrum)
* *pyphot* (required to do photometry on spectra)
* *MySQL* (including the libmysqlclient and python-mysql bindings -- currently only required by the 4GP unit tests, so not very important)

## Ubuntu installation instructions

Under Ubuntu Linux, you can install all these packages with a single command, as follows:

Ubuntu 16.04:

```
apt-get install git python-sqlite3 mysql-server libmysqlclient-dev python-virtualenv libhealpix-cxx-dev libchealpix-dev libcfitsio3-dev python-healpy gfortran python-tk
```

Ubuntu 14.04:

```
apt-get install git python-sqlite mysql-server libmysqlclient-dev python-virtualenv libcfitsio3-dev libblas-dev liblapack-dev libblas3gf liblapack3gf gfortran python-tk
```

## Installing 4GP in a python virtual environment

4GP is distributed with a standard `setuptools` installation script `setup.py` which allows you to install its
constituent modules into your local python environment.

We strongly recommended that you install them in a python virtual
environment, rather than tampering with your
system-wide python installation.

Follow these steps in a Linux shell to do this:

```
# Check out code from GitHub
git clone https://github.com/dcf21/4most-4gp.git
 
# Set up a python virtual environment
virtualenv virtualenv
source virtualenv/bin/activate
pip install numpy scipy astropy MySQL-python flask tables
 
# Install 4GP code
cd src/pythonModules/fourgp_speclib
python setup.py install
cd ../fourgp_cannon
python setup.py install<commit_msg>Updated docs for installing on 14.04<commit_after># 4most-4gp

This is the code for the IWG7 Galactic pipeline for the 4MOST multi-object
spectrograph. It comprises a collection of Python modules which use a common data format to
store and manipulate spectra and their associated metadata. It makes it easy to pass spectra
between a range of spectral synthesis and processing tools including Turbospectrum, the
4MOST Facility Simulator (4FS) and the Cannon, without the need for manual data format
conversion. It includes the ability to store spectra in libraries and search for them by arbitrary
metadata constraints, making it easy to create new tests on subgroups of stars filtered from
larger samples.

In addition, a simple web interface allows the contents of spectrum libraries to be searched
and viewed quickly for diagnostic purposes.

The framework is available in two repositories on GitHub, and includes step-by-step
installation instructions. The first repository contains the Python modules which provide
programmatic interfaces for creating and manipulating libraries of spectra, including wrappers
for passing them to various analysis tools:

[https://github.com/dcf21/4most-4gp]

The second repository contains python scripts which utilise these modules to synthesise the
spectra and perform the tests described in this report:

[https://github.com/dcf21/4most-4gp-scripts]

This code is under active development, and stable releases are periodically
made.

Visiting the GitHub URLs above will present you with the `master` branch of our
code, which should always correspond to the latest stable release. If you click
on the "branches" dropdown menu, you can select a different version of the code
to download.

Stable releases are given date stamps, for example, `release-2018-01-09-1`. The
master branch points to the most recent release. The `dev` branch is not stable
and should be used with extreme caution.

# Code structure

The code is organised into a collection of python modules, most of which do not
depend on each other. These can be found in the directory `src/pythonModules`.

The code is divided up in this way since each module has different dependencies,
and it allows functions to be used without installing every dependency.

It is strongly recommended that they be installed in the python virtual
environment as described below, and that you do not tamper with your
system-wide python installation.

The python modules are as follows:

**fourgp_speclib** - Defines core classes representing spectra and libraries to keep them in. Functionality to search for spectra within libraries is provided using both SQLite and MySQL. All other 4GP modules depend on this core module.

**fourgp_specsynth** -- Provides a wrapper for synthesising spectra with given stellar parameters using Turbospectrum.

**fourgp_fourfs** -- Provides a wrapper for degrading spectra using 4FS; specifically, using the 4MOST Exposure Time Calculator (ETC).

**fourgp_cannon** -- Provides a wrapper for passing arrays of spectra to the Cannon.

**fourgp_rv** -- Provides a simple MCMC code for estimating the radial velocities of spectra.

**fourgp_degrade** -- Provides very simple functions for convolving spectra with Gaussians and for interpolating them onto new wavelength rasters.

**fourgp_telescope_data** -- Provides some basic data about 4MOST, including its wavelength coverage and spectral resolution.

# Getting 4GP to do useful things

All of this code in this repository is simply a collection of python modules which provide a programmatic interface for handling spectra. There is no example code here to show how to use these modules in practice.

Such examples can be found separately in the respository [4most-4gp-scripts](https://github.com/dcf21/4most-4gp-scripts).

# Installing 4GP

For more complete instructions, visit the [Wiki](https://github.com/dcf21/4most-4gp/wiki/dependencies) on our GitHib page.

## Dependencies

Before you start, you need to have all the dependencies you need to build Turbospectrum, 4FS, and the other tools that 4GP wraps.

4GP currently only runs on python 2.7.

These packages and libraries are required:

* *git* (required to check the code out from GitHub)
* *SQLite3* (including the python-sqlite3 binding; you can test for this by typing `import sqlite3` into a python terminal)
* *python-tk* (required to use the 4GP Spectrum Browser and the Cannon; you can test for this by typing `import tkinter` into a python terminal)

The following packages are strongly recommended:

* *python-virtualenv* (required to set up a python virtual environment)

The following packages are needed to run certain parts of 4GP:

* *libchealpix-dev* (required to build and install 4FS)
* *python-healpy* (required to build and install 4FS)
* *libcfitsio3-dev* (required to build and install 4FS)
* *gfortran* (required to build and install Turbospectrum)
* *pyphot* (required to do photometry on spectra)
* *MySQL* (including the libmysqlclient and python-mysql bindings -- currently only required by the 4GP unit tests, so not very important)

## Ubuntu installation instructions

Under Ubuntu Linux, you can install all these packages with a single command, as follows:

Ubuntu 16.04:

```
apt-get install git python-sqlite mysql-server libmysqlclient-dev python-virtualenv libhealpix-cxx-dev libchealpix-dev libcfitsio3-dev python-healpy gfortran python-tk python-matplotlib sqlite3 python-dev
```

Ubuntu 14.04:

```
apt-get install git python-sqlite mysql-server libmysqlclient-dev python-virtualenv libcfitsio3-dev libblas-dev liblapack-dev libblas3gf liblapack3gf gfortran python-tk python-matplotlib sqlite3 python-dev
apt-get build-dep python-matplotlib
```

Note that owing to this issue described on [StackOverflow](https://stackoverflow.com/questions/17426087/why-pythonsqlite3-is-extremely-slow), the pipeline runs very slowly on Ubuntu 14.04.


## Installing 4GP in a python virtual environment

4GP is distributed with a standard `setuptools` installation script `setup.py` which allows you to install its
constituent modules into your local python environment.

We strongly recommended that you install them in a python virtual
environment, rather than tampering with your
system-wide python installation.

Follow these steps in a Linux shell to do this:

```
# Check out code from GitHub
git clone https://github.com/dcf21/4most-4gp.git
 
# Set up a python virtual environment
virtualenv virtualenv
source virtualenv/bin/activate
pip install numpy scipy astropy MySQL-python flask matplotlib tables
 
# Install 4GP code
cd src/pythonModules/fourgp_speclib
python setup.py install
cd ../fourgp_cannon
python setup.py install
{'fraction_non_alphanumeric': 0.05134099616858238, 'fraction_numerical': 0.012720306513409961, 'mean_word_length': 4.86561797752809, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '272130', 'n_tokens_mistral': 3519, 'n_tokens_neox': 3286, 'n_words': 1765}