<commit_before># GomPlex

GomPlex is a machine learning toolkit designed for predictive modeling of
complex-valued predictions. The key of success is to find a tailormade function
that maps the inputs to the targets. In this sense, linear functions undoubtedly
would be too simple to solve the problem. Nevertheless, non-linear functions
with much flexibility are mostly concerned. In fact, there are uncountable ways
to define an non-linear function, and it's generally hard to tell which class of
mathematical functions specifically works for a problem. A machine-learning kind
of approach is to approximate such a function by 'supervising' data and
'learning' patterns. In statistics, this is traditionally coined
'regression'. Although such a data-driven function can be obtained through
optimization, the optimized models tend to lose generality, which is technically
regarded as 'overfitting'.

To prevent from overfitting, one feasible approach is to carry out Bayesian
inference over 'the distribution of functions'. In this sense, our desired
function should be a sample of certain function space. A strictforward approach
would be defining the distribution of parameters that can specify the class of
function. Another way would be taking advantages of well-established stochastic
processes. Due to mathematical brevity and elegance, Gaussian process was
employed to describe the distribution over functions. [<NAME> and
<NAME>](http://www.gaussianprocess.org/gpml/), who pioneer
and popularize the idea of using Gaussian processes for machine learning tasks,
emphasize that one of the greatest advantages of Gaussian process is that we can
integrate all possible functions over the function distribution (Gaussian
process), and obtain an analytical solution because of nice properties of
Gaussian. It is pinpointed that this Bayesian routine is prefered over
optimization on a certain estimate of function.

The idea of GomPlex came from [Quasi-Monte Carlo Feature Maps](http://jmlr.org/papers/volume17/14-538/14-538.pdf)
and [Fourier features](https://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf)
, in which the feature maps are obtained by sampling on the frequency distribution corresponding to a pre-determined
kernel function. In this setup, the randomized feature maps is no more than an approximation method.
Yet, in GomPlex, we take a new scope for the association of feature maps and kernel function.
We treat the feature maps as hyperparameters, and result in optimization of the mapping on the Gaussian process regression likelihood.
In this sence, we optimize the kernel properties without explicitly define a kernel.
One significant hurdle of this approach is the explosive amount of hyperparameters,
which in turns require careful regularization on optimization.

### Highlights of GomPlex

- GomPlex optimizes
the Fourier features so as to "learn" a tailmade covariance matrix from the data. 
This removes the necessity of deciding which kernel function to use in different problems.

- GomPlex implements a variety of formulation to transform the optimized Fourier features to covariance matrix, including the typical sin-cos concatenation introduced by [Miguel](http://www.jmlr.org/papers/v11/lazaro-gredilla10a.html), and the generalized approach described by [Yarin](http://jmlr.org/proceedings/papers/v37/galb15.html).

- GomPlex uses low-rank frequency matrix for sparse approximation of Fourier features. It is 
intended to show that low-rank frequency matrix is able to lower the computational 
burden in each step of optimization, and also render faster convergence and a stabler result.

- Compared with other state-of-the-art regressors, GomPlex usually gives the most accurate prediction on the benchmark datasets of regression.

# Installation
   
### GomPlex

To install GomPlex, clone this repo:

    $ git clone https://github.com/MaxInGaussian/GomPlex.git
    $ python setup.py install
   
# Try GomPlex with Only 3 Lines of Code
```python
from GomPlex import *
# <>: necessary inputs, {}: optional inputs
gp = GomPlex(<sparsity>)
gp.fit(<X>, <y>, {opt_rate}, {max_iter}, {iter_tol}, {nlml_tol})
predict_mean, predict_std = gp.predict(<new_X>)
```

# Analyze Training Process on Real Time by Plot
## Training on One-dimensional Inputs
```python
gp.fit(X, y, plot=True)
```
# Toy Example: Custom 1-d Function
## True Function
<p align="center">
<img src ="http://www.sciweavers.org/upload/Tex2Img_1493629203/eqn.png" />
</p>
<img src ="tests/demo_3d_plot_true_function.png" />

## Synthesize Data
<p align="center"><commit_msg>commit from Maxs-Research-Lab.local by MaxLam<commit_after># GomPlex

GomPlex is a machine learning toolkit designed for predictive modeling of
complex-valued predictions. The key of success is to find a tailormade function
that maps the inputs to the targets. In this sense, linear functions undoubtedly
would be too simple to solve the problem. Nevertheless, non-linear functions
with much flexibility are mostly concerned. In fact, there are uncountable ways
to define an non-linear function, and it's generally hard to tell which class of
mathematical functions specifically works for a problem. A machine-learning kind
of approach is to approximate such a function by 'supervising' data and
'learning' patterns. In statistics, this is traditionally coined
'regression'. Although such a data-driven function can be obtained through
optimization, the optimized models tend to lose generality, which is technically
regarded as 'overfitting'.

To prevent from overfitting, one feasible approach is to carry out Bayesian
inference over 'the distribution of functions'. In this sense, our desired
function should be a sample of certain function space. A strictforward approach
would be defining the distribution of parameters that can specify the class of
function. Another way would be taking advantages of well-established stochastic
processes. Due to mathematical brevity and elegance, Gaussian process was
employed to describe the distribution over functions. [<NAME>en and
<NAME>](http://www.gaussianprocess.org/gpml/), who pioneer
and popularize the idea of using Gaussian processes for machine learning tasks,
emphasize that one of the greatest advantages of Gaussian process is that we can
integrate all possible functions over the function distribution (Gaussian
process), and obtain an analytical solution because of nice properties of
Gaussian. It is pinpointed that this Bayesian routine is prefered over
optimization on a certain estimate of function.

The idea of GomPlex came from [Quasi-Monte Carlo Feature Maps](http://jmlr.org/papers/volume17/14-538/14-538.pdf)
and [Fourier features](https://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf)
, in which the feature maps are obtained by sampling on the frequency distribution corresponding to a pre-determined
kernel function. In this setup, the randomized feature maps is no more than an approximation method.
Yet, in GomPlex, we take a new scope for the association of feature maps and kernel function.
We treat the feature maps as hyperparameters, and result in optimization of the mapping on the Gaussian process regression likelihood.
In this sence, we optimize the kernel properties without explicitly define a kernel.
One significant hurdle of this approach is the explosive amount of hyperparameters,
which in turns require careful regularization on optimization.

### Highlights of GomPlex

- GomPlex optimizes
the Fourier features so as to "learn" a tailmade covariance matrix from the data. 
This removes the necessity of deciding which kernel function to use in different problems.

- GomPlex implements a variety of formulation to transform the optimized Fourier features to covariance matrix, including the typical sin-cos concatenation introduced by [Miguel](http://www.jmlr.org/papers/v11/lazaro-gredilla10a.html), and the generalized approach described by [Yarin](http://jmlr.org/proceedings/papers/v37/galb15.html).

- GomPlex uses low-rank frequency matrix for sparse approximation of Fourier features. It is 
intended to show that low-rank frequency matrix is able to lower the computational 
burden in each step of optimization, and also render faster convergence and a stabler result.

- Compared with other state-of-the-art regressors, GomPlex usually gives the most accurate prediction on the benchmark datasets of regression.

# Installation
   
### GomPlex

To install GomPlex, clone this repo:

    $ git clone https://github.com/MaxInGaussian/GomPlex.git
    $ python setup.py install
   
# Try GomPlex with Only 3 Lines of Code
```python
from GomPlex import *
# <>: necessary inputs, {}: optional inputs
gp = GomPlex(<sparsity>)
gp.fit(<X>, <y>, {opt_rate}, {max_iter}, {iter_tol}, {nlml_tol})
predict_mean, predict_std = gp.predict(<new_X>)
```
## Analyze Training Process by Real-Time Plot
```python
gp.fit(X, y, plot=True)
```
# Toy Example: Custom 1-d Function
## True Function
<p align="center">
<img src ="http://www.sciweavers.org/upload/Tex2Img_1493629203/eqn.png" />
</p>
<img src ="tests/demo_3d_plot_true_function.png" />

## Synthesize Data
<p align="center">
{'fraction_non_alphanumeric': 0.06326774649418415, 'fraction_numerical': 0.008261767583432982, 'mean_word_length': 5.036745406824147, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '545660', 'n_tokens_mistral': 2508, 'n_tokens_neox': 2312, 'n_words': 1173}