<commit_before>{
 "metadata": {
  "name": "Regression"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Regression Models"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "By <NAME> - <a href='https://github.com/Saurabh7'>github.com/Saurabh7</a>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook demonstrates various regression methods provided in shogun. Linear models like Ridge regression, Least Square regression, Least Angle regression. etc and also kernel based methods like Kernel Ridge regression are discussed along with real life examples."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Regression analysis is mainly used to estimate relationship and dependence between variables. This finds applications in prediction and hence in machine Learning. The goal  is to determine the values of parameters for a function that cause the function to best fit a set of data observations that you provide. In linear regression, the function is a linear (straight-line) equation.\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Mathematical Formulation (skip if you just want code examples)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A simple regression model can be defined as $\\textrm y =$  $\\textbf {w.x} $ $+ b$. Here $\\text y$ is the predicted value, $\\text x$ the independent variable and $\\text w$ the so called weights.</br> We need best fit a line to this data. One way to do this is using [Ordinary Least Sqaures method](http://en.wikipedia.org/wiki/Ordinary_least_squares). This method minimizes the sum of squared vertical distances between the observed responses in the dataset and the responses predicted by the linear approximation. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The vertical distances called residues have to minimized. This can be represented as:$$\\frac{1}{2}\\left(\\sum_{i=1}^N(y_i-{\\bf w}\\cdot {\\bf x}_i)^2\\right)$$\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ridge regression is like least squares but shrinks the estimated coe\u000efficients towards zero. In the [Ridge regression](http://en.wikipedia.org/wiki/Tikhonov_regularization) form, following system has to be minimized:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$$\\frac{1}{2}\\left(\\sum_{i=1}^N(y_i-{\\bf w}\\cdot {\\bf x}_i)^2 + \\tau||{\\bf w}||^2\\right)$$\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here $\\tau$ is some suitably chosen Tikhonov matrix. In many cases, this matrix is chosen as a multiple of the identity matrix $\\tau$ $\\bf I$ . It imposes a penalty on the weights.</br> Another method is [Least Angle Regression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastAngleRegression.html) (LARS) which is used to solve:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " $$ \\min \\|X^T\\beta - y\\|^2 + \\lambda\\|\\beta\\|_1\n",
      "$$</br>which is L1-Regularized form of the Least Square Regression where:$$\\|\\beta\\|_1 = \\sum_i|\\beta_i|\n",
      "$$</br>In Shogun, following equivalent form is solved:\n",
      "$$\\min \\|X^T\\beta - y\\|^2 \\quad s.t. \\|\\beta\\|_1 \\leq C\n",
      "$$This regularized form solved by using LARS algorithm is known as LASSO (Least Absolute Shrinkage and Selection Operator)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Prediction using Least Squares"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Regression using Least squares is demonstrated below on toy data. Shogun provides the tool to do it using [CLeastSquaresRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastSquaresRegression.html) class. The data is a straight line with lot of noise and having slope 3. Comparing with the mathematical equation above we thus expect $\\text w$ to be around 3 for a good prediction. Once the data is converted to Shogun format, we are ready to train the machine. To label the training data [CRegressionLabels](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CRegressionLabels.html) are used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# import all shogun classes\n",
      "from modshogun import *\n",
      "slope=3\n",
      "\n",
      "X_train=rand(30)*10\n",
      "y_train=slope*(X_train)+random.randn(30)*2+2\n",
      "y_true=slope*(X_train)+2\n",
      "X_test=concatenate((linspace(0,10, 50),X_train))\n",
      "\n",
      "#Convert data to shogun format features\n",
      "feats_train=RealFeatures(X_train.reshape(1,len(X_train)))\n",
      "feats_test=RealFeatures(X_test.reshape(1,len(X_test)))\n",
      "labels_train=RegressionLabels(y_train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Training and generating weights"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[LeastSquaresRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastSquaresRegression.html) has to be initialised with the training features and training labels. Once that is done to learn from data we `train()` it. This also generates the $\\text w$ from the general equation described above. To access $\\text w$ use `get_w()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ls=LeastSquaresRegression(feats_train, labels_train)\n",
      "ls.train()\n",
      "w=ls.get_w()\n",
      "print 'Weights:'\n",
      "print w"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This value of $\\text w$ is pretty close to 2, which certifies a pretty good fit for the training data. Now let's `apply` this trained machine to our test data to get the ouput values."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out=ls.apply(feats_test).get_labels()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As an aid to visualisation, a plot of the output and also of the residues is shown. The sum of the squares of these residues is minimised. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "figure(figsize=(20,5))\n",
      "#Regression and true plot\n",
      "pl1=subplot(131)\n",
      "title('Regression')\n",
      "_=plot(X_train,labels_train, 'ro')\n",
      "_=plot(X_test,out, color='blue')\n",
      "_=plot(X_train, y_true, color='green')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "p3 = Rectangle((0, 0), 1, 1, fc=\"g\")\n",
      "pl1.legend((p1, p2, p3), [\"Training samples\", \"Predicted output\", \"True relationship\"], loc=2)\n",
      "\n",
      "#plot residues\n",
      "pl2=subplot(132)\n",
      "title(\"Squared error and output\")\n",
      "_=plot(X_test,out, linewidth=2)\n",
      "gray()\n",
      "_=scatter(X_train,labels_train,c=ones(50) ,cmap=gray(), s=40)\n",
      "for i in range(50,80):\n",
      "    plot([X_test[i],X_test[i]],[out[i],y_train[i-50]] , linewidth=2, color='red')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "pl2.legend((p1, p2), [\"Error/residue to be squared\", \"Predicted output\"], loc=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Ridge Regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ridge regression can be performed in Shogun using [CLinearRidgeRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLinearRidgeRegression.html) class. It takes the regularization constant $\\tau$ as an additional argument. Let us see the basic regression example solved using the same."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tau=0.8\n",
      "rr=LinearRidgeRegression(tau, feats_train, labels_train)\n",
      "rr.train()\n",
      "w=rr.get_w()\n",
      "print w\n",
      "out=rr.apply(feats_test).get_labels()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "figure(figsize=(20,5))\n",
      "#Regression and true plot\n",
      "pl1=subplot(131)\n",
      "title('Ridge Regression')\n",
      "_=plot(X_train,labels_train, 'ro')\n",
      "_=plot(X_test, out, color='blue')\n",
      "_=plot(X_train, y_true, color='green')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "p3 = Rectangle((0, 0), 1, 1, fc=\"g\")\n",
      "pl1.legend((p1, p2, p3), [\"Training samples\", \"Predicted output\", \"True relationship\"], loc=2)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Relationship between weights and regularization"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The prediction in the basic regression example was simliar to that of least squares one. To actually see ridge regression's forte, we analyse how the weights change along with the regularization constant. Data with slightly higher dimensions is sampled from the standard normal distribution to do this. Here `set_tau()` method is used to set the necessary parameter."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#sample some data\n",
      "X=rand(10)*1.5\n",
      "for i in range(9):\n",
      "    x=random.standard_normal(10)*0.0005\n",
      "    X=vstack((X, x))\n",
      "y=ones(10)\n",
      "print '(Dim, shape) of training data;'\n",
      "print X.shape\n",
      "\n",
      "feats_train=RealFeatures(X)\n",
      "labels_train=RegressionLabels(y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "n = 200\n",
      "taus = logspace(-8, -2, n)\n",
      "weights=[]\n",
      "rr=LinearRidgeRegression(tau, feats_train, labels_train)\n",
      "\n",
      "#vary regularization\n",
      "for t in taus:\n",
      "    rr.set_tau(t)\n",
      "    rr.train()\n",
      "    weights.append(rr.get_w())\n",
      "    \n",
      "figure(figsize=(12,6))\n",
      "title('Relationship of weights and regularisation')\n",
      "ax = gca()\n",
      "ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])\n",
      "\n",
      "ax.plot(taus, weights, linewidth=2)\n",
      "xlabel('Tau', fontsize=12)\n",
      "ylabel('Weights', fontsize=12)\n",
      "ax.set_xscale('log')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is the famous ridge trace that is the signature of this technique. The plot is really very straight forward to read. It presents the standardized regression coefficients(weights) on the vertical axis and various values of tau(Regularisation constant) along the horizontal axis. Since the values of tau($\\tau$) span several orders of magnitude, we adopt a logarithmic scale along this axis. As tau is increased, the values of the regression estimates change, often wildly at first. At some point, the coefficients seem to settle down and then gradually drift towards zero. The value of tau for which these coefficients are at their stable values is often the best one.\n"
     ]<commit_msg>lars section<commit_after>{
 "metadata": {
  "name": "Regression"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Regression Models"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "By <NAME> - <a href='https://github.com/Saurabh7'>github.com/Saurabh7</a>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook demonstrates various regression methods provided in shogun. Linear models like Ridge regression, Least Square regression, Least Angle regression, etc. and also kernel based methods like Kernel Ridge regression are discussed along with real life examples."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Regression analysis is mainly used to estimate relationship and dependence between variables. This finds applications in prediction and hence in machine Learning. The goal  is to determine the values of parameters for a function that cause the function to best fit a set of data observations that you provide. In linear regression, the function is a linear (straight-line) equation.\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Mathematical Formulation (skip if you just want code examples)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A simple regression model can be defined as $\\textrm y =$  $\\textbf {w.x} $ $+ b$. Here $\\text y$ is the predicted value, $\\text x$ the independent variable and $\\text w$ the so called weights.</br> We need best fit a line to this data. One way to do this is using [Ordinary Least Sqaures method](http://en.wikipedia.org/wiki/Ordinary_least_squares). This method minimizes the sum of squared vertical distances between the observed responses in the dataset and the responses predicted by the linear approximation. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The vertical distances called residues have to minimized. This can be represented as:$$\\frac{1}{2}\\left(\\sum_{i=1}^N(y_i-{\\bf w}\\cdot {\\bf x}_i)^2\\right)$$\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ridge regression is like least squares but shrinks the estimated coe\u000efficients towards zero. In the [Ridge regression](http://en.wikipedia.org/wiki/Tikhonov_regularization) form, following system has to be minimized:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$$\\frac{1}{2}\\left(\\sum_{i=1}^N(y_i-{\\bf w}\\cdot {\\bf x}_i)^2 + \\tau||{\\bf w}||^2\\right)$$\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here $\\tau$ is some suitably chosen Tikhonov matrix. In many cases, this matrix is chosen as a multiple of the identity matrix $\\tau$ $\\bf I$ . It imposes a penalty on the weights.</br> Another method is [Least Angle Regression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastAngleRegression.html)(LARS). LARS is essentially [forward stagewise](http://en.wikipedia.org/wiki/Stepwise_regression) made fast. Instead of making tiny hops in the direction of one variable at a time, LARS makes optimally-sized leaps in optimal directions. These directions are chosen to make equal angles (equal correlations) with each of the variables currently in our model(equiangular).  LARS is used to solve:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " $$ \\min \\|X^T\\beta - y\\|^2 + \\lambda\\|\\beta\\|_1\n",
      "$$</br>which is L1-Regularized form of the Least Square Regression where:$$\\|\\beta\\|_1 = \\sum_i|\\beta_i|\n",
      "$$</br>In Shogun, following equivalent form is solved:\n",
      "$$\\min \\|X^T\\beta - y\\|^2 \\quad s.t. \\|\\beta\\|_1 \\leq C\n",
      "$$This regularized form solved by using LARS algorithm is known as LASSO (Least Absolute Shrinkage and Selection Operator)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Prediction using Least Squares"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Regression using Least squares is demonstrated below on toy data. Shogun provides the tool to do it using [CLeastSquaresRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastSquaresRegression.html) class. The data is a straight line with lot of noise and having slope 3. Comparing with the mathematical equation above we thus expect $\\text w$ to be around 3 for a good prediction. Once the data is converted to Shogun format, we are ready to train the machine. To label the training data [CRegressionLabels](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CRegressionLabels.html) are used."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# import all shogun classes\n",
      "from modshogun import *\n",
      "slope=3\n",
      "\n",
      "X_train=rand(30)*10\n",
      "y_train=slope*(X_train)+random.randn(30)*2+2\n",
      "y_true=slope*(X_train)+2\n",
      "X_test=concatenate((linspace(0,10, 50),X_train))\n",
      "\n",
      "#Convert data to shogun format features\n",
      "feats_train=RealFeatures(X_train.reshape(1,len(X_train)))\n",
      "feats_test=RealFeatures(X_test.reshape(1,len(X_test)))\n",
      "labels_train=RegressionLabels(y_train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Training and generating weights"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[LeastSquaresRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastSquaresRegression.html) has to be initialised with the training features and training labels. Once that is done to learn from data we `train()` it. This also generates the $\\text w$ from the general equation described above. To access $\\text w$ use `get_w()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ls=LeastSquaresRegression(feats_train, labels_train)\n",
      "ls.train()\n",
      "w=ls.get_w()\n",
      "print 'Weights:'\n",
      "print w"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This value of $\\text w$ is pretty close to 2, which certifies a pretty good fit for the training data. Now let's `apply` this trained machine to our test data to get the ouput values."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "out=ls.apply(feats_test).get_labels()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As an aid to visualisation, a plot of the output and also of the residues is shown. The sum of the squares of these residues is minimised. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "figure(figsize=(20,5))\n",
      "#Regression and true plot\n",
      "pl1=subplot(131)\n",
      "title('Regression')\n",
      "_=plot(X_train,labels_train, 'ro')\n",
      "_=plot(X_test,out, color='blue')\n",
      "_=plot(X_train, y_true, color='green')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "p3 = Rectangle((0, 0), 1, 1, fc=\"g\")\n",
      "pl1.legend((p1, p2, p3), [\"Training samples\", \"Predicted output\", \"True relationship\"], loc=2)\n",
      "\n",
      "#plot residues\n",
      "pl2=subplot(132)\n",
      "title(\"Squared error and output\")\n",
      "_=plot(X_test,out, linewidth=2)\n",
      "gray()\n",
      "_=scatter(X_train,labels_train,c=ones(50) ,cmap=gray(), s=40)\n",
      "for i in range(50,80):\n",
      "    plot([X_test[i],X_test[i]],[out[i],y_train[i-50]] , linewidth=2, color='red')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "pl2.legend((p1, p2), [\"Error/residue to be squared\", \"Predicted output\"], loc=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Ridge Regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ridge regression can be performed in Shogun using [CLinearRidgeRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLinearRidgeRegression.html) class. It takes the regularization constant $\\tau$ as an additional argument. Let us see the basic regression example solved using the same."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tau=0.8\n",
      "rr=LinearRidgeRegression(tau, feats_train, labels_train)\n",
      "rr.train()\n",
      "w=rr.get_w()\n",
      "print w\n",
      "out=rr.apply(feats_test).get_labels()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "figure(figsize=(20,5))\n",
      "#Regression and true plot\n",
      "pl1=subplot(131)\n",
      "title('Ridge Regression')\n",
      "_=plot(X_train,labels_train, 'ro')\n",
      "_=plot(X_test, out, color='blue')\n",
      "_=plot(X_train, y_true, color='green')\n",
      "p1 = Rectangle((0, 0), 1, 1, fc=\"r\")\n",
      "p2 = Rectangle((0, 0), 1, 1, fc=\"b\")\n",
      "p3 = Rectangle((0, 0), 1, 1, fc=\"g\")\n",
      "pl1.legend((p1, p2, p3), [\"Training samples\", \"Predicted output\", \"True relationship\"], loc=2)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Relationship between weights and regularization"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The prediction in the basic regression example was simliar to that of least squares one. To actually see ridge regression's forte, we analyse how the weights change along with the regularization constant. Data with slightly higher dimensions is sampled from the standard normal distribution to do this. Here `set_tau()` method is used to set the necessary parameter."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#sample some data\n",
      "X=rand(10)*1.5\n",
      "for i in range(9):\n",
      "    x=random.standard_normal(10)*0.0005\n",
      "    X=vstack((X, x))\n",
      "y=ones(10)\n",
      "print '(No. of attributes, No. of samples) of training data;'\n",
      "print X.shape\n",
      "\n",
      "feats_train=RealFeatures(X)\n",
      "labels_train=RegressionLabels(y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "n = 200\n",
      "taus = logspace(-8, -2, n)\n",
      "weights=[]\n",
      "rr=LinearRidgeRegression(tau, feats_train, labels_train)\n",
      "\n",
      "#vary regularization\n",
      "for t in taus:\n",
      "    rr.set_tau(t)\n",
      "    rr.train()\n",
      "    weights.append(rr.get_w())\n",
      "    \n",
      "figure(figsize=(12,6))\n",
      "title('Relationship of weights and regularisation')\n",
      "ax = gca()\n",
      "ax.set_color_cycle(['b', 'r', 'g', 'c', 'k', 'y', 'm'])\n",
      "\n",
      "ax.plot(taus, weights, linewidth=2)\n",
      "xlabel('Tau', fontsize=12)\n",
      "ylabel('Weights', fontsize=12)\n",
      "ax.set_xscale('log')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is the famous ridge trace that is the signature of this technique. The plot is really very straight forward to read. It presents the standardized regression coefficients(weights) on the vertical axis and various values of tau(Regularisation constant) along the horizontal axis. Since the values of tau($\\tau$) span several orders of magnitude, we adopt a logarithmic scale along this axis. As tau is increased, the values of the regression estimates change, often wildly at first. At some point, the coefficients seem to settle down and then gradually drift towards zero. The value of tau for which these coefficients are at their stable values is often the best one.\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Least Angle Regression and LASSO"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Shogun provides tools for Least angle regression(LARS) and lasso using [CLeastAngleRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastAngleRegression.html) class. As explained in the mathematical formaulation, LARS is just like [Stepwise Regression](http://en.wikipedia.org/wiki/Stepwise_regression) but increases the estimated variables in a direction equiangular to each one's correlations with the residual. The working of this is shown below by plotting the LASSO path. Data is generated in a similar way to the previous section."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#sample some data\n",
      "X=rand(10)*1.5\n",
      "for i in range(9):\n",
      "    x=random.standard_normal(10)*0.5\n",
      "    X=vstack((X, x))\n",
      "y=ones(10)\n",
      "\n",
      "feats_train=RealFeatures(X)\n",
      "labels_train=RegressionLabels(y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[CLeastAngleRegression](http://shogun-toolbox.org/doc/en/latest/classshogun_1_1CLeastAngleRegression.html) requires the features to be normalized with a zero mean and unit norm. Hence we use two preprocessors: [PruneVarSubMean](http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CPruneVarSubMean.html) and [NormOne](http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CPruneVarSubMean.html)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Preprocess data\n",
      "preproc=PruneVarSubMean()\n",
      "preproc.init(feats_train)\n",
      "feats_train.add_preprocessor(preproc)\n",
      "feats_train.apply_preprocessor()\n",
      "\n",
      "preprocessor=NormOne()\n",
      "preprocessor.init(feats_train)\n",
      "feats_train.add_preprocessor(preprocessor)\n",
      "feats_train.apply_preprocessor()\n",
      "\n",
      "print \"(No. of attributes, No. of samples) of data:\"\n",
      "print feats_train.get_feature_matrix().shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we train on the data. Keeping in mind that we had 10 attributes/dimensions in our data, let us have a look at the size of LASSO path which is obtained readily using `get_path_size()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Train and generate weights\n",
      "la=LeastAngleRegression()\n",
      "la.set_labels(labels_train)\n",
      "la.train(feats_train)\n",
      "\n",
      "size=la.get_path_size()\n",
      "print (\"Size of path is %s\" %size)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The weights generated ($\\beta_i$) and their norm ($\\sum_i|\\beta_i|$) change with each step. This is when a new variable is added to path. To get the weights at each of these steps `get_w_for_var()` method is used. The argument is the index of the variable which should be in the range [0, path_size)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#calculate weights\n",
      "weights=[]\n",
      "for i in range(size):\n",
      "    weights.append(la.get_w_for_var(i))  \n",
      "s = sum(abs(array(weights)), axis=1)\n",
      "print ('Max. norm is %s' %s[-1])\n",
      "\n",
      "figure(figsize(30,7))\n",
      "#plot 1\n",
      "ax=subplot(131)\n",
      "title('Lasso path')\n",
      "ax.plot(s, weights, linewidth=2)\n",
      "ymin, ymax = ylim()\n",
      "ax.vlines(s[1:-1], ymin, ymax, linestyle='dashed')\n",
      "xlabel(\"Norm\")\n",
      "ylabel(\"weights\")\n",
      "\n",
      "#Restrict norm to half for early termination\n",
      "la.set_max_l1_norm(s[-1]*0.5)\n",
      "la.train(feats_train)\n",
      "size=la.get_path_size()\n",
      "weights=[]\n",
      "for i in range(size):\n",
      "    weights.append(la.get_w_for_var(i))\n",
      "s = sum(abs(array(weights)), axis=1)\n",
      "\n",
      "#plot 2\n",
      "ax2=subplot(132)\n",
      "title('Lasso path with restricted norm')\n",
      "ax2.plot(s, weights, linewidth=2)\n",
      "ax2.vlines(s[1:-1], ymin, ymax, linestyle='dashed')\n",
      "xlabel(\"Norm\")\n",
      "ylabel(\"weights\")\n",
      "print ('Restricted norm is  %s' %(s[-1]))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each color in the plot represents a coefficient and the vertical lines denote steps. It is clear that the weights are piecewise linear function of the norm.  "
     ]
{'fraction_non_alphanumeric': 0.17297000258598397, 'fraction_numerical': 0.012832945435738298, 'mean_word_length': 2.6538325262784928, 'pattern_counts': {'":': 305, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3427879', 'n_tokens_mistral': 10717, 'n_tokens_neox': 10270, 'n_words': 2742}