<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Alignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `align` function projects 2 or more datasets with different coordinate systems into a common space.  By default it uses the hyperalignment algorithm ([Haxby et al, 2011](http://haxbylab.dartmouth.edu/publications/HGC+11.pdf)), but also provides the option to use the Shared Response Model (SRM) for alignment, if preferred, via the Brain Imaging Analysis Kit ([brainiak](http://brainiak.org/)).\n",
    "\n",
    "Alignment can be particularly useful in exploring statistical properties and/or similarities of datasets that are not in the same coordinate system (such as fMRI data from visual areas of participants watching a movie, and the movie data itself).\n",
    "\n",
    "Alignment algorithms use linear transformations to rotate and scale your datasets so they match as best as possible. For example, take these three distinct datasets. Each has a similar shape (an S), but are scaled and rotated differently. Aligning these datasets finds the transformation that minimizes the distance between them.\n",
    "\n",
    "<img src=\"img/alignment.png\",width=600>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Hypertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import hypertools as hyp\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll load one of the sample datasets. This dataset is a list of 2 `numpy` arrays, each containing average brain activity (fMRI) from 18 subjects listening to the same story, fit using Hierarchical Topographic Factor Analysis (HTFA) with 100 nodes.  The rows are timepoints and the columns are fMRI components. \n",
    "\n",
    "See the [full dataset](http://dataspace.princeton.edu/jspui/handle/88435/dsp015d86p269k) or the [HTFA article](https://www.biorxiv.org/content/early/2017/02/07/106690) for more info on the data and HTFA, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = hyp.load('weights')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize unaligned data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we can see how the first hundred data points from two arrays in the weights data look when plotted together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average into two groups\n",
    "group1 = np.mean(data[:17], 0)\n",
    "group2 = np.mean(data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning data with Hyperalignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can align the two datasets (using [hyperalignment](http://papers.nips.cc/paper/5855-a-reduced-dimension-fmri-shared-response-model.pdf)) and visualize the aligned data. Note that the two datasets are now much more similar to each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aligned_data = hyp.align(data)\n",
    "\n",
    "# average into two groups\n",
    "group1 = np.mean(aligned_data[:17], 0)\n",
    "group2 = np.mean(aligned_data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning data with the Shared Response Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may use the Shared Response Model for alignment by setting `align` to 'SRM'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aligned_data = hyp.align(data, align='SRM')\n",
    "\n",
    "# average into two groups\n",
    "group1 = np.mean(aligned_data[:17], 0)\n",
    "group2 = np.mean(aligned_data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }<commit_msg>updating path to align picture:<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Alignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `align` function projects 2 or more datasets with different coordinate systems into a common space.  By default it uses the hyperalignment algorithm ([Haxby et al, 2011](http://haxbylab.dartmouth.edu/publications/HGC+11.pdf)), but also provides the option to use the Shared Response Model (SRM) for alignment, if preferred, via the Brain Imaging Analysis Kit ([brainiak](http://brainiak.org/)).\n",
    "\n",
    "Alignment can be particularly useful in exploring statistical properties and/or similarities of datasets that are not in the same coordinate system (such as fMRI data from visual areas of participants watching a movie, and the movie data itself).\n",
    "\n",
    "Alignment algorithms use linear transformations to rotate and scale your datasets so they match as best as possible. For example, take these three distinct datasets. Each has a similar shape (an S), but are scaled and rotated differently. Aligning these datasets finds the transformation that minimizes the distance between them.\n",
    "\n",
    "<img src=\"https://github.com/ContextLab/hypertools/raw/master/docs/tutorials/img/alignment.png\", width=600>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Hypertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import hypertools as hyp\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll load one of the sample datasets. This dataset is a list of 2 `numpy` arrays, each containing average brain activity (fMRI) from 18 subjects listening to the same story, fit using Hierarchical Topographic Factor Analysis (HTFA) with 100 nodes.  The rows are timepoints and the columns are fMRI components. \n",
    "\n",
    "See the [full dataset](http://dataspace.princeton.edu/jspui/handle/88435/dsp015d86p269k) or the [HTFA article](https://www.biorxiv.org/content/early/2017/02/07/106690) for more info on the data and HTFA, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = hyp.load('weights')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize unaligned data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we can see how the first hundred data points from two arrays in the weights data look when plotted together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# average into two groups\n",
    "group1 = np.mean(data[:17], 0)\n",
    "group2 = np.mean(data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning data with Hyperalignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can align the two datasets (using [hyperalignment](http://papers.nips.cc/paper/5855-a-reduced-dimension-fmri-shared-response-model.pdf)) and visualize the aligned data. Note that the two datasets are now much more similar to each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "aligned_data = hyp.align(data)\n",
    "\n",
    "# average into two groups\n",
    "group1 = np.mean(aligned_data[:17], 0)\n",
    "group2 = np.mean(aligned_data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning data with the Shared Response Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may use the Shared Response Model for alignment by setting `align` to 'SRM'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "aligned_data = hyp.align(data, align='SRM')\n",
    "\n",
    "# average into two groups\n",
    "group1 = np.mean(aligned_data[:17], 0)\n",
    "group2 = np.mean(aligned_data[18:], 0)\n",
    "\n",
    "# plot\n",
    "geo = hyp.plot([group1[:100, :], group2[:100, :]])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
{'fraction_non_alphanumeric': 0.17516479255525397, 'fraction_numerical': 0.019775106630476928, 'mean_word_length': 3.065011820330969, 'pattern_counts': {'":': 155, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5771480', 'n_tokens_mistral': 3687, 'n_tokens_neox': 3475, 'n_words': 955}