<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Technical Overview\n",
    "\n",
    "This page provides very high level technical information about microstructure discretization, 2-point statistics, MKS Localization and MKS Homogenization. For more detailed information see the [MINED research group's website](http://mined.gatech.edu/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Microstructure Discretization\n",
    "\n",
    "The first step in all of the MKS work-flows is to discretize the microstructures. To do this we create a probabilistic description of the microstructure by introducing the continuous local state variable $h$, the local state space $H$ and the microstructure function $m(h, x)$. The local state space $H$ can be thought of as all of the thermodynamic state variables that are needed to uniquely define the material structure at a given location. The local state variable $h$ is one instance of the local state space, or one configuration of state variables. The microstructure function $m(h, x)$ is a probability density function of finding a a local state $h$ at location $x$. For instance let $\\mu(x)$ be a microstructure that we plan to discretize, then $\\mu$ is the expectation of the microstructure function.\n",
    "\n",
    "$$ \\mu(x) = \\int_H h m(h, x) dh $$\n",
    "\n",
    "Now we will discretize the microstructure in space by averaging over small cubic domains in the microstructure function. The local state can be discretized using two methods one is to bin the microstructure using the primitive basis $\\Lambda$\n",
    "\n",
    "\n",
    "$$  \\frac{1}{\\Delta x} \\int_{H} \\int_{s} \\Lambda(h - l) m(h, x) dx dh = m[l, s] $$\n",
    "\n",
    "the other is to using a spectral representation using some orthogonal basis function $\\xi$\n",
    "\n",
    "$$ \\frac{1}{\\Delta x}  \\int_{s} m(h, x) dx dt = \\sum_{l=0}^{L-1} m[l, s] \\xi_l (h) $$\n",
    "\n",
    "In the notation above all of the round brackets are used continuous variables and the square variables are the discrete variables. The variables $s$ and $S$ represent a discrete position and the total volume, while $l$ and $L$ represent the discrete versions of $h$ and $H$. In PyMKS the Legendre polynomials are currently the only orthgonal basis functions available.\n",
    "\n",
    "Either of these two discretization methods used to discretize the microstructure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##2-Point Statistics\n",
    "\n",
    "### n-Point Spatial Correlations\n",
    "\n",
    "####1-Point Spatial Correlations (or 1-point statistics)\n",
    "\n",
    "n-point spatial correlations provide a way rigorously quantify material structure using statistics. As an introduction n-point spatial correlations, let's first discuss 1-point statistics. 1-point statistics are the probability that a specified local state  will be found in any randomly selected spatial bin in a microstructure [1][2][3]. In this checkerboard example discussed here, there are two possible local states, one is colored white and the other is colored black. 1-point statistics compute the volume fractions of the local states in the microstructure. 1-point statistics are computed as\n",
    "\n",
    "$$ f[l] = \\frac{1}{S} \\sum_s m[s,l] $$\n",
    "\n",
    "In this equation, $f[l]$ is the probability of finding the local state $l$ in any randomly selected spatial bin in the microstructure, $m[s, l]$ is the microstructure function (the digital representation of the microstructure), $S$ is the total number of spatial bins in the microstructure and $s$ refers to a specific spatial bin. \n",
    "\n",
    "While 1-point statistics provide information on the relative amounts of the different local states, it does not provide any information about how those local states are spatially arranged in the microstructure. Therefore, 1-point statistics are a limited set of metrics to describe the structure of materials.\n",
    "\n",
    "####2-Point Spatial Correlations\n",
    "\n",
    "2-point spatial correlations (also known as 2-point statistics) contain information about the fractions of local states as well as the first order information on how the different local states are distributed in the microstructure. \n",
    "\n",
    "2-point statistics can be thought of as the probability of having a vector placed randomly in the microstructure and having one end of the vector be on one specified local state and the other end on another specified local state. This vector could have any length or orientation that the discrete microstructure allows. The equation for 2-point statistics can found below.\n",
    "\n",
    "$$ f[r \\vert l, l'] = \\frac{1}{S} \\sum_s m[s, l] m[s + r, l'] $$\n",
    "\n",
    "In this equation $ f[r \\vert l, l']$ is the conditional probability of finding the local states $l$ and $l'$ at a distance and orientation away from each other defined by the vector $r$. All other variables are the same as those in the 1-point statistics equation. In the case that we have an eigen microstructure function (it only contains values of 0 or 1) and we are using an indicator basis, the the $r=0$ vector will recover the 1-point statistics. \n",
    "\n",
    "When the 2 local states are the same $l = l'$, it is referred to as a autocorrelation. If the 2 local states are not the same it is referred to as a cross-correlation. \n",
    "\n",
    "####Higher Order Spatial Statistics\n",
    "\n",
    "Higher order spatial statistics are similar to 2-point statistics, in that they can be thought of in terms of conditional probabilities of finding specified local states separated by a prescribed set of vectors. 3-point statistics are the probability of finding three specified local states at the ends of a triangle (defined by 2 vectors) placed randomly in the material structure. 4-point statistics describes the probability of finding 4 local states at 4 locations (defined using 3 vectors) and so on. \n",
    "\n",
    "While higher order statistics are a better metric to quantify the material structure, the 2-point statistics can be computed much faster than higher order spatial statistics, and still provide information about how the local states are distributed. For this reason, only 2-point statistics are implemented into PyMKS. Let us look at an example of computing the 2-point statistics for a checkerboard microstructure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##MKS Homogenization\n",
    "\n",
    "Homogenization can be used to determine effective or homogenized properties a material, and provides a way to multiscale from the bottom up. Below is some technical information on the MKS Homogenization work flow. \n",
    "\n",
    "###2-Point Statistics\n",
    "\n",
    "The first step in MKS homogenization is to compute the 2-point statistics for each of the microstructures in the calibration dataset. For more information about 2-point statistics see the section above.\n",
    "\n",
    "###Dimensionality Reduction\n",
    "\n",
    "Once we have computed the 2-point statistics for every microstructure in our calibration dataset, we need to determine which of these microstructure features are most important and need to be pass to the higher length scale. We can do this using dimensionality reduction techniques from machine learning. \n",
    "\n",
    "[PCA](http://en.wikipedia.org/wiki/Principal_component_analysis) is the most common the dimensionality reduction technique used in the MKS Homogenization work flow, and provides an efficient way to find a small number of microstructure descriptors that capture most of the variance in the microstructures.\n",
    "\n",
    "###Regression Model\n",<commit_msg>Remove reference to checkerboard example<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Technical Overview\n",
    "\n",
    "This page provides very high level technical information about microstructure discretization, 2-point statistics, MKS Localization and MKS Homogenization. For more detailed information see the [MINED research group's website](http://mined.gatech.edu/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Microstructure Discretization\n",
    "\n",
    "The first step in all of the MKS work-flows is to discretize the microstructures. To do this we create a probabilistic description of the microstructure by introducing the continuous local state variable $h$, the local state space $H$ and the microstructure function $m(h, x)$. The local state space $H$ can be thought of as all of the thermodynamic state variables that are needed to uniquely define the material structure at a given location. The local state variable $h$ is one instance of the local state space, or one configuration of state variables. The microstructure function $m(h, x)$ is a probability density function of finding a a local state $h$ at location $x$. For instance let $\\mu(x)$ be a microstructure that we plan to discretize, then $\\mu$ is the expectation of the microstructure function.\n",
    "\n",
    "$$ \\mu(x) = \\int_H h m(h, x) dh $$\n",
    "\n",
    "Now we will discretize the microstructure in space by averaging over small cubic domains in the microstructure function. The local state can be discretized using two methods one is to bin the microstructure using the primitive basis $\\Lambda$\n",
    "\n",
    "\n",
    "$$  \\frac{1}{\\Delta x} \\int_{H} \\int_{s} \\Lambda(h - l) m(h, x) dx dh = m[l, s] $$\n",
    "\n",
    "the other is to using a spectral representation using some orthogonal basis function $\\xi$\n",
    "\n",
    "$$ \\frac{1}{\\Delta x}  \\int_{s} m(h, x) dx dt = \\sum_{l=0}^{L-1} m[l, s] \\xi_l (h) $$\n",
    "\n",
    "In the notation above all of the round brackets are used continuous variables and the square variables are the discrete variables. The variables $s$ and $S$ represent a discrete position and the total volume, while $l$ and $L$ represent the discrete versions of $h$ and $H$. In PyMKS the Legendre polynomials are currently the only orthgonal basis functions available.\n",
    "\n",
    "Either of these two discretization methods used to discretize the microstructure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##2-Point Statistics\n",
    "\n",
    "### n-Point Spatial Correlations\n",
    "\n",
    "####1-Point Spatial Correlations (or 1-point statistics)\n",
    "\n",
    "n-point spatial correlations provide a way rigorously quantify material structure using statistics. As an introduction n-point spatial correlations, let's first discuss 1-point statistics. 1-point statistics are the probability that a specified local state  will be found in any randomly selected spatial bin in a microstructure [1][2][3]. 1-point statistics compute the volume fractions of the local states in the microstructure. 1-point statistics are computed as\n",
    "\n",
    "$$ f[l] = \\frac{1}{S} \\sum_s m[s,l] $$\n",
    "\n",
    "In this equation, $f[l]$ is the probability of finding the local state $l$ in any randomly selected spatial bin in the microstructure, $m[s, l]$ is the microstructure function (the digital representation of the microstructure), $S$ is the total number of spatial bins in the microstructure and $s$ refers to a specific spatial bin. \n",
    "\n",
    "While 1-point statistics provide information on the relative amounts of the different local states, it does not provide any information about how those local states are spatially arranged in the microstructure. Therefore, 1-point statistics are a limited set of metrics to describe the structure of materials.\n",
    "\n",
    "####2-Point Spatial Correlations\n",
    "\n",
    "2-point spatial correlations (also known as 2-point statistics) contain information about the fractions of local states as well as the first order information on how the different local states are distributed in the microstructure. \n",
    "\n",
    "2-point statistics can be thought of as the probability of having a vector placed randomly in the microstructure and having one end of the vector be on one specified local state and the other end on another specified local state. This vector could have any length or orientation that the discrete microstructure allows. The equation for 2-point statistics can found below.\n",
    "\n",
    "$$ f[r \\vert l, l'] = \\frac{1}{S} \\sum_s m[s, l] m[s + r, l'] $$\n",
    "\n",
    "In this equation $ f[r \\vert l, l']$ is the conditional probability of finding the local states $l$ and $l'$ at a distance and orientation away from each other defined by the vector $r$. All other variables are the same as those in the 1-point statistics equation. In the case that we have an eigen microstructure function (it only contains values of 0 or 1) and we are using an indicator basis, the the $r=0$ vector will recover the 1-point statistics. \n",
    "\n",
    "When the 2 local states are the same $l = l'$, it is referred to as a autocorrelation. If the 2 local states are not the same it is referred to as a cross-correlation. \n",
    "\n",
    "####Higher Order Spatial Statistics\n",
    "\n",
    "Higher order spatial statistics are similar to 2-point statistics, in that they can be thought of in terms of conditional probabilities of finding specified local states separated by a prescribed set of vectors. 3-point statistics are the probability of finding three specified local states at the ends of a triangle (defined by 2 vectors) placed randomly in the material structure. 4-point statistics describes the probability of finding 4 local states at 4 locations (defined using 3 vectors) and so on. \n",
    "\n",
    "While higher order statistics are a better metric to quantify the material structure, the 2-point statistics can be computed much faster than higher order spatial statistics, and still provide information about how the local states are distributed. For this reason, only 2-point statistics are implemented into PyMKS. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##MKS Homogenization\n",
    "\n",
    "Homogenization can be used to determine effective or homogenized properties a material, and provides a way to multiscale from the bottom up. Below is some technical information on the MKS Homogenization work flow. \n",
    "\n",
    "###2-Point Statistics\n",
    "\n",
    "The first step in MKS homogenization is to compute the 2-point statistics for each of the microstructures in the calibration dataset. For more information about 2-point statistics see the section above.\n",
    "\n",
    "###Dimensionality Reduction\n",
    "\n",
    "Once we have computed the 2-point statistics for every microstructure in our calibration dataset, we need to determine which of these microstructure features are most important and need to be pass to the higher length scale. We can do this using dimensionality reduction techniques from machine learning. \n",
    "\n",
    "[PCA](http://en.wikipedia.org/wiki/Principal_component_analysis) is the most common the dimensionality reduction technique used in the MKS Homogenization work flow, and provides an efficient way to find a small number of microstructure descriptors that capture most of the variance in the microstructures.\n",
    "\n",
    "###Regression Model\n",
{'fraction_non_alphanumeric': 0.09373000639795266, 'fraction_numerical': 0.0056941778630838135, 'mean_word_length': 3.7138118214716527, 'pattern_counts': {'":': 28, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7222704', 'n_tokens_mistral': 4167, 'n_tokens_neox': 3993, 'n_words': 2178}