<commit_before>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logging\n",
    "\n",
    "For most purposes, we are going to want to set our logging level to `INFO`, since some commands are going to run for a long time, and we would like periodic updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "reload(logging)\n",
    "logging.basicConfig(format='%(asctime)s %(levelname)s:%(message)s', level=logging.INFO, datefmt='%I:%M:%S')\n",
    "#logging.debug('This is a debug message')\n",
    "#logging.basicConfig(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up an `HDF5` database\n",
    "\n",
    "The first step is to create a dataset. This process is mostly abstracted away for you using `a3.Dataset` objects. Your responsibility is to specify how your data is stored and structured. We'll start by specifying some structure. \n",
    "\n",
    "### Keys\n",
    "\n",
    "For the example dataset, there is only a single subject. For this dataset, a combination of a study ID and frame ID is sufficient to pick out a unique datapoint.  We will encode this database key as a list of the form `['study','frame']`. While you might construct a key of additional fields (ex. `subject`), the last element will usually be frame, since that is usually the minimal unit of analysis. \n",
    "\n",
    "Note that these names are arbitrary, as long as you are consistant. Of course using informative names is best practice, since `autotres` has some sensible defaults that rely on certain keys; for example, the default code for extracting images from video relies on a `'frame'` key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "keys = ['study', 'frame']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types\n",
    "\n",
    "Next, we need to be able to tell autotres about the types of files we have, the types of data they represent, and what levels of the key heirarchy each piece of data should be associated with. This will mostly be accomplished with regular expressions. We create a dict of data types. The keys of the dict represent data type names. Note that we will want one type for each type of data we want, not for each type of file we have. `autotres` is perfectly happy pulling more than one type of information from a file. \n",
    "\n",
    "These are again arbitrary labels, but autotres can provide some sensible default behaviors for certain labels. Each dict contains information about that type:\n",
    "\n",
    "#### `'conflict'`: How to deal with multiple files of the same type appearing for same combination of identifiers. \n",
    "\n",
    "For example, If I have multiple tracers on my team, then I would expect to have multiple 'trace' files for each combination of 'subject' and 'frame'. In this case, I would like to keep all of the traces for the same image, so that I can do something sensible with it, like interpolate. In this case I will set the value for 'conflict' to 'list'. \n",
    "\n",
    "Similarly, if I have conducted multiple studies with my dataset, one looking at coronals and one looking at fricatives, I would expect to have multiple copies of the images for coronal fricatives. However, unlike the instance with multiple traces, `fricative_frame-00042.png` and `coronal_frame-00042.png` should be identical files. I can use the 'hash' option to specify that I should ignore duplicates as long as they are the same, but should raise an exception if they don't.\n",
    "\n",
    "Finally, there are some situations that I simply expect not to happen. For example, if a single subject is associated with more that one 'audio' file, then perhaps it is most likely that somebdy mislabeled something. In this case, I would not set 'conflict' to anything, and if there is a conflict, autotres will raise an exception automatically\n",
    "\n",
    "#### `'regex'`: How to associate each file with a combination of heirachical levels.\n",
    "\n",
    "This is a regular expression that will match a filename in the dataset. We use the `(?P<label>...)` syntax to capture parts of the filename that are informative. Specifically, we need to be able to infer all the relevent heirarchical information from the file name. This should be a left-substring of the keys list. \n",
    "\n",
    "Note that the regex is matched to the entire pathname, relative to whatever path we give it (see below). Here, we have also used the `(?x)` flag to allow us to break the regex over multiple lines, and to include comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "\n",
    "types = {\n",
    "    'trace': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<study>\\d+\\w+)              # in the example dataset, a 'study' is encoded in the image name as the substring preceding an '_'\n",
    "            _(?P<frame>\\d+)\\.(?:jpg|png)   # the frame number\n",
    "            \\.(?P<tracer>\\w+)              # the tracer id\n",
    "            \\.traced\\.txt$\"\"\",\n",
    "        'conflict': 'list'\n",
    "        },\n",
    "    'image': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<study>\\d+\\w+)\n",
    "            _(?P<frame>\\d+)\n",
    "            \\.(?P<ext>jpg|png)$\"\"\",\n",
    "        'conflict': 'hash'\n",
    "        },\n",
    "    'name': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<fname>(?P<study>\\d+\\w+)\n",
    "                _(?P<frame>\\d+)\n",
    "                \\.(?P<ext>jpg|png)\n",
    "            )$\"\"\",\n",
    "        }\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the dataset\n",
    "\n",
    "We will now set up our dataset. The `roi`, `n_points`, and `scale` `kwargs` will be passed down to the default data extraction callbacks (see [`a3/dataset.py`](../a3/dataset.py) documentation). Custom callbacks can be provided by putting a callable in the ds.callbacks dict. These should return a numpy array of type `float32`. \n",
    "\n",
    "If you don't have CUDA properly installed, importing a3 will throw some errors about nvcc (nvidia cuda compiler) not being found. This is fine so long as you are fine with only using the CPU (instead of the GPU) to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import a3\n",
    "ds = a3.Dataset('example.hdf5',roi=(140.,320.,250.,580.),n_points=32,scale=.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directory containing our data is `HS_diverse_800_50` using our types and our key heirarchy. You can scan multiple directories if you need to, possibly with different type definitions, but watch out for file conflicts! Your heirarchy should be the same accross calls to `scan_directory`. For large datasets may take a while to complete, since it is doing a full walk of the file heirarchy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d = 'example_data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds.scan_directory(d,types,keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you can inspect what data sources you have by looking at the `ds.sources` dict. This `dict` can get very large, so be cautious about printing the whole of it to `stdout`."
   ]<commit_msg>Minor update to example<commit_after>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logging\n",
    "\n",
    "For most purposes, we are going to want to set our logging level to `INFO`, since some commands are going to run for a long time, and we would like periodic updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "reload(logging)\n",
    "logging.basicConfig(format='%(asctime)s %(levelname)s:%(message)s', level=logging.INFO, datefmt='%I:%M:%S')\n",
    "#logging.debug('This is a debug message')\n",
    "#logging.basicConfig(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up an `HDF5` database\n",
    "\n",
    "The first step is to create a dataset. This process is mostly abstracted away for you using `a3.Dataset` objects. Your responsibility is to specify how your data is stored and structured. We'll start by specifying some structure. \n",
    "\n",
    "### Keys\n",
    "\n",
    "For the example dataset, there is only a single subject. For this dataset, a combination of a study ID and frame ID is sufficient to pick out a unique datapoint.  We will encode this database key as a list of the form `['study','frame']`. While you might construct a key of additional fields (ex. `subject`), the last element will usually be frame, since that is usually the minimal unit of analysis. \n",
    "\n",
    "Note that these names are arbitrary, as long as you are consistant. Of course using informative names is best practice, since `autotres` has some sensible defaults that rely on certain keys; for example, the default code for extracting images from video relies on a `'frame'` key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "keys = ['study', 'frame']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types\n",
    "\n",
    "Next, we need to be able to tell autotres about the types of files we have, the types of data they represent, and what levels of the key heirarchy each piece of data should be associated with. This will mostly be accomplished with regular expressions. We create a dict of data types. The keys of the dict represent data type names. Note that we will want one type for each type of data we want, not for each type of file we have. `autotres` is perfectly happy pulling more than one type of information from a file. \n",
    "\n",
    "These are again arbitrary labels, but autotres can provide some sensible default behaviors for certain labels. Each dict contains information about that type:\n",
    "\n",
    "#### `'conflict'`: How to deal with multiple files of the same type appearing for same combination of identifiers. \n",
    "\n",
    "For example, If I have multiple tracers on my team, then I would expect to have multiple 'trace' files for each combination of 'subject' and 'frame'. In this case, I would like to keep all of the traces for the same image, so that I can do something sensible with it, like interpolate. In this case I will set the value for 'conflict' to 'list'. \n",
    "\n",
    "Similarly, if I have conducted multiple studies with my dataset, one looking at coronals and one looking at fricatives, I would expect to have multiple copies of the images for coronal fricatives. However, unlike the instance with multiple traces, `fricative_frame-00042.png` and `coronal_frame-00042.png` should be identical files. I can use the 'hash' option to specify that I should ignore duplicates as long as they are the same, but should raise an exception if they don't.\n",
    "\n",
    "Finally, there are some situations that I simply expect not to happen. For example, if a single subject is associated with more that one 'audio' file, then perhaps it is most likely that somebdy mislabeled something. In this case, I would not set 'conflict' to anything, and if there is a conflict, autotres will raise an exception automatically\n",
    "\n",
    "#### `'regex'`: How to associate each file with a combination of heirachical levels.\n",
    "\n",
    "This is a regular expression that will match a filename in the dataset. We use the `(?P<label>...)` syntax to capture parts of the filename that are informative. Specifically, we need to be able to infer all the relevent heirarchical information from the file name. This should be a left-substring of the keys list. \n",
    "\n",
    "Note that the regex is matched to the entire pathname, relative to whatever path we give it (see below). Here, we have also used the `(?x)` flag to allow us to break the regex over multiple lines, and to include comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "\n",
    "types = {\n",
    "    'trace': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<study>\\d+\\w+)              # in the example dataset, a 'study' is encoded in the image name as the substring preceding an '_'\n",
    "            _(?P<frame>\\d+)\\.(?:jpg|png)   # the frame number\n",
    "            \\.(?P<tracer>\\w+)              # the tracer id\n",
    "            \\.traced\\.txt$\"\"\",\n",
    "        'conflict': 'list'\n",
    "        },\n",
    "    'image': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<study>\\d+\\w+)\n",
    "            _(?P<frame>\\d+)\n",
    "            \\.(?P<ext>jpg|png)$\"\"\",\n",
    "        'conflict': 'hash'\n",
    "        },\n",
    "    'name': {\n",
    "        'regex': r\"\"\"(?x)\n",
    "            (?P<fname>(?P<study>\\d+\\w+)\n",
    "                _(?P<frame>\\d+)\n",
    "                \\.(?P<ext>jpg|png)\n",
    "            )$\"\"\",\n",
    "        }\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the dataset\n",
    "\n",
    "We will now set up our dataset. The `roi`, `n_points`, and `scale` `kwargs` will be passed down to the default data extraction callbacks (see [`a3/dataset.py`](../a3/dataset.py) documentation). Custom callbacks can be provided by putting a callable in the ds.callbacks dict. These should return a numpy array of type `float32`. \n",
    "\n",
    "If you don't have CUDA properly installed, importing a3 will throw some errors about nvcc (nvidia cuda compiler) not being found. This is fine so long as you are fine with only using the CPU (instead of the GPU) to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import a3\n",
    "ds = a3.Dataset('example.hdf5',roi=(140.,320.,250.,580.),n_points=32,scale=.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directory containing our data is `example_data`. You can scan multiple directories if you need to, possibly with different type definitions, but watch out for file conflicts! Your heirarchy should be the same accross calls to `scan_directory`. For large datasets may take a while to complete, since it is doing a full walk of the file heirarchy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d = 'example_data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds.scan_directory(d,types,keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you can inspect what data sources you have by looking at the `ds.sources` dict. This `dict` can get very large, so be cautious about printing the whole of it to `stdout`."
   ]
{'fraction_non_alphanumeric': 0.14266288281896652, 'fraction_numerical': 0.006207875133025896, 'mean_word_length': 2.7875055978504255, 'pattern_counts': {'":': 136, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3856471', 'n_tokens_mistral': 5326, 'n_tokens_neox': 5115, 'n_words': 2075}