<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling of experimental metadata using odMLtables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook presents the interaction with metadata from an experiment using odMLtables. Here we use the metadata of a complex electrophysiology experiment published in *[<NAME>. et al. Massively parallel recordings in macaque motor cortex during an instructed delayed reach-to-grasp task. Sci. Data 5:180055](https://www.doi.org10.1038/sdata.2018.55)*. In the first step, we show how to extract a subset of information from the complete metadata collection for visualization. In the second step, we compare numerous similar values available with in one metadata file and generate an overview sheet, eg. for addition into a lab book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download of published metadata\n",
    "The published [datasets](https://web.gin.g-node.org/INT/multielectrode_grasp) are hosted on [GIN](https://web.gin.g-node.org) and can be accessed directly via the webinterface or the local gin client. Here we download the metadata files of the two datasets directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# helper function by <NAME>, https://stackoverflow.com/questions/7243750/download-file-from-web-in-python-3\n",
    "from requests import get\n",
    "def download(url, file_name):\n",
    "    # open in binary mode\n",
    "    with open(file_name, \"wb\") as file:\n",
    "        # get request\n",
    "        response = get(url)\n",
    "        # write to file\n",
    "        file.write(response.content)\n",
    "\n",
    "# location of the repository and metadata files on GIN\n",
    "gin_repo = \"https://web.gin.g-node.org/INT/multielectrode_grasp/raw/master/\"\n",
    "filenames = [\"i140703-001.odml\", \"l101210-001.odml\"]\n",
    "filepaths = [\"datasets/\" + f for f in filenames]\n",
    "\n",
    "# download metadata files from GIN\n",
    "for filepath, filename in zip(filepaths, filenames):\n",
    "    download(gin_repo + filepath, filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extraction of a subset of metadata\n",
    "The metadata collections of the electrophysiological experiment contain due to the complexity of the experiment thousands of entries organized in a hierarchical odML structure. This is of advantage for the storage of the metadata in an organized fashion, but for human interaction and visualization the amout of data needs to be reduced to a subset relevant for the current question. This can be achieved using the filter functionality of odMLtables. Here we extract all metadata related to the subject performing the task, which includes information such as the species, name, birthdate and handedness.\n",
    "\n",
    "Since the publication of the datasets odML was developed further, so an updated odML version exists. In a first step we are converting the metadata files from odML file version 1.0 to file version 1.1 using the version conversion tool provided by odML.\n",
    "In a second step we use odMLtables to select only information related to the subject of the experiment and store this in a separate metadata file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inplace conversion of odMl files to latest odML version\n",
    "from odml.tools.version_converter import VersionConverter\n",
    "for filename in filenames:\n",
    "    converter = VersionConverter(filename)\n",
    "    converter.convert()\n",
    "    converter.write_to_file(filename)\n",
    "\n",
    "\n",
    "# load odml file using odmltables and extract subset of information\n",
    "import odmltables as odt\n",
    "\n",
    "# extract only directly related subject information\n",
    "for filename in filenames:\n",
    "    odmlfile = odt.OdmlTable(filename)\n",
    "    odmlfile.filter(SectionName='Subject')\n",
    "    new_filename = filename.split('.')[0] + '_filtered.odml'\n",
    "    odmlfile.write2odml(new_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated odML files, which contain only a subset of the metadata can now be visualized in a browser using the [odML style sheet](https://raw.githubusercontent.com/G-Node/odml-terminologies/master/v1.1/odml.xsl). For this just place the style sheet in the same folder as your odML file and open the odML file in your browser. Here we use a helper function to display the odML content using the odML style sheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is utility code for displaying the odML file as html representation here.\n",
    "# You can also just open the odML file in your browser having the style sheet in the same location as your odML file and\n",
    "# will get the same result\n",
    "from IPython.display import display, HTML\n",
    "import lxml.etree as ET\n",
    "\n",
    "def display_odML_as_html(odML_file, xsl_file='odml.xsl'):\n",
    "    # generate html representation from odML file and style sheet\n",
    "    dom = ET.parse(odML_file)\n",
    "    xslt = ET.parse(xsl_file)\n",
    "    transform = ET.XSLT(xslt)\n",
    "    newdom = transform(dom)\n",
    "    # display html\n",
    "    display(HTML(ET.tostring(newdom, pretty_print=True).decode()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the extracted subsection for one of the datasets\n",
    "display_odML_as_html(filenames[0].split('.')[0] + '_filtered.odml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison of metadata\n",
    "In many cases metadata collections contain repeating structures, containing similar information for different contexts. Comparing these information located in different parts of the odML structure in a single table helps gaining an overview of the metadata and is sometimes required to be tracked in lab book tables. odMLtables offers a comparison function within an odML document, which generates such an overview table for a selected number of subsections.\n",
    "The published metadata collection contains information about 96 active recording electrodes, characterizing each in a separate odML section. The properties in these sections are always the same and only the structure of the subsections differs depending on the preprocessing (spikesorting) of the data. The basic electrode properties are a variety of different IDs associated with each electrode, it's impedance and length.\n",
    "\n",
    "The odMLtables comparison generates a xls or csv table listing the shared properties against the sections to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in filenames:\n",
    "    # create an odMLtables table for comparison with functionality to write to xls\n",
    "    comparetable = odt.CompareSectionXlsTable()\n",
    "    comparetable.load_from_file(filename)\n",
    "    # specify which section to compare, here the first 5 of 97 electrodes\n",<commit_msg>[doc] Add missing image in tutorial2<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling of experimental metadata using odMLtables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook presents the interaction with metadata from an experiment using odMLtables. Here we use the metadata of a complex electrophysiology experiment published in *[Bro<NAME>. et al. Massively parallel recordings in macaque motor cortex during an instructed delayed reach-to-grasp task. Sci. Data 5:180055](https://www.doi.org10.1038/sdata.2018.55)*. In the first step, we show how to extract a subset of information from the complete metadata collection for visualization. In the second step, we compare numerous similar values available with in one metadata file and generate an overview sheet, eg. for addition into a lab book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download of published metadata\n",
    "The published [datasets](https://web.gin.g-node.org/INT/multielectrode_grasp) are hosted on [GIN](https://web.gin.g-node.org) and can be accessed directly via the webinterface or the local gin client. Here we download the metadata files of the two datasets directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function by <NAME>, https://stackoverflow.com/questions/7243750/download-file-from-web-in-python-3\n",
    "from requests import get\n",
    "def download(url, file_name):\n",
    "    # open in binary mode\n",
    "    with open(file_name, \"wb\") as file:\n",
    "        # get request\n",
    "        response = get(url)\n",
    "        # write to file\n",
    "        file.write(response.content)\n",
    "\n",
    "# location of the repository and metadata files on GIN\n",
    "gin_repo = \"https://web.gin.g-node.org/INT/multielectrode_grasp/raw/master/\"\n",
    "filenames = [\"i140703-001.odml\", \"l101210-001.odml\"]\n",
    "filepaths = [\"datasets/\" + f for f in filenames]\n",
    "\n",
    "# download metadata files from GIN\n",
    "for filepath, filename in zip(filepaths, filenames):\n",
    "    download(gin_repo + filepath, filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extraction of a subset of metadata\n",
    "The metadata collections of the electrophysiological experiment contain due to the complexity of the experiment thousands of entries organized in a hierarchical odML structure. This is of advantage for the storage of the metadata in an organized fashion, but for human interaction and visualization the amout of data needs to be reduced to a subset relevant for the current question. This can be achieved using the filter functionality of odMLtables. Here we extract all metadata related to the subject performing the task, which includes information such as the species, name, birthdate and handedness.\n",
    "\n",
    "Since the publication of the datasets odML was developed further, so an updated odML version exists. In a first step we are converting the metadata files from odML file version 1.0 to file version 1.1 using the version conversion tool provided by odML.\n",
    "In a second step we use odMLtables to select only information related to the subject of the experiment and store this in a separate metadata file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# inplace conversion of odMl files to latest odML version\n",
    "from odml.tools.version_converter import VersionConverter\n",
    "for filename in filenames:\n",
    "    converter = VersionConverter(filename)\n",
    "    converter.convert()\n",
    "    converter.write_to_file(filename)\n",
    "\n",
    "\n",
    "# load odml file using odmltables and extract subset of information\n",
    "import odmltables as odt\n",
    "\n",
    "# extract only directly related subject information\n",
    "for filename in filenames:\n",
    "    odmlfile = odt.OdmlTable(filename)\n",
    "    odmlfile.filter(SectionName='Subject')\n",
    "    new_filename = filename.split('.')[0] + '_filtered.odml'\n",
    "    odmlfile.write2odml(new_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated odML files, which contain only a subset of the metadata can now be visualized in a browser using the [odML style sheet](https://raw.githubusercontent.com/G-Node/odml-terminologies/master/v1.1/odml.xsl). For this just place the style sheet in the same folder as your odML file and open the odML file in your browser. Here we use a helper function to display the odML content using the odML style sheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is utility code for displaying the odML file as html representation here.\n",
    "# You can also just open the odML file in your browser having the style sheet in the same location as your odML file and\n",
    "# will get the same result\n",
    "from IPython.display import display, HTML\n",
    "import lxml.etree as ET\n",
    "\n",
    "def display_odML_as_html(odML_file, xsl_file='odml.xsl'):\n",
    "    # generate html representation from odML file and style sheet\n",
    "    dom = ET.parse(odML_file)\n",
    "    xslt = ET.parse(xsl_file)\n",
    "    transform = ET.XSLT(xslt)\n",
    "    newdom = transform(dom)\n",
    "    # display html\n",
    "    display(HTML(ET.tostring(newdom, pretty_print=True).decode()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html xmlns:fn=\"http://www.w3.org/2005/02/xpath-functions\" xmlns:odml=\"http://www.g-node.org/odml\">\n",
       "  <style type=\"text/css\">\n",
       "            body { margin-left:2%; margin-top:10px; padding:0;} div { border:0px solid #888; }\n",
       "\n",
       "            #navigationContainer { left:10%; width:95%;}\n",
       "\n",
       "            #contentContainer { left:10%; width:95%;}\n",
       "         </style>\n",
       "  <body>\n",
       "    <a name=\"top\" style=\"color:#336699\">\n",
       "      <h1>odML - Metadata</h1>\n",
       "    </a>\n",
       "    <div id=\"navigationContainer\">\n",
       "      <p><hr style=\"color:yellow; background-color:#336699; height:4px; margin-right:0; text-align:right; border:1px dashed black;\"/><h2>Document info</h2><b>Author: </b><NAME><br/><b>Date: </b>2017-07-14<br/><b>Version: </b>0.1<br/><b>Repository: </b><br/></p>\n",
       "      <hr style=\"color:yellow; background-color:#336699; height:4px; margin-right:0; text-align:right; border:1px dashed black;\"/>\n",
       "      <h2>Structure</h2>\n",
       "      <font size=\"-1\">\n",
       "        <li>\n",
       "          <ol style=\"compact\">\n",
       "            <font size=\"normal\">\n",
       "              <a href=\"#Sec1\">Subject (type: subject)\n",
       "               </a>\n",
       "            </font>\n",
       "          </ol>\n",
       "        </li>\n",
       "      </font>\n",
       "    </div>\n",
       "    <div id=\"contentContainer\">\n",
       "      <hr style=\"color:yellow; background-color:#336699; height:4px; margin-right:0; text-align:right; border:1px dashed black;\"/>\n",
       "      <h2>Content</h2>\n",
       "      <a name=\"Sec1\">\n",
       "        <h3>Section: Subject</h3>\n",
       "      </a>\n",
       "      <p><b>Type: </b>subject<br/><b>Repository: </b><br/><b>Link: </b><br/><b>Include:</b><br/><b>Definition:</b>Information on the experimental subject (animal or person) under investigation<br/></p>\n",
       "      <table border=\"1\" rules=\"rows\" width=\"100%\">\n",
       "        <font size=\"-1\">\n",
       "          <tr bgcolor=\"#336699\" valign=\"middle\" align=\"left\">\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Name</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Value</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Uncertainty</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Unit</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Type</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Reference</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Definition</b>\n",
       "              </font>\n",
       "            </th>\n",
       "            <th>\n",
       "              <font size=\"+1\" color=\"white\">\n",
       "                <b>Value origin</b>\n",
       "              </font>\n",
       "            </th>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"ActiveHand\"/>\n",
       "              <p>ActiveHand</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>left</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Trained hand (left and/or right)</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Birthday\"/>\n",
       "              <p>Birthday</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>2008-05-17</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>date</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Date of birth</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Character\"/>\n",
       "              <p>Character</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>Calm, but overall not really motivated. Less attentive.</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>text</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Comment on the general character/personality</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Disabilities\"/>\n",
       "              <p>Disabilities</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>-</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>text</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Comment on existing disabilities</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Gender\"/>\n",
       "              <p>Gender</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>male</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Gender (male or female)</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"GivenName\"/>\n",
       "              <p>GivenName</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>monkey_N</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Given name</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Identifier\"/>\n",
       "              <p>Identifier</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>i</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Short identifier for monkey (e.g., used in file names)</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"Species\"/>\n",
       "              <p>Species</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>Macaca mulatta</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Binomial species name (genus, species within genus)</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "          <tr>\n",
       "            <td width=\"15%\">\n",
       "              <a name=\"TrivialName\"/>\n",
       "              <p>TrivialName</p>\n",
       "            </td>\n",
       "            <td width=\"10%\">\n",
       "              <p>Rhesus monkey</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p>string</p>\n",
       "            </td>\n",
       "            <td width=\"5%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "            <td width=\"22.5%\">\n",
       "              <p>Commonly used (trivial) species name</p>\n",
       "            </td>\n",
       "            <td width=\"7.5%%\">\n",
       "              <p/>\n",
       "            </td>\n",
       "          </tr>\n",
       "        </font>\n",
       "      </table>\n",
       "      <a href=\"#top\">\n",
       "        <tiny>top</tiny>\n",
       "      </a>\n",
       "      <hr style=\"background-color:#336699; height:1px; margin-right:0; text-align:right;\"/>\n",
       "    </div>\n",
       "  </body>\n",
       "</html>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# display the extracted subsection for one of the datasets\n",
    "display_odML_as_html(filenames[0].split('.')[0] + '_filtered.odml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison of metadata\n",
    "In many cases metadata collections contain repeating structures, containing similar information for different contexts. Comparing these information located in different parts of the odML structure in a single table helps gaining an overview of the metadata and is sometimes required to be tracked in lab book tables. odMLtables offers a comparison function within an odML document, which generates such an overview table for a selected number of subsections.\n",
    "The published metadata collection contains information about 96 active recording electrodes, characterizing each in a separate odML section. The properties in these sections are always the same and only the structure of the subsections differs depending on the preprocessing (spikesorting) of the data. The basic electrode properties are a variety of different IDs associated with each electrode, it's impedance and length.\n",
    "\n",
    "The odMLtables comparison generates a xls or csv table listing the shared properties against the sections to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "for filename in filenames:\n",
    "    # create an odMLtables table for comparison with functionality to write to xls\n",
    "    comparetable = odt.CompareSectionXlsTable()\n",
    "    comparetable.load_from_file(filename)\n",
    "    # specify which section to compare, here the first 5 of 97 electrodes\n",
{'fraction_non_alphanumeric': 0.17566337604144788, 'fraction_numerical': 0.01249737450115522, 'mean_word_length': 1.557704360282926, 'pattern_counts': {'":': 97, '<': 415, '<?xml version=': 0, '>': 415, 'https://': 12, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6890089', 'n_tokens_mistral': 8958, 'n_tokens_neox': 8714, 'n_words': 2201}