<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# G-Mode filtering and inspection using pycroscopy\n",
    "### <NAME> and <NAME>\n",
    "The Center for Nanophase Materials Science and The Institute for Functional Imaging for Materials <br>\n",
    "Oak Ridge National Laboratory<br>\n",
    "5/05/2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ensure python 3 compatibility\n",
    "from __future__ import division, print_function, absolute_import, unicode_literals\n",
    "\n",
    "# Import necessary libraries:\n",
    "# General utilities:\n",
    "from os import path\n",
    "\n",
    "# Computation:\n",
    "import numpy as np\n",
    "import h5py\n",
    "\n",
    "# Visualization:\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Finally, pycroscopy itself\n",
    "import pycroscopy as px\n",
    "\n",
    "# set up notebook to show plots within the notebook\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ui_file_window = False\n",
    "try:\n",
    "    from PyQt5 import QtWidgets\n",
    "    \n",
    "    def uiGetFile(filter='H5 file (*.h5)', caption='Select File'):\n",
    "        \"\"\"\n",
    "        Presents a File dialog used for selecting the .mat file\n",
    "        and returns the absolute filepath of the selecte file\\n\n",
    "        Parameters\n",
    "        ----------\n",
    "        extension : String or list of strings\n",
    "            file extensions to look for\n",
    "        caption : (Optional) String\n",
    "            Title for the file browser window\n",
    "        Returns\n",
    "        -------\n",
    "        file_path : String\n",
    "            Absolute path of the chosen file\n",
    "        \"\"\"\n",
    "        app = QtWidgets.QApplication([])\n",
    "        path = QtWidgets.QFileDialog.getOpenFileName(caption=caption, filter=filter)[0]\n",
    "        app.exit()\n",
    "        del app\n",
    "\n",
    "        return str(path)\n",
    "    \n",
    "    ui_file_window = True\n",
    "except ImportError:\n",
    "    print('***********************************************************')\n",
    "    print('*                                                         *')\n",
    "    print('*  You will need to specify the file path manually below  *')\n",
    "    print('*                                                         *')\n",
    "    print('***********************************************************')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make the data pycroscopy compatible\n",
    "Converting the raw data into a pycroscopy compatible hierarchical data format (HDF or .h5) file gives you access to the fast fitting algorithms and powerful analysis functions within pycroscopy\n",
    "\n",
    "#### H5 files:\n",
    "* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.\n",
    "* are readily compatible with high-performance computing facilities\n",
    "* scale very efficiently from few kilobytes to several terabytes\n",
    "* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.\n",
    "\n",
    "#### You can load either of the following:\n",
    "* Any .mat or .txt parameter file from the original experiment\n",
    "* A .h5 file generated from the raw data using pycroscopy - skips translation\n",
    "\n",
    "You can select desired file type by choosing the second option in the pull down menu on the bottom right of the file window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if ui_file_window:\n",
    "    input_file_path = uiGetFile(caption='Select translated .h5 file or raw experiment data',\n",
    "                                filter='Parameters for raw G-Line data (*.txt);; \\\n",
    "                                        Translated file (*.h5)')\n",
    "else:\n",
    "    input_file_path = '/Volumes/IFgroup/SPM software development/Raw_Data/G_mode/GVS/2015_04_08_PZT_AuCu_nanocaps/GLine_8V_10kHz_256x256_0001/GLine_8V_10kHz_256x256.h5'\n",
    "\n",
    "folder_path, _ = path.split(input_file_path)\n",
    "\n",
    "if input_file_path.endswith('.txt'):\n",
    "    print('Translating raw data to h5. Please wait')\n",
    "    tran = px.GLineTranslator()\n",
    "    h5_path = tran.translate(file_path)\n",
    "else:\n",
    "    h5_path = input_file_path\n",
    "\n",
    "print('Working on:\\n' + h5_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open the .h5 file and extract some basic parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdf = px.ioHDF5(h5_path)\n",
    "h5_main = px.hdf_utils.getDataSet(hdf.file, 'Raw_Data')[-1]\n",
    "parms_dict = h5_main.parent.parent.attrs\n",
    "\n",
    "samp_rate = parms_dict['IO_rate_[Hz]']\n",
    "ex_freq = parms_dict['BE_center_frequency_[Hz]']\n",
    "\n",
    "h5_spec_vals = px.hdf_utils.getAuxData(h5_main, auxDataName='Spectroscopic_Values')[0]\n",
    "pixel_ex_wfm = h5_spec_vals[0, :int(h5_spec_vals.shape[1]/parms_dict['grid_num_cols'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Inspect the contents of this h5 data file\n",
    "The file contents are stored in a tree structure, just like files on a conventional computer.\n",
    "The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data. Thus, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0.... and the data for each position is stored as it was chronologically collected  \n",
    "\n",
    "The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset.\n",
    "\n",
    "Note that G-mode data is acquired line-by-line rather than pixel-by-pixel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Datasets and datagroups within the file:\\n------------------------------------')\n",
    "px.io.hdf_utils.print_tree(hdf.file)\n",
    " \n",
    "print('\\nThe main dataset:\\n------------------------------------')\n",
    "print(h5_main)\n",
    "print('\\nThe ancillary datasets:\\n------------------------------------')\n",
    "print(hdf.file['/Measurement_000/Channel_000/Position_Indices'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Position_Values'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Spectroscopic_Indices'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Spectroscopic_Values'])\n",
    "\n",
    "print('\\nMetadata or attributes in a datagroup\\n------------------------------------')\n",
    "for key in hdf.file['/Measurement_000'].attrs:\n",
    "    print('{} : {}'.format(key, hdf.file['/Measurement_000'].attrs[key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the raw data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_ind = 40\n",
    "raw_row = h5_main[row_ind].reshape(-1, pixel_ex_wfm.size)\n",
    "\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, raw_row, x_label='Bias (V)', title='Raw Measurement',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try different FFT filters on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_parms = dict()\n",
    "filter_parms['noise_threshold'] = 1E-4\n",
    "filter_parms['comb_[Hz]'] = [ex_freq, 1E+3, 10]\n",
    "# filter_parms['LPF_cutOff_[Hz]'] = -1\n",
    "# Noise frequencies - 15.6 kHz ~ 14-17.5, 7.8-8.8, 45-49.9 ~ 48.9414 kHz\n",
    "# filter_parms['band_filt_[Hz]'] = None  # [[8.3E+3, 15.6E+3, 48.9414E+3], [1E+3, 0.5E+3, 0.1E+3]]\n",
    "# filter_parms['phase_[rad]'] = 0\n",
    "filter_parms['samp_rate_[Hz]'] = samp_rate\n",
    "filter_parms['num_pix'] = 1\n",
    "\n",
    "# Test filter on a single line:\n",
    "row_ind = 40\n",
    "filt_line, fig_filt, axes_filt = px.processing.gmode_utils.test_filter(h5_main[row_ind], filter_parms, samp_rate,\n",
    "                                                                      show_plots=True, use_rainbow_plots=False)\n",
    "fig_filt.savefig(path.join(folder_path, 'FFT_filter_on_line_{}.png'.format(row_ind)), format='png', dpi=300)\n",
    "\n",
    "filt_row = filt_line.reshape(-1, pixel_ex_wfm.size)\n",
    "\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, filt_row, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')\n",
    "# fig.savefig(path.join(folder_path, 'FFT_filtered_loops_on_line_{}.png'.format(row_ind)), format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply selected filter to entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# h5_filt_grp = px.hdf_utils.findH5group(h5_main, 'FFT_Filtering')[-1]\n",
    "h5_filt_grp = px.processing.gmode_utils.fft_filter_dataset(h5_main, filter_parms, write_filtered=True)\n",
    "h5_filt = h5_filt_grp['Filtered_Data']\n",
    "\n",
    "# Test to make sure the filter gave the same results\n",
    "filt_row = h5_filt[row_ind].reshape(-1, pixel_ex_wfm.size)\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, filt_row, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now break up the filtered lines into \"pixels\"\n",
    "Also visualize loops from different pixels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# h5_resh = h5_filt_grp['Filtered_Data-Reshape_000/Reshaped_Data']\n",
    "h5_resh = px.processing.gmode_utils.reshape_from_lines_to_pixels(h5_filt, pixel_ex_wfm.size, 1)\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, h5_resh, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=5, y_label='Deflection (a.u.)')\n",
    "# fig.savefig(path.join(folder_path, 'FFT_filtered_loops_on_line_{}.png'.format(row_ind)), format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
<commit_msg>Fixing a bug in G_mode_filtering notebook<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# G-Mode filtering and inspection using pycroscopy\n",
    "### <NAME> and <NAME>\n",
    "The Center for Nanophase Materials Science and The Institute for Functional Imaging for Materials <br>\n",
    "Oak Ridge National Laboratory<br>\n",
    "5/05/2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ensure python 3 compatibility\n",
    "from __future__ import division, print_function, absolute_import, unicode_literals\n",
    "\n",
    "# Import necessary libraries:\n",
    "# General utilities:\n",
    "from os import path\n",
    "\n",
    "# Computation:\n",
    "import numpy as np\n",
    "import h5py\n",
    "\n",
    "# Visualization:\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Finally, pycroscopy itself\n",
    "import pycroscopy as px\n",
    "\n",
    "# set up notebook to show plots within the notebook\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ui_file_window = False\n",
    "try:\n",
    "    from PyQt5 import QtWidgets\n",
    "    \n",
    "    def uiGetFile(filter='H5 file (*.h5)', caption='Select File'):\n",
    "        \"\"\"\n",
    "        Presents a File dialog used for selecting the .mat file\n",
    "        and returns the absolute filepath of the selecte file\\n\n",
    "        Parameters\n",
    "        ----------\n",
    "        extension : String or list of strings\n",
    "            file extensions to look for\n",
    "        caption : (Optional) String\n",
    "            Title for the file browser window\n",
    "        Returns\n",
    "        -------\n",
    "        file_path : String\n",
    "            Absolute path of the chosen file\n",
    "        \"\"\"\n",
    "        app = QtWidgets.QApplication([])\n",
    "        path = QtWidgets.QFileDialog.getOpenFileName(caption=caption, filter=filter)[0]\n",
    "        app.exit()\n",
    "        del app\n",
    "\n",
    "        return str(path)\n",
    "    \n",
    "    ui_file_window = True\n",
    "except ImportError:\n",
    "    print('***********************************************************')\n",
    "    print('*                                                         *')\n",
    "    print('*  You will need to specify the file path manually below  *')\n",
    "    print('*                                                         *')\n",
    "    print('***********************************************************')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make the data pycroscopy compatible\n",
    "Converting the raw data into a pycroscopy compatible hierarchical data format (HDF or .h5) file gives you access to the fast fitting algorithms and powerful analysis functions within pycroscopy\n",
    "\n",
    "#### H5 files:\n",
    "* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.\n",
    "* are readily compatible with high-performance computing facilities\n",
    "* scale very efficiently from few kilobytes to several terabytes\n",
    "* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.\n",
    "\n",
    "#### You can load either of the following:\n",
    "* Any .mat or .txt parameter file from the original experiment\n",
    "* A .h5 file generated from the raw data using pycroscopy - skips translation\n",
    "\n",
    "You can select desired file type by choosing the second option in the pull down menu on the bottom right of the file window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if ui_file_window:\n",
    "    input_file_path = uiGetFile(caption='Select translated .h5 file or raw experiment data',\n",
    "                                filter='Parameters for raw G-Line data (*.txt);; \\\n",
    "                                        Translated file (*.h5)')\n",
    "else:\n",
    "    input_file_path = '/Volumes/IFgroup/SPM software development/Raw_Data/G_mode/GVS/2015_04_08_PZT_AuCu_nanocaps/GLine_8V_10kHz_256x256_0001/GLine_8V_10kHz_256x256.h5'\n",
    "\n",
    "folder_path, _ = path.split(input_file_path)\n",
    "\n",
    "if input_file_path.endswith('.txt'):\n",
    "    print('Translating raw data to h5. Please wait')\n",
    "    tran = px.GLineTranslator()\n",
    "    h5_path = tran.translate(input_file_path)\n",
    "else:\n",
    "    h5_path = input_file_path\n",
    "\n",
    "print('Working on:\\n' + h5_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open the .h5 file and extract some basic parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdf = px.ioHDF5(h5_path)\n",
    "h5_main = px.hdf_utils.getDataSet(hdf.file, 'Raw_Data')[-1]\n",
    "parms_dict = h5_main.parent.parent.attrs\n",
    "\n",
    "samp_rate = parms_dict['IO_rate_[Hz]']\n",
    "ex_freq = parms_dict['BE_center_frequency_[Hz]']\n",
    "\n",
    "h5_spec_vals = px.hdf_utils.getAuxData(h5_main, auxDataName='Spectroscopic_Values')[0]\n",
    "pixel_ex_wfm = h5_spec_vals[0, :int(h5_spec_vals.shape[1]/parms_dict['grid_num_cols'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Inspect the contents of this h5 data file\n",
    "The file contents are stored in a tree structure, just like files on a conventional computer.\n",
    "The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data. Thus, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0.... and the data for each position is stored as it was chronologically collected  \n",
    "\n",
    "The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset.\n",
    "\n",
    "Note that G-mode data is acquired line-by-line rather than pixel-by-pixel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Datasets and datagroups within the file:\\n------------------------------------')\n",
    "px.io.hdf_utils.print_tree(hdf.file)\n",
    " \n",
    "print('\\nThe main dataset:\\n------------------------------------')\n",
    "print(h5_main)\n",
    "print('\\nThe ancillary datasets:\\n------------------------------------')\n",
    "print(hdf.file['/Measurement_000/Channel_000/Position_Indices'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Position_Values'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Spectroscopic_Indices'])\n",
    "print(hdf.file['/Measurement_000/Channel_000/Spectroscopic_Values'])\n",
    "\n",
    "print('\\nMetadata or attributes in a datagroup\\n------------------------------------')\n",
    "for key in hdf.file['/Measurement_000'].attrs:\n",
    "    print('{} : {}'.format(key, hdf.file['/Measurement_000'].attrs[key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the raw data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_ind = 40\n",
    "raw_row = h5_main[row_ind].reshape(-1, pixel_ex_wfm.size)\n",
    "\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, raw_row, x_label='Bias (V)', title='Raw Measurement',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try different FFT filters on the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_parms = dict()\n",
    "filter_parms['noise_threshold'] = 1E-4\n",
    "filter_parms['comb_[Hz]'] = [ex_freq, 1E+3, 10]\n",
    "# filter_parms['LPF_cutOff_[Hz]'] = -1\n",
    "# Noise frequencies - 15.6 kHz ~ 14-17.5, 7.8-8.8, 45-49.9 ~ 48.9414 kHz\n",
    "# filter_parms['band_filt_[Hz]'] = None  # [[8.3E+3, 15.6E+3, 48.9414E+3], [1E+3, 0.5E+3, 0.1E+3]]\n",
    "# filter_parms['phase_[rad]'] = 0\n",
    "filter_parms['samp_rate_[Hz]'] = samp_rate\n",
    "filter_parms['num_pix'] = 1\n",
    "\n",
    "# Test filter on a single line:\n",
    "row_ind = 40\n",
    "filt_line, fig_filt, axes_filt = px.processing.gmode_utils.test_filter(h5_main[row_ind], filter_parms, samp_rate,\n",
    "                                                                      show_plots=True, use_rainbow_plots=False)\n",
    "fig_filt.savefig(path.join(folder_path, 'FFT_filter_on_line_{}.png'.format(row_ind)), format='png', dpi=300)\n",
    "\n",
    "filt_row = filt_line.reshape(-1, pixel_ex_wfm.size)\n",
    "\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, filt_row, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')\n",
    "# fig.savefig(path.join(folder_path, 'FFT_filtered_loops_on_line_{}.png'.format(row_ind)), format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply selected filter to entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# h5_filt_grp = px.hdf_utils.findH5group(h5_main, 'FFT_Filtering')[-1]\n",
    "h5_filt_grp = px.processing.gmode_utils.fft_filter_dataset(h5_main, filter_parms, write_filtered=True)\n",
    "h5_filt = h5_filt_grp['Filtered_Data']\n",
    "\n",
    "# Test to make sure the filter gave the same results\n",
    "filt_row = h5_filt[row_ind].reshape(-1, pixel_ex_wfm.size)\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, filt_row, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=4, y_label='Deflection (a.u.)',\n",
    "                                     subtitles='Row: ' + str(row_ind) + ' Col:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now break up the filtered lines into \"pixels\"\n",
    "Also visualize loops from different pixels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# h5_resh = h5_filt_grp['Filtered_Data-Reshape_000/Reshaped_Data']\n",
    "h5_resh = px.processing.gmode_utils.reshape_from_lines_to_pixels(h5_filt, pixel_ex_wfm.size, 1)\n",
    "fig, axes = px.plot_utils.plot_loops(pixel_ex_wfm, h5_resh, x_label='Bias (V)', title='FFT Filtering',\n",
    "                                     plots_on_side=5, y_label='Deflection (a.u.)')\n",
    "# fig.savefig(path.join(folder_path, 'FFT_filtered_loops_on_line_{}.png'.format(row_ind)), format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}

{'fraction_non_alphanumeric': 0.19444218737303973, 'fraction_numerical': 0.01698220524904526, 'mean_word_length': 2.49347147317627, 'pattern_counts': {'":': 208, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 48, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6258031', 'n_tokens_mistral': 8982, 'n_tokens_neox': 8655, 'n_words': 1793}