<jupyter_start><jupyter_text>Introduction to Setting Up the Environment and Importing DataThis notebook is part of the [SachsLab Workshop for Intracranial Neurophysiology and Deep Learning](https://github.com/SachsLab/IntracranialNeurophysDL).      Run in Google Colab        View source on GitHub    Normalizing EnvironmentsLet's try to unify the Google Colab environment and the local environment.If you are running on Google Colab, then in the next cell you'll be presentedwith a widget. Use it to browse to the kaggle.json file.(Note: You might have to show hidden files/folders in the file browser. `CMD + SHIFT + .` on Mac)It will also clone the workshop repository and change into its directory.If you are running locally then this will simply change to the workshop root directory.<jupyter_code>from pathlib import Path
import os

# Standard block to equalize local and Colab.
try:
    # See if we are running on google.colab
    import google.colab
    from google.colab import files
    if not (Path.home() / '.kaggle').is_dir():
        # Configure kaggle
        files.upload()  # Find the kaggle.json file in your ~/.kaggle directory.
        !pip install -q kaggle
        !mkdir -p ~/.kaggle
        !mv kaggle.json ~/.kaggle/
        !chmod 600 ~/.kaggle/kaggle.json
    
    if Path.cwd().stem != 'IntracranialNeurophysDL':
        if not (Path.cwd() / 'IntracranialNeurophysDL').is_dir():
            # Download the workshop repo and change to its directory
            !git clone --recursive https://github.com/SachsLab/IntracranialNeurophysDL.git
        os.chdir('IntracranialNeurophysDL')
    IN_COLAB = True
except ModuleNotFoundError:
    IN_COLAB = False
    import sys
    if Path.cwd().stem == 'notebooks':
        os.chdir(Path.cwd().parent)
    # Make sure the kaggle executable is on the PATH
    os.environ['PATH'] = os.environ['PATH'] + ';' + str(Path(sys.executable).parent / 'Scripts')
    
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Get Data DownloadNow that your system is configured to use kaggle, whether local or running on colab,we check for the existence of the datadir, and if it's not there we download the data(2.2 GB) and unzip it into the correct folder.While running on google colab, this takes about a minute to download and another minute to unzip.While running locally, this can take a long time depending on your internet connection. ThePyCharm notebook interface doesn't give feedback about download or unzip status, so you maywant to download and extract from an Anaconda prompt, in the repository parent directory:```kaggle datasets download --unzip --path data/kjm_ecog/converted/faces_basic cboulay/kjm-ecog-faces-basic```If you are getting errors related to access restriction then you may need to download a new Kaggle API token and reset your Colab instance to upload the new .json file.<jupyter_code>datadir = Path.cwd() / 'data' / 'kjm_ecog'
if not (datadir / 'converted').is_dir():
    !kaggle datasets download --unzip --path {str(datadir / 'converted' / 'faces_basic')} cboulay/kjm-ecog-faces-basic
    print("Finished downloading and extracting data.")
else:
    print("Data directory already exists. Skipping download.")<jupyter_output><empty_output><jupyter_text>Import FileWe can now start working with the data.We will import a single file that contains processed band-power data.If you are curious about how the data were processed, the script can be found [here](https://github.com/SachsLab/IntracranialNeurophysDL/blob/master/data/kjm_ecog/03_convert.py).<jupyter_code>from data.utils.fileio import from_neuropype_h5

SUB_ID = 'de'

test_file = datadir / 'converted' / 'faces_basic' / (SUB_ID + '_bp.h5')
chunks = from_neuropype_h5(test_file)

print("Chunks found: {}".format([_[0] for _ in chunks]))<jupyter_output><empty_output><jupyter_text>Data Exploration Print ContentsLet's quickly inspect the data to see what we have.<jupyter_code>import pandas as pd

# Get the 'signals' chunk
chunk_names = [_[0] for _ in chunks]
chunk = chunks[chunk_names.index('signals')][1]
ax_types = [_['type'] for _ in chunk['axes']]

print("The 'signals' chunk has data with shape {}.".format(chunk['data'].shape))
print("The axes types are {}".format(ax_types))

time_axis = chunk['axes'][ax_types.index('time')]
t_vec = time_axis['times']
print("Each trial has {} samples, ranging from {} to {} s.".format(len(t_vec), min(t_vec), max(t_vec)))

instance_axis = chunk['axes'][ax_types.index('instance')]
print("The trial label frequencies are \n{}".format(pd.value_counts(instance_axis['data']['Marker'])))<jupyter_output><empty_output><jupyter_text>Simple PlottingThe data contain 603 trials, with each trial having17 samples in time, and 31 channels.Of the 603 trials, 303 or inter-stimulus intervals, and the remaining300 are split between `face` and `house`.An N-dimensional data container is also called a "tensor".The most common forms are 1-D tensors, also known as "vectors",and 2-D tensors, also known as "matrices".This is a 3-D tensor.We will plot a few different slices of the tensor.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 22})

fig = plt.figure(figsize=(12, 6), facecolor='white')

# Plot a single trial
tr_idx = 20  # trial index
dat = np.copy(chunk['data'][tr_idx, :, :])  # Slice a single-trial
dat -= np.arange(dat.shape[-1])[None, :]  # Separate channels for visibility.
plt.subplot(1, 2, 1)
plt.plot(t_vec, dat)
plt.title('Single Trial')
plt.ylabel('Channel')
plt.xlabel('Time (s)')

# Plot a few trials from a single channel.
ch_idx = 11  # Also try 22
dat = np.copy(chunk['data'][:, :, ch_idx])
tr_idx_isi = np.where(instance_axis['data']['Marker'] == 'ISI')[0][:5]
plt.subplot(1, 2, 2)
plt.plot(t_vec, dat[tr_idx_isi].T, 'k')
tr_idx_houses = np.where(instance_axis['data']['Marker'] == 'house')[0][:5]
plt.plot(t_vec, dat[tr_idx_houses].T, 'b')
tr_idx_faces = np.where(instance_axis['data']['Marker'] == 'face')[0][:5]
plt.plot(t_vec, dat[tr_idx_faces].T, 'r')
plt.title('Trials for for Chan {}'.format(ch_idx))
plt.ylabel('Broadband Power (z)')
plt.xlabel('Time After Stim (s)')

plt.show()<jupyter_output><empty_output><jupyter_text>Tensor DecompositionLet's use a tensor-decomposition tool to get a simpler view of the tensor contents.[Source](https://pyramidal.stanford.edu/publications/Williams2018_Neuron.pdf)![simple linear network](https://github.com/SachsLab/IntracranialNeurophysDL/blob/master/notebooks/img/tca.png?raw=1) In the below plot, each row is a different tensor component and each column is a different axis/dimension.The output changes each time it is run.Look for a component where the 1st column clearly shows two different groups of trials.Then in the 2nd column we can see the time-course of that component, and in the 3rd column we see the channels that contributed to that component.<jupyter_code>if IN_COLAB:
    !pip install git+https://github.com/ahwillia/tensortools
import tensortools as tt

U = tt.cp_als(chunk['data'], rank=3, verbose=True)
fig, ax, po = tt.plot_factors(U.factors, plots=['scatter', 'line', 'bar'], figsize=(12, 12))<jupyter_output><empty_output><jupyter_text>Tensor Decomposition in NeuropypeThe next cell does the same thing with a slightly better plot, but is for Neuropype users only, and only those who have the latest version. I don't yet have a workflow for getting Neuropype on the cloud so this one part is for local use only.As it uses plotly, it won't show up in PyCharm (should be fixed in v2019.2) but it will in a browser.<jupyter_code># Add my local copy of Neuropype to the PATH.
import sys
sys.path.append(str(Path.cwd().parent / 'Intheon' / 'cpe'))
import neuropype.nodes as nn

pkt = nn.ImportH5(filename=str(test_file))()
tt_res = nn.TensorDecomposition(num_components=3, aggregate_axes=['instance'])(
    data=pkt, return_outputs='all')
type_map = {'time': 'lines', 'space': 'bars', 'instance': 'markers'}
plt = nn.TensorDecompositionPlot(output_mode='notebook', iv_field='Marker',
                                 type_map=type_map)(data=tt_res['model'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11448598130841121, 'fraction_numerical': 0.009468765371372355, 'mean_word_length': 4.821760916249105, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '510516', 'license': 'MIT', 'n_tokens_mistral': 2691, 'n_tokens_neox': 2549, 'n_words': 856, 'path': 'notebooks/01_02_data_import.ipynb', 'repo_name': 'stfnrpplngr/IntracranialNeurophysDL', 'size': '8132'}