<jupyter_start><jupyter_text>Dataset objects in rsatoolboxThese exercises show how to load and structure a dataset object.In this demo, we will first provide a walkthrough for loading a single-subject dataset from a .mat file and arranging it into an rsatoolbox dataset object.We then demonstrate how to create dataset objects using data from multiple subjects.<jupyter_code># relevant imports
import numpy as np
from scipy import io
import matplotlib.pyplot as plt
import rsatoolbox
import rsatoolbox.data as rsd # abbreviation to deal with dataset<jupyter_output><empty_output><jupyter_text>1. Single-subject dataset example Getting started We will use a dataset where one subject was presented with 92 different visual stimuli while brain responses were measured in 100 voxels.The different visual stimuli (each row) are the conditions, and the voxels (each column) are the measurement channels.<jupyter_code># import the measurements for the dataset
measurements = io.matlab.loadmat('92imageData/simTruePatterns.mat')
measurements = measurements['simTruePatterns']
nCond = measurements.shape[0]
nVox = measurements.shape[1]

# plot the imported data
plt.imshow(measurements,cmap='gray') 
plt.xlabel('Voxels')
plt.ylabel('Conditions')
plt.title('Measurements')<jupyter_output><empty_output><jupyter_text>Creating the dataset objectWe will now arrange the loaded data into a dataset object for use in rsatoolbox.A dataset object contains all the information needed to calculate a representational dissimilarity matrix (RDM). Therefore, the dataest must include: - measurements: [NxP] numpy.ndarray. These are the observations (N) from each measurement channel (P). - obs_descriptors: dict that defines the condition label associated with each observation in measurementsBecause we also want to include helpful information about this dataset, we include the additional information: - descriptors: dict with metadata about this dataset object (e.g. experiment session , subject , experiment name). Basically general descriptions - channel_descriptors: dict that identifies each column (channel) in measurementsTo start, we will note the session  (e.g. the first scanning session) and the subject  for this dataset. In addition, we will create labels for each of the 92 conditions and 100 voxels. Finally, we package this information into an rsatoolbox dataset object.<jupyter_code># now create a  dataset object
des = {'session': 1, 'subj': 1}
obs_des = {'conds': np.array(['cond_' + str(x) for x in np.arange(nCond)])}
chn_des = {'voxels': np.array(['voxel_' + str(x) for x in np.arange(nVox)])}
#obs_des = {'conds': np.array(['cond_' + str(x) for x in np.arange(1,nCond+1)])} # indices from 1
#chn_des = {'conds': np.array(['voxel' + str(x) for x in np.arange(1,nVox+1)])} # indices from 1
data = rsd.Dataset(measurements=measurements,
                           descriptors=des,
                           obs_descriptors=obs_des,
                           channel_descriptors=chn_des)
print(data)<jupyter_output>rsatoolbox.data.Dataset
measurements = 
[[-2.28467120e+16 -2.89393925e+16  1.98175667e+16 -1.52670137e+16
   6.16739860e+14  2.10873889e+16  7.54719220e+15  3.91406464e+15
   2.06459753e+15  1.71183930e+15  9.25183442e+15  6.25203471e+14
  -2.61395909e+16  3.30336292e+16  9.85160565e+15  7.08693907e+15
   4.56990621e+15  1.43744616e+16  7.94155302e+15 -9.69347003e+15
   4.54740006e+13  2.52235447e+16  1.09579543e+16  2.01266145e+16
  -8.27816988e+15  1.78844114e+16 -8.56003159e+15  3.99141304e+15
   6.61310188e+14 -3.44699674e+16 -1.02998653e+16  2.17767629e+16
   8.36071759e+15  3.87267836e+16 -7.68930294e+13  2.92524184e+14
   3.61891025e+16  2.03835675e+16 -1.48229413e+16 -1.79885027e+16
   1.38548750e+16 -1.48847681e+16  8.07503594e+15 -6.08550537e+15
  -3.95331143e+16  1.47461837e+15  1.82758194e+16  2.49558214e+16
   5.57712653e+15  3.05545911e+16 -1.34602130e+16  6.08982232e+15
  -1.46623702e+16  9.11955876e+15 -1.48137929e+15 -1.00568477e+16
   2.26715542e+16 -2.28012042e+15 -8[...]<jupyter_text>Sometimes we wish to consider only a subset of data - either a subset of observations (conditions), or subset of measurement channels.  This might be to only consider the measurement channels where all the subjects have data, or conditions which occur across all subjects / sessions. Using dataset functionality, we can subset the datasets according to a subset of the conditions or channels via 'subset_obs' and 'subset_channel', respectively.<jupyter_code># create an example dataset with random data, subset some conditions
nChannel = 50
nObs = 12
randomData = np.random.rand(nObs, nChannel)
des = {'session': 1, 'subj': 1}
obs_des = {'conds': np.array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5])}
chn_des = {'voxels': np.array(['voxel_' + str(x) for x in np.arange(nChannel)])}
data = rsd.Dataset(measurements=randomData,
                        descriptors=des,
                        obs_descriptors=obs_des,
                        channel_descriptors=chn_des
                        )
# select a subset of the dataset: select data only from conditions 0:4
sub_data = data.subset_obs(by='conds', value=[0,1,2,3,4])
print(sub_data)<jupyter_output>rsatoolbox.data.Dataset
measurements = 
[[0.91863267 0.4846205  0.33593029 0.75119766 0.19551113 0.6198097
  0.50750475 0.12601606 0.47426061 0.5989329  0.0637164  0.94045044
  0.48710383 0.11057852 0.51008086 0.63837393 0.22056416 0.74386461
  0.94673388 0.98933934 0.80076636 0.43450966 0.09145941 0.48246455
  0.97571431 0.0635358  0.78716189 0.14841272 0.97617542 0.84753924
  0.86312071 0.30659987 0.56769031 0.83987536 0.2818678  0.61745352
  0.20236613 0.44194742 0.20136011 0.70478929 0.81369772 0.64082679
  0.58836394 0.4405892  0.58239543 0.88344139 0.14956161 0.73649931
  0.07653746 0.6726465 ]
 [0.80857067 0.62109056 0.88911834 0.41281895 0.2843382  0.72506928
  0.24971596 0.74377068 0.44387595 0.95477941 0.11043901 0.20102694
  0.4691736  0.81217928 0.22625212 0.68598281 0.69745939 0.49944524
  0.09048105 0.17302212 0.03534    0.60489602 0.78473337 0.04339872
  0.12817508 0.97542835 0.73614003 0.68608745 0.60052989 0.45384906
  0.7319914  0.9108467  0.6451067  0.72519248 0.4503[...]<jupyter_text>Additionally, you might want to split the data in a certain way and analyze the splits as separate datasets. For instance, if your data is organized such that there are different ROIs, you might wish to perform the subsequent analyses separately for each ROI. Similarly, you could split the observations. This is supported with 'split_obs' and 'split_channel' options on the dataset object.<jupyter_code># Split by channels
nChannel = 3 
nChannelVox = 10 # three ROIs, each with 10 voxels
nObs = 4
randomData = np.random.rand(nObs, nChannel*nChannelVox)
des = {'session': 1, 'subj': 1}
obs_des = {'conds': np.array([0, 1, 2, 3])}
chn_des = ['ROI1', 'ROI2', 'ROI3'] * nChannelVox
chn_des = {'ROIs': np.array(chn_des)}
data = rsd.Dataset(measurements=randomData,
                        descriptors=des,
                        obs_descriptors=obs_des,
                        channel_descriptors=chn_des
                        )
split_data = data.split_channel(by='ROIs')
print(split_data)<jupyter_output>[rsatoolbox.data.Dataset(
measurements = 
[[2.72517419e-01 6.29869066e-01 1.61217030e-04 6.91723973e-01
  7.05569326e-01 9.50770933e-01 7.89382007e-01 9.78685869e-01
  2.90501341e-01 7.03136292e-01]
 [2.01005582e-01 5.46987762e-01 4.84408356e-01 5.88773463e-01
  8.38763394e-01 5.84621828e-01 8.63289698e-03 4.01037703e-01
  4.55127908e-01 3.64116665e-01]
 [1.69157574e-01 8.80178106e-01 4.75415837e-01 4.66419825e-01
  4.85189987e-02 9.62232648e-01 7.74986919e-01 9.11800689e-01
  7.01191521e-01 3.20210323e-02]
 [9.20122144e-01 5.68003364e-02 2.01368202e-01 2.44601779e-01
  1.52852082e-01 5.83556175e-01 6.09733866e-01 2.05660245e-01
  8.96234776e-01 3.81140491e-01]]
descriptors = 
{'session': 1, 'subj': 1, 'ROIs': 'ROI1'}
obs_descriptors = 
{'conds': array([0, 1, 2, 3])}
channel_descriptors = 
{'ROIs': ['ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1', 'ROI1']}
, rsatoolbox.data.Dataset(
measurements = 
[[0.48887379 0.94426476 0.79195955 0.11657703 0.44759874 0.493549[...]<jupyter_text>2. Multi-subject dataset exampleFirst, we generate random data for a number of subjects. For simplicity, here we set each subject to have the same number of voxels and conditions.<jupyter_code># create a datasets with random data
nVox = 50 # 50 voxels/electrodes/measurement channels
nCond = 10 # 10 conditions
nSubj = 5 # 5 different subjects
randomData = np.random.rand(nCond, nVox, nSubj)<jupyter_output><empty_output><jupyter_text>We can then create a list of dataset objects by appending each dataset for each subject.<jupyter_code>obs_des = {'conds': np.array(['cond_' + str(x) for x in np.arange(nCond)])}
chn_des = {'voxels': np.array(['voxel_' + str(x) for x in np.arange(nVox)])}

data = [] # list of dataset objects
for i in np.arange(nSubj):
    des = {'session': 1, 'subj': i+1}
    # append the dataset object to the data list
    data.append(rsd.Dataset(measurements=randomData[:,:,i],
                        descriptors=des,
                        obs_descriptors=obs_des,
                        channel_descriptors=chn_des
                        )
               )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10586598158852494, 'fraction_numerical': 0.21076857204024835, 'mean_word_length': 3.9909188034188032, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '519769', 'license': 'MIT', 'n_tokens_mistral': 4583, 'n_tokens_neox': 3457, 'n_words': 907, 'path': 'demos/demo_dataset.ipynb', 'repo_name': 'smazurchuk/rsatoolbox', 'size': '9342'}