<jupyter_start><jupyter_text>We first define the input data. The input data is has the following structure: We have a group of subjects (say S1 and S2) and each subject has completed the decision-making task multiple times (multiple blocks), say each subject has completed the task two times in this example, i.e., we have two blocks of data for each subject. The data within each block is a ditonary containing three numpy arrays: 'action', 'state', 'reward'. 'action' containig the actions taken by the subject on each trial and it should be a non-zero integer or -1. If the action is -1 it will coded by a zero vector and corresponds to no-action. Dimesionlity of 'action' is B x |T| in which |T| is the number of trials.'state' contains the state of the environment each trial. Its dimesionlity is B x |T| x |S| in which |T| is the number of trials, and |S| is the lenght of state vector. 'reward' contains the reward received after taking each actions. Its dimesionlity is B x |T| in which |T| is the number of trials.For example, if subject S1 has completed 6 trials in the firt block and 4 trials in the second block and subject 2 has completed 5, 6 trials in the first and second blocks respectivly, then the data structure can look like this:<jupyter_code>#  Data
import numpy as np
data = {'S1':
        [
            {
                'action': np.array([
                    [0, 1, 0, 1, 0, 1]
                ]),
                'state': np.array([
                    [[0, 0], [0, 1], [0.1, 0.2], [0.5, 0.6], [0, 0], [1, 1]]
                ]),
                'reward': np.array([
                    [0, 1, 0, 1, 0, 1]
                ]),
                'id': 'S1',
                
                'block': 0
            },
            {
                'action': np.array([
                    [1, 1, 0, -1]
                ]),
                'state': np.array([
                    [[0, 1], [1, -1], [0.1, 0], [0, 0.6]]
                ]),
                'reward': np.array([
                    [0, 0, 0, 0]
                ]),
                'id': 'S1',
                
                'block': 1
            }
            
        ],

    'S2':
        [
            {
                'action': np.array([
                    [0, 1, 0, 1, 0]
                ]),
                'state': np.array([
                    [[0, 0], [0, 1], [0.1, 0.2], [0.5, 0.6], [0, 0]]
                ]),
                'reward': np.array([
                    [0, 1, 0, 1, 0]
                ]),
                'id': 'S2',
                
                'block': 0
                
            },
            {
                'action': np.array([
                    [1, 1, 0, 1, 1, 1]
                ]),
                'state': np.array([
                    [[0, 0], [0, -1], [0.1, 0.2], [0.5, 0.6], [0, 1], [-1, -1]]
                ]),
                'reward': np.array([
                    [0, 0, 0, 0, 0, 1]
                ]),
                'id': 'S2',
                
                'block': 1
                
            }
            
        ]

        }
print(data['S1'][0]['action'].shape,data['S1'][0]['state'].shape,data['S1'][0]['reward'].shape)
data<jupyter_output><empty_output><jupyter_text>In the above example, |A|=2 (there are two actions coded as 0 and 1), and |S|=2 (the state vector has two elements). For example, if there are three stimuli in the environment, they can be coded as [1, 0, 0], [0, 1, 0], [0, 0, 1] state vectors. In this case |S|=3.<jupyter_code>cd ..
from actionflow.rnn.lstm_beh import LSTMBeh
worker = LSTMBeh(a_size=2, s_size=2, n_cells=5)<jupyter_output>c:\users\user\documents\surf 2021\rl rnn surf\venv\lib\site-packages\tensorflow\python\framework\dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\users\user\documents\surf 2021\rl rnn surf\venv\lib\site-packages\tensorflow\python\framework\dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\users\user\documents\surf 2021\rl rnn surf\venv\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\users\user\documents\surf[...]<jupyter_text>Finally, we train the model:<jupyter_code>from actionflow.rnn.opt_beh import OptBEH
from actionflow.util.logger import LogFile

output_path = '../results/'
with LogFile(output_path, 'run.log'):
    OptBEH.optimise(worker, output_path, data, None,
                    learning_rate=0.01,
                    global_iters=50,
                    load_model_path=None
                    )
from actionflow.data.data_process import DataProcess
train_merged = DataProcess.merge_data(data)
train_merged<jupyter_output><empty_output><jupyter_text>And then the merged data can be used for training the model as before. The test data can also be passed to the training method, in order to test the mmodel on the training data in regular intervals. Say the test data is as follows:<jupyter_code>test_data = {'S4':
        [
            {
                'action': np.array([
                    [1, 1, 0, 1, 0, 1]
                ]),
                'state': np.array([
                    [[0, 0], [0, 1], [0.0, 0.2], [0.1, 0.2], [1, 1], [1, 1]]
                ]),
                'reward': np.array([
                    [1, 3, 0, 1, 0, 1]
                ]),
                'id': 'S4',
                
                'block': 0
            },
            {
                'action': np.array([
                    [0, 1, -1, 0]
                ]),
                'state': np.array([
                    [[0, 1], [1, -1], [0.1, 0], [0, 0.6]]
                ]),
                'reward': np.array([
                    [0, 1, 0, 1]
                ]),
                'id': 'S4',
                
                'block': 1
            }
            
        ],

    'S5':
        [
            {
                'action': np.array([
                    [1, 1, 0, 1, 1]
                ]),
                'state': np.array([
                    [[0, 0], [0, 0], [0.1, 0.6], [0.5, 0.6], [1, 1]]
                ]),
                'reward': np.array([
                    [0, 1, 0, 1, 0]
                ]),
                'id': 'S5',
                
                'block': 0
                
            },
            {
                'action': np.array([
                    [1, 1, 0, 1, 1, 1]
                ]),
                'state': np.array([
                    [[0, 0], [0, -1], [0.1, 0.2], [0.5, 0.6], [0, 1], [-1, -1]]
                ]),
                'reward': np.array([
                    [0, 0, 0, 0, 0, 1]
                ]),
                'id': 'S5',
                
                'block': 1
                
            }
            
        ]

        }<jupyter_output><empty_output><jupyter_text>and we want to test the model every 10 iterations:<jupyter_code>with LogFile(output_path, 'run.log'):
    OptBEH.optimise(worker, output_path, data, test_data,
                    learning_rate=0.01,
                    global_iters=50,
                    load_model_path=None,
                    test_period=10
                    )<jupyter_output>2021-06-28 13:14:03,107 - DeepRL - DEBUG - version control: (None, None)
2021-06-28 13:14:03,108 - DeepRL - DEBUG - learning rate: 0.01
2021-06-28 13:14:03,109 - DeepRL - DEBUG - global iters: 50
2021-06-28 13:14:03,110 - DeepRL - DEBUG - training data-points: 2
2021-06-28 13:14:03,111 - DeepRL - DEBUG - test data-points: 2
2021-06-28 13:14:03,546 - DeepRL - DEBUG - opt started...
2021-06-28 13:14:03,547 - DeepRL - DEBUG - started testing...
2021-06-28 13:14:03,975 - DeepRL - DEBUG - finished testing.
{'fraction_non_alphanumeric': 0.1409247417609444, 'fraction_numerical': 0.0558288243974422, 'mean_word_length': 1.4364889155182745, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '112352', 'license': 'Apache-2.0', 'n_tokens_mistral': 2902, 'n_tokens_neox': 2512, 'n_words': 793, 'path': 'src/examples/sample.ipynb', 'repo_name': 'kylmcgr/RL-RNN-SURF', 'size': '8132'}