<jupyter_start><jupyter_text>Author[<NAME>](https://koenaerts.ca/) @ [Mobia Technology Innovations](https://mobia.io)[myOpenHealth](https://mobia.io/healthcare/)  IntroThis notebook reads all the CSV files that were generated by notebook "02_import_mitdb_data" and combines them all in 3 new CSV files that can be used for training your model. More specifically the following is done:* Read all CSV files generated by the "02_import_mitdb_data" notebook.* Shuffle all records randomly.* Save 60% of the records in a training CSV file.* Save 20% of the records in a validation CSV file.* Save 20% of the records in a test CSV file.  InitializeImport dependencies.<jupyter_code>from glob import glob
import numpy as np<jupyter_output><empty_output><jupyter_text>Load dataRead all the CSV files into memory.<jupyter_code>alldata = np.empty(shape=[0, 188])
print(alldata.shape)
paths = glob('data_ecg/*.csv')
for path in paths:
    print('Loading ', path)
    csvrows = np.loadtxt(path, delimiter=',')
    alldata = np.append(alldata, csvrows, axis=0)<jupyter_output><empty_output><jupyter_text>Shuffle and separate the dataAll records are shuffled randomly, then split into training, validation, and testing groups.<jupyter_code># Randomly mix rows
np.random.shuffle(alldata)
totrows = len(alldata)
trainrows = int((totrows * 3 / 5) + 0.5) # 60%
testrows = int((totrows * 1 / 5) + 0.5) # 20%
validaterows = totrows - trainrows - testrows # 20%
mark1 = trainrows
mark2 = mark1 + testrows<jupyter_output><empty_output><jupyter_text>Save dataData is saved in 3 separate CSV files: training, validation, testing.<jupyter_code>with open('train.csv', "wb") as fin:
    np.savetxt(fin, alldata[:mark1], delimiter=",", fmt='%f')
with open('test.csv', "wb") as fin:
    np.savetxt(fin, alldata[mark1:mark2], delimiter=",", fmt='%f')
with open('validate.csv', "wb") as fin:
    np.savetxt(fin, alldata[mark2:], delimiter=",", fmt='%f')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12807377049180327, 'fraction_numerical': 0.0194672131147541, 'mean_word_length': 5.73448275862069, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '161132', 'license': 'MIT', 'n_tokens_mistral': 685, 'n_tokens_neox': 646, 'n_words': 198, 'path': '04_prep_imported_data.ipynb', 'repo_name': 'JiaminXu1019/ECG_ML', 'size': '1956'}