<jupyter_start><jupyter_text>Week 01 Assignment epileptic seizureWelcome to week one of this course programming 1. You will learn about data wrangling with `pandas` and `numpy` and you will learn to visualize with `bokeh`. Concretely, you will preprocess the brain EEG data in an appropiate format in order to conduct statistical and visual analysis. Learning outcomes:- load a tabular dataset and and rename the column names- split a text column into two columns in Pandas DataFrame- inspect the dataset for quality and metadata information- reshape the dataset into a format suitable for visual and statistical analysis- perform visual analysis- perform statistical analysisThe assignment consists of 6 parts:- [part 1: load the data](0)     - [Exercise 1.1](ex-11)- [part 2: prepare for inspection](1)     - [Exercise 2.1](ex-21)- [part 3: inspect the data](2)     - [Exercise 3.1](ex-31)- [part 4: reshape the data](3)     - [Exercise 4.1](ex-41)     - [Exercise 4.3](ex-43)- [part 5: visualize the data](4)     - [Exercise 5.1](ex-51)- [part 6: statistical analysis](5)     - [Exercise 6.1](ex-61)Part 1 and 5 are mandatory, part 6 is optional (bonus)To pass the assingnment you need to a score of 60%.   The DataYou will use a dataset called `eeg_data.csv`. This is a preprocessed dataset. The original dataset from the reference consists of 5 different folders, each with 100 files, with each file representing a single subject/person. Each file is a recording of brain activity for around 23 seconds. Each data point is the value of the EEG recording at a different point in time. There are in total 500 patients with each has 4097 datapoints.The originial data was shuffled into 23 chunks per patient, each chunk contains 178 data points. The first column is an identifier with the chunk number (X1 ..X23) and the Patient number (for example V1.791)The EEG recordings are X1, X2, ..., X178. The last column represents the label y {1,2,3,4,5}5 - eyes open, means when they were recording the EEG signal of the brain the patient had their eyes open4 - eyes closed, means when they were recording the EEG signal the patient had their eyes closed3 - Yes they identify where the region of the tumor was in the brain and recording the EEG activity from the healthy brain area2 - They recorder the EEG from the area where the tumor was located1 - Recording of seizure activityAll subjects falling in classes 2, 3, 4, and 5 are subjects who did not have epileptic seizure. Only subjects in class 1 have epileptic seizure. Source: <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME> (2001) Indications of nonlinear deterministic and finite dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state, Phys. Rev. E, 64, 061907 Run the cell below to load in the packages.<jupyter_code>import pandas as pd
import numpy as np
from bokeh.io import output_notebook
output_notebook()<jupyter_output><empty_output><jupyter_text>Part 1: Load the dataInstructions: Load the data and rename the unnamed column containing the identifier with the chunck number and the patientID into the column `ID`        Hintspandas.read_csv() method can be used to read a csv file    pandas.DataFrame.head() method is often used to inspect the dataframe    pandas.DataFrame.rename() method can be used to rename columns  1.1 Code your solution<jupyter_code># change this code
df = pd.read_csv('eeg_data.csv')
df = df.rename(columns = {'Unnamed: 0':'ID' })<jupyter_output><empty_output><jupyter_text>---  1.2 Test your solution<jupyter_code>#do not modify this cell, run cell to test your code
try:
    df.info()
except:
    print("df not properly loaded")<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 11500 entries, 0 to 11499
Columns: 180 entries, ID to y
dtypes: int64(179), object(1)
memory usage: 15.8+ MB<jupyter_text>Expected outcome:<jupyter_code><class 'pandas.core.frame.DataFrame'>
    RangeIndex: 11500 entries, 0 to 11499
    Columns: 180 entries, ID to y
    dtypes: int64(179), object(1)
    memory usage: 15.8+ MB<jupyter_output><empty_output><jupyter_text>The dataset should have 11500 entries (500patients x 23 rows per patient) and 180 columns (178 recordings, an identifier and the class variable y) ---  Part 2: Prepare for inspection Although the number of records suggest that there should be 500 patients with 23 record each we would like to check this. Furthermore we do not know whether the y label is the same for each patient. We also do not know whether there are missing values. In order to answer the questions above it is more convient to split up the identifier containing a combination of chunck number and patient ID into two separate columns. After that we can do some inspections.Instructions: - change the `ID` column into two separate columns `idx` and `patient_id`- change the column `idx` into a number (you need to stripe the X first)- remove the `ID` column from the DataFrame        Hintspandas.DataFrame.str.split() method can be used to split    pandas.DataFrame.astype(int) method can be used to typecast to a number    pandas.DataFrame.drop() method can be used to drop a column 2.1 Code your solution<jupyter_code># YOUR CODE HERE
df[['idx','patient_id']] = df["ID"].str.split("." ,1 , expand=True)
df['idx'] = df['idx'].str.strip('X')
df['idx'] = df['idx'].astype('int64')
df = df.drop(columns=['ID'])

# print(df.head(2))<jupyter_output><empty_output><jupyter_text>---  2.2 Test your solution<jupyter_code>#do not modify this cell, run cell to test your code
try:
    print(df.dtypes)
except:
    print("df not properly loaded")<jupyter_output>X1             int64
X2             int64
X3             int64
X4             int64
X5             int64
               ...  
X177           int64
X178           int64
y              int64
idx            int64
patient_id    object
Length: 181, dtype: object<jupyter_text>expected outcome:<jupyter_code>X1             int64
X2             int64
X3             int64
X4             int64
X5             int64
               ...  
X177           int64
X178           int64
y              int64
idx            int64
patient_id    object
Length: 181, dtype: object<jupyter_output><empty_output><jupyter_text>---  Part 3: Inspect the dataNow that we prepared the data we are going to inspect the data to get more familiar with the data. You are required to do the following- inspect how many patients are in the dataset (should be 500 different)- inspect if patient V1.924 has 23 records- check if a patient has more than 1 Y value- check for how many patients are there per level- check if there are missing valuesThe code for the inspection of the chunk numbers is given.        Hintspandas.DataFrame.groupby()    (df[df['columnname'] == value])    if you sum the expression ['y'].min() == ['y'].max() when grouped by patient it should return 500  3.1 Code your solution<jupyter_code># should have 23 chunck numbers
y = len(pd.DataFrame(df.groupby('idx')))
x = len(pd.DataFrame(df.groupby('patient_id')))

print(x, y)
#YOUR CODE HERE to 
# print(df.head(4))
### - inspect if patient V1.924 has 23 records
# df[df['patient_id'] == 'V1.924'].sort_values(['idx'])

### - check if a patient has more than 1 Y value
# pd.DataFrame(df.groupby(['patient_id', 'y']))

### - check for how many patients are there per level
# df['y'].value_counts() / 23

### - check if there are missing values
# df.isnull().values.any() #or
# df.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>---  Part 4: Reshape the dataWe would like to plot the patient data. Therefore we would need to reshape the data. For plotting purpose the data of the 23 chunks need to be ordered by patient and chunk number first. And then we need to get the X1..X178 data (a matrix with shape 23, 178) to be reshaped to one array of 178 * 23 = 4094 datapoints. (shape 1, 4094). The 4094 datapoint array is the EEG signal data of a particular patient to be plotted on the y-axis. On the x-axis we simply can create a sequence from 1 to 4094. Instructions:- set the index to `patient_id` and `idx`- sort the index- create a timesequence from 1 to 4094- finish the code to select a patient- finish the code to get the data of a patient and reshape it into an one dimensional array        Hintsdf.set_index    df.sort_index    use np.arange to create the sequence and use tolist to transfer it to a list    check pandas.DataFrame.iloc and slicing options for selecting a subset of the data    with numpy.reshape you can reshape a matrix 4.1 Code your solution<jupyter_code>#YOUR CODE HERE
cols = list(df.columns.values)
df = df[ [cols[-1]] + [cols[-2]] + cols[0:179]]
df  = df.sort_values(['patient_id', 'idx'])
df = df.reset_index()
df = df.drop(columns=['index'])
df = df.set_index(['patient_id', 'idx']).sort_index()

timeseq = np.arange(1, 4095) #change this, should return a list with values 1..4094
timeseq = timeseq.tolist()<jupyter_output><empty_output><jupyter_text>---  4.2 Test your solution<jupyter_code>#check outcome df and timeseq, do not modify this cell, run to test
print(df.head(2))
print(f'the length of timeseq [{np.array(timeseq).min()}..{np.array(timeseq).max()}] is {len(timeseq)}')<jupyter_output>X1  X2  X3  X4  X5  X6  X7  X8  X9  X10  ...  X170  X171  \
patient_id idx                                           ...               
V1         1    12  22  35  45  69  74  79  78  66   43  ...   -18   -32   
           2   -41 -50 -53 -49 -35 -28 -15  -2  14   18  ...    34    22   

                X172  X173  X174  X175  X176  X177  X178  y  
patient_id idx                                               
V1         1     -47   -53   -48   -40   -17   -23   -32  5  
           2       4   -18   -31   -27   -26   -21   -30  5  

[2 rows x 179 columns]
the length of timeseq [1..4094] is 4094<jupyter_text>expected outcome:<jupyter_code>X1  X2  X3  X4  X5  X6  X7  X8  X9  X10  ...  X170  X171  \
patient_id idx                                           ...               
V1         1    12  22  35  45  69  74  79  78  66   43  ...   -18   -32   
           2   -41 -50 -53 -49 -35 -28 -15  -2  14   18  ...    34    22   

                X172  X173  X174  X175  X176  X177  X178  y  
patient_id idx                                               
V1         1     -47   -53   -48   -40   -17   -23   -32  5  
           2       4   -18   -31   -27   -26   -21   -30  5  

[2 rows x 179 columns]
the length of timeseq [1..4094] is 4094<jupyter_output><empty_output><jupyter_text>---Now that we reorganized the patients we could use the new dataframe to select a patient, load the values in an numpy matrix and reshape that to a one dimensional array. To do such complete the code below  4.3 Code your solution<jupyter_code>#get the data of one patient
def get_patient(df, patient):
    ##########################
    dfpat = df[(df.index.get_level_values('patient_id') == patient)] # change this line to select only the records from the particular patient
    #########################
    return dfpat.sort_index()


#fetch the values part and put in numpy array
def get_values(df):
    ###############################
    X = np.array(df.iloc[:,0:178]) # change this line to load value columns and the rows of the patient
    ###############################
    n = int(len(df) / 23)
    X = X.reshape(n, 4094)
    return X<jupyter_output><empty_output><jupyter_text>--- 4.4 Test your solution<jupyter_code>#check outcome of array, do not modify this cell, run to test
test_patient = get_values(get_patient(df,'V1.334'))[0]
print(f'the length of patient V1.334 array {test_patient} = {len(test_patient)}')<jupyter_output>the length of patient V1.334 array [ -18  -55 -126 ... -210 -222 -224] = 4094<jupyter_text>expected outcome<jupyter_code>the length of patient V1.334 array [ -18  -55 -126 ... -210 -222 -224] = 4094<jupyter_output><empty_output><jupyter_text>---  Part 5: Plot the dataIn this part we will first select some patients per class level in the function `get_data(level)`. Then we are going to plot two plots per level using bokeh. Your job is to finish the code of the bokeh plot. For each patient eeg signal should be plotted on the y-axis given the timesequence on the x-axis<jupyter_code>#code to select some patients per level, do not modify this cell, run this cell
#it is allowed though to add some print statements in order to understand what is happening
def get_data(level):
    df_level = df[df['y']==level]
    s = df_level.index.unique().tolist()
    patients = np.array(sorted(set([x[0] for x in s]))).tolist()
    return list(patients)[3:5]

p1 = get_data(1)
p2 = get_data(2)
p3 = get_data(3)
p4 = get_data(4)
p5 = get_data(5)<jupyter_output><empty_output><jupyter_text>5.1 Code your solution<jupyter_code>from bokeh.plotting import figure, show
from bokeh.layouts import gridplot
from bokeh.models import Range1d

def make_plot(title, y, pat_values, timeseq, color):
    p = figure(title=title, background_fill_color="#fafafa")
    p.y_range=Range1d(-1500, 1500)
    
    ###########################################
    p.line(timeseq,pat_values, color='blue') # CHANGE THIS LINE
    ###########################################

    p.xaxis.axis_label = 'time'
    p.yaxis.axis_label = 'signal'
    p.grid.grid_line_color="white"
    return p


def make_plot_array(g, pa, y):
    d = {1:'seizure', 2:'tumor location', 3:'non-tumor location', 4:'eyes closed', 5:'eyes open'}
    for i in pa:
        v = get_values(get_patient(df,i))[0]
        p = make_plot(title = f'brain EEG for patient {i} {d[y]}', y=y, pat_values=v, timeseq=timeseq, color='black')
        g.append(p)
    return g

g = []
g = make_plot_array(g, p1, 1)
g = make_plot_array(g, p2, 2)
g = make_plot_array(g, p3, 3)
g = make_plot_array(g, p4, 4)
g = make_plot_array(g, p5, 5)
    
show(gridplot(g, ncols=2, plot_width=450, plot_height=300))<jupyter_output>BokehDeprecationWarning: plot_width and plot_height was deprecated in Bokeh 2.4.0 and will be removed, use width or height instead.<jupyter_text>---  Part 6: Statistical analysis (bonus points) From the graphical analysis is seems that there is significant difference between level 1 and the other levels. We can use `pandas.DataFrame.melt` to prepare the data for statistical analysis. Your job is to give statistical evidence of a significant difference see also [gitbook example](https://fennaf.gitbook.io/bfvm19prog1/data-wrangling/reshape-with-melt)  6.1 Code your solution<jupyter_code>#YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10380670475666001, 'fraction_numerical': 0.05100846699249673, 'mean_word_length': 3.3084223013048635, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 72, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '442724', 'license': 'MIT', 'n_tokens_mistral': 4957, 'n_tokens_neox': 4441, 'n_words': 1830, 'path': 'Programming_1/Assignment_week_01_SM.ipynb', 'repo_name': 'StevetheGreek97/Master_DSLS', 'size': '14544'}