<jupyter_start><jupyter_text>Data analysis with Python Software Carpentries Workshop - 25 Jan 2022  Load in the data<jupyter_code>import numpy as np
my_data = np.loadtxt("data/inflammation-01.csv", delimiter=',')<jupyter_output><empty_output><jupyter_text>Data type and shape<jupyter_code>print(my_data)
print(my_data.dtype)
print(np.shape(my_data))<jupyter_output>[[0. 0. 1. ... 3. 0. 0.]
 [0. 1. 2. ... 1. 0. 1.]
 [0. 1. 1. ... 2. 1. 1.]
 ...
 [0. 1. 1. ... 1. 1. 1.]
 [0. 0. 0. ... 0. 2. 0.]
 [0. 0. 1. ... 1. 1. 0.]]
float64
(60, 40)<jupyter_text>Mean value for a subset of data<jupyter_code>mean_patient = np.mean(my_data, axis = 1)
mean_day = np.mean(my_data, axis = 0)

print(mean_patient.shape)
print(mean_day.shape)<jupyter_output>(60,)
(40,)<jupyter_text>Sorting out references<jupyter_code>first, second = 'Grace', 'Hopper'
third, fourth = second, first
print(third, fourth)<jupyter_output><NAME><jupyter_text>Change in inflammation<jupyter_code>my_diff = np.diff(my_data, axis = 1)
print(my_diff.shape)<jupyter_output>(60, 39)<jupyter_text>Largest inflammation change in each patient<jupyter_code>largest_change = np.amax(np.abs(my_diff), axis = 1)
print(largest_change)
# print('Number of patients: {}'.format(largest_change.shape))<jupyter_output>[12. 14. 11. 13. 11. 13. 10. 12. 10. 10. 10. 12. 13. 10. 11. 10. 12. 13.
  9. 10. 13.  9. 12.  9. 12. 11. 10. 13.  9. 13. 11. 11.  8. 11. 12. 13.
  9. 10. 13. 11. 11. 13. 11. 13. 13. 10.  9. 10. 10.  9.  9. 13. 10.  9.
 10. 11. 13. 10. 10. 12.]<jupyter_text>Visualising data<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

image = plt.imshow(my_data)
fig = plt.figure(figsize=(10.0, 3.0))

axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)

axes1.set_ylabel('average')
axes1.plot(np.mean(my_data, axis=0), drawstyle='steps-mid')

axes2.set_ylabel('max')
axes2.plot(np.max(my_data, axis=0), drawstyle='steps-mid')

min_data = np.min(my_data, axis=0)
axes3.set_ylabel('min')
axes3.set_ylim(np.min(min_data), np.max(min_data) * 1.1)

axes3.plot(np.min(my_data, axis=0), drawstyle='steps-mid')

fig.tight_layout()

plt.show()<jupyter_output><empty_output><jupyter_text>Make your own plot<jupyter_code>my_sd = np.std(my_data, axis=0)
# print(my_sd.shape)
fig2 = plt.figure(figsize=(4.0, 3.0))

axes2a = fig2.add_subplot(1, 1, 1)

axes2a.plot(my_sd)
axes2a.set_xlabel('days')
axes2a.set_ylabel('standard deviation')

plt.show()<jupyter_output><empty_output><jupyter_text>Analysing data from multiple files<jupyter_code>import glob
print(glob.glob('data/inflammation*.csv'))
filenames = sorted(glob.glob('data/inflammation*.csv'))
filenames = filenames[0:3]
for f in filenames:
    print(f)

    my_data = np.loadtxt(fname=f, delimiter=',')

    fig = plt.figure(figsize=(10.0, 3.0))

    axes1 = fig.add_subplot(1, 3, 1)
    axes2 = fig.add_subplot(1, 3, 2)
    axes3 = fig.add_subplot(1, 3, 3)

    axes1.set_ylabel('average')
    axes1.plot(np.mean(my_data, axis=0))

    axes2.set_ylabel('max')
    axes2.plot(np.max(my_data, axis=0))

    axes3.set_ylabel('min')
    axes3.plot(np.min(my_data, axis=0), drawstyle='steps-mid')

    fig.tight_layout()
    plt.show()<jupyter_output>data/inflammation-01.csv<jupyter_text>Plotting differences<jupyter_code>data0 = np.loadtxt(fname=filenames[0], delimiter=',')
data1 = np.loadtxt(fname=filenames[1], delimiter=',')

fig = plt.figure(figsize=(10.0, 3.0))

plt.ylabel('Difference in average')
plt.plot(data0.mean(axis=0) - data1.mean(axis=0))

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Generate composite statistics<jupyter_code>composite_data = np.zeros((60,40))

for f in filenames:
    data = np.loadtxt(fname = f, delimiter=',')
    composite_data += data

composite_data/=len(filenames)

fig = plt.figure(figsize=(10.0, 3.0))

axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)

axes1.set_ylabel('average')
axes1.plot(np.mean(composite_data, axis=0))

axes2.set_ylabel('max')
axes2.plot(np.max(composite_data, axis=0))

axes3.set_ylabel('min')
axes3.plot(np.min(composite_data, axis=0))

fig.tight_layout()

plt.show()<jupyter_output><empty_output><jupyter_text>Filtering data  Checking the maxima<jupyter_code>for f in filenames:
    print(f)

    my_data = np.loadtxt(fname=f, delimiter=',')
    
    data_max = np.max(my_data, axis=0)
    data_min = np.min(my_data, axis=0)
    
    count = 0
    for i in range(0,21):
        if data_max[i] != count:
            print('Suspicious looking maxima on day {}'.format(count))
        # else:
        #     print('Seems OK!')  
        count +=1
        
    if data_min.sum() == 0:
        print('Healthy patient has snuck into the study')<jupyter_output>Seems OK!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Seems OK!
Suspicious looking maxima!
Suspicious looking maxima!
Seems OK!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima!
Suspicious looking maxima[...]
{'fraction_non_alphanumeric': 0.1480388753904894, 'fraction_numerical': 0.05154460256855259, 'mean_word_length': 4.705940594059406, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '405929', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2611, 'n_tokens_neox': 2282, 'n_words': 491, 'path': '2022-01-25-software-carpentries.ipynb', 'repo_name': 'jnywong/python-data-analysis', 'size': '5768'}