<jupyter_start><jupyter_text>Programming with Python Episode 2 - Analyzing Data from Multiple FilesTeaching: 20 min, Exercises: 20 min   Objectives- Use a library function to get a list of filenames that match a wildcard pattern.- How can I do the same operations on many different files?- Write a for loop to process multiple files. We now have almost everything we need to process all our data files. The only thing that’s missing is a library with a rather unpleasant name `glob`, let's import it.```import glob```<jupyter_code>import glob<jupyter_output><empty_output><jupyter_text>The `glob` library contains a function, also called `glob`, that finds files and directories whose names match a pattern. We provide those patterns as strings which may include *wildcards*: - the wildcard character `*` matches zero or more of any character- the wildcard character `?` matches a single character.We can use this to get the names of all the CSV files in the `data` directory:```print(glob.glob('data/inflammation*.csv'))``` the wild card character means you don't have to put the full name in (i.e. if you forget the full name??)<jupyter_code>print(glob.glob('data/inflamm*.csv'))
filelist = glob.glob('data/inflamm*.csv')

print (filelist)<jupyter_output>['data/inflammation-10.csv', 'data/inflammation-03.csv', 'data/inflammation-07.csv', 'data/inflammation-01.csv', 'data/inflammation-05.csv', 'data/inflammation-12.csv', 'data/inflammation-02.csv', 'data/inflammation-08.csv', 'data/inflammation-06.csv', 'data/inflammation-04.csv', 'data/inflammation-11.csv', 'data/inflammation-09.csv']
['data/inflammation-10.csv', 'data/inflammation-03.csv', 'data/inflammation-07.csv', 'data/inflammation-01.csv', 'data/inflammation-05.csv', 'data/inflammation-12.csv', 'data/inflammation-02.csv', 'data/inflammation-08.csv', 'data/inflammation-06.csv', 'data/inflammation-04.csv', 'data/inflammation-11.csv', 'data/inflammation-09.csv']<jupyter_text>`glob.glob`’s result is a list of matching filenames and directory paths (in arbitrary order). This means we can loop over it to do something with each filename in turn. In our case, the “something” we want to do is generate a set of plots for each file in our complete inflammation dataset. If we want to start by analyzing just the first three files in alphabetical order, we can use the built-in `sorted` function to generate a new sorted list from the `glob.glob` output:```import numpyimport matplotlib.pyplotfilenames = sorted(glob.glob('data/inflammation*.csv'))filenames = filenames[0:3]print(filenames)```<jupyter_code>import numpy
import matplotlib.pyplot
import glob

filenames = sorted(glob.glob('data/inflammation*.csv'))
filenames = filenames[0:3]

for filenames in filenames:
   # print(filenames)
    data = numpy.loadtxt(fname=filenames, delimiter=',')
    print (filenames, 'total sum', numpy.sum (data))
    print (filenames, 'sum per day', numpy.sum (data, axis=0))
    print(data)
#print('data after the loop is finished:\n', data)
#numpy.sum(data)<jupyter_output>data/inflammation-01.csv total sum 14757.0
data/inflammation-01.csv sum per day [  0.  27.  67. 105. 146. 189. 228. 233. 314. 331. 357. 354. 501. 464.
 502. 570. 575. 638. 694. 741. 795. 718. 662. 610. 600. 520. 549. 435.
 440. 395. 364. 357. 307. 216. 198. 214. 149.  90.  68.  34.]
[[0. 0. 1. ... 3. 0. 0.]
 [0. 1. 2. ... 1. 0. 1.]
 [0. 1. 1. ... 2. 1. 1.]
 ...
 [0. 1. 1. ... 1. 1. 1.]
 [0. 0. 0. ... 0. 2. 0.]
 [0. 0. 1. ... 1. 1. 0.]]
data/inflammation-02.csv total sum 14378.0
data/inflammation-02.csv sum per day [  0.  30.  56.  95. 158. 160. 215. 248. 287. 338. 373. 390. 438. 454.
 485. 558. 566. 603. 627. 691. 763. 667. 648. 600. 614. 532. 482. 492.
 456. 386. 368. 333. 298. 220. 230. 179. 148.  90.  68.  32.]
[[0. 0. 0. ... 1. 1. 0.]
 [0. 0. 2. ... 0. 2. 1.]
 [0. 1. 2. ... 0. 2. 1.]
 ...
 [0. 0. 2. ... 0. 2. 0.]
 [0. 0. 0. ... 2. 2. 0.]
 [0. 1. 2. ... 3. 2. 0.]]
data/inflammation-03.csv total sum 10091.0
data/inflammation-03.csv sum per day [  0.   0.  35.  92.  24. 144. 194. 216.[...]<jupyter_text>and now we can loop over each filename in turn using the code from an earlier episode to product a set of plots for each.```for f in filenames:    print(f)    data = numpy.loadtxt(fname=f, delimiter=',')    fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))    axes1 = fig.add_subplot(1, 3, 1)    axes2 = fig.add_subplot(1, 3, 2)    axes3 = fig.add_subplot(1, 3, 3)    axes1.set_ylabel('average')    axes1.plot(numpy.mean(data, axis=0))    axes2.set_ylabel('max')    axes2.plot(numpy.max(data, axis=0))    axes3.set_ylabel('min')    axes3.plot(numpy.min(data, axis=0))    fig.tight_layout()    matplotlib.pyplot.show()```<jupyter_code>for f in filenames:
    print(f)

    data = numpy.loadtxt(fname=filenames, delimiter=',')

    fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))

    axes1 = fig.add_subplot(1, 3, 1)
    axes2 = fig.add_subplot(1, 3, 2)
    axes3 = fig.add_subplot(1, 3, 3)

    axes1.set_ylabel('average')
    axes1.plot(numpy.mean(data, axis=0))

    axes2.set_ylabel('max')
    axes2.plot(numpy.max(data, axis=0))

    axes3.set_ylabel('min')
    axes3.plot(numpy.min(data, axis=0))

    fig.tight_layout()
    matplotlib.pyplot.show()<jupyter_output>d<jupyter_text>Sure enough, the maxima of the first two data sets show exactly the same ramp as the first, and their minima show the same staircase structure; a different situation has been revealed in the third dataset, where the maxima are a bit less regular, but the minima are consistently zero - probably indicating an issue in our data.  Excercises  Plotting differences between data filesPlot the difference between the average of the first dataset and the average of the second dataset, i.e., the difference between the leftmost plot of the first two figures.<jupyter_code>import glob
import numpy as np
import matplotlib.pyplot as plt

# Grab all the filenames and sort them 
filenames = sorted(glob.glob('data/inflammation*.csv'))

print(filenames)

# load the first 2 data files
data0 = np.loadtxt(fname=filenames[0], delimiter=',')
data1 = np.loadtxt(fname=filenames[1], delimiter=',')

# now compute the means - np.mean()


data0_mean = np.mean(data0,axis = 0)
data1_mean = np.mean(data1,axis = 0)

print(data0_mean)
print(data1_mean)

data_mean_diff = data1_mean - data0_mean

print(data_mean_diff)


# create a figure
#either this way with subplots or below with plt.plot
fig = plt.figure(figsize=(10.0, 3.0))
subplot1 = fig.add_subplot(1,1,1,)
subplot1.plot(data_mean_diff)

# and plot the difference between the meansof the first and second data set - plt.plt()
#plt.plot (data_mean_diff)

# add a label to Y axis
plt.ylabel('Difference in average')


# and show it
plt.show()<jupyter_output>['data/inflammation-01.csv', 'data/inflammation-02.csv', 'data/inflammation-03.csv', 'data/inflammation-04.csv', 'data/inflammation-05.csv', 'data/inflammation-06.csv', 'data/inflammation-07.csv', 'data/inflammation-08.csv', 'data/inflammation-09.csv', 'data/inflammation-10.csv', 'data/inflammation-11.csv', 'data/inflammation-12.csv']
[ 0.          0.45        1.11666667  1.75        2.43333333  3.15
  3.8         3.88333333  5.23333333  5.51666667  5.95        5.9
  8.35        7.73333333  8.36666667  9.5         9.58333333 10.63333333
 11.56666667 12.35       13.25       11.96666667 11.03333333 10.16666667
 10.          8.66666667  9.15        7.25        7.33333333  6.58333333
  6.06666667  5.95        5.11666667  3.6         3.3         3.56666667
  2.48333333  1.5         1.13333333  0.56666667]
[ 0.          0.5         0.93333333  1.58333333  2.63333333  2.66666667
  3.58333333  4.13333333  4.78333333  5.63333333  6.21666667  6.5
  7.3         7.56666667  8.08333333  9.3        [...]<jupyter_text>Generate Composite StatisticsUse each of the files once, to generate a dataset containing values averaged over all patients:<jupyter_code>import glob
import numpy as np
import matplotlib.pyplot as plt

filenames = glob.glob('data/inflammation*.csv')
composite_data = np.zeros((60,40))

for f in filenames:
    #
    # read each file (np.loadtxt) and add it to the composite_data vector
    #
    
    
# and rescale it
composite_data/=len(filenames)

print(composite_data)<jupyter_output><empty_output><jupyter_text>and now plot the stats for the composite date it using `matplotlib`<jupyter_code>fig = plt.figure(figsize=(10.0, 3.0))

axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)

axes1.set_ylabel('average')
axes1.plot(np.mean(composite_data, axis=0))

axes2.set_ylabel('max')
axes2.plot(np.max(composite_data, axis=0))

axes3.set_ylabel('min')
axes3.plot(np.min(composite_data, axis=0))

fig.tight_layout()

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13714742491720908, 'fraction_numerical': 0.10174717368961973, 'mean_word_length': 3.5401762571280457, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 13, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '11111', 'license': 'MIT', 'n_tokens_mistral': 3825, 'n_tokens_neox': 3150, 'n_words': 930, 'path': 'lessons/python/ep4-multiple-files.ipynb', 'repo_name': 'StephHowarth/2019-07-24-intro-to-python-workshop', 'size': '8757'}