<jupyter_start><jupyter_text>Iteration: for-Loops, lists, simple file input Exercises 4: Using for-loops and lists in PythonIt is often necessary to deal with data consisting of collections of valuese.g.  might compute trajectory of missile as its position (x;y coords) or at a series of time points. This gives a sequence of values.In Python, such data is stored as a **list**. A *list* is a single variable which can hold many values. A single Python list may mix together values of different types, e.g. strings and integers such as **["some string", 3.55, "this", 2]**, and it is inherently ordered. The *empty list* (i.e. which contains no items) is **[ ]**. List items can be accessed by their *position* in the list, with index values from *0* to *(n - 1)* for a list of length *n*. As you can see in the cell below, accessing a non-existent position results in an error.<jupyter_code>x = ['this', 55, 'that']
print (x[0])
print (x[1])
print (x[3])<jupyter_output><empty_output><jupyter_text>Time series analysisIn this notebook, we will practice using Python **for**-loops and lists. As a practical exercise, we will do some processing of time-series data. Along with the iterative numerical techniques we looked at in the previous notebook, time series analysis has many applications in Science and Engineering. Time series data are measurements that change with time. An example of a time series is a piece of music, which is stored in an MP3 player as a sequence of numbers representing the sound intensity at evenly spaced times. In Science and Engineering applications, we often want to obtain information from a time series by filtering or other processing steps. In this notebook, we will write a Python program to recover a signal from a time series that has been corrupted with noise.1) Python Lists and for-loopsAs the example below shows, a list can be changed, with values at existing positions being overwritten, or with a list being made shorter or longer, as in the example (where **append** is used to 'grow' a list).<jupyter_code># Run this cell to see how to modify the number of elements in the list or how to replace one of them

x = ['aa', 'bb', 'cc']
x[1] = 33
print (x)

x.append('dd')
print (x)<jupyter_output><empty_output><jupyter_text>We can use + to compute the concatenation of two lists.<jupyter_code>x = ['the', 'cat', 'sat']
y = ['on', 'the', 'mat']
z = x + y
print (z)<jupyter_output><empty_output><jupyter_text>We can also take a slice of a list, using two indices: **[i:j]** A slice starts with item at index *i*, plus items up to (but not including) *j*.<jupyter_code>z = ['the', 'cat', 'sat', 'on', 'the', 'mat']
print (z[3:10])
print (z[8:10])<jupyter_output><empty_output><jupyter_text>There is an alternative sequence type in Python, the **tuple**. This is written with round brackets    ('this', 55)They are like Python list, ordered and allow access by index **but** they cannot be changed, i.e. you cannot assign new value to a position in an existing tuple and you cannot append to an existing tuple. They are more memory efficient (though that is not a big concern here). for-loops Recall the two types of loops, *conditional* ("while") and *counting* ("for"), introduced in the previous notebook to allow for repetition control structures. The **for loop** construct widely used to implement counting loops. In most languages, for loop has an explicit loop variable, whose value counts in fixed steps from an initial value to a final value. When the final value is reached, the loop stops. For example, var *i*, counting 0, 1, . . . , 9. A loop variable may be used within the loop e.g. as an index, to access successive elements of an array, but there is no explicit counting in Python. Often, **for**-loops are used together with a list in the pattern below. This loop will repeat as many times as there are items in **LIST**, with **VAR** being assigned each member of **LIST** in turn as its value. This kind of *iteration* is fine for many purposes. However, to *change* the values in the list, we must access the positions by *index*, for which we use the **range** function, to generate a list of the index positions.                  for VAR in LIST:            CODE-BLOCK                    If we look at the informal *while*-loop supermarket shopping pseudo code from the previous notebook    1. Get a trolley    2. While there are items on shopping list        2.1 Read first item on shopping list        2.2 Get that item from shelf        2.3 Put item in trolley        2.4 Cross item off shopping list    3. Pay at checkout    we can rewrite this with the corresponding *for* loop pseudo code    1. Get a trolley    2. For (each) item on shopping list        2.1 Get item from shelf        2.2 Put item in trolley    3. Pay at checkout<jupyter_code># Run this cell and experiment, so you understand what's going on

values = [875, 23, 451]
for val in values:
    print ('-->', val)

print ("\n")<jupyter_output><empty_output><jupyter_text>We can iterate over other types, such as *tuples* or *strings*:<jupyter_code>for char in "Yes":
    print (char)<jupyter_output><empty_output><jupyter_text>More generally, various types demonstrate iterable behaviour, and can appear in a *for* loop.While this way of accessing accessing items in a list is sufficient for many tasks, such as scanning list to search for a particular value or computing sum of values in list of numbers:<jupyter_code># total += val means same as total = total + val

values = [3, 12, 9]
total = 0
for val in values:
    total += val
print ('TOTAL:', total)<jupyter_output><empty_output><jupyter_text>Sometimes, it will be necessary to address list items by index, e.g. if have a list of numbers and want to increment each by 1, i.e. so can change list values. For this, we havethe *range* function and use it to generate list of integers for index positions.<jupyter_code># Python 3.x: range() function got its own type

# range has to be converted to a list
print (list(range(3)))        # single arg = 'end' value
print (list(range(2,6)))      # `start' value also given
print (list(range(1,10,3)))   # 3rd arg specifies 'step' size

# Incrementing values in a list by 1
vals = [8, 12, 10, 34]
print (len(vals))
print (list(range(len(vals))))
for i in range(len(vals)):
    vals[i] = vals[i] + 1
print (vals)

# What does this do?
for i in range(len(vals)):
    vals[i] = vals[i] * 2
print (vals)<jupyter_output><empty_output><jupyter_text>Loop control commands **break** and **continue** also work with *for* loops and modify the normal flow of a loop. A *break* statement in a loop immediately terminates the current iteration and ends the loop overall. A *continue* statement in a loop immediately terminatesthe current iteration and starts the next iteration - i.e. in a for loop, this causes the loop to move on to next item of iteration. For example, we could use a *for* loop to scan for sought item, and use break to end scan if it is found. What does the following do and why is 14 not output?<jupyter_code>nums = [1,3,8,7,15,14]

for n in nums:
    if n % 7 == 0:
        print ('found:', n)
        break<jupyter_output><empty_output><jupyter_text>2) Introductory exercisesI) Sum of a list of numbers/Triangular numberDefine a function **sum_list**, which takes one argument, a list of numbers, and *which uses* a **for**-*loop* to compute (and return) the sum of these numbers, e.g. so that **sum_list([3,4,5])** should return 12.<jupyter_code># Insert here your definition for sum_list function<jupyter_output><empty_output><jupyter_text>Define a function *which uses* a **for**-*loop* to compute (and return) *triangular numbers*. For a positive integer *n*, this is the sum of values from n down to one, i.e. *n + (n - 1) + ... + 2 + 1*.)<jupyter_code># Insert here your code for triangular numbers<jupyter_output><empty_output><jupyter_text>II) List of squares / triangularsDefine a function **square_list**, which takes a list of numbers as its argument, and which computes (and returns) a list like the first, but with the values squared, e.g. so that **square_list([3,4,5])** returns **[9,16,25]**. **NOTE**: your definition *should not change the value of the original list*, i.e. if we call **square_list(x)** when **x = [3,4,5]**, then **x** should still be **[3,4,5]** after the call (check this!). We can copy a list using the **list** function, e.g. **y = list(x)** creates a copy of list **x**.<jupyter_code># Insert here your definition for square_list function<jupyter_output><empty_output><jupyter_text>Next define a function **triangular_list**, which is similar to **square_list**, except that the numbers in the new list are the triangular numbers of the original values. Your definition should compute the triangular numbers by calling the function you have defined in the cell above.<jupyter_code># Insert here your definition for triangular_list function<jupyter_output><empty_output><jupyter_text>3) Simple graph plotting using Pylab**Pylab** is a library of code (a *module*) which provides lots of useful stuff, including *graph plotting*. The basic plotting function takes two arguments: a list of the *x*-coords of the points to be plotted, and a list of the corresponding *y*-coords. To plot a graph with *(x, y)* points (0, 1.2), (1, 2.2) and (2, 1.8), for example, we form a list of the *x* values [0, 1, 2] and *y* values [1.2, 2.2, 1.8].We could then plot this graph with the code below. The **plot** command takes the lists of *x* and *y* coord values as stated above. The **show** command causes the graph to be actually drawn and displayed.<jupyter_code>%matplotlib inline

# Run this cell to see how Pylab plots graphs
import pylab as pyl
Xs = [0, 1, 2]
Ys = [1.2, 2.2, 1.8]
pyl.plot(Xs,Ys)
pyl.show()<jupyter_output><empty_output><jupyter_text>By default, we get a continuous line in a random colour, but **plot** takes an *optional* third value: a string to control the line format. In **plot(Xs,Ys,'ro-')**, the format **'ro-'** gives a red **('r')** continuous line **('-')** with circles **('o')** at the data points, but we could instead have blue **('b')** or green **('g')**, asterisks **('\*')** or crosses **('x')**, or a line that is dashed **('--')**, dot-dashed **('-.')** or absent. Other functions **(xlabel,ylabel)** assign labels to the *x/y* axes (e.g. **xlabel('time')**), give the figure a title **(title)**, or name a file in which the figure is saved **(savefig)** for later use.We can have several lines on a graph by having several calls to **plot** before calling **show**. A call to **figure** between **plot** calls causes a new figure to be started (so multiple figures are displayed: see first example below). The **subplot** function arranges multiple graphs within the same figure (see second example below). (See: **http://matplotlib.org/users/pyplot_tutorial.html**)<jupyter_code># Multiple figures are displayed in this case (first two figures below)
print ("Multiple figures")
Xs = [0, 1, 2]
Ys1 = [1.2, 2.2, 1.8]
Ys2 = [1.5, 2.0, 2.6]
pyl.plot(Xs,Ys1)
pyl.figure()
pyl.plot(Xs,Ys2)
pyl.show()
 
print ("\n", "Subplot function")    
# One figure containing multiple graphs displayed (last figure)    
Xs = [0, 1, 2]
Ys1 = [1.2, 2.2, 1.8]
Ys2 = [1.5, 2.0, 2.6]
pyl.subplot(211)
pyl.plot(Xs,Ys1)
pyl.subplot(212)
pyl.plot(Xs,Ys2)
pyl.show()<jupyter_output><empty_output><jupyter_text>I) Plotting functions using for loopsNext, plot a graph of the function *$f(x) = x^2 + 20$*, for integer values of *$x$* in the range 0-20. We can build the lists of *$x$* and *$y$* values with a **for** loop, with a call such as **"for i in range(N):"** to produce a suitable number of iterations. Build the list of *$y$* values by starting with an empty list **(Ys = [ ])** and appending values to it (e.g. **"Ys.append(V)"**). Having created the lists of *$x$* and *$y$* values, plot the graph as shown above. As a further exercise (which you might leave until later), plot the function *$g(x) = (x/2)^3 - 100$*, for the same range of *$x$* values, to appear alongside the first line on the same graph.<jupyter_code># Insert here your code for plotting function f
# Insert your code for plotting function g<jupyter_output><empty_output><jupyter_text>4) Task: Recovering signal from noiseOur task is to analyse some data that represents a noisy signal, and to extract the original 'pure signal' from it, to which noise has been added. The data is stored in the file **noisy_signal.txt**. This noisy signal was created as follows. The initial *pure* signal is a 5 Hz sine wave. To this, two different forms of noise were added. Firstly, some 50 Hz sine wave: the frequency of the mains electrical supply in the UK. 50 Hz ''mains hum" interference is always a problem when we try to record small amplitude electrical signals (see http://en.wikipedia.org/wiki/Mains_hum). Secondly, some entirely random noise was also added to the signal. (The data files for these three separate components, **pure_signal.txt**, **mains_noise.txt** and **random_noise.txt**, have also been provided, in case you want to display them.)I) Loading the data from fileOur first task is to load the noisy signal data from the file **noisy_signal.txt**. If you inspect the file, you will see that each line has two numbers, of which the first is a time point, and the second is an intensity value. The time points run at *millisecond* intervals across a 3 second period.In Python, the command     open()    opens a connection to a named file, for reading. It creates and returns a file object, which stores connection information and assign it to a variable, so we can use it later (any variable name okay). For example    in_fs = open('foo.txt')We can then read lines from file using a **for** loop:    in_fs = open('foo.txt')    for line in in_fs:        print (line, end = " ")with each iteration of the *for*-loop, loop variable (line) is assigned the next line of text from the file. This is a clean and readable way to code reading from a file.Consider the following, which reads lines from a file and prints them out with a *line number* added to the start of each line.<jupyter_code>in_fs = open('files/rhyme.txt')
linenum = 0
for line in in_fs:
    linenum += 1
    print (linenum, line)<jupyter_output><empty_output><jupyter_text>Thus to read from **noisy_signal.txt**, we first use a command such as:                 in_fs = open('noisy_signal.txt')            This call to the **open** command creates a *filestream object*, for reading from the named file, which we assign to the variable **in_fs** (which is just an ordinary variable). We can read the lines of text from the file by using a simple **for** loop to *iterate* over the filestream:                 for line in in_fs:        do_something(line)                    Here **line** is just the loop variable (its name has no special significance), and **in_fs** is the 'filestream object' created above. As the loop proceeds, the loop var **line** is assigned each successive line of text from the file in turn. It ends when there are no more lines to read.For example, the code below will read and print the first 10 lines of the file, and then exit due to the **break** command (without which, the loop would print all 3000 lines before finishing).<jupyter_code># Run this cell to see how reading data from a file works
# When reading from files, the location is txt/notebookx/name_of_the_file, where x is the number of the notebook

in_fs = open('txt/notebook4/noisy_signal.txt')
c = 0
for line in in_fs:
    print (line)
    c = c + 1
    if c > 9: 
        break<jupyter_output><empty_output><jupyter_text>We can use this approach to read the data into the program, but what it delivers are *lines of text*, i.e. *strings*, rather than the numbers we want. As shown below, we can get the values by first using **split** (which splits a string into a list of substrings at positions where spaces occur), and then **float**, which converts a string to a floating point number (provided it *looks like* one). Using this approach, load the noisy signal data from the file. Store the loaded values into lists suitable for plotting (i.e. one for the time points, and another for the intensities), and then plot the signal. If you have time, you might also load and plot the data for the original pure signal, mains noise and random noise, for comparison.<jupyter_code># Run this cell to see how the two functions mentioned in the last paragraph work

line = '0.33 0.5'
vals = line.split()
print (vals)

t = float(vals[0])
v = float(vals[1])
print (t + v)<jupyter_output><empty_output><jupyter_text>II) Implementing a moving average filterThe simplest way to remove high frequency noise from a digitised signal is to smooth it by calculating the moving average (see http://en.wikipedia.org/wiki/Moving_average). In this part of the notebook, we are going to implement a moving average filter.The moving average is calculated by looping through the data, and at each point calculating the average value of the signal within *a window* of specified length. Using 'a window' here means that, for a position *i*, with a window size *W*, we include all positions from *i - W/2* through to *i + W/2*. Python allows a simple means to access such a sequence of values from the list, by using *slicing*. In particular, we can specify *two* values, as in **x[i:j]**, as a way to access a *subsequence* of a list, known as a *slice*, which is itself returned as a list, as the following illustrates:<jupyter_code># Slicing function for lists example

x = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
x[2:5]<jupyter_output><empty_output><jupyter_text>For values that are sensibly in range, the slice **x[i:i+j]** returns the item **x[i]** plus further items to its right, so as to return a total of j items. Try this yourself in the example above (and also investigate what happens if the first index is less than 0, or the second greater than the list length). The function **sum** computes the sum of a list of numbers, e.g. **sum([5,7,9])** returns 21. Using slicing and **sum**, it is easy to compute the sum of values from a window of the signal.In the cell below write code to compute a moving average for the signal data you have loaded. Start by creating a list of zero values in to which the moving average values can be written (which must of the same length as the list of time points for plotting to be possible.) Compute the moving average values, using a window size of 20. Plot the filtered signal against time, to see if/how well the original 5 Hz 'pure signal' is recovered.<jupyter_code># Insert the code that computes the moving average for the loaded files here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09607382908353782, 'fraction_numerical': 0.013656246665955404, 'mean_word_length': 4.321317059324439, 'pattern_counts': {'":': 1, '<': 85, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '58744', 'license': 'MIT', 'n_tokens_mistral': 5528, 'n_tokens_neox': 5269, 'n_words': 2849, 'path': '_Part 1 Notebook 4 - Iteration - for-Loops, lists, simple file input.ipynb', 'repo_name': 'SheffieldDataAnalytics/Azure-DataAnalyticsPython', 'size': '18746'}