<jupyter_start><jupyter_text>Please download the new class notes. Step 1 : Navigate to the directory where your files are stored.  Open a terminal. Using `cd`, navigate to *inside* the ILAS_Python_for_everyone folder on your computer.  Step 3 : Update the course notes by downloading the changesIn the terminal type:>`git add -Agit commit -m "commit"git fetch upstreamgit merge -X theirs upstream/master`  Reading Input and Generating and Storing Output Reading and Writing Text Files      &emsp;&emsp; Writing Files      &emsp;&emsp; Reading Files      &emsp;&emsp; Appending Files      &emsp;&emsp; Automatically Closing Files using `with`      &emsp;&emsp; Reading and Editing Files      &emsp;&emsp; Binary Files ("Pickling" Data) Plotting      &emsp;&emsp; Matplotlib     &emsp;&emsp; Line and Scatter Plots          &emsp;&emsp; &emsp;&emsp; Adding Labels, Legend and Title          &emsp;&emsp; &emsp;&emsp; Setting the Axis Limits      &emsp;&emsp; Multiple Plots      &emsp;&emsp; Saving a Plot      &emsp;&emsp; Importing Data from Delimited Files      &emsp;&emsp; Bar Charts and Histograms          &emsp;&emsp; &emsp;&emsp; Bar Charts           &emsp;&emsp; &emsp;&emsp;Histograms  &emsp;&emsp; Visualising 2D Arrays Summary  Test-Yourself Exercises  Lesson GoalRead information from external files.Store output from your programs as:- text files - graphs or plots.Read information from external files. Fundamental programming concepts - Importing and exporting data from your Python programs. Until now, we have only considered data that remains within the Python program. When we run a program, variables are created, used and no longer exist when the program finishes or is terminated.  We cannot access the information from outside of the Python program. It can be useful to access data created by a program.In a game we might want to:- store a player's score to continue playing the game later- store a leaderboard of top scoresFor an assingment or project it can be useful to store output from your program as:- a graph- text In last week's class we learnt to import files containing data in the form of:- sound- images  Reading and Writing Text FilesPython has a collection of easy to use functions for reading and writing files: - `open()`- `read()`- `write()`- `close()`  Before a file can be read or written to, it must be opened using the `open()` function. __Function arguments:__1. Path to the file (filename and location)2. The *mode specifier* with which to open the file:      - `r`: open an existing file to read   - `w`: open an existing file to write to. If no file exists: creates a new file.If file exists : over-writes previous contents.    - `a`: open an existing file to write to. If no file exists: creates a new file.If file exists : appends text to end of file. - `r+`: open a text file to read from __or__ write to. File must already exist.If file exists : over-writes previous contents.- `w+`: open a text file to read from __or__ write to.If no file exists: creates a new file.If file exists : over-writes previous contents.- `a+` : open a text file to read from __or__ write to.If no file exists: creates a new file.If file exists : appends text to end of file. Once the file is open, it creates a *file object*.  We can interrogate the file for various details including whether it is open or closed: - `name`: Name of the opened file - `mode`: Mode specifier with which the file was opened   - `closed`: Boolean True or False - `readable( )`: Read permission, Boolean True or False - `writable( )`: Write permission, Boolean True or False __Example__Once a file has been opened it's contents can be read, overwritten, or added to, depnding on the mode specifier used to open it. We are going to open an example file as if we were going to *write* some data to it. The mode specifier is `w`.<jupyter_code>file = open("sample_data/my_file.txt", "w" )<jupyter_output><empty_output><jupyter_text>Once the file is open, we can *interrogate* it for information.<jupyter_code># Interrrogate the file:
print( "File Name:",  file.name )

print( "Open Mode:",  file.mode )

print( "Readable:",  file.readable())

print( "Writable:",  file.writable())<jupyter_output>File Name: sample_data/my_file.txt
Open Mode: w
Readable: False
Writable: True<jupyter_text>An open file must then always be closed again by calling the close() method.Let's first write a function to show us if a file is open or closed.<jupyter_code># Write a function to determine the file's status:
def get_status( f ):
    if ( f.closed != False ) :
        return "Closed"
    else :
        return "Open"<jupyter_output><empty_output><jupyter_text>Why do we need to close a file? 1. Python does not automatically close the file. Leaving it open means that you risk overwriting information.1. Closing the file saves any changes to it.1. Depending on your computer's operating system, you may not be able to open a file simultaneously for reading and writing. __Example:__ If a program attempts to open a file that is already open (has not been closed), an error may be generated. Let's try out our function.<jupyter_code>print( "File Status:" , get_status(file))  

file.close()

print( "File Status:" , get_status(file))<jupyter_output>File Status: Open
File Status: Closed<jupyter_text>__Try it yourself__1. Save the previous 4 code cells contents to a file, `interrogate_file.py`. 1. Correct the following line to specify the path to `my_file.txt`, *relative* to the location of `interrogate.py`:>`file = open("sample_data/my_file.txt’", "w" )`1. Run `interrogate_file.py` from the terminal to see the file get created and display information about `my_file.txt`.   Writing FilesData can be written to a files in a number of ways.This time, let's actually write some data to the file. Let's first create a string to write to a text file:The inclusion of `\n` anywhere in the string creates a line break.<jupyter_code>poem = 'I never saw a man who looked\nWith such a wistful eye\n'
print(poem)<jupyter_output>I never saw a man who looked
With such a wistful eye<jupyter_text>A string my be *concatenated* (stuck together end to end) using the `+=` arithmetic operator.This allows us to build the string over the course of the program.<jupyter_code>poem += "Upon that little tent of blue\n" 

poem += "Which prisoners call the sky\n"

print(poem)<jupyter_output>I never saw a man who looked
With such a wistful eye
Upon that little tent of blue
Which prisoners call the sky
Upon that little tent of blue
Which prisoners call the sky<jupyter_text>Create an object to __write__ the string to a file:<jupyter_code>file = open("sample_data/poem.txt", "w" )<jupyter_output><empty_output><jupyter_text>Write the string to the file, then close it.<jupyter_code>file.write(poem)

file.close()<jupyter_output><empty_output><jupyter_text>You can open the file using a text editor to confirm it's contents.  Reading FilesCreate an object to __read__ the string from the containing file.(Note: As we have finished using the object name `file` to open `poem.txt` for writing we can recycle it to open the file for reading).<jupyter_code>file = open("sample_data/poem.txt", "r" )<jupyter_output><empty_output><jupyter_text>Read everything in the file:<jupyter_code>print(file.read())
file.close()<jupyter_output>I never saw a man who looked
With such a wistful eye
Upon that little tent of blue
Which prisoners call the sky<jupyter_text>Iteration can be used to quickly and efficiently read file contents line by line.In a `for` loop, the lines of a file are treated like the elements of a list.i.e. In the example below, `file` and `line` can be replaced with any variable name.<jupyter_code>file = open("sample_data/poem.txt", "r" )

for line in file:
    print(line, end="")
    
file.close()<jupyter_output>I never saw a man who looked
With such a wistful eye
Upon that little tent of blue
Which prisoners call the sky<jupyter_text>We can also look at individual words of a file using `split()`:<jupyter_code>file = open("sample_data/poem.txt", "r" )
print(file.read().split())
file.close()<jupyter_output>['I', 'never', 'saw', 'a', 'man', 'who', 'looked', 'With', 'such', 'a', 'wistful', 'eye', 'Upon', 'that', 'little', 'tent', 'of', 'blue', 'Which', 'prisoners', 'call', 'the', 'sky']<jupyter_text>The items are returned as a list, so we can use a for loop to print each word in a more readable format.<jupyter_code>file = open("sample_data/poem.txt", "r" )

for word in file.read().split() :
    print(word, end=" ")
    
file.close()<jupyter_output>I never saw a man who looked With such a wistful eye Upon that little tent of blue Which prisoners call the sky<jupyter_text>What does this code do?>`print(word, end=" ")` Note that the default ending for each print statement is changed to a space. Otherwise each word prints on a new line. `split` is also useful when we use the `input` function.<jupyter_code>user_input = input("enter 3 words : ")<jupyter_output>enter 3 words : hemma f th<jupyter_text>The words are treated as a single string.e.g. element 0 is the first character<jupyter_code>print(user_input[0])<jupyter_output><empty_output><jupyter_text>`split` allows us to access the individual words.<jupyter_code>for word in user_input.split() :
    print(word, end=" ")<jupyter_output><empty_output><jupyter_text>Appending Files *Add* data to a file (rather than overwriting).<jupyter_code>file = open( "sample_data/poem.txt" , "a" )  

file.write( "(by <NAME>)" )<jupyter_output><empty_output><jupyter_text>Notice that if you open `poem.txt` in a text editor at this stage, your changes have not appeared yet.Run:<jupyter_code>file.close()<jupyter_output><empty_output><jupyter_text>The changes should have now been saved an will appear if you re-open the file in a text editor.  This means you can update the same file by writing data at multiple points in your program.<jupyter_code># File append 1
file = open("sample_data/appended_file.csv", "a" )
file.write('word\n')
file.close()

# File append 2
file = open("sample_data/test_now2.csv", "a" )
file.write('1, 3, 5\n')
file.close()

# File append 3
file = open("sample_data/test_now2.csv", "a" )
file.write('word\n')
file.write('1, 3, 5')
file.close()<jupyter_output><empty_output><jupyter_text>__Try it yourself__Practise writing and reading data by:- Creating a Python file, `write_and_read.py`. - Use `write_and_read.py` to produce a .txt file, `w_and_r.txt` in the sample_data directory.- Edit `write_and_read.py` and run the file to write a string to `w_and_r.txt`.- Edit `write_and_read.py` to read and print the contents of `w_and_r.txt`.  Automatically Closing Files Using `with`.  `with`It is good practice to use the Python with keyword to group file operations within a block.  This automatically closes the file is properly closed after operations in the block have been completed, even if an exception or error occurs within the block.  Example : `with`  Take an example string...<jupyter_code>twister = "How can a clam cram in a clean cream can?"<jupyter_output><empty_output><jupyter_text>In the `with` block: - Write `twister` into a file. - Display the file’s current status.<jupyter_code>with open("sample_data/twister.txt" , "w" ) as file :  
    file.write(twister)
    print(f"File Now Closed?: {file.closed}")
    
print(f"File Now Closed?: {file.closed}")<jupyter_output>File Now Closed?: False
File Now Closed?: True<jupyter_text>Reading and Editing FilesWe will now study some more functions to read and edit a file that has been opened using the `read` file specifier.- `read()`- `seek()`- `tell()`- `truncate()`   `read()`We studied read earlier in today's class.By default `read()` will read the entire contents of the file from the beginning (index 0) to the end (index -1).  An optional integer argument can be used to specify *how many characters to read*.   `seek()`The position within a file: - from which to read or... - at which to writecan be selected using `seek()`.An integer argument is used to specify *how many characters to offset* the start position from the start of the file.  `tell()`The current position within a file can be returned an integer location index at any time using `tell()`.  `truncate()`Deletes all characters after the current position. __Example :__`read`, `seek()`, `tell()` and `truncate()`.Let's first write some data to a file:<jupyter_code>with open("sample_data/twister.txt" , "w" ) as file :  
    file.write("How can a clam cram in a clean cream can?")<jupyter_output><empty_output><jupyter_text>Now we are going to read and edit the data.We first:- open the file using a mode specifier to read *and* write- print the current position in the file:<jupyter_code>file = open("sample_data/twister.txt" , "r+")

print("Position In File Now:", file.tell())<jupyter_output>Position In File Now: 0<jupyter_text>The position changes, once the file has been read.<jupyter_code># reading the file moves the position 
print(file.read())
print("Position In File Now:", file.tell())<jupyter_output>How can a clam cram in a clean cream can?
Position In File Now: 41<jupyter_text>Let's move to a specific position, 31 characters into the file:*(This looks for a specific __position__. In next week's class we will learn to look for a specific __word__.)*<jupyter_code># seek
file.seek( 31 )
print("Position In File Now:", file.tell())<jupyter_output>Position In File Now: 31<jupyter_text>Position 31 is just before the words `' cream can?'`.We can see this by reading the file at this point.<jupyter_code>print(file.read())<jupyter_output>cream can?<jupyter_text>We are going to replace:    'cream can?'...with:    'caravan?' We are now at the end of the file.Let's go back to position 31 and write `'caravan'`.<jupyter_code># write starting at new position
file.seek( 31 )
file.write("caravan?")<jupyter_output><empty_output><jupyter_text>What does the whole file look like now? We can go back to position 0 and read the file to check:<jupyter_code># return to position 0 and print the file contents
file.seek(  0  )
print(file.read())<jupyter_output>How can a clam cram in a clean caravan?n?<jupyter_text>`'cream can?'` has been *overwritten* with  `'caravan?'`.However, `'cream can?'` contains more letters than `'caravan?'` so the additional letters are note overwritten.   We can use `truncate` to remove any traling charcters:<jupyter_code>print(len('caravan?'))
file.seek( 31 + len('caravan?'))
# truncate deletes all characters from current position onwards
file.truncate()<jupyter_output>8<jupyter_text>What does the whole file look like now? We can go back to position 0 and read the file to check:<jupyter_code># return to position 0 and print the file contents
file.seek( 0 )
print(file.read())<jupyter_output>How can a clam cram in a clean caravan?<jupyter_text>Finally, remember to close the file:<jupyter_code>file.close()<jupyter_output><empty_output><jupyter_text>The full example, using `with`:<jupyter_code># write file
with open("sample_data/twister.txt" , "w" ) as file :  
    file.write("How can a clam cram in a clean cream can?")
    

# read and edit
with open("sample_data/twister.txt" , "r+" ) as file : 
    
    # tell 
    print("Position In File Now:", file.tell()) 
    
    # reading the file moves the position 
    print(file.read())
    print("Position In File Now:", file.tell())
    
    # seek
    file.seek( 31 )
    print("Position In File Now:", file.tell())
    
    # write starting at new position
    file.write("caravan?")
    
    # truncate deletes all characters from current position onwards
    file.truncate()    
    
    # move back to start poistion
    file.seek( 0 )
    print("Position In File Now:", file.tell())
    
    # print updated string
    print(file.read())<jupyter_output>Position In File Now: 0
How can a clam cram in a clean cream can?
Position In File Now: 41
Position In File Now: 31
Position In File Now: 0
How can a clam cram in a clean caravan?<jupyter_text>Alternatively, if we know the word to replace we can use the function `replace()`:<jupyter_code>with open("sample_data/twister.txt" , "w+" ) as file : 
    # write contents
    file.write("How can a clam cram in a clean cream can?")
    
    # go to start of file
    file.seek( 0 )
    
    # create a new string with the words replaced 
    data = file.read().replace("cream can", "caravan")
    
    # go to start of file
    file.seek( 0 )
    
    # over-write the old string
    file.write(data)
    file.truncate()
    
    # go to start of file
    file.seek( 0 )
    print(file.read())<jupyter_output>How can a clam cram in a clean caravan?<jupyter_text>A more efficient way to do this, without the need to truncate the extra characters is to loop through each line of the file and replace any instances of the variable.<jupyter_code>with open("sample_data/twister.txt" , "w+" ) as file : 
    # write contents
    file.write("How can a clam cram in a clean cream can?")
    
    # go to start of file
    file.seek( 0 )
    
    # replace any instances in each line of the file
    for line in file:
        line.replace("cream can", "caravan")
    
    # go to start of file
    file.seek( 0 )
    
    # go to start of file
    file.seek( 0 )
    print(file.read())<jupyter_output>How can a clam cram in a clean cream can?<jupyter_text>A maximum file sixe can be specified when using truncate:<jupyter_code>with open("sample_data/twister.txt" , "w+" ) as file : 
    # write contents
    file.write("How can a clam cram in a clean cream can?")
    
    # set maximum length of string
    file.truncate(10)
    
    # go to start of file
    file.seek( 0 )
    print(file.read())<jupyter_output>How can a<jupyter_text>Importing Data from Delimited Text FilesTo read data from text files such as .csv, .txt, .dat, we can use  `numpy.load.txt`. __Remember__The file to be loaded must be in the same directory as your python program.Otherwise you must specify the the full path to the data file using / to seperate directory names. The filename (or path plus filename) needs to be between "" quotation marks.  To import the data from `sample_data/data.dat`:>`0.000 1.053 2.105 3.158 4.21174.452 48.348 68.733 59.796 54.123`<jupyter_code>import numpy as np
A = np.loadtxt('sample_data/sample_data.dat')
print(A)
print(type(A))<jupyter_output>[[ 1.053  2.105  3.158  4.211  6.065]
 [48.348 68.733 59.796 54.123 74.452]]
<class 'numpy.ndarray'><jupyter_text>Purely numerical data is imported as a Numpy array.This is a data structure for storing numerical values. Individual elements, rows or columns can be returned using the index<jupyter_code>print(A[0][1])  # row 0, col 1
print(A[0, :])  # row 0
print(A[:, 1])  # col 1<jupyter_output>2.105
[1.053 2.105 3.158 4.211 6.065]
[ 2.105 68.733]<jupyter_text>The __delimiter__ should (sometimes) be specified.This tells Python how to separate the data into the individual elements of an array. The default delimiter is a space.(Data separated by spaces will be automatically be assigned different indices). The __data type__ should (sometimes) be specified.This tells Python how to separate the data into the individual elements of an array. The default data type is `float`.If your data contains items that cannot be expressed as a float, importing it will cause an error unless the data-type is specified.Mixed data types can be imported as `string` values.  Data in `sample_data/sample_student_data.txt`:- __delimiter__ : tab (`/t`) - __data type__ : string (float in some columns)```PythonSubject	Sex	DOB	Height	Weight	BP(ID)	M/F	dd/mm/yy	m	kg	mmHgJW-1	M	19/12/1995	1.82	92.4	119/76JW-2	M	11/01/1996	1.77	80.9	114/73JW-3	F	02/10/1995	1.68	69.7	124/79JW-6	M	06/07/1995	1.72	75.5	110/60JW-7	F	28/03/1996	1.66	72.4	-JW-9	F	11/12/1995	1.78	82.1	115/75JW-10	F	07/04/1996	1.6	45	-/-JW-11	M	22/08/1995	1.72	77.2	97/63JW-12	M	23/05/1996	1.83	88.9	105/70JW-14	F	12/01/1996	1.56	56.3	108/72JW-15	F	01/06/1996	1.64	65	99/67JW-16	M	10/09/1995	1.63	73	131/84JW-17	M	17/02/1996	1.67	89.8	101/76JW-18	M	31/07/1996	1.66	75.1	-/-JW-19	F	30/10/1995	1.59	67.3	103/69JW-22	F	09/03/1996	1.7	45	119/80JW-23	M	15/05/1995	1.97	89.2	124/82JW-24	F	01/12/1995	1.66	63.8	100/78JW-25	F	25/10/1995	1.63	64.4	-/-JW-26	M	17/04/1996	1.69	55	121/82```<jupyter_code>np.loadtxt('sample_data/sample_student_data.txt', delimiter="\t", dtype=str)<jupyter_output><empty_output><jupyter_text>Regions can be selected, for example to select only numerical data. `skiprows` skips the first n lines. `usecols` specifies which columns to read, with 0 being the first. `usecols = (1, 4, 5)` : extracts the 2nd, 5th and 6th columns.`usecols = (3, 4)` : extracts the 4th and 5th columns (height and weight data).<jupyter_code>students = np.loadtxt('sample_data/sample_student_data.txt', skiprows=9, usecols=(3,4))
print(students)<jupyter_output>[[ 1.78 82.1 ]
 [ 1.6  45.  ]
 [ 1.72 77.2 ]
 [ 1.83 88.9 ]
 [ 1.56 56.3 ]
 [ 1.64 65.  ]
 [ 1.63 73.  ]
 [ 1.67 89.8 ]
 [ 1.66 75.1 ]
 [ 1.59 67.3 ]
 [ 1.7  45.  ]
 [ 1.97 89.2 ]
 [ 1.66 63.8 ]
 [ 1.63 64.4 ]
 [ 1.69 55.  ]]<jupyter_text>You can now use the imported data as a regular Numpy array.<jupyter_code># Use indexing to select which ROWS and which COLUMNS to plot
plt.plot(students[:, 0], students[:, 1], 'rx', label="students")

# Legend
plt.legend(loc='best', fontsize=12)

# Axes labels
plt.xlabel('Height (m)', fontsize=12)
plt.ylabel('Weight (kg)', fontsize=12)

# Title
plt.title("Height to weight relationship for students in class A", fontsize=18);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10208255857195983, 'fraction_numerical': 0.032261063592413534, 'mean_word_length': 4.25604690935744, 'pattern_counts': {'":': 0, '<': 139, '<?xml version=': 0, '>': 143, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '151682', 'license': 'MIT', 'n_tokens_mistral': 7467, 'n_tokens_neox': 6884, 'n_words': 2790, 'path': '.ipynb_checkpoints 2/08_InputOutput-checkpoint.ipynb', 'repo_name': 'hphilamore/ILAS_PyEv2019', 'size': '21517'}