<jupyter_start><jupyter_text>Reading and Writing Files in Python In this section, we will learn some basic opearation about reading and writing files. Moreover, as a data scientist, building an accurate machine learning model is not the end of the project. We will showing you how to save and load your machine learning model in Python.This allows you to save your model to file and load it later in order to make predictions.  Read txt file<jupyter_code>txt_file_url = "../data/files/Python.txt"
f = open(txt_file_url, "r") #opens file with name of "Python.txt"
# read and print the entire file
print(f.read())
# remember to colse the file
f.close()<jupyter_output>Life is short,
Use Python!<jupyter_text>Used the **readline()** method twice, we would get the first 2 lines because of Python's reading process.<jupyter_code>f = open(txt_file_url, "r") #opens file with name of "Python.txt"
# read the 1st line
print(f.readline())
# read the next line
print(f.readline())
f.close()
#opens file with name of "Python.txt"
f = open("files/Python.txt", "r") 
myList = []
for line in f:
    myList.append(line)
f.close()
    
print(myList)
print(myList[0])
print(myList[1])<jupyter_output>['Life is short,\n', 'Use Python!']
Life is short,

Use Python!<jupyter_text>Write txt file<jupyter_code># Write file with name of "test.txt"
f = open("../data/files/test.txt","w")  
f.write("I love Python.\n")
f.write("I will be a Python master.\n")
f.write("I need to keep learning!")
f.close()

# read and see the test.txt file
f = open("../data/files/test.txt","r") 
print(f.read())
f.close()<jupyter_output>I love Python.
I will be a Python master.
I need to keep learning!<jupyter_text>Read csv file<jupyter_code>import csv
csvFile = open("../data/files/test.csv", "r") 
reader = csv.reader(csvFile, delimiter=',')
# load the data in a dictionary 
result = {}
for item in reader:
    # ignore the first line
    if reader.line_num == 1:
        continue    
    result[item[0]] = item[1]
csvFile.close()

print(result)<jupyter_output>{'Ali': '25', 'Bob': '24', 'Chirs': '29'}<jupyter_text>Write csv file<jupyter_code>import csv
fileHeader = ["name", "age"]

d1 = ["Chris", "27"]
d2 = ["Ming", "26"]

csvFile = open("../data/files/write.csv", "w")
writer = csv.writer(csvFile)
writer = csv.writer(csvFile)

# write the head and data
writer.writerow(fileHeader)
writer.writerow(d1)
writer.writerow(d2)

# Here is another command 
# writer.writerows([fileHeader, d1, d2])

csvFile.close()

# go to see the "write.csv" file.<jupyter_output><empty_output><jupyter_text>You can find more information from the [documentation](https://docs.python.org/3.6/library/csv.html)  Using Pandas to Read CSV file<jupyter_code>import pandas as pd
import numpy as np
data = pd.read_csv("../data/files/test.csv")
# data is data
print(data) 

# extract the age data
Age = np.array(data.Age, dtype = 'double')
print(Age)

# reshap this age vector
Age = np.reshape(Age, [3,1])
print(Age)<jupyter_output>Name  Age
0    Ali   25
1    Bob   24
2  Chirs   29
[25. 24. 29.]
[[25.]
 [24.]
 [29.]]<jupyter_text>Find more operation about Pandas in the [documentation](https://pandas.pydata.org/) and [cheatsheet](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf)  Read Matlab file The functions --scipy.io.loadmat-- and --scipy.io.savemat-- allow you to read and write MATLAB files. You can read about them in the [documentation](https://docs.scipy.org/doc/scipy/reference/io.html).<jupyter_code>import numpy as np
from scipy.io import loadmat, savemat

data = loadmat("../data/files/magic.mat");
print(data);
print(data['magic'])<jupyter_output>{'__header__': b'MATLAB 5.0 MAT-file, Platform: PCWIN64, Created on: Sun Jan 28 21:23:19 2018', '__version__': '1.0', '__globals__': [], 'magic': array([[17, 24,  1,  8, 15],
       [23,  5,  7, 14, 16],
       [ 4,  6, 13, 20, 22],
       [10, 12, 19, 21,  3],
       [11, 18, 25,  2,  9]], dtype=uint8)}
[[17 24  1  8 15]
 [23  5  7 14 16]
 [ 4  6 13 20 22]
 [10 12 19 21  3]
 [11 18 25  2  9]]<jupyter_text>Write Matlab file<jupyter_code>X = np.array(data['magic'])
# Do some calculation
X = X*2
# Dictionary from which to save matfile variables.
data = {'magic2': X}
# save the data
savemat("../data/files/magic2.mat", data)
# Go to matlab and check the data

data = loadmat("../data/files/magic2.mat");
print(data['magic2'])<jupyter_output>[[34 48  2 16 30]
 [46 10 14 28 32]
 [ 8 12 26 40 44]
 [20 24 38 42  6]
 [22 36 50  4 18]]<jupyter_text>Save and Load file by Pickle The Pickle pacakge is used for serializing and de-serializing a Python object structure. Any object in python can be pickled so that it can be saved on disk and loaded back to continue the work. You can read about them in the [documentation](https://docs.python.org/3.6/library/pickle.html?highlight=picklemodule-pickle).<jupyter_code>import numpy as np
import pickle
X = np.eye(5)
print(X)
# Save the matirx X
with open('../data/files/X.pickle', 'wb') as f:
    pickle.dump(X, f)
# Change the value of the original X    
X =  X + 4
print(X)
# load the matrix 
with open('files/X.pickle', 'rb') as f:
    X = pickle.load(f)
print(X)<jupyter_output>[[1. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0.]
 [0. 0. 1. 0. 0.]
 [0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 1.]]<jupyter_text>One Example  In this demonstration, we will use a Logistic Regression Model and the Iris dataset.<jupyter_code>from sklearn.linear_model import LogisticRegression  
from sklearn.datasets import load_iris  
from sklearn.model_selection import train_test_split

# Load and split data
data = load_iris()  
Xtrain, Xtest, Ytrain, Ytest = train_test_split(data.data, data.target, test_size=0.3, random_state=4)  

# Create a model
model = LogisticRegression(C=0.1,  
                           max_iter=2000, 
                           fit_intercept=True 
                           )
model.fit(Xtrain, Ytrain)  
print(model);<jupyter_output>LogisticRegression(C=0.1, max_iter=2000)<jupyter_text>In the following few lines of code, the model which we created in the previous step is saved to file, and then loaded as a new object called pickled_model. The loaded model is then used to calculate the accuracy score and predict outcomes on new unseen (test) data.<jupyter_code>import pickle

#
# Create your model here (same as above)
#

# Save to file in the current working directory
pkl_filename = "../data/pickle_model.pkl"  
with open(pkl_filename, 'wb') as file:  
    pickle.dump(model, file)

# Load from file
with open(pkl_filename, 'rb') as file:  
    pickle_model = pickle.load(file)

# Calculate the accuracy score and predict target values
score = pickle_model.score(Xtest, Ytest)  
print("Test score: {0:.2f} %".format(100 * score))  
Ypredict = pickle_model.predict(Xtest)<jupyter_output>Test score: 97.78 %<jupyter_text>Use python to read and write the `yaml` file.<jupyter_code>import yaml
yaml_url = "../data/test.yaml"
with open(yaml_url, encoding='utf-8') as file:
    data = yaml.safe_load(file)
    print(data)
    print(data['case1']['json'])
    print(data['case1']['json']['username'])
 
import yaml

content = {
    'id': 1,
    'text': 'programming languages',
    'members': ['java', 'python', 'python', 'c', 'go', 'shell'],
    'next': {'a':1,'b':2}
}

save_path = '../data/test_save.yaml'
with open(save_path, 'w', encoding='utf-8') as file:
    yaml.dump(content, file, default_flow_style=False, encoding='utf-8', allow_unicode=True)

with open(save_path, encoding='utf-8') as file:
    data = yaml.safe_load(file)
    print(data)
# import pyyaml module
import yaml
from yaml.loader import SafeLoader

# Open the file and load the file
with open(save_path) as f:
    data = yaml.load(f, Loader=SafeLoader)
    print(data)
import yaml

from yaml.loader import SafeLoader

with open(save_path, 'r') as f:
    data = list(yaml.load_all(f, Loader=SafeLoader))
    print(data)
import yaml

# dict object
members = [{'name': 'Zoey', 'occupation': 'Doctor'},
           {'name': 'Zaara', 'occupation': 'Dentist'}]

# Convert Python dictionary into a YAML document
print(yaml.dump(members))
import yaml

save_path = '../data/model_config.yaml'

user_details = { 'model_name':'vgg19',
                'w1': [1, 2, 3, 4, 5.0],
                'AccessKeys': ['EmployeeTable',
                               'SoftwaresList',
                               'HardwareList']}

with open(save_path, 'w') as f:
    data = yaml.dump(user_details, f, sort_keys=False, default_flow_style=False)

with open(save_path, 'r') as f:
    # data = yaml.load_all(f, Loader=SafeLoader)
    data = yaml.safe_load(f)
    print(data)    
type(data['w1'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13917466188879898, 'fraction_numerical': 0.031441451855276845, 'mean_word_length': 3.8853754940711465, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '500052', 'license': 'MIT', 'n_tokens_mistral': 3286, 'n_tokens_neox': 3042, 'n_words': 941, 'path': '01.Python/Python_03_Read_and_Write_Files.ipynb', 'repo_name': 'cuicaihao/Data_Science_Python', 'size': '8651'}