<jupyter_start><jupyter_text>What you will learn- What is a CSV file- Reading and writting on a csv file  CSV = Comma seperated valuesChances are you have worked with .csv files before. There are simply values sperated by commas ... Note- All files used or created will be stored under week 3 in a folder called "data"Here is some exampe output:<jupyter_code>data,timeStamp,dataType,units,sensorType,supervisorID
35.18,03/22/2020-23:54:04,Humidity,%,BME280,2
840.78,03/22/2020-23:54:04,Pressure,hPa,BME280,2
1546.75,03/22/2020-23:54:04,Elevation,m,BME280,2
176.6784452,03/22/2020-17:54:04,solar irradiance,w/m^2,ADC1115_Pyra,1
0.035,03/22/2020-17:54:04,raw Voltage,V,ADC1115_Pyra,1
17.25,03/22/2020-23:54:05,Temperature,C,BME280,2
35.18,03/22/2020-23:54:05,Humidity,%,BME280,2
840.84,03/22/2020-23:54:05,Pressure,hPa,BME280,2
1546.24,03/22/2020-23:54:05,Elevation,m,BME280,2
175.4164563,03/22/2020-17:54:06,solar irradiance,w/m^2,ADC1115_Pyra,1
0.03475,03/22/2020-17:54:06,raw Voltage,V,ADC1115_Pyra,1
17.23,03/22/2020-23:54:06,Temperature,C,BME280,2
35.16,03/22/2020-23:54:06,Humidity,%,BME280,2
840.73,03/22/2020-23:54:06,Pressure,hPa,BME280,2
1547.14,03/22/2020-23:54:06,Elevation,m,BME280,2
176.0474508,03/22/2020-17:54:07,solar irradiance,w/m^2,ADC1115_Pyra,1
0.034875,03/22/2020-17:54:07,raw Voltage,V,ADC1115_Pyra,1
17.24,03/22/2020-23:54:07,Temperature,C,BME280,2
34.98,03/22/2020-23:54:07,Humidity,%,BME280,2
840.67,03/22/2020-23:54:07,Pressure,hPa,BME280,2<jupyter_output><empty_output><jupyter_text>If this kinda looks like data in excel that is because excel spread-sheets can be exported as .csv files.Above each column for a given row has a distinction. The first row shown above are the column headers - i.e. the name of each column. We'll show how to work with this data but first the basics: Reading a CSV file<jupyter_code>import csv
import os

def run():
    with open("./data/example1.csv", "r") as csvFile: # the "r" means read
        fileReader = csv.reader(csvFile)

        i = 0
        for row in fileReader:
            print(row)
            i += 1

            if i == 10:
                break
            
if __name__=="__main__":
    run()<jupyter_output>['data', 'timeStamp', 'dataType', 'units', 'sensorType', 'supervisorID']
['218.3240787', '03/22/2020-17:38:38', 'solar irradiance', 'w/m^2', 'ADC1115_Pyra', '1']
['0.04325', '03/22/2020-17:38:38', 'raw Voltage', 'V', 'ADC1115_Pyra', '1']
['17.67', '03/22/2020-23:38:39', 'Temperature', 'C', 'BME280', '2']
['35.05', '03/22/2020-23:38:39', 'Humidity', '%', 'BME280', '2']
['840.43', '03/22/2020-23:38:39', 'Pressure', 'hPa', 'BME280', '2']
['1550.4', '03/22/2020-23:38:39', 'Elevation', 'm', 'BME280', '2']
['218.3240787', '03/22/2020-17:38:39', 'solar irradiance', 'w/m^2', 'ADC1115_Pyra', '1']
['0.04325', '03/22/2020-17:38:39', 'raw Voltage', 'V', 'ADC1115_Pyra', '1']
['17.68', '03/22/2020-23:38:40', 'Temperature', 'C', 'BME280', '2']<jupyter_text>Note- The "if i == 10" break is just so all 5001 rows are not printed- the "r" tells the file to be opened for reading only  What is the "with open"To operate (meaning to read or to write) on a file that file has to be accessed. This involves opening the file. But what happens when you are done? That file needs to be closed. This is where "with" comes in. It is a context manager meaning that when you fall out of the with indentation the file will automatically be closed.You cannot write to a file that another program is using - this is like two people trying to use the remote at the same time; chaoas and confusion will result. Context mangers allow clean closing of files without having to explicitly state that the file closes.We could write the above code as seen below - but THIS IS BAD FORM!<jupyter_code># this is bad code
csvFile = open("./data/example1.csv")

fileReader = csv.reader(csvFile)

i = 0
for row in fileReader:
    print(row)
    i += 1

    if i == 10:
        break

csvFile.close()<jupyter_output>['data', 'timeStamp', 'dataType', 'units', 'sensorType', 'supervisorID']
['218.3240787', '03/22/2020-17:38:38', 'solar irradiance', 'w/m^2', 'ADC1115_Pyra', '1']
['0.04325', '03/22/2020-17:38:38', 'raw Voltage', 'V', 'ADC1115_Pyra', '1']
['17.67', '03/22/2020-23:38:39', 'Temperature', 'C', 'BME280', '2']
['35.05', '03/22/2020-23:38:39', 'Humidity', '%', 'BME280', '2']
['840.43', '03/22/2020-23:38:39', 'Pressure', 'hPa', 'BME280', '2']
['1550.4', '03/22/2020-23:38:39', 'Elevation', 'm', 'BME280', '2']
['218.3240787', '03/22/2020-17:38:39', 'solar irradiance', 'w/m^2', 'ADC1115_Pyra', '1']
['0.04325', '03/22/2020-17:38:39', 'raw Voltage', 'V', 'ADC1115_Pyra', '1']
['17.68', '03/22/2020-23:38:40', 'Temperature', 'C', 'BME280', '2']<jupyter_text>Writing to a CSV fileAbove we used "r" to read from a file. Well now that becomes "w" to write. ImportantIf you write to a file that already exists then the file will be overwritten. There will be no warning as with many office tools saying you are about to overwrite something.If you with to add something to a csv file we will cover that below.<jupyter_code>import csv
import os
import time
import random

HEADERS = ['data', 'timeStamp', 'dataType']

def makeRow(): # random data to put in csv
    return [random.randint(0,101), time.time(), "random Float"]
    

def makeFile(): # makes a csv
    with open("./data/example2.csv", "w") as csvFile: # the "w" means write
        fileWriter = csv.writer(csvFile)
        
        fileWriter.writerow(HEADERS)
        
        for i in range(10):
            fileWriter.writerow(makeRow())
            

def readFile(): # read from the csv you created to see it!
    with open("./data/example2.csv", "r") as csvFile:
        fileReader = csv.reader(csvFile)

        for row in fileReader:
            print(row)
            
if __name__=="__main__":
    makeFile()
    readFile()<jupyter_output>['data', 'timeStamp', 'dataType']
['100', '1586128830.8706567', 'random Float']
['7', '1586128830.870685', 'random Float']
['23', '1586128830.8707', 'random Float']
['95', '1586128830.8707128', 'random Float']
['17', '1586128830.8707256', 'random Float']
['25', '1586128830.870738', 'random Float']
['27', '1586128830.8707497', 'random Float']
['1', '1586128830.8707626', 'random Float']
['3', '1586128830.8707747', 'random Float']
['50', '1586128830.870787', 'random Float']<jupyter_text>Note- If you re-run the above bit of code you will notice that the all the values change- This is, again, because the "w" mode will overwrite your files!- Please don't accedently overwrite the results of an experiment have to redo it ...   From csv to list of dicsOften it is good to organize data in a csv to a list of dics. This is easier to later operate on<jupyter_code># we'll take the values that you just created in the above code and turn them into the list of dics!

def csvToDict(csvData):
    headers = next(csvData)  # next will push the iterable object one forword
    
    data = list(map(lambda row: dict(zip(headers, row)) , csvData))
    
    return data


def run(): # read from the csv you created to see it!
    with open("./data/example2.csv", "r") as csvFile:
        fileReader = csv.reader(csvFile)
        listOfDics = csvToDict(fileReader)
        
        print(listOfDics)
        
            
if __name__=="__main__":
    run()<jupyter_output>[{'data': '77', 'timeStamp': '1586127437.9168706', 'dataType': 'random Float'}, {'data': '99', 'timeStamp': '1586127437.9168973', 'dataType': 'random Float'}, {'data': '71', 'timeStamp': '1586127437.9169118', 'dataType': 'random Float'}, {'data': '72', 'timeStamp': '1586127437.9169242', 'dataType': 'random Float'}, {'data': '9', 'timeStamp': '1586127437.9169366', 'dataType': 'random Float'}, {'data': '85', 'timeStamp': '1586127437.9169497', 'dataType': 'random Float'}, {'data': '75', 'timeStamp': '1586127437.9169614', 'dataType': 'random Float'}, {'data': '21', 'timeStamp': '1586127437.916973', 'dataType': 'random Float'}, {'data': '75', 'timeStamp': '1586127437.9169855', 'dataType': 'random Float'}, {'data': '47', 'timeStamp': '1586127437.9169977', 'dataType': 'random Float'}]<jupyter_text>Appending a csv fileLet's take the csv file you created above and append it with 10 more rows<jupyter_code>import csv
import os
import time
import random

HEADERS = ['data', 'timeStamp', 'dataType']

def makeRow(): # random data to put in csv
    return [random.randint(0,101), time.time(), "random Float"]
    

def makeFile(): # makes a csv
    with open("./data/example2.csv", "a") as csvFile: # the "a" means append
        fileWriter = csv.writer(csvFile)
        
        for i in range(10):
            fileWriter.writerow(makeRow())
            

def readFile(): # read from the csv you created to see it!
    with open("./data/example2.csv", "r") as csvFile:
        fileReader = csv.reader(csvFile)

        for row in fileReader:
            print(row)
            
if __name__=="__main__":
    makeFile()
    readFile()<jupyter_output>['data', 'timeStamp', 'dataType']
['100', '1586128830.8706567', 'random Float']
['7', '1586128830.870685', 'random Float']
['23', '1586128830.8707', 'random Float']
['95', '1586128830.8707128', 'random Float']
['17', '1586128830.8707256', 'random Float']
['25', '1586128830.870738', 'random Float']
['27', '1586128830.8707497', 'random Float']
['1', '1586128830.8707626', 'random Float']
['3', '1586128830.8707747', 'random Float']
['50', '1586128830.870787', 'random Float']
['29', '1586128842.588033', 'random Float']
['31', '1586128842.5880682', 'random Float']
['37', '1586128842.588084', 'random Float']
['21', '1586128842.588097', 'random Float']
['82', '1586128842.5881093', 'random Float']
['56', '1586128842.5881224', 'random Float']
['58', '1586128842.5881343', 'random Float']
['22', '1586128842.5881467', 'random Float']
['97', '1586128842.5881586', 'random Float']
['57', '1586128842.588171', 'random Float']
{'fraction_non_alphanumeric': 0.17299919159256266, 'fraction_numerical': 0.17178658043654002, 'mean_word_length': 4.446890478811227, 'pattern_counts': {'":': 4, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '10952', 'license': 'MIT', 'n_tokens_mistral': 4709, 'n_tokens_neox': 3724, 'n_words': 1020, 'path': 'Python_Jupyter_Training/Week_3/.ipynb_checkpoints/4_Input_Output_CSV_Files-checkpoint.ipynb', 'repo_name': 'brendo61-byte/pythonOnBoarding', 'size': '9896'}