<jupyter_start><jupyter_text>Reading from and Writing to Files using Python![](https://i.imgur.com/rv8wZ7l.png)  Interacting with the OS and filesystemThe `os` module in Python provides many functions for interacting with the OS and the filesystem. Let's import it and try out some examples.<jupyter_code>import os<jupyter_output><empty_output><jupyter_text>We can check the present working directory using the `os.getcwd` function.<jupyter_code>os.getcwd()<jupyter_output><empty_output><jupyter_text>To get the list of files in a directory, use `os.listdir`. You pass an absolute or relative path of a directory as the argument to the function.<jupyter_code>help(os.listdir)
os.listdir('.') #relative path
os.listdir('/usr') # absolute path<jupyter_output><empty_output><jupyter_text>You can create a new directory using `os.makedirs`. Let's create a new directory called `data`, where we'll later download some files.<jupyter_code>os.makedirs('./data', exist_ok=True)<jupyter_output><empty_output><jupyter_text>Can you figure out what the argument `exist_ok` does? Try using the `help` function or [read the documentation](https://docs.python.org/3/library/os.htmlos.makedirs).Let's verify that the directory was created and is currently empty.<jupyter_code>'data' in os.listdir('.')
os.listdir('./data')<jupyter_output><empty_output><jupyter_text>Let us download some files into the `data` directory using the `urllib` module.<jupyter_code>url1 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans1.txt'
url2 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans2.txt'
url3 = 'https://gist.githubusercontent.com/aakashns/257f6e6c8719c17d0e498ea287d1a386/raw/7def9ef4234ddf0bc82f855ad67dac8b971852ef/loans3.txt'
from urllib.request import urlretrieve
urlretrieve(url1, './data/loans1.txt')
urlretrieve(url2, './data/loans2.txt')
urlretrieve(url3, './data/loans3.txt')<jupyter_output><empty_output><jupyter_text>Let's verify that the files were downloaded.<jupyter_code>os.listdir('./data')<jupyter_output><empty_output><jupyter_text>You can also use the [`requests`](https://docs.python-requests.org/en/master/) library to dowload URLs, although you'll need to [write some additional code](https://stackoverflow.com/questions/44699682/how-to-save-a-file-downloaded-from-requests-to-another-directory) to save the contents of the page to a file.  Reading from a file To read the contents of a file, we first need to open the file using the built-in `open` function. The `open` function returns a file object and provides several methods for interacting with the file's contents.<jupyter_code>file1 = open('./data/loans1.txt', mode='r')<jupyter_output><empty_output><jupyter_text>The `open` function also accepts a `mode` argument to specifies how we can interact with the file. The following options are supported:```    ========= ===============================================================    Character Meaning    --------- ---------------------------------------------------------------    'r'       open for reading (default)    'w'       open for writing, truncating the file first    'x'       create a new file and open it for writing    'a'       open for writing, appending to the end of the file if it exists    'b'       binary mode    't'       text mode (default)    '+'       open a disk file for updating (reading and writing)    'U'       universal newline mode (deprecated)    ========= ===============================================================```To view the contents of the file, we can use the `read` method of the file object.<jupyter_code>file1_contents = file1.read()
print(file1_contents)<jupyter_output>amount,duration,rate,down_payment
100000,36,0.08,20000
200000,12,0.1,
628400,120,0.12,100000
4637400,240,0.06,
42900,90,0.07,8900
916000,16,0.13,
45230,48,0.08,4300
991360,99,0.08,
423000,27,0.09,47200<jupyter_text>The file contains information about loans. It is a set of comma-separated values (CSV). > **CSVs**: A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. (Wikipedia)The first line of the file is the header, indicating what each of the numbers on the remaining lines represents. Each of the remaining lines provides information about a loan. Thus, the second line `10000,36,0.08,20000` represents a loan with:* an *amount* of `$10000`, * *duration* of `36` months, * *rate of interest* of `8%` per annum, and * a down payment of `$20000`The CSV is a standard file format used for sharing data for analysis and visualization. Over the course of this tutorial, we will read the data from these CSV files, process it, and write the results back to files. Before we continue, let's close the file using the `close` method (otherwise, Python will continue to hold the entire file in the RAM)<jupyter_code>file1.close()<jupyter_output><empty_output><jupyter_text>Once a file is closed, you can no longer read from it.<jupyter_code>file1.read()<jupyter_output><empty_output><jupyter_text>Closing files automatically using `with`To close a file automatically after you've processed it, you can open it using the `with` statement.<jupyter_code>with open('./data/loans2.txt') as file2:
    file2_contents = file2.read()
    print(file2_contents)<jupyter_output>amount,duration,rate,down_payment
828400,120,0.11,100000
4633400,240,0.06,
42900,90,0.08,8900
983000,16,0.14,
15230,48,0.07,4300<jupyter_text>Once the statements within the `with` block are executed, the `.close` method on `file2` is automatically invoked. Let's verify this by trying to read from the file object again.<jupyter_code>file2.read()<jupyter_output><empty_output><jupyter_text>Reading a file line by lineFile objects provide a `readlines` method to read a file line-by-line.<jupyter_code>with open('./data/loans3.txt', 'r') as file3:
    file3_lines = file3.readlines()
file3_lines<jupyter_output><empty_output><jupyter_text>Processing data from filesBefore performing any operations on the data stored in a file, we need to convert the file's contents from one large string into Python data types. For the file `loans1.txt` containing information about loans in a CSV format, we can do the following:* Read the file line by line* Parse the first line to get a list of the column names or headers* Split each remaining line and convert each value into a float* Create a dictionary for each loan using the headers as keys* Create a list of dictionaries to keep track of all the loansSince we will perform the same operations for multiple files, it would be useful to define a function `read_csv`. We'll also define some helper functions to build up the functionality step by step. Let's start by defining a function `parse_header` that takes a line as input and returns a list of column headers.<jupyter_code>def parse_headers(header_line):
    return header_line.strip().split(',')<jupyter_output><empty_output><jupyter_text>The `strip` method removes any extra spaces and the newline character `\n`. The `split` method breaks a string into a list using the given separator (`,` in this case).<jupyter_code>file3_lines[0]
headers = parse_headers(file3_lines[0])
headers<jupyter_output><empty_output><jupyter_text>Next, let's define a function `parse_values` that takes a line containing some data and returns a list of floating-point numbers.<jupyter_code>def parse_values(data_line):
    values = []
    for item in data_line.strip().split(','):
        values.append(float(item))
    return values
file3_lines[1]
parse_values(file3_lines[1])
file3_lines[2]
parse_values(file3_lines[2])<jupyter_output><empty_output><jupyter_text>The code above leads to a `ValueError` because the empty string `''` cannot be converted to a float. We can enhance the `parse_values` function to handle this *edge case*. We will also handle the case where the value is not a float.<jupyter_code>def parse_values(data_line):
    values = []
    for item in data_line.st<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11612284069097889, 'fraction_numerical': 0.04858445297504799, 'mean_word_length': 5.359267734553776, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '304666', 'license': 'MIT', 'n_tokens_mistral': 2769, 'n_tokens_neox': 2522, 'n_words': 894, 'path': 'src/Lesson 3 - Numerical Computing with Numpy/python-os-and-filesystem.ipynb', 'repo_name': 'sahanashetty31/Data-Analysis-with-Python-Zero-to-Pandas', 'size': '8336'}