<jupyter_start><jupyter_text>Directories and PathsUp till now we've seen how to use python to solve simple math problems, store data in lists etc. But what if we wanted to make something more practical? Something that would help us crate scripts that automate some mundane tasks, such as copying, moving, renaming or deleting files and folders.An important aspect of file organization in operating systems are **paths**. Paths are strings that tell us where a certain file or folder can be found inside a the OS's directory structure. These files are typically organized in a hierarchical tree structure.For example in **Windows** systems, in the root of a boot partition one can find the following folders:- `\Program Files`: Is where the programs are installed- `\ProgramData`: Contains program data that are expected to be accessed by computer programs regardless of the user account in the context of which they run.- `\Users`: Is the user profile folder containing one subfolder for each user.- `\Windows`: Windows is installed in this folder.- `\System`: Is where the DLLs, that implement the core features of Windows are stored.  etc.A path in a windows like system looks like:C:\Users\Thanos\Projects\Python\This means that my user name is *`Thanos`* and that I have a folder called *`Python`* in another folder called *`Projects`* in my user directory. The *`C:`* part in the beginning means that my root directory is the partition *`C`*.In **UNIX** based systems, have their own directory structure:- `/bin`: Contains the binaries for certain fundamental utilities.- `/boot`: Contains all the files needed for booting.- `/home`: Contains the a subfolder for each user (similar to window's *`\Users`* folder).- `/lib`: Contains essential libraries needed for the programs in *`/bin`*.  etc.Unix type paths use regular slashes (`/`) to separate directories.  The equivalent path in Unix would be:/home/thanos/projects/python/The first slash in the beginning indicates the root directory.Paths are used extensively in computer science to represent the directory/file relationships common in modern operating systems, and are essential in the construction of Uniform Resource Locators (**URL**s). Resources can be represented by either **absolute** or **relative** paths.- An **absolute** or full path points to the same location in a file system, regardless of the current working directory. To do that, it must include start from root directory.- A **relative** path starts from some given working directory, avoiding the need to provide the full absolute path. A relative path **never** starts with a slash.From now on, we'll assume a UNIX based operating system. OSPython's OS module in Python provides a way of using operating system dependent functionality. Let's dive in.<jupyter_code>from __future__ import print_function
import os

os.getcwd()<jupyter_output><empty_output><jupyter_text>This command returns the path of the current working directory (in this case the directory from which the notebook was launched).If we wanted to, say, get a list of all items inside our working directory?<jupyter_code>os.listdir('.')<jupyter_output><empty_output><jupyter_text>The previous command needs a path as an argument: `os.listdir(path)`  To refer to the working directory we use the dot (`.`)Say we want to change the working directory to match the example above (`/home/thanos/projects/python/`).<jupyter_code>cwd = os.getcwd()  # store the previous WD so that we can return to it afterwards
os.chdir('/home/thanos/projects/python/')<jupyter_output><empty_output><jupyter_text>Oops... the directory doesn't exist. We'll need to create it.Let's change to a one we know exists.<jupyter_code>os.chdir('/home/thanos/')  # change directory
os.getcwd()                # to confirm
# if you want to make this work on your pc you need to change /home/thanos to a valid path in your pc<jupyter_output><empty_output><jupyter_text>Now we need to create a directory named `projects` in our working directory.<jupyter_code>os.mkdir('projects')  # make directory
os.chdir('projects')  # change directory
os.getcwd()           # confirm<jupyter_output><empty_output><jupyter_text>Note that we are using relative paths! This means that we don't have to write the full path each time (`/home/thanos/projects`). In short there are two options when wanting to use paths:- Absolute paths: We instruct python to create a directory in `/home/thanos/projects`.- Relative paths: We tell python to create a directory called `projects`. Where? Where I currently am working from (`/home/thanos/`)The difference between the two is that absolute paths **always** start with a slash (`/`).Another choice we have is to use our operating system's own commands. We can do this easily:```pythonos.system(command)```where command is a string containing the command we want to give to our os. For instance in UNIX systems `mkdir /path/name` is the command used to make a directory.<jupyter_code>os.system('mkdir python')  # this should work in windows systems too<jupyter_output><empty_output><jupyter_text>If these worked correctrly we should have created a directory with the path: `/home/thanos/projects/python`. We can easily confirm this.<jupyter_code>os.chdir('python')
os.getcwd()<jupyter_output><empty_output><jupyter_text>A last interesting function of the os module is `os.walk()`. This method generates the file names in a directory tree by walking the tree either top-down or bottom-up.```pythonos.walk(top[, topdown=True[, onerror=None[, followlinks=False]]])```- top: Each directory rooted at directory, yields 3-tuples, i.e., (dirpath, dirnames, filenames)- topdown: If optional argument topdown is True or not specified, directories are scanned from top-down. If topdown is set to False, directories are scanned from bottom-up.- onerror: This can show error to continue with the walk, or raise the exception to abort the walk.- followlinks: This visits directories pointed to by symlinks, if set to true.<jupyter_code>def recurrsive_listdir(path):
    for root, dirs, files in os.walk(path, topdown=False):
        for name in files:
            print(os.path.join(root, name))
        for name in dirs:
            print(os.path.join(root, name))
            
p = '/home/thanos/projects'
recurrsive_listdir(p)<jupyter_output>/home/thanos/projects/python<jupyter_text>The above code will scan all the directories and subdirectories in `/home/thanos/projects/` bottom-to-up. If `topdown` was set to `True`, the code will scan the directories top-to-bottom.The OS module also offers a lot more functionality like renaming or removing files, manipulating users, groups, environmental variables, process id's etc. Important! Don't use:  `from os import *`This will cause the `os.open()` function to override the built-in `open()` function which operates differently! os.pathThis is the os module's submodule that contains functions on pathnames.In order to showcase some of this module's functionality we'll first create a file. Remember we're still working off of  `/home/thanos/projects/python`.<jupyter_code>with open('test', 'w') as f:
    f.write('bla bla bla')<jupyter_output><empty_output><jupyter_text>Now let's confirm that actually created this file.<jupyter_code>os.listdir('.')<jupyter_output><empty_output><jupyter_text>Note that, we can't tell if this is a file, a directory or even a link. How can we make such a check?<jupyter_code>print('file:       ', os.path.isfile('test'))   # checks if 'test' is a file
print('directory:  ', os.path.isdir('test'))    # checks if 'test' is a directory
print('link:       ', os.path.islink('test'))   # checks if 'test' is a link
print('mount point:', os.path.ismount('test'))  # checks if 'test' is a mount point<jupyter_output>file:        True
directory:   False
link:        False
mount point: False<jupyter_text>We can learn other things about our file, such as it's size or the time from it's last modification:<jupyter_code>print('last modified:', os.path.getmtime('test'))
print('file size:    ', os.path.getsize('test'))<jupyter_output>last modified: 1553786881.7497778
file size:     11<jupyter_text>The first one returns the time of the last modification (in seconds) since the epoch. The second one is the filesize in bytes.We can also make a check if the path is an absolute path or not.<jupyter_code>print(os.path.isabs('/absolute/path'))
print(os.path.isabs('relative/path'))<jupyter_output>True
False<jupyter_text>This, in UNIX systems, just checks if there is a slash in front of the path. It **doesn't ** check if this actual path is valid.For this, we use the `os.path.exists(path)` function.<jupyter_code>print(os.path.exists('/home/thanos/projects/python/test'))
print(os.path.exists('/home/thanos/projects/python/wrong'))<jupyter_output>True
False<jupyter_text>We can also convert a relative to an absolute path.<jupyter_code>print(os.path.abspath('test'))<jupyter_output>/home/thanos/projects/python/test<jupyter_text>This just joins the **working directory** to the relative path: absolute = cwd + relative Alternatively, if we know where a file is located, we can join that with the filename to create the absolute path.<jupyter_code>file_location = '/an/absolute/path'
filename = 'a_file'
os.path.join(file_location, filename)<jupyter_output><empty_output><jupyter_text>Other functions of the `os.path` module can split paths into lists, join lists into paths etc. pathlibWhile the `os` module provides most of the functionality we might need, some commands might get a bit overwhelming. For example to get a list of the *absolute* paths in everything under a certain path, probably the easiest way would be:```pythonpath = '/a/path/to/list/its/contents'[os.path.join(path, x) for x in os.listdir(path)]```Ok, that's not so hard. But what if we wanted to list **only** the python scripts (the ones that end with *.py*) under that path.```pythonpath = '/a/path/to/list/its/contents'[os.path.join(path, x) for x in os.listdir(path) if x.endswith('.py')]```How about searhing all directories one level below `path` for python scripts.```pythonscripts = []for s in [os.path.join(path, d) for d in os.listdir(path) if os.path.isdir(os.path.join(path, d))]:    scripts += [os.path.join(path, s, x) for x in os.listdir(s) if x.endswith('.py')]```Things get out of hand really quick. **Pathlib** creates an easy way to handle paths in python. Pathlib serves as a higher-level API to `os.path` and allows for an easier and more intuituve handling of paths in python.<jupyter_code>from pathlib import Path

p = Path()

p<jupyter_output><empty_output><jupyter_text>Since we didn't initialize `p` with anything, it took the default value of the *cwd* (i.e. `'.'`). We can easily convert this to an absolute path if we wish.<jupyter_code>p.absolute()<jupyter_output><empty_output><jupyter_text>To use these paths in other functions you might have to convert them to strgins. For example:```pythonstr(p.absolute())```Pathlib allows us to join two or more paths very easily with the slash sign (`/`).<jupyter_code>p2 = p.absolute() / 'second/part/of/the/path' / 'final/path' / 'a_file.txt'
p2<jupyter_output><empty_output><jupyter_text>The equivalent if with `os.path` would be:```pythonp = '.'   assuming p is the relative path to the CWDp2 = os.path.join(os.path.absolute(p), 'second/part/of/the/path', 'final/path', 'a_file.txt')```Joining a list of directories in to a path is also very easy:```pythonPath.home().joinpath(*['projects', 'python'])   equivalent to the path we were before```Concerning a specific path we can easily useful parts from it, such as:<jupyter_code># Parent directory:
print('parent directory: ', p2.parent)

# File name (compete name)
print('file name:        ', p2.name)

# File name without suffix
print('file stem:        ', p2.stem)

# File's suffix
print('file suffix:      ', p2.suffix)<jupyter_output>parent directory:  /home/thanos/projects/python/second/part/of/the/path/final/path
file name:         a_file.txt
file stem:         a_file
file suffix:       .txt<jupyter_text>*pathlib* also offers a generator for accessing a path's parents one by one:<jupyter_code>for parent in p2.parents:
    print(parent)<jupyter_output>/home/thanos/projects/python/second/part/of/the/path/final/path
/home/thanos/projects/python/second/part/of/the/path/final
/home/thanos/projects/python/second/part/of/the/path
/home/thanos/projects/python/second/part/of/the
/home/thanos/projects/python/second/part/of
/home/thanos/projects/python/second/part
/home/thanos/projects/python/second
/home/thanos/projects/python
/home/thanos/projects
/home/thanos
/home
/<jupyter_text>Obviously we could also use the above as a list if we wish:```pythonlist(p2.parents)```We can also check for the same conditions like with `os.path`.<jupyter_code>p3 = p / 'test'

print('checks performed on \'p3\' (a file):')
print('Exists: ', p3.exists())
print('Is dir: ', p3.is_dir())
print('Is file:', p3.is_file())

print('\nchecks performed on \'p2\' (a non-existant file):')
print('Exists: ', p2.exists())
print('Is dir: ', p2.is_dir())
print('Is file:', p2.is_file())

print('\nchecks performed on \'p\' (the CWD):')
print('Exists: ', p.exists())
print('Is dir: ', p.is_dir())
print('Is file:', p.is_file())<jupyter_output>checks performed on 'p3' (a file):
Exists:  True
Is dir:  False
Is file: True

checks performed on 'p2' (a non-existant file):
Exists:  False
Is dir:  False
Is file: False

checks performed on 'p' (the CWD):
Exists:  True
Is dir:  True
Is file: False<jupyter_text>*pathlib* supports many more operations such as inspecting file **stats** (e.g. date & time modified/created, size) **renaming** files and many more.The most interesting functionality is the support of [glob patterns][1], which offer a series of **wildcard characters** that allow us to define filename patterns and list only files that match our description.  [1]: https://en.wikipedia.org/wiki/Glob_%28programming%29  We'll go back to the original directory, which has a few files we can play with. The most common wildcard character is `'*'` which allows for **any number of any characters**.<jupyter_code>tmp = p.absolute()  # store where we currently are so that we can return later
os.chdir(cwd)       # move back to original CWD
list(p.glob('*'))   # p points at '.' so this command lists all files in the CWD<jupyter_output><empty_output><jupyter_text>By using this character in a string, we can create all sorts of interesting patterns! For example:```pythonp.glob('*A')      lists all files ending with the letter 'A'p.glob('A*')      lists all files staring with the letter 'A'p.glob('*A*')     lists all files that contain the letter 'A' somewhere in their namesp.glob('A*B')     lists all files that start with an 'A' and end with a 'B'p.glob('*A*B*')   lists all files that contain both letters 'A' and 'B' (in that order) etc...```A few examples in practice:<jupyter_code>list(p.glob('2*'))  # lists all files starting with 2
list(p.glob('*.py'))  # lists all files ending with .py
list(p.glob('*module*'))  # lists all files containing the word 'module'
list(p.glob('*module*.py'))  # lists all '.py' files containing the word 'module'<jupyter_output><empty_output><jupyter_text>Another popular wildcard character is `'?'`, which substitutes **a single character**. For example:```pythonp.glob('?A')      lists all files with two letters, the second of which is 'A'p.glob('A?')      lists all files with two letters, the first of which is 'A'p.glob('*A*')     lists all files with three letters, the second of which is 'A' etc..```In practice:<jupyter_code>list(p.glob('?5*'))  # lists all files whose second character is 5  
list(p.glob('?5*'))  # lists all files whose second character is 5  
list(p.glob('?' * 35))  # lists all files with exactly 35 characters
list(p.glob('?' * 30 + '*'))  # lists all files with at least 30 characters<jupyter_output><empty_output><jupyter_text>Lastly there are the square brackets `[...]` which substitute certain characters only, but we won't go much into these.*pathlib* also allows for the recursive listing of items that match a pattern through `.rglob()`.<jupyter_code>up = p / '..'           # directrory that sits exactly one level up the hierarchy
list(up.rglob('*.py'))  # lists all .py files under path 'up' (looks in subdirectories too)<jupyter_output><empty_output><jupyter_text>The above would be equivalent to:```pythonlist(up.rglob('**/*.py'))```Let's move back to the previous directory we created so that we can continue with *shutil*.<jupyter_code>os.chdir(tmp)<jupyter_output><empty_output><jupyter_text>shutilThe shutil module offers a number of high-level operations on files and collections of files. In particular, functions are provided which support file copying and removal.  The shutil module helps you automate copying files and directories. This saves the steps of opening, reading, writing and closing files when there is no actual processing.```pythonshutil.copy(source, destination)```Copies file from path `source` to `destination`.<jupyter_code>os.getcwd()
import shutil

shutil.copy('test', 'test2')                                                            # relative paths
shutil.copy('/home/thanos/projects/python/test', '/home/thanos/projects/python/test3')  # absolute paths
os.listdir('.')<jupyter_output><empty_output><jupyter_text>There are also more specific `shutil` functions that allow us to copy a file's metadata, contents or permissions.Another useful feature of `shutil` is the `copytree` function.```pythonshutil.copytree(src, dst, symlinks=False, ignore=None)```- `src` is the source path.- `dst` is the destination path.- If `symlinks` is `True`, symbolic links in the source tree are represented as symbolic links in the new tree, but the metadata of the original links is not copied; if false or omitted, the contents and metadata of the linked files are copied to the new tree.- `ignore` is a list of paths (as returned from `os.listdir()`) that won't be copied.<jupyter_code>shutil.copytree('../../projects', '../projects2')
# copies /home/thanos/projects to /home/thanos/projects/projects2
recurrsive_listdir('..')
# prints all the directory tree along with it's contents in /home/thanos/projects<jupyter_output>../projects2/python/test3
../projects2/python/test
../projects2/python/test2
../projects2/python
../python/test3
../python/test
../python/test2
../projects2
../python<jupyter_text>First of all what we did with the `shutil.copytree()` command was to copy the whole `projects` directory to a subdirectory called `projects2` inside `projects`. Secondly, we saw a new way of referring to paths: the two dots `..`  The two dots refer to the parent directory, or the directory one level up in the hierarchy.So if our working directory is `/home/thanos/projects/python/`:- `..` is `/home/thanos/projects/`- `../..` is `/home/thanos/`etcWith shutil we can also move a file or directory from a location `src` to another `dst`.<jupyter_code>shutil.move('test3', '..')
# moves /home/thanos/projects/python/test3 to /home/thanos/projects/test3
print(os.listdir('..'))
# lists files and directories in /home/thanos/projects
print(os.listdir('.'))
# lists files and directories in /home/thanos/projects/python<jupyter_output>['test3', 'projects2', 'python']
['test', 'test2']<jupyter_text>We can confirm it has been moved.This module also offers tools for removing directory trees.<jupyter_code>shutil.rmtree('/home/thanos/projects')
# removes everything under /home/thanos/projects
os.chdir(cwd)  
# returns to the original directory<jupyter_output><empty_output><jupyter_text>We can confirm it's deleted.<jupyter_code>os.listdir('projects')
# The error is pretty self-explanatory.<jupyter_output><empty_output><jupyter_text>Another important module for automating computer tasks is the `zipfile` module that helps for compressing and extracting files to and from zip files. sysThis module provides helps with command line input/output operations.In shell environments there are three ways the shell interacts with the user. We call these I/O connections **standard streams**:- stdin: Is the standard input. This is what we type to the terminal.- stdout: Is the standard output. This is what we see as output.- stderr: Is the standard error. This is an output as well, but it is used for error messages. stdinWe have used `sys.stdin` in the past without knowing it! This module is called for the execution of `input()`.  stderr & stdoutThese two can be used to manipulate the two output channels.<jupyter_code>import sys
sys.stderr.write('This is an error message!\n')
sys.stderr.flush()
sys.stdout.write('This is an output!\n')<jupyter_output>This is an error message!
{'fraction_non_alphanumeric': 0.1096313453001351, 'fraction_numerical': 0.003860258637328701, 'mean_word_length': 4.821629213483146, 'pattern_counts': {'":': 0, '<': 121, '<?xml version=': 0, '>': 121, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '607934', 'license': 'MIT', 'n_tokens_mistral': 6178, 'n_tokens_neox': 5983, 'n_words': 2451, 'path': 'notebooks/14_os_sys_shutil.ipynb', 'repo_name': 'sniafas/python_ml_tutorial', 'size': '20724'}