<jupyter_start><jupyter_text>File system interaction<jupyter_code>from datetime import datetime
import fileinput
import os
import pathlib
from pathlib import Path
import re
import shutil
import stat
import tempfile<jupyter_output><empty_output><jupyter_text>Contents of directories & properties Iterate over the contents of the current working directory, and for each file, print its name, size, and last access time.<jupyter_code>for item in Path.cwd().iterdir():
    if item.is_file() and item.name.endswith('.py'):
        lstat = item.lstat()
        access_time = datetime.strftime(datetime.fromtimestamp(lstat.st_atime),
                                        '%Y-%m-%d %H:%M:%S')
        print(f'{item.name}: {lstat.st_size} bytes, last accessed: {access_time}')<jupyter_output>subprocess_environment.py: 858 bytes, last accessed: 2018-11-12 13:25:04
sys_info.py: 2439 bytes, last accessed: 2018-11-12 13:32:16<jupyter_text>File permissions Create a new directory, don't throw an error if it exists. Note that the mode must be specified in octal.<jupyter_code>tmp_dir = Path.cwd() / 'tmp'
tmp_dir.mkdir(mode=0o700, exist_ok=True)<jupyter_output><empty_output><jupyter_text>More specifically, testing whether the current user can access a file is also quite useful.<jupyter_code>if pathlib.os.access(tmp_dir, pathlib.os.W_OK):
    print(f'current user can write in {tmp_dir.name}')<jupyter_output>current user can write in tmp<jupyter_text>Set the access mode a bit more liberal and check the permissions.<jupyter_code>tmp_dir.chmod(0o755)
print(oct(tmp_dir.lstat().st_mode))<jupyter_output>0o40755<jupyter_text>Detailed tests on access permissions can be written using the bitmasks defined in the `stat` module.<jupyter_code>mode = tmp_dir.lstat().st_mode
if mode & stat.S_IREAD and mode & stat.S_IWRITE and mode & stat.S_IEXEC:
    print(f'owner has full permissions on {tmp_dir.name}')
if not (mode & stat.S_IWGRP):
    print(f'group can not write in {tmp_dir.name}')<jupyter_output>group can not write in tmp<jupyter_text>Creating files & globbing Creating some file in the `tmp` directory can be done by creating a `Path` object for each file, and opening it directly.<jupyter_code>readme = tmp_dir / 'README.md'
with readme.open('w') as readme_file:
    print('# Contents', file=readme_file, end='\n\n')
    nr_files = 5
    for i in range(1, nr_files + 1):
        file = tmp_dir / f'data_{i:03d}.txt'
        with file.open('w') as data_file:
            print(f'data {i}', file=data_file)
        print(f'  * `{file.name}`: data file {i}', file=readme_file)
!cat tmp/README.md
for file in tmp_dir.glob('*.txt'):
    print(f'{file.name}: ', end='')
    with file.open('r') as data_file:
        data = data_file.readline().rstrip()
        print(data)<jupyter_output>data_003.txt: data 3
data_004.txt: data 4
data_001.txt: data 1
data_002.txt: data 2
data_005.txt: data 5<jupyter_text>Dissecting paths Splitting a path into its various functional parts is of course OS dependent, pathlib lets you do this in a platform independent way.  Files<jupyter_code>readme.name
readme.suffix
readme.parent
readme.drive<jupyter_output><empty_output><jupyter_text>Directories<jupyter_code>tmp_dir.name
tmp_dir.suffix
tmp_dir.parent<jupyter_output><empty_output><jupyter_text>Copying, moving, deleting Some operations can be done using `pathlib` such as moving or deleting files.<jupyter_code>!ls tmp
readme.unlink()
!ls tmp<jupyter_output>data_001.txt  data_002.txt  data_003.txt  data_004.txt	data_005.txt<jupyter_text>Renaming files is straightforward.<jupyter_code>for file in tmp_dir.glob('*.txt'):
    new_file = file.parent / file.name.replace('.txt', '.dat')
    file.rename(new_file)
!ls tmp<jupyter_output>data_001.dat  data_002.dat  data_003.dat  data_004.dat	data_005.dat<jupyter_text>`pathlib` has no facilities for copying files, however `shutil` does.<jupyter_code>for file in tmp_dir.glob('*.dat'):
    new_file = file.parent / file.name.replace('.dat', '.bak')
    shutil.copy(file, new_file)
!ls tmp<jupyter_output>data_001.bak  data_002.bak  data_003.bak  data_004.bak	data_005.bak
data_001.dat  data_002.dat  data_003.dat  data_004.dat	data_005.dat<jupyter_text>Move the `.bak` files into a subdirectory `bak`.<jupyter_code>bak_dir = tmp_dir / 'bak'
bak_dir.mkdir()
for file in tmp_dir.glob('*.bak'):
    file.rename(bak_dir / file.name)<jupyter_output><empty_output><jupyter_text>Print the sizes of all the files in `tmp`, and also compute the total size.<jupyter_code>total_size = 0
for dir_name, _, files in os.walk(tmp_dir, topdown=False):
    dir_size = 0
    print(f'{dir_name}:')
    for file in map(lambda f: Path(dir_name) / f, files):
        file_size = file.lstat().st_size
        dir_size += file_size
        print(f'\t{file.name}: {file_size} bytes')
    print(f'\tTotal: {dir_size} bytes')
    total_size += dir_size
print(f'Total: {total_size} bytes')<jupyter_output>/home/gjb/Documents/Projects/training-material/Python/SystemsProgramming/tmp/bak:
	data_002.bak: 7 bytes
	data_005.bak: 7 bytes
	data_001.bak: 7 bytes
	data_003.bak: 7 bytes
	data_004.bak: 7 bytes
	Total: 35 bytes
/home/gjb/Documents/Projects/training-material/Python/SystemsProgramming/tmp:
	data_002.dat: 7 bytes
	data_004.dat: 7 bytes
	data_005.dat: 7 bytes
	data_003.dat: 7 bytes
	data_001.dat: 7 bytes
	Total: 35 bytes
Total: 70 bytes<jupyter_text>`Path`'s `rmdir` method can be used to remove an empty directory, however, if the directory is not empty, the `rmtree` function in `shutil` can be used.<jupyter_code>shutil.rmtree(tmp_dir)<jupyter_output><empty_output><jupyter_text>Temporary files and directories, `inputfile` Especially when running application concurrently, it is important to ensure that names for temporary files are unique.  The module `tempfile` implements this. The temporary directory is created in `/tmp` and will be deleted with all its contents as soon as the end of the context is reached.  Files are created in that directory with random names, but will not be deleted. The default mode for temporary files is `w+b`, in this case we choose only `w` since the files are permanent, and we open them for reading later.<jupyter_code>nr_files = 3
nr_lines = 4
with tempfile.TemporaryDirectory(prefix='data_', dir='/tmp') as tmp_dir:
    print(f'director {tmp_dir} exists: {Path(tmp_dir).exists()}')
    # write the data files, use random names
    for file_nr in range(nr_files):
        with tempfile.NamedTemporaryFile(mode='w', prefix='data_', suffix='.txt',
                                         dir=tmp_dir, delete=False) as tmp_file:
            for i in range(nr_lines):
                print(f'line {i + 1} of {file_nr + 1}', file=tmp_file)
    # for each data file, print the name, and, indented, all the lines
    for tmp_file in Path(tmp_dir).iterdir():
        print(tmp_file.name)
        with tmp_file.open('r') as file:
            for line in file:
                print(f'\t{line.rstrip()}')
    # treat all the data files as one input, and compute the sum of all the numbers in the file
    total = 0
    for line in fileinput.FileInput(Path(tmp_dir).glob('*.txt')):
        total += sum(map(int, re.findall(r'\d+', line)))
    print(f'total = {total}')
print(f'director {tmp_dir} exists: {Path(tmp_dir).exists()}')<jupyter_output>director /tmp/data_dm6_6mf1 exists: True
data_xf0siwln.txt
	line 1 of 1
	line 2 of 1
	line 3 of 1
	line 4 of 1
data_rrsq3__a.txt
	line 1 of 2
	line 2 of 2
	line 3 of 2
	line 4 of 2
data_czdumvnm.txt
	line 1 of 3
	line 2 of 3
	line 3 of 3
	line 4 of 3
total = 54
director /tmp/data_dm6_6mf1 exists: False<jupyter_text>Miscellaneous Get the current working directory.<jupyter_code>Path.cwd()<jupyter_output><empty_output><jupyter_text>Get the current user's home directory.<jupyter_code>Path.home()<jupyter_output><empty_output><jupyter_text>Get the system's temporary directory.<jupyter_code>tempfile.gettempdir()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10870113493064312, 'fraction_numerical': 0.027868852459016394, 'mean_word_length': 4.5038167938931295, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '113038', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2936, 'n_tokens_neox': 2743, 'n_words': 773, 'path': 'Python/SystemsProgramming/filesystem_interaction.ipynb', 'repo_name': 'Gjacquenot/training-material', 'size': '7930'}