<jupyter_start><jupyter_text>The filesystem module `datamol.fs` is not strictly related to molecule but it makes it very convenient to work with files both locally and remotely (AWS S3, GCS, HTTP, FTP, Git, etc) in a smooth and transparent manner. Under the hood the Datamol `fs` module is built on top of the library [fsspec](https://filesystem-spec.readthedocs.io/en/latest/).<jupyter_code>import tempfile

import datamol as dm<jupyter_output><empty_output><jupyter_text>Destructive path manipulationThe below examples are made locally for the purpose of the demo but all the function supports remote path (S3, GCS, etc). First let's get temp directory<jupyter_code>temp_dir = tempfile.mkdtemp()
dm.fs.exists(temp_dir)<jupyter_output><empty_output><jupyter_text>Create a directory and check if it has correctly been created.<jupyter_code>subdir_path = dm.fs.join(temp_dir, "subdir1", "subsubdir293")
dm.fs.mkdir(subdir_path, exist_ok=True)

dm.fs.exists(subdir_path)<jupyter_output><empty_output><jupyter_text>Copy a file from a source path to a destination path<jupyter_code>destination_path = dm.fs.join(subdir_path, "Compound_000000001_000500000.sdf.gz")
dm.fs.copy_file(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz",
    destination=destination_path,
    progress=True,
    force=True,
)<jupyter_output><empty_output><jupyter_text>Now, we would like to copy an full directory tree to a given destination.<jupyter_code>subdir2_path = dm.fs.join(temp_dir, "subdir2")
dm.fs.copy_dir(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/specifications/",
    destination=subdir2_path,
    progress=True,
)<jupyter_output><empty_output><jupyter_text>Let's check the files have been copied correctly.<jupyter_code>dm.fs.glob(dm.fs.join(subdir2_path, "**"))<jupyter_output><empty_output><jupyter_text>Non destructive path manipulation Retrieve the paths matching a path pattern<jupyter_code>dm.fs.glob("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/**")[:5]<jupyter_output><empty_output><jupyter_text>Get the name of the file or directory for a given path<jupyter_code>dm.fs.get_basename(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)
dm.fs.get_basename("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03")<jupyter_output><empty_output><jupyter_text>Get the extension of a given path<jupyter_code>dm.fs.get_extension("s3://an-s3-bucket-random/subdir1/subdir2/hello.txt")<jupyter_output><empty_output><jupyter_text>Check whether a file or a directory exists<jupyter_code>dm.fs.exists("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/")<jupyter_output><empty_output><jupyter_text>Check whether a file or a directory exists<jupyter_code>dm.fs.exists("gs://a-gcs-bucket-random/subdir1/subdir2/hello.txt")<jupyter_output><empty_output><jupyter_text>Check whether a path is a file and exists<jupyter_code>dm.fs.is_file(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)<jupyter_output><empty_output><jupyter_text>Check whether a path is a directory and exists<jupyter_code>dm.fs.is_dir("gs://a-gcs-bucket-random/subdir1/subdir2/")<jupyter_output><empty_output><jupyter_text>Check whether a path is local or remote<jupyter_code>dm.fs.is_local_path("/home/hello/a_subdir")<jupyter_output><empty_output><jupyter_text>Join paths together<jupyter_code>data_dir = "gs://awesome-data-bucket/data_dir"
filename = "molecules.sdf"
dm.fs.join(data_dir, filename)<jupyter_output><empty_output><jupyter_text>Get the size of a file (in byte)<jupyter_code>dm.fs.get_size(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz"
)<jupyter_output><empty_output><jupyter_text>Get the MD5 checksum of a file<jupyter_code>dm.fs.md5("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/README-Compound")<jupyter_output><empty_output><jupyter_text>Cache directoryIt's often convenient to get the path of a persistent cache folder. Unfortunately this path will change depending on the OS you're working on. Datamol offers a function to easily retrieve the path of the "official" cache directory on which it's running.<jupyter_code>dm.fs.get_cache_dir(app_name="datamol-demo", suffix="subdir1", create=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12271480804387569, 'fraction_numerical': 0.031078610603290677, 'mean_word_length': 6.987226277372263, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '629782', 'license': 'Apache-2.0', 'n_tokens_mistral': 1729, 'n_tokens_neox': 1579, 'n_words': 283, 'path': 'docs/tutorials/Filesystem.ipynb', 'repo_name': 'datamol-org/datamol', 'size': '4376'}