<jupyter_start><jupyter_text>Reading/Writing Parquet Files From/to OCI Object Storage with PandasThe setup for this notebook is simple. I used the `generalml_p37_cpu_v1` conda environment. This environment has ADS,pyarrow, pandas, snappy, and fastparquet pre-installed. I also upgraded the version of ADS. In this notebook ADS version 2.5.8 was used. To read/write files to Object Storage, use `ocifs` : * github.com/oracle/ocifs* docs: https://docs.oracle.com/en-us/iaas/tools/ocifs-sdk/latest/unix-operations.html<jupyter_code>#!pip install oracle-ads --upgrade
import ads 
from ads.common.auth import default_signer
import pandas as pd 
import fsspec
from ocifs import OCIFileSystem

# Using resource principal auth method: 
print(ads.__version__)
ads.set_auth(auth="resource_principal")
# object storage bucket + data 
# this bucket is publicly available 
bucket = "hosted-ds-datasets"
namespace = "bigdatadatasciencelarge"<jupyter_output><empty_output><jupyter_text>Single Large File<jupyter_code># ~ 440 MB, 14M rows. 
large_files = ["nyc_tlc/2009/01/data.parquet"] # NYC Taxi dataset 
for f in large_files: 
    df = pd.read_parquet(f"oci://{bucket}@{namespace}/{f}", 
                     storage_options=default_signer(),
                     engine="pyarrow")
    print(f"file {f}")
    print(df.head())
    print(f"size {df.shape}")<jupyter_output><empty_output><jupyter_text>Multiple Large Files<jupyter_code>from oci.auth.signers import get_resource_principals_signer
# Using resource principal for authn. 
fs = OCIFileSystem(signer=get_resource_principals_signer())
# This is a small sample of the overall NYC Taxi dataset. There are 4 files in total, each of size ~ 450MB. 
relevant_files = fs.ls(f"{bucket}@{namespace}/nyc_tlc/2009/")
%%time 
# this operation takes about 50 secs: 
df = pd.concat((pd.read_parquet(f"oci://{f}/data.parquet", storage_options=default_signer(), engine="pyarrow") for f in relevant_files), 
               ignore_index=True)
# ~ 56M rows 
df.shape<jupyter_output><empty_output><jupyter_text>Write Parquet Files to Object Storage:<jupyter_code>%%time 

# Writing the dataframe as a parquet file to Object Storage. 
# Insert the name of your bucket and namespace you want to write parquet. 
# this operation takes about 2 mins
your_bucket = ""
your_namespace = ""

df.to_parquet(f"oci://{your_bucket}@{your_namespace}/taxi-data.parquet", 
                     storage_options=default_signer())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10583941605839416, 'fraction_numerical': 0.012165450121654502, 'mean_word_length': 4.77751756440281, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '454503', 'license': 'Apache-2.0', 'n_tokens_mistral': 845, 'n_tokens_neox': 799, 'n_words': 211, 'path': 'parquet/parquet-from-oss.ipynb', 'repo_name': 'jrgauthier01/oci-ds-samples', 'size': '2466'}