<jupyter_start><jupyter_text>Accessing files directly from ADLS Gen-2 without mounting to DBFS In the below section we will see how to access ADLS Gen-2 storage accout directly using OAuth2.0 using Service Principal. We can access any ADLS Gen-2 account that the service principal has access to.<jupyter_code>spark.conf.set("fs.azure.account.auth.type.cookbookadlsgen2storage.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.cookbookadlsgen2storage.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.cookbookadlsgen2storage.dfs.core.windows.net", clientID)
spark.conf.set("fs.azure.account.oauth2.client.secret.cookbookadlsgen2storage.dfs.core.windows.net", clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.cookbookadlsgen2storage.dfs.core.windows.net", oauth2Endpoint)
#Directly accessing the ADLS Gen-2 fileystem and reading data from specific folder 
df_direct = spark.read.format("csv").option("header",True).schema(cust_schema).load("abfss://<EMAIL>data@<EMAIL>.net/Customer/csvFiles")
display(df_direct.limit(10))
parquetCustomerDestDirect = "abfss://<EMAIL>data@<EMAIL>.net/Customer/parquetFiles"
df_direct_repart=df_direct.repartition(10)
df_direct_repart.write.mode("overwrite").option("header", "true").parquet(parquetCustomerDestDirect)
display(dbutils.fs.ls(parquetCustomerDestDirect))
#Reading the parquet files created in preceding step
df_parquet = spark.read.format("parquet").option("header",True).schema(cust_schema).load("abfss://<EMAIL>@<EMAIL>/Customer/parquetFiles")
display(df_parquet.limit(10))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13416075650118203, 'fraction_numerical': 0.013002364066193853, 'mean_word_length': 7.72680412371134, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '395400', 'license': 'MIT', 'n_tokens_mistral': 577, 'n_tokens_neox': 565, 'n_words': 85, 'path': 'Chapter02/Code/2_3.Reading and Writing from and to ADLS Gen-2 File System.ipynb', 'repo_name': 'ssprsiva/cloned_databricks', 'size': '1787'}