<jupyter_start><jupyter_text>Runs this to start from scratchBoth should return an error if no credentials were previously set and your are using the service account of the instance.<jupyter_code>!gcloud auth revoke --quiet
!gcloud auth application-default revoke --quiet<jupyter_output>[1;31mERROR:[0m (gcloud.auth.application-default.revoke) Application Default Credentials have not been set up, nothing was revoked.<jupyter_text>Authentication 1. **As a developer, I want to interact with GCP via gcloud.**    ```gcloud auth login``` (run from a Notebook terminal)    This obtains your credentials via a web flow and stores them in ```/root/.config/gcloud/credentials.db``` and for backward compatibility in ```/root/.config/gcloud/legacy_credentials/[YOUR_EMAIL]/adc.json```    Now:    - **gcloud** commands runs from the Notebook's cells finds your credentials automatically.     - **Other code or SDK** (Python, Java,...) not automatically picks up those credentials.    Reference: https://cloud.google.com/sdk/gcloud/reference/auth/login1. **As a developer, I want my code to interact with GCP via SDK.**    ```gcloud auth application-default login``` (run using the GCP option in the navigation menu)    This obtains your credentials via a web flow and stores them in ```/root/.config/gcloud/application_default_credentials.json```.     Now:    - **Other code or SDK** (Python, Java,...) finds the credentials automatically.     - Can run code locally which would normally run on a server without the need of a credentials file.    Reference: https://cloud.google.com/sdk/gcloud/reference/auth/application-default/loginFor more information, you can read the Google documentation or this excellent [blog post](https://jpassing.com/2020/01/14/google-application-default-credentials-vs-your-personal-gcloud-credentials/).  Authenticate code running in the Notebook<jupyter_code># General
import google.auth
credentials, project_id = google.auth.default()
# Transparent for google.cloud libraries
from google.cloud import bigquery
client = bigquery.Client()
# If you try to run a query, this gets updated with values.
client.__dict__['_credentials'].__dict__<jupyter_output><empty_output><jupyter_text>---**NOTE**For BigQuery, if you run a query using the following, your identity should have the following IAM roles or similar:- *roles/bigquery.jobUser* (Lower resource is Project) that includes the *bigquery.jobs.create* permission.- *roles/bigquery.dataViewer* (Lower resource is Dataset) that includes *bigquery.tables.getData* permission.```pyquery_job = client.query(QUERY) rows = query_job.result()```---  Authenticate SparkYou can authenticate Spark using the credentials file or its content. Although you could use the file directly, workers would not have it locally because ```gcloud auth application-default login``` runs only for the Master. It means that the application_default_credentials.json file is only created on the Master node.We have 3 options:- **Option 1 [Recommended]: Read the file and pass the value as a string.**- Option 2: Have the add-on to write the file to the master and workers. Requires proper permissions.- Option 3: Manually copy the file using a gcloud scp for example. Requires proper firewall access.<jupyter_code>import base64

CREDENTIALS_FILE = "/root/.config/gcloud/application_default_credentials.json"

def get_credentials_text():
    if not os.path.isfile(CREDENTIALS_FILE):
        print("\x1b[31m\nNo credentials defined. Run gcloud auth application-default login.\n\x1b[0m")
        return
    
    return open(CREDENTIALS_FILE, "r").read()

credentials_txt = get_credentials_text()
credentials_b64 = base64.b64encode(credentials_txt.encode('utf-8')).decode('utf-8')

# Can not have both credentialsFile and credentials set.
spark.conf.unset("credentialsFile")
spark.conf.set("credentials", credentials_b64)

print("\x1b[32m\nSpark is now authenticated on this Master node.\n\x1b[0m")<jupyter_output>[32m
Spark is now authenticated on this Master node.
[0m
{'fraction_non_alphanumeric': 0.09808314662683594, 'fraction_numerical': 0.010704505850136919, 'mean_word_length': 4.988077496274218, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '400294', 'license': 'Apache-2.0', 'n_tokens_mistral': 1156, 'n_tokens_neox': 1105, 'n_words': 418, 'path': 'dataproc-hub-example/build/infrastructure-builder/mig/files/gcs_working_folder/examples/Environment Checks/00 - Authentication.ipynb', 'repo_name': 'sbadame/ai-notebooks-extended', 'size': '4017'}