Q:

Application Default Credential Is Not Working on Google Container Engine

I am trying to use the Application Default Credentials on Container Engine, for the python IAM API.  But I am getting the following error pointing to insufficient authentication scopes.  My project has the IAM API enabled, and the code works locally.  So, I am not sure what I am missing.
My Error Message:
22:26:16.000
ERROR:root:<HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
{
 metadata: {…}   
 textPayload: "ERROR:root:<HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
"   
 insertId: "116wpgtg3n4zndx"   
 log: "simplekubeserver"   
}

22:26:16.000
HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
{
 metadata: {…}   
 textPayload: "HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
"   
 insertId: "116wpgtg3n4znej"   
 log: "simplekubeserver"   
}

My Code, which works locally but not on GKE:
from oauth2client.client import GoogleCredentials

def _iam_service():
  credentials = GoogleCredentials.get_application_default()
  return discovery.build(serviceName='iam', 
                         version='v1',
                         credentials=credentials)

def list_keys(project_id, service_account_id):
  full_name = 'projects/{0}/serviceAccounts/{1}'.format(project_id, service_account_id)
  keys = _iam_service().projects().serviceAccounts().keys()
  request = keys.list(name=full_name)
  return request.execute()

One thing I did to troubleshoot is to get which service account is being used.
print credentials.service_account_email

Locally, this shows the correct service account I am using.  Whereas on GKE, I get None, but expected something like 123456789-compute@developer.gserviceaccount.com
In the source code, I see:
_get_application_default_credential_GCE()          
_get_application_default_credential_GAE()

But nothing explicit for GKE.  So, I am assuming the one for GCE is used.
This doc mentions that this should be working on Container Engine.
Application Default Credentials are best suited for cases
when the call needs to have the same identity and authorization level
for the application independent of the user. This is the recommended
approach to authorize calls to Google Cloud Platform APIs, particularly
when you're building an application that is deployed to Google App
Engine, **Google Container Engine**, or Google Compute Engine virtual
machines.

A:

The IAM Service Accounts API requires either the https://www.googleapis.com/auth/iam or https://www.googleapis.com/auth/cloud-platform scope. The scopes on your GKE cluster's nodes are defined at cluster creation (or node pool creation) time. If you created your cluster through the Cloud Console or through gcloud, the default scopes do not include those.
In the Cloud Console, you can add the Cloud-Platform scope to a new cluster by clicking the "More" link and setting "Cloud Platform" to "Enabled"
If you are using gcloud, you can specify scopes by passing the --scopes flag to gcloud container clusters create or gcloud container node-pools create

