Q:

Hadoop, Spark: java.lang.NoSuchFieldError: TOKEN_KIND

I want to share an interesting error I've caught up recently:
Exception in thread "main" java.lang.NoSuchFieldError: TOKEN_KIND
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.handleKind(KMSClientProvider.java:166)
at org.apache.hadoop.security.token.Token.getRenewer(Token.java:351)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at org.apache.spark.deploy.security.HadoopFSCredentialProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$5$$anonfun$apply$1.apply$mcJ$sp(HadoopFSDelegationTokeProvider.scala:119)

I was trying to spark2-submit a job to a remote driver host on Cloudera cluster like this:
spark = SparkSession.builder
   .master("yarn")
   .config("cluster")
   .config("spark.driver.host", "remote_driver_host")
   .config("spark.yarn.keytab", "path_to_pricnipar.keytab")
   .config("spark.yarn.principal", "principal.name") \
   .config("spark.driver.bindAddress", "0.0.0.0") \
   .getOrCreate()

The Apache spark and Hadoop versions on Cloudera cluster are: 2.3.0 and 2.6.0 accordingly.

A:

So the cause of issue was quite trivial, it is spark local binaries vs remote spark driver version mismatch. 
Locally I had installed spark 2.4.5 and on Cloudera it was 2.3.0, after aligning the versions to 2.3.0, the issue resolved and the spark job completed successfully. 

