Q:

Change Hadoop version for Spark

How can I set a Hadoop version for the Spark application without submitting a jar and defining specific Hadoop binary? And is it even possible?
I am just not really sure how can Hadoop version be changed while submitting Spark application.
Something like this does not work:
  val sparkSession = SparkSession
    .builder
    .master("local[*]")
    .appName("SparkJobHDFSApp")
    .getOrCreate()
  sparkSession.sparkContext.hadoopConfiguration.set("hadoop.common.configuration.version", "2.7.4")

A:

It can't be. The Spark Master and Workers each have their own Hadoop JARs on the classpath with which your own application must be compatible with 

