Q:

Pass list of column values to spark dataframe as new column

I am trying to add a new column to spark dataframe as below:
val abc = [a,b,c,d]   ---  List of columns

I am trying to pass above list of column values as new column to dataframe and trying to do sha2 on that new column and trying to do a varchar(64).
source = source.withColumn("newcolumn", sha2(col(abc), 256).cast('varchar(64)'))

It complied and the runtime error I am getting as:
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'abc' given input 
columns:

The expected output should be a dataframe with newcolum as column name and the column value as varchar64 with sha2 of concatenate of Array of string with ||.
Please suggest.

A:

We can use map and concat_ws || to create new column and apply sha2() on the concat data.
val abc = Seq("a","b","c","d")
val df=Seq(((1),(2),(3),(4))).toDF("a","b","c","d")
df.withColumn("newColumn",sha2(concat_ws("||",  abc.map(c=> col(c)):_*),256)).show(false)
//+---+---+---+---+----------------------------------------------------------------+
//|a  |b  |c  |d  |newColumn                                                       |
//+---+---+---+---+----------------------------------------------------------------+
//|1  |2  |3  |4  |20a5b7415fb63243c5dbacc9b30375de49636051bda91859e392d3c6785557c9|
//+---+---+---+---+----------------------------------------------------------------+

