Q:

select column of a dataframe using udf

I use spark-shell and want to create a dataframe (df2) from another dataframe (df1) using select and udf. But there is an error when I want to show the df2 ==> df2.show(1)
  var df1 = sql(s"select * from table_1")
  val slice = udf ((items: Array[String]) => if (items == null) items 
  else {
     if (items.size <= 20)
       items
     else
       items.slice(0, 20)
  })
  var df2 = df1.select($"col1", slice($"col2"))

and the df1 schema is:
scala> df1.printSchema
root
  |-- col1: string (nullable = true)
  |-- col2: array (nullable = true)
  |    |-- element: string (containsNull = true)

 scala> df2.printSchema
 root
   |-- col1: string (nullable = true)
   |-- UDF(col2): array (nullable = true)
   |    |-- element: string (containsNull = true)

error:
 Failed to execute user defined function($anonfun$1: (array<string>) => array<string>)

A:

Used Seq[String] instead of Array[String] in the udf and the issue is resolved.

