Q:

create scala dataframe out of custom object

I am a newbie in scala. I will try to be as clear as possible.I have the following code:
case class Session (bf: Array[File]) 
case class File(s: s, a: Option[a], b: Option[b], c: Option[c])

case class s(s1:Int, s2:String) 
case class a(a1:Int, a2:String) 
case class b(b1:Int, b2:String)  
case class c(c1:Int, c2:String)

val x = Session(...) // some values here, many session objects grouped in a  dataset collection i.e. Dataset[Sessions]

I want to know how to create dataframes from a Dataset[Sessions]. I do not 
know how to manipulate such a complex structure. 
how to create a dataframe from Dataset[sessions] only containing the custom
object "a".

Thank you

A:

A Spark DataSet works much like a regular Scala collection. It has a toDF() operation to create a DataFrame out of it. Now you just need to extract the right data out of it using different transformations.

flatMap it into a DataSet of File
filter every File for a non-empty a
map every remaining File to a
call toDF() to create a DataFrame

In code this would be:
val ds: DataSet[Session] = ...
ds.flatMap(_.bf)
  .filter(_.a.isDefined)
  .map(_.a.get)
  .toDF()

In Scala you can also combine the filter and map to a collect, which would lead to the following code:
ds.flatMap(_.bf).collect({ case File(_, Some(a), _, _) => a }).toDF()

