Q:

spark scala merge multi rdd by key

I have got multi rdds
genderRDD
(1713926427,{gender={f=3327, m=1945, unknown=897}})

And
actionRDD
(1713926427,{actionType={repost=2927, comment=2345, like=897}})

And
deviceRDD
(1713926427,{deviceType={iphone=2999, android=12321}})

They are of the format RDD(id, HashMap[String, HashMap[String, Integer]])
I want to merge the two rdds by id which should generate:
(1713926427,{gender={f=3327, m=1945, unknown=897},actionType={repost=2927, comment=2345, like=897},device={iphone=2999, android=12321}})

So that I can store it to the database. How to do that typically? 

A:

I'm assuming that {deviceType={iphone=2999, android=12321}} represents a Map[String, Map[String, Int]], in that case - you should simply use joins and then some simple mapping to "flatten" the result:
// some sample data:
val rdd1 = sc.parallelize(Seq((1713926427, Map("gender" -> Map("f" -> 3327, "m" ->1945, "unknown" -> 897)))))
val rdd2 = sc.parallelize(Seq((1713926427, Map("actionType" -> Map("repost" -> 2927, "comment" -> 2345, "like" -> 897)))))
val rdd3 = sc.parallelize(Seq((1713926427, Map("deviceType" -> Map("iphone" -> 2999, "android" -> 12321)))))

// join all three RDDs and map to flatten the value:
val result = rdd1
  .join(rdd2)
  .join(rdd3)
  .map { case (id, ((gender, action), device)) => (id, (gender ++ action ++ device)) }

// result has type RDD[(Int, Map[String, Map[String, Int]])]
result.foreach(println)
// prints:
// (1713926427,Map(gender -> Map(f -> 3327, m -> 1945, unknown -> 897), actionType -> Map(repost -> 2927, comment -> 2345, like -> 897), deviceType -> Map(iphone -> 2999, android -> 12321)))

