Q:

spark cross join eliminate duplicate

I'm trying to figure out how to eliminate duplicates in cross join given two dataset.
For example client (clientDs)
clientDs = sc.parallelize(List('c1', 'c2', 'c3'))

clientMatrixDs = clientDs.join(clientDs)

clientProduct.show()

 -- output

 c1, c1
 c1, c2
 c1, c3
 c2, c1
 c2, c2
 c2, c3
 c3, c1
 c3, c2
 c3, c3

In this case,
both (c1,c2) and (c2,c1) are duplicates and I need to de-dupe it
I'm not sure how to do this
Looking for some ideas here.

A:

In your example, you have a bunch of Tuple2[String, String]s, and the problem is that you want an equality that Tuple2 just doesn't have where (a, b) == (b, a). This is why distinct doesn't work for you. You therefore have to provide your own custom equality.
The thing is you don't want to override Tuple2's version of equals because that can be dangerous, so you can provide a custom equals somewhere:
def customEquals(tuple1: (String, String), tuple2: (String, String)) = {
  tuple1 == tuple2 || (tuple1._1 == tuple2._2 && tuple1._2 == tuple2._1)
}

Then you can use this function in a filter to get rid of your custom definition of duplicates:
val deduped = clientMatrixDs.filter {
  var seq = Seq.empty[(String, String)]
  tuple =>
    if (seq.exists(customEquals(tuple, _))) {
      false
    } else {
      seq :+= tuple
      true
    }
}

