Q:

cogroup IN PySpark

The tutorial suggests this:
>>> x = sc.parallelize([("a", 1), ("b", 4)])
>>> y = sc.parallelize([("a", 2)])
>>> sorted(x.cogroup(y).collect())
[('a', ([1], [2])), ('b', ([4], []))]

However, on running this I get the following output:
('a', (<pyspark.resultiterable.ResultIterable object at 0x1d8b190>, <pyspark.resultiterable.ResultIterable object at 0x1d8b150>))
('b', (<pyspark.resultiterable.ResultIterable object at 0x1d8b210>, <pyspark.resultiterable.ResultIterable object at 0x1d8b1d0>))

This has 3 level of nesting, if I store the output in 'r' and do this:
for i in r:
     for j in i[1]:
         print list(j)

I get the correct cogrouped numbers:
1) Why does Cogroup not return numbers like rightjoin/leftouterjoin etc. in PySpark?
2) Why can't I replicate the example on my PySpark shell?

A:

Easy answer, because that's what cogroup is supposed to return
  /**
   * For each key k in `this` or `other`, return a resulting RDD that contains a tuple with the
   * list of values for that key in `this` as well as `other`.
   */
  def cogroup[W](other: RDD[(K, W)]): RDD[(K, (Iterable[V], Iterable[W]))]

Joins in spark are actually implemented with cogroup, basically the join just breaks the iterables from cogroup into tuples. Here is the implantation of join from spark.
  def join[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (V, W))] = {
    this.cogroup(other, partitioner).flatMapValues { case (vs, ws) =>
      for (v <- vs; w <- ws) yield (v, w)
    }
  }

As for the slight difference in interpreter output (keep in mind the output is the same except the pyspark Iterable doesn't show it's contents), I can't be sure unless I see the tutorial. The tutorial may be showing the output to be clearer even if that's not actually what appears. One more thing I ran a similar script in the scala shell, and it shows all the output.
Array((a,(ArrayBuffer(1),ArrayBuffer(2))), (b,(ArrayBuffer(4),ArrayBuffer())))

