Q:

Spark: Is merging transformation usefull?

I have a graph to itarate over. Sometimes I need to call several times in a row map() function. I have the possibility to merge this nodes and call only one map().
My question is whether it is usefull in terms of runtime? Is it expensive to call a Transformation (maybe overhead)?  Or does Spark optimize that with the help of the lineage graph?
Thnx!

A:

Most of the time the answer is negative. Chaining operations on the underlaying local collections should be more than enough. Details vary from language to language but you can assume that Spark does a pretty good job and avoids creating intermediate objects when possible.
However, there are cases when you can benefit from composing functions or chaining transformations directly using mapPartitions. Remember that each transformation extends the lineage, and as a result increases a cost of handling the DAG. Normally this can be addressed with cutting the lineage withs checkpoints, but it is not cheap.
So if you have long chain of calls:
rdd
  .map(f1)
  .map(f2)
  ...
  .map(fn)

you can rewrite it as:
rdd
  .mapPartitions(iter => iter
    .map(f1)
    .map(f2)
    ...
    .map(fn))

This is worth the shot only if the number of chained local transformations is large. With a few maps (filters, flatMaps, etc.) it will only make your code less readable without bringing any real improvements.

