We built on this MapPartitions operation and create a SparkML transformer, called the “CNTKModel”, that distributes, loads, and executes arbitrary CNTK computation graphs inside of SparkML pipelines. We expose parameters that give users control over the CNTK model object, mini-batch size, and evaluation of subgraphs of the CNTK graph. The ultimate is useful for transfer learning, where one is interested in inner activations of a network [1]}
