df_y = sqlContext.read.json('/user/test.json')
udf_dt = func.udf(lambda x: datetime.strptime(x, '%Y%m%d%H%M%S'),
    TimestampType())
df = df_y.withColumn('datetime', udf_dt(df_y.date))
Get the distinct elements of each group by other field on a Spark 1.6 Dataframe