new_col = sqlContext.createDataFrame(zip(list(range(1, 5)), [20] * 4), ('rn',
    'new_col'))
sparkdf.registerTempTable('df')
sparkdf_indexed = sqlContext.sql(
    'SELECT row_number() OVER (ORDER BY a, b, c) AS rn, * FROM df')
sparkdf_indexed.join(new_col, new_col.rn == sparkdf_indexed.rn).drop(new_col.rn
    )
How do you add a numpy.array as a new column to a pyspark.SQL DataFrame?