Q:

How to split rows of a pyspark dataframe based on a list of row numbers?

I've created a pyspark dataframe that figuratively looks like this:-
>>> df

f1 | f2 | ... | fn | row_num
------------------------------
10 | 50 | ... |100 |    1
20 | 50 | ... |200 |    2
30 | 50 | ... |300 |    3
40 | 50 | ... |400 |    4
50 | 50 | ... |500 |    5

I also then created randomized list of row numbers as follows:-
my_list = [[2, 5],[4, 1, 3]]

Depending on the structure of this list, I wish to create pyspark dataframes as follows:-
>>> split_df[0]

    f1 | f2 | ... | fn | row_num
    ------------------------------
    20 | 50 | ... |200 |    2
    50 | 50 | ... |500 |    5

>>> split_df[1]

    f1 | f2 | ... | fn | row_num
    ------------------------------
    40 | 50 | ... |400 |    4
    10 | 50 | ... |100 |    1
    30 | 50 | ... |300 |    3

How do I split my df according to my_list of row numbers?
I do not wish to use df.RandomSplit() method.

A:

It seems like you want to split your DataFrame into a list, based on the values of my_list.
You can use pyspark.sql.Column.isin in a list comprehension:
from pyspark.sql.functions import col
split_df = [df.where(col('row_num').isin(x)) for x in my_list]

