num_rows = 100000
all = list(range(num_rows))
some = np.random.choice(all, replace=False, size=num_rows / 2)
some.sort()
trainer_df = pd.read_csv(path, skiprows=some)
rest = [i for i in all if i not in some]
rest.sort()
df = pd.read_csv(path, skiprows=rest)
Splitting a large Pandas Dataframe with minimal memory footprint