Q:

Dividing a big dataset into smaller sub-sets in python

I have a large time-based data set. it has 120,496 data points. I want to have smaller subsets each with ,say, 600 data points (the problem is with the final series, since the length is not a multiple of 600), I can do this with a lengthy and dirty for loop but I wanted to ask if there are any simple functions that would carry this task out? or if someone could direct towards the right direction as I am a bit lost w this.
I am assuming that this definitely does exist, since from what I know, some audio file analysis methodologies sort of divide the voice sample into smaller pieces.
Any help would be greatly appreciated.

A:

np.array_split(df, n) splits the dataframe into n sub-dataframes, while splitDataFrameIntoSmaller(df, chunkSize = n) splits the dataframe every chunkSize rows.
# input - df: a Dataframe, chunkSize: the chunk size
# output - a list of DataFrame
# purpose - splits the DataFrame into smaller chunks
df = pd.DataFrame([1,2,3,4,5,6,7,8,9,10,11], columns=['TEST'])
df_split = np.array_split(df, 3)

You get 3 sub-dataframes:
df_split[0] # 1, 2, 3, 4
df_split[1] # 5, 6, 7, 8
df_split[2] # 9, 10, 11

With:
def splitDataFrameIntoSmaller(df, chunkSize = 10): #10 for default 
    listOfDf = list()
    numberChunks = len(df) // chunkSize + 1
    for i in range(numberChunks):
        listOfDf.append(df[i*chunkSize:(i+1)*chunkSize])
    return listOfDf

df_split2 = splitDataFrameIntoSmaller(df, chunkSize = 3)

You get 4 sub-dataframes:
df_split2[0] # 1, 2, 3
df_split2[1] # 4, 5, 6
df_split2[2] # 7, 8, 9
df_split2[3] # 10, 11

I hope this help you.

