Q:

Memory Blow Up when using Dask compute or persist with Dask Delayed

I am trying to process the data of several subjects all in one dataframe. There are >30 subjects and 14 computations per subject it is a large data set but any more than 5 blows up the memory on the scheduler node with out running any workers on the same node as the scheduler it has 128gb of memory? Any ideas how I can get around this or if im doing something wrong? code bellow. 
def channel_select(chn,sub):

    subject = pd.DataFrame(df.loc[df['sub'] == sub])
    subject['s0'] = subject[chn]
    val = []
    for x in range(13):
        for i in range(len(subject)):
            val.append(subject['s0'].values[i-x])
        name = 's' + str(x+1)
        subject[name] = val
        val = []
    return subject

subs = df['sub'].unique()
subs = np.delete(subs, [34,33])

for s in subs:
    for c in chn:
        chn_del.append(delayed(channel_select)(c,subs[s]))

results = e.persist(pred)

I have the code shown to run all the subjects but anymore than 5 at a time and I run out of memory 

A:

You're telling the computer to keep almost 1,000 GB of memory.
But you knew that already (:

