I am a beginner at programming. I was able to build the thing below, which achieves what I want with a small dataset. With larger datasets, my RAM gets swamped bringing the computer to a halt (2014 Macbook Pro with 16GB RAM). Can I simplify my process somehow?# This code starts from a co-occurrence list of pairs with dates in the first column,     #like this:# # Jan-20; Monkey; Dog# Jan-21; Dog; Horse# Jan-22; Monkey; Cat# Jan-23; Monkey; Dog# # That is, these animals occurred together on these specific days.## This code cleans out the list, keeping only those lines that have co-occurrences # including an animal below a certain age in the dataset. Let's say 2 days. Like this:## Monkey occurred the first time on Jan-20, meaning that lines including 'Monkey' should # be kept only if they are dated Jan-# 20 or Jan-21. Cat occurred the first time on Jan-21,# meaning that lines including 'Cat' should be kept in the dataset only # if they are dated # Jan-21 or Jan-22.# Gather data on dates of earliest occurrence for all included animals# Set a time window# Extract lines based on 1 and 2import pandas# STEP 1: Gather data on earliest occurence ('entrydates') for all included items## Set column namescolnames=['Date','Item1','Item2']## Read csv adding column namesdata = pandas.read_csv('/Users/Simon/Dropbox/Work/Datasets/idlehash.csv', names=colnames)## Create a dataframe with info on dates for first columndatelist1 = data[['Date', 'Item1']]## Create a dataframe with info on dates for second columndatelist2 = data[['Date', 'Item2']]## Join the two dataframes into oneentrydates = datelist1.append(datelist2)## Melt the resulting dataframe into two columnsentrydates = pandas.melt(entrydates, id_vars='Date')[['Date','value']]## Sort the dataframe by Date and k eep only the earliest occurence of a value## drop_duplicates considers the column 'value' and keeps only first occurenceentrydates = entrydates.sort('Date').drop_duplicates(cols=['value'])# STEP 2: Calculate item ages in dataset at each co-occurence event## Create a dataframe with co-occurrence pairs and the entrydates of Item1 in each pairmatrix = pandas.merge(left=data, right=entrydates, left_on='Item1', right_on='value')## Create a dataframe with co-occurrence pairs and the entrydates of Item2 in each pairmatrix2 = pandas.merge(left=data, right=entrydates, left_on='Item2', right_on='value')## Rename some of the columns for claritymatrix = matrix.rename(columns={'Date_x':'co-oc date', 'Date_y':'entrydate of item 1',  'value':'Item1 (check)'})matrix2 = matrix2.rename(columns={'Date_x':'co-oc date', 'Date_y':'entrydate of item 2',   'value':'Item2 (check)'})## Sort themmatrix = matrix.sort(['co-oc date','entrydate of item 1'], ascending=False)matrix2 = matrix2.sort(['co-oc date','entrydate of item 2'], ascending=False)## Join themgorillaking = pandas.merge(matrix, matrix2, on='Item2', how='outer')## Build dataframe with selected columns from gorillakinggorillaking = pandas.concat([gorillaking['co-oc date_x'], gorillaking['Item1_x'],   gorillaking['Item2'], gorillaking['entrydate of item 1'], gorillaking['entrydate of item 2']],    axis=1, keys=['date', 'item1', 'item2', 'item1 birth', 'item2 birth'])## Add a column calculating the age of Item 1 on the occasion of each co-occurrencegorillaking['item1 age'] = gorillaking['date'] - gorillaking['item1 birth']## Add a column calculating the age of Item 2 on the occasion of each co-occurrencegorillaking['item2 age'] = gorillaking['date'] - gorillaking['item2 birth']# STEP 3: Select only the co-occurrences that happen in a certain window of time## Set a timewindowtimewindow = 7## Extract only the rows where the age of Item 1## is less than or equal to the user defined timewindow## That is: ('date' - 'item1 birth') <= timewindowmask = (gorillaking['date'] - gorillaking['item1 birth'] <= timewindow)keptpairs = gorillaking.loc[mask]## Output kept pairs to a filedataset = keptpairs[['item1', 'item2']]dataset.to_csv('/Users/Simon/Dropbox/Work/Datasets/#keptpairs.csv', sep='\t', encoding='utf-8',    index=False, header=False)## Print resultprint datasetStarted testing it line by line myself while checking the Activity Monitor in OSX, and it all works fine up until this line:## Join themgorillaking = pandas.merge(matrix, matrix2, on='Item2', how='outer')That's where RAM pressure starts mushrooming.
Simplifying Python Pandas code for selecting co-occurrences in a window of time
python;performance;beginner;pandas