<jupyter_start><jupyter_text>Vectorized Feature Extraction Construct a DataFrame from the Raw Data Files<jupyter_code>folders = [f for f in os.listdir('data') if not os.path.isfile(os.path.join('data', f)) and 
           f.find('_MODEL') < 0]
def parse_metadata_from_file_name(file_name):
    subject_id = file_name.split('-')[-1].split('.txt')[0]
    gender = 'Male' if subject_id[0] == 'm' else 'Female'
    timestamp = ''
    for i in file_name.split('-')[1:7]:
        timestamp += i
    return subject_id, gender, timestamp
data = pd.DataFrame()
for folder in folders:
    files = [f for f in os.listdir('data/'+folder) if 
             os.path.isfile(os.path.join('data/'+folder, f))]
    for file_name in files:
        raw_data = pd.read_csv('data/{0}/{1}'.format(folder, file_name), sep=' ', header=0)
        raw_data.columns = ['X', 'Y', 'Z']
        subject_id, gender, timestamp = parse_metadata_from_file_name(file_name)
        raw_data['SubjectID'] = subject_id
        raw_data['Gender'] = gender
        raw_data['Timestamp'] = timestamp
        raw_data['ADL'] = folder

        # entityType, content, event, malware, eventTime= parse_metadata_from_file_name(file_name)
        # raw_data['entityType'] = entityType
        # raw_data['content'] = content
        # raw_data['event'] = event
        # raw_data['malware'] = malware
        data = data.append(raw_data, ignore_index=True)
print: data.shape
data.to_csv('data/adl_data.csv', sep=',', header=True, index=False)<jupyter_output><empty_output><jupyter_text>The Groupby Operation The pandas groupby operation lets you apply a function to subsets of a DataFrame without using a for loop. This is very similar to applying an aggregate function with a group by clause in SQL, if you are familiar with the database language. If you are not, now is a great time to start experimenting with set-based logic.For the [ADL data set](https://archive.ics.uci.edu/ml/datasets/Dataset+for+ADL+Recognition+with+Wrist-worn+Accelerometer), we want to calculate a set of features for every axis in every file. We will apply a groupby operation that aggregates by timestamp because it is unique to every file. Since we want the other indicators (subject_id and gender) and the label (ADL) to be in the output, and these are all guaranteed to be constant for a given timestamp, we will include those columns in the groupby specification. The order does not matter.<jupyter_code>grouped_data = data.groupby(['SubjectID', 'Gender', 'Timestamp', 'ADL'])<jupyter_output><empty_output><jupyter_text>Now we can easily generate a feature like the mean for every group in the set:<jupyter_code>grouped_data[['X', 'Y', 'Z']].mean().head(20)<jupyter_output><empty_output><jupyter_text>With the agg() function, you can generate many different features in one pass through the original DataFrame. I will leave the verification of increased efficiency to you as an exercise.<jupyter_code>from numpy import mean, std, min, max

features = grouped_data['X'].agg({
        'MeanX': mean,
        'StdX': std,
        'MinX': min,
        'MaxX': max,
        'LenX': len
    })
features.head(20)<jupyter_output><empty_output><jupyter_text>Define Feature Extraction Functions Now, we would like to do this for every acclerometer axis with only a single pass through the DataFrame. Defining a set of functions will allow us to do that while also maintaining control over the named columns. This might seem like a lot of overhead for such readily available built-in functions like mean and std, but it will be become very useful when you write your own feature extraction algorithms.<jupyter_code>def Length(data):
    """Returns the number of samples in a time series"""
    return len(data)

def Mean(data):
    """Returns the mean of a time series"""
    return data.mean()

def Std(data):
    """Returns the standard deviation a time series"""
    return data.std()

def Min(data):
    """Returns the mean of a time series"""
    return data.min()

def Max(data):
    """Returns the standard deviation a time series"""
    return data.max()
functions = [Mean, Std, Min, Max, Length]

features = grouped_data.agg({
        'X': functions, 
        'Y': functions,
        'Z': functions
    })

features.head(20)<jupyter_output><empty_output><jupyter_text>Put the Functions in a Module and Apply to the DataFrame If you find yourself repeatedly analyzing time series data, e.g. for wearable sensor application development, this technique will come in handy over and over again. I have a feature function library (features.py) that you are welcome to use and add to: https://github.com/mhbuehler/feature_extractionThis is how you import and apply the functions:<jupyter_code>from features import *

functions = [Mean, Median, Std, IQR, Length, Min, Max, DominantFrequency]
features = grouped_data.agg({
        'X': functions, 
        'Y': functions,
        'Z': functions
    })

features.head(20)<jupyter_output><empty_output><jupyter_text>Create the Flat Feature FileNow let's reformat the index and column names and save it to CSV. This file will be the source of training examples for our machine learning models.<jupyter_code>features.reset_index(inplace=True)
features.columns = ['{0}{1}'.format(a, b) if b else a for a, b in features.columns]

features.head(20)
features.to_csv('data/features.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10437523173896923, 'fraction_numerical': 0.00389321468298109, 'mean_word_length': 4.237864077669903, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '18974', 'license': 'MIT', 'n_tokens_mistral': 1626, 'n_tokens_neox': 1569, 'n_words': 586, 'path': 'Vectorized Feature Extraction.ipynb', 'repo_name': 'Simonlukyamuzi/improvednfe_research', 'size': '5394'}