I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...Author provided: The goal of my ...The goal of my ... END OF TRANSCRIPTThe goal of my ... END, SPONSORED BY COMPANY XYZThe goal of my ... SPONSORED: COMPANY XYZ, All rights reserved, date: 10/21This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?
Detecting boilerplate in text samples
text mining;data cleaning;data wrangling
This might get you started. Phrase length is determined by the range() function. Basically this tokenizes and creates n-grams. Then it counts each token. Tokens with a high mean over all documents (occurs often across documents) is printed out in the last line.from sklearn.feature_extraction.text import CountVectorizerimport numpy as npimport nltktext = DESCRIPTION PROVIDED BY AUTHOR: The goal of my a...Author provided: The goal of my b...The goal of my c... END OF TRANSCRIPTThe goal of my d... END SPONSORED BY COMPANY XYZThe goal of my e... SPONSORED: COMPANY XYZ All rights reserved date: 10/21def todocuments(lines):    for line in lines:        words = line.lower().split(' ')        doc =         for n in range(3, 6):            ts = nltk.ngrams(words, n)            for t in ts: doc = doc +   + str.join('_', t)         yield doccv = CountVectorizer(min_df=.5)fit = cv.fit_transform(todocuments(text.splitlines()))vocab_idx = {b: a for a, b in cv.vocabulary_.items()}means = fit.mean(axis=0)arr = np.squeeze(np.asarray(means))[vocab_idx[idx] for idx in np.where(arr > .95)[0]]# ['goal_of_my', 'the_goal_of', 'the_goal_of_my']