Q:

Error when trying to create n-grams from a column DataFrame

Given a DataFrame that has a single column Text:
      Text
0     chest  pain  nstemi  this  84-year  old  man  present  on  26/5  with  
      chest  pain  associate  with  profuse  sweating  and  nausea

I would like to create two new columns with unigrams and bigrams generated for the previous DataFrame.
This is the method I am using for generating the ngrams:
    def generate_ngrams(self, s, n):
        # Convert to lowercases
        s = s.lower()

        # Replace all none alphanumeric characters with spaces
        s = re.sub(r'[^a-zA-Z0-9\s]', ' ', s)

        # Break sentence in the token, remove empty tokens
        tokens = [token for token in s.split(" ") if token != ""]

        # Use the zip function to help us generate n-grams
        # Concatentate the tokens into ngrams and return
        ngrams = zip(*[tokens[i:] for i in range(n)])
        return [" ".join(ngram) for ngram in ngrams]

And that's how I am trying to populate my DataFrame:
    for index, row in featuresDF.iterrows():
        featuresDF.at[index, '1-gram'] = generate_ngrams(infoDF.at[index, 'Text'], 1)
        featuresDF.at[index, '2-gram'] = generate_ngrams(infoDF.at[index, 'Text'], 2)

When I run it, I get the following error : ValueError: setting an array element with a sequence.
Here's the Traceback:
Traceback (most recent call last):

  File "<ipython-input-64-e014e2e1c7e2>", line 3, in <module>
    featuresDF.at[index, '1-gram'] = featureExtraction.generate_ngrams(infoDF.at[index, 'Text'], 1)

  File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\indexing.py", line 2287, in __setitem__
    self.obj._set_value(*key, takeable=self._takeable)

  File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py", line 2815, in _set_value
    engine.set_value(series._values, index, value)

  File "pandas/_libs/index.pyx", line 95, in pandas._libs.index.IndexEngine.set_value

  File "pandas/_libs/index.pyx", line 106, in pandas._libs.index.IndexEngine.set_value

I understand that it is an issue when I am assigning the unigrams and bigrams to the DataFrame, right? But I am not sure how to solve it. Thanks!

A:

generate_ngrams() should return a string, but it returns a list like:
['chest', 'pain', .....] 

Before returning list you cant convert it to comma separated string like:
chest,pain, .....

by adding these lines:
ngramList = [" ".join(ngram) for ngram in ngrams]        
return ','.join(ngramList)

Additionally you can use CountVectorizer for finding N-grams:
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(ngram_range=(2,2)) # 2,2 means 2-gram, 1,1 is unigram
corpus = ['the boy is gone !']
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names()) # this will print the list containing gram values

