After thoroughly profiling my program, I have been able to pinpoint that it is being slowed down by the vectorizer.I am working on text data, and two lines of simple tfidf unigram vectorization is taking up 99.2% of the total time the code takes to execute. I am extending the default TfidfVectorizer to add a stemmer.Here is a runnable example (this will download a 3mb training file to your disk, omit the urllib parts to run on your own sample):###################################### Loading Data#####################################import urllibfrom sklearn.feature_extraction.text import TfidfVectorizerimport nltk.stem  raw = urllib.urlopen(https://s3.amazonaws.com/hr-testcases/597/assets/trainingdata.txt).read()file = open(to_delete.txt,w).write(raw)###def extract_training():    f = open(to_delete.txt)    N = int(f.readline())    X = []    y = []    for i in xrange(N):        line  = f.readline()        label,text =  int(line[0]), line[2:]        X.append(text)        y.append(label)    return X,yX_train, y_train =  extract_training()    ############################################## Extending Tfidf to have only stemmed features#############################################english_stemmer = nltk.stem.SnowballStemmer('english')class StemmedTfidfVectorizer(TfidfVectorizer):    def build_analyzer(self):        analyzer = super(TfidfVectorizer, self).build_analyzer()        return lambda doc: (english_stemmer.stem(w) for w in analyzer(doc))tfidf = StemmedTfidfVectorizer(min_df=1, stop_words='english', analyzer='word', ngram_range=(1,1))############################################## Line below takes 6-7 seconds on my machine#############################################Xv = tfidf.fit_transform(X_train) I tried converting the list X_train into an np.array but there was no difference in performance.
Speeding up this tfidf vectorizer
python;performance