natural language toolkit positive naive bayes classifier c 2012 nltk project alessandro presta alessandro prestagmail com url https www nltk org for license information see license txt a variant of the naive bayes classifier that performs binary classification with partiallylabeled training sets in other words assume we want to build a classifier that assigns each example to one of two complementary classes e g male names and female names if we have a training set with labeled examples for both classes we can use a standard naive bayes classifier however consider the case when we only have labeled examples for one of the classes and other unlabeled examples then assuming a prior distribution on the two labels we can use the unlabeled set to estimate the frequencies of the various features let the two possible labels be 1 and 0 and let s say we only have examples labeled 1 and unlabeled examples we are also given an estimate of p1 we compute pfeature1 exactly as in the standard case to compute pfeature0 we first estimate pfeature from the unlabeled set we are assuming that the unlabeled examples are drawn according to the given prior distribution and then express the conditional probability as pfeature pfeature1 p1 pfeature0 p0 example from nltk classify import positivenaivebayesclassifier some sentences about sports sportssentences the team dominated the game they lost the ball the game was intense the goalkeeper catched the ball the other team controlled the ball mixed topics including sports varioussentences the president did not comment i lost the keys the team won the game sara has two kids the ball went off the court they had the ball for the whole game the show is over the features of a sentence are simply the words it contains def featuressentence words sentence lower split return dict containss w true for w in words we use the sports sentences as positive examples the mixed ones ad unlabeled examples positivefeaturesets mapfeatures sportssentences unlabeledfeaturesets mapfeatures varioussentences classifier positivenaivebayesclassifier trainpositivefeaturesets unlabeledfeaturesets is the following sentence about sports classifier classifyfeatures the cat is on the table false what about this one classifier classifyfeatures my team lost the game true positive naive bayes classifier param positivefeaturesets an iterable of featuresets that are known as positive examples i e their label is true param unlabeledfeaturesets an iterable of featuresets whose label is unknown param positiveprobprior a prior estimate of the probability of the label true default 0 5 count up how many times each feature value occurred in positive examples count up how many times each feature value occurred in unlabeled examples if a feature didn t have a value given for an instance then we assume that it gets the implicit value none create the plabel distribution create the pfvallabel fname distribution todo we need to add some kind of smoothing here instead of setting negative probabilities to zero and normalizing demo natural language toolkit positive naive bayes classifier c 2012 nltk project alessandro presta alessandro presta gmail com url https www nltk org for license information see license txt a variant of the naive bayes classifier that performs binary classification with partially labeled training sets in other words assume we want to build a classifier that assigns each example to one of two complementary classes e g male names and female names if we have a training set with labeled examples for both classes we can use a standard naive bayes classifier however consider the case when we only have labeled examples for one of the classes and other unlabeled examples then assuming a prior distribution on the two labels we can use the unlabeled set to estimate the frequencies of the various features let the two possible labels be 1 and 0 and let s say we only have examples labeled 1 and unlabeled examples we are also given an estimate of p 1 we compute p feature 1 exactly as in the standard case to compute p feature 0 we first estimate p feature from the unlabeled set we are assuming that the unlabeled examples are drawn according to the given prior distribution and then express the conditional probability as p feature p feature 1 p 1 p feature 0 p 0 example from nltk classify import positivenaivebayesclassifier some sentences about sports sports_sentences the team dominated the game they lost the ball the game was intense the goalkeeper catched the ball the other team controlled the ball mixed topics including sports various_sentences the president did not comment i lost the keys the team won the game sara has two kids the ball went off the court they had the ball for the whole game the show is over the features of a sentence are simply the words it contains def features sentence words sentence lower split return dict contains s w true for w in words we use the sports sentences as positive examples the mixed ones ad unlabeled examples positive_featuresets map features sports_sentences unlabeled_featuresets map features various_sentences classifier positivenaivebayesclassifier train positive_featuresets unlabeled_featuresets is the following sentence about sports classifier classify features the cat is on the table false what about this one classifier classify features my team lost the game true positive naive bayes classifier param positive_featuresets an iterable of featuresets that are known as positive examples i e their label is true param unlabeled_featuresets an iterable of featuresets whose label is unknown param positive_prob_prior a prior estimate of the probability of the label true default 0 5 count up how many times each feature value occurred in positive examples count up how many times each feature value occurred in unlabeled examples if a feature didn t have a value given for an instance then we assume that it gets the implicit value none create the p label distribution create the p fval label fname distribution todo we need to add some kind of smoothing here instead of setting negative probabilities to zero and normalizing demo
from collections import defaultdict

from nltk.classify.naivebayes import NaiveBayesClassifier
from nltk.probability import DictionaryProbDist, ELEProbDist, FreqDist






class PositiveNaiveBayesClassifier(NaiveBayesClassifier):
    @staticmethod
    def train(
        positive_featuresets,
        unlabeled_featuresets,
        positive_prob_prior=0.5,
        estimator=ELEProbDist,
    ):
        
        positive_feature_freqdist = defaultdict(FreqDist)
        unlabeled_feature_freqdist = defaultdict(FreqDist)
        feature_values = defaultdict(set)
        fnames = set()

        
        num_positive_examples = 0
        for featureset in positive_featuresets:
            for fname, fval in featureset.items():
                positive_feature_freqdist[fname][fval] += 1
                feature_values[fname].add(fval)
                fnames.add(fname)
            num_positive_examples += 1

        
        num_unlabeled_examples = 0
        for featureset in unlabeled_featuresets:
            for fname, fval in featureset.items():
                unlabeled_feature_freqdist[fname][fval] += 1
                feature_values[fname].add(fval)
                fnames.add(fname)
            num_unlabeled_examples += 1

        
        
        for fname in fnames:
            count = positive_feature_freqdist[fname].N()
            positive_feature_freqdist[fname][None] += num_positive_examples - count
            feature_values[fname].add(None)

        for fname in fnames:
            count = unlabeled_feature_freqdist[fname].N()
            unlabeled_feature_freqdist[fname][None] += num_unlabeled_examples - count
            feature_values[fname].add(None)

        negative_prob_prior = 1.0 - positive_prob_prior

        
        label_probdist = DictionaryProbDist(
            {True: positive_prob_prior, False: negative_prob_prior}
        )

        
        feature_probdist = {}
        for fname, freqdist in positive_feature_freqdist.items():
            probdist = estimator(freqdist, bins=len(feature_values[fname]))
            feature_probdist[True, fname] = probdist

        for fname, freqdist in unlabeled_feature_freqdist.items():
            global_probdist = estimator(freqdist, bins=len(feature_values[fname]))
            negative_feature_probs = {}
            for fval in feature_values[fname]:
                prob = (
                    global_probdist.prob(fval)
                    - positive_prob_prior * feature_probdist[True, fname].prob(fval)
                ) / negative_prob_prior
                
                
                negative_feature_probs[fval] = max(prob, 0.0)
            feature_probdist[False, fname] = DictionaryProbDist(
                negative_feature_probs, normalize=True
            )

        return PositiveNaiveBayesClassifier(label_probdist, feature_probdist)







def demo():
    from nltk.classify.util import partial_names_demo

    classifier = partial_names_demo(PositiveNaiveBayesClassifier.train)
    classifier.show_most_informative_features()