<jupyter_start><jupyter_text>Document classifier Praktisch wenn eine Menge Dokumente sortiert werden muss  Daten- Wir brauchen zuerst daten um unser Modell zu trainieren<jupyter_code>#!pip3 install -U textblob
from textblob.classifiers import NaiveBayesClassifier

train = [
    ('I love this sandwich.', 'pos'),
    ('this is an amazing place!', 'pos'),
    ('I feel very good about these beers.', 'pos'),
    ('this is my best work.', 'pos'),
    ("what an awesome view", 'pos'),
    ('I do not like this restaurant', 'neg'),
    ('I am tired of this stuff.', 'neg'),
    ("I can't deal with this", 'neg'),
    ('He is my sworn enemy!', 'neg'),
    ('My boss is horrible.', 'neg')
]
test = [
    ('The beer was good.', 'pos'),
    ('I do not enjoy my job', 'neg'),
    ("I ain't feeling dandy today.", 'neg'),
    ("I feel amazing!", 'pos'),
    ('Gary is a friend of mine.', 'pos'),
    ("I can't believe I'm doing this.", 'neg')
]<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>a = NaiveBayesClassifier(train)<jupyter_output><empty_output><jupyter_text>Test- Wie gut performed unser Modell bei Daten die es noch nie gesehen hat?<jupyter_code>a.accuracy(test)<jupyter_output><empty_output><jupyter_text>- Zu 80% korrekt, ok für mich :)  Features- Welche wörter sorgen am meisten dafür dass etwas positiv oder negativ klassifiziert wird?<jupyter_code>a.show_informative_features(5)<jupyter_output>Most Informative Features
            contains(an) = False             neg : pos    =      1.6 : 1.0
             contains(I) = False             pos : neg    =      1.4 : 1.0
             contains(I) = True              neg : pos    =      1.4 : 1.0
            contains(He) = False             pos : neg    =      1.2 : 1.0
            contains(My) = False             pos : neg    =      1.2 : 1.0<jupyter_text>Er ist der meinung wenn "this" vorkommt ist es eher positiv, was natürlich quatsch ist, aber das hat er nun mal so gelernt, deswegen braucht ihr gute trainingsdaten.   Klassifizierung<jupyter_code>a.classify("their burgers are amazing")  # "pos"
a.classify("I don't like their pizza.")  # "neg"
a.classify("I love my job.")<jupyter_output><empty_output><jupyter_text>Klassizierung nach Sätzen<jupyter_code>from textblob import TextBlob
blob = TextBlob("The beer was amazing. "
                "But the hangover was horrible. My boss was not happy.",
                classifier=a)
for sentence in blob.sentences:
    print(("%s (%s)") % (sentence,sentence.classify()))<jupyter_output>The beer was amazing. (pos)
But the hangover was horrible. (neg)
My boss was not happy. (neg)<jupyter_text>Mit schweizer Songtexten Kommentare klassifizieren- http://www.falleri.ch<jupyter_code>import os,glob
from nltk.tokenize import sent_tokenize
from nltk.tokenize import word_tokenize


from io import open


train = []
countries = ["schweiz", "deutschland"]
for country in countries:
    out = []
    folder_path = 'songtexte/%s' % country 
    for filename in glob.glob(os.path.join(folder_path, '*.txt')):
        with open(filename, 'r') as f:
            text = f.read()
            words = word_tokenize(text)
            words=[word.lower() for word in words if word.isalpha()]
            for word in words:
                out.append(word)
    out = set(out)
    for word in out:
        train.append((word,country))
        #print (filename)
        #print (len(text))
train
#len(train)
from textblob.classifiers import NaiveBayesClassifier
c2 = NaiveBayesClassifier(train)
c2.classify("Ich gehe durch den Wald")  # "deutsch"
c2.classify("Häsch es guet")  # "deutsch"
c2.classify("Ich fahre mit meinem Porsche auf der Autobahn.")
c2.show_informative_features(5)<jupyter_output>Most Informative Features
            contains(am) = True           schwei : deutsc =      1.3 : 1.0
           contains(bei) = True           schwei : deutsc =      1.3 : 1.0
           contains(das) = True           schwei : deutsc =      1.3 : 1.0
           contains(dir) = True           schwei : deutsc =      1.3 : 1.0
            contains(du) = True           schwei : deutsc =      1.3 : 1.0<jupyter_text>Orakel  Ihr könnt natürlich jetzt Euer eigenes Orakel bauen wie hier: - http://home.datacomm.ch/cgi-heeb/dialect/chochi.pl?Hand=Hand&nicht=net&heute=hit&Fenster=Feischter&gestern=gescht&Abend=Abend&gehorchen=folge&Mond=Manat&jeweils=abe&Holzsplitter=Schepfa&Senden=Jetzt+analysieren%21  Hardcore Beispiel mit Film-review daten mit NLTK- https://www.nltk.org/book/ch06.html- Wir nutzen nur noch die 100 häufigsten Wörter in den Texten und schauen ob sie bei positiv oder negativ vorkommen<jupyter_code>import random
import nltk
nltk.download("movie_reviews")
from nltk.corpus import movie_reviews
documents = [(list(movie_reviews.words(fileid)), category)
              for category in movie_reviews.categories()
              for fileid in movie_reviews.fileids(category)]
random.shuffle(documents)
len(documents)
(" ").join(documents[1][0])
all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words())
word_features = list(all_words)[:2000]
word_features[0:-10]
all_words
all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words())
word_features = list(all_words)[:2000]

def document_features(document): 
    document_words = set(document)
    features = {}
    for word in word_features:
        features['contains({})'.format(word)] = (word in document_words)
    return features
print(document_features(movie_reviews.words('pos/cv957_8737.txt')))
featuresets = [(document_features(d), c) for (d,c) in documents]
train_set, test_set = featuresets[100:], featuresets[:100]
b = nltk.NaiveBayesClassifier.train(train_set)
#document_features("a movie with bad actors".split(" "))
b.classify(document_features("a movie with bad actors".split(" ")))
b.classify(document_features("an uplifting movie with russel crowe".split(" ")))
b.show_most_informative_features(10)
b.accuracy(test_set)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12134453781512605, 'fraction_numerical': 0.014285714285714285, 'mean_word_length': 3.525475285171103, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '9371', 'license': 'MIT', 'n_tokens_mistral': 2077, 'n_tokens_neox': 1998, 'n_words': 531, 'path': '16 Textanalyse Verfiefung Teil 2/2.4 Classifying Text.ipynb', 'repo_name': 'Tamibobi/kurs_21_22', 'size': '5950'}