<jupyter_start><jupyter_text>**Accuracy based on 0-1 occurence is 99.55% which is less than full-word-4-gram model while if we include total frequency of each token, then the accuracy reduces to 91.7% .**<jupyter_code># Test few cases where this model didn't work
num_case = 0
for i in range(len(output_labels)):
    if output_labels[i] != test_labels[i]:
        print(new_data[i])
        print("Actual Language {0} | Predicted Language {1}".format(test_labels[i], output_labels[i]))
        count = {}
        for key in lang_dict_n_gram.keys():
            count[key] = 0
        for token in new_data[i].split():
            for key in lang_dict_n_gram.keys():
                if token in lang_dict_n_gram[key]:
                    count[key] += 1
        print(count)
        print("--------------------------------")
        num_case += 1
        if num_case == 5:
            break<jupyter_output>co jsme tím tehdy tehd ehdy dali najevo naje ajev jevo najev ajevo
Actual Language cs | Predicted Language fi
{'pl': 1, 'el': 0, 'fr': 4, 'bg': 0, 'nl': 4, 'ro': 4, 'pt': 4, 'de': 4, 'sv': 4, 'da': 4, 'lv': 2, 'cs': 5, 'lt': 2, 'en': 4, 'sk': 4, 'sl': 4, 'it': 4, 'et': 4, 'hu': 2, 'fi': 6, 'es': 4}
--------------------------------
děkuji děk ěku �kuj kuji děku ěkuj �kuji vám pane battene batt atte tten tene batte atten ttene
Actual Language cs | Predicted Language en
{'pl': 2, 'el': 1, 'fr': 5, 'bg': 1, 'nl': 6, 'ro': 2, 'pt': 2, 'de': 6, 'sv': 6, 'da': 6, 'lv': 2, 'cs': 3, 'lt': 1, 'en': 7, 'sk': 1, 'sl': 1, 'it': 6, 'et': 4, 'hu': 6, 'fi': 5, 'es': 3}
--------------------------------
jmenovali jmen meno enov nova oval vali jmeno menov enova noval ovali se william will illi llia liam willi illia lliam meyer meye eyer bernard bern erna rnar nard berna ernar rnard starie star tari arie stari tarie reginald regi egin gina inal nald regin egina ginal inald pike thomas thom homa omas thoma[...]<jupyter_text>**If we actually notice the difference between actual language and predicted language, we see very less difference in terms of occurence of words/trigrams which gives a sense of similarity in those languages. In some cases they are equal.**<jupyter_code># Lets check which language has most wrong predictions
wrong_pred = {}
for key in lang_dict_n_gram.keys():
    wrong_pred[key] = 0
for i in range(len(output_labels)):
    if output_labels[i] != test_labels[i]:
        wrong_pred[test_labels[i]] += 1
print(wrong_pred)
%matplotlib inline
import matplotlib.pyplot as plt
plt.bar(wrong_pred.keys(), wrong_pred.values())
plt.show()
# Convert the classes to numbers
classes_map = {}
classes = []
for ind, key in enumerate(count.keys()):
    classes_map[key] = ind
    classes.append(key)

test_labels_n, output_labels_n = [], []
for i in range(len(test_labels)):
    test_labels_n.append(classes_map[test_labels[i]])
    output_labels_n.append(classes_map[output_labels[i]])
# Plot confusion matrix
from sklearn.metrics import confusion_matrix
cnf_matrix = confusion_matrix(test_labels_n, output_labels_n)

import seaborn as sns
fig = plt.figure(figsize=(16,12))
ax = fig.add_subplot(111)
sns.heatmap(cnf_matrix, annot=True, fmt='g',ax=ax)
ax.xaxis.set_ticklabels(classes)
ax.yaxis.set_ticklabels(classes)
plt.xlabel('Predicted')
plt.ylabel('True')<jupyter_output><empty_output><jupyter_text>From confusion matrix we can seee that 42 English(en) sentences are classified as Dutch(nl). Lets see what are those sentences.<jupyter_code>counter = 0
for i in range(len(output_labels)):
    if (output_labels[i] == 'nl') and (test_labels[i] == 'en'):
        print(test_data[i])
        print(new_data[i])
        print("----------------------------------------------")
        counter += 1
    if counter == 5:
        break<jupyter_output>the same has happened under other circumstances 
the same has happened happ appe ppen pene ened happe appen ppene pened under unde nder other othe ther circumstances circ ircu rcum cums umst msta stan tanc ance nces circu ircum rcums cumst umsta mstan stanc tance ances
----------------------------------------------
we must help others to help themselves 
we must help others othe ther hers other thers to help themselves them hems emse msel selv elve lves thems hemse emsel mselv selve elves
----------------------------------------------
{'fraction_non_alphanumeric': 0.1665891112145184, 'fraction_numerical': 0.01698464402047464, 'mean_word_length': 3.1296829971181555, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '127034', 'license': 'MIT', 'n_tokens_mistral': 1508, 'n_tokens_neox': 1426, 'n_words': 498, 'path': 'src/n-gram/full-word-4-5-gram-model.ipynb', 'repo_name': 'amitmac/Language-Detection', 'size': '4298'}