<jupyter_start><jupyter_text>Changed the code to also output the fit_predict array which is a 2D array where the index corresponds to the line of the text and the value at that index corresponds to the cluster valuepreviously looked at the dictionary type and replicating each dictionary index value for the number of items in the cluster but then would have duplicates<jupyter_code>def cluster_sentences(sentences, nb_of_clusters):
    tfidf_vectorizer = TfidfVectorizer(tokenizer=word_tokenizer,
                                    stop_words=stopwords.words('english'),
                                     max_df=0.9,
                                    min_df=0.1,
                                    lowercase=True)
            #builds a tf-idf matrix for the sentences
    tfidf_matrix = tfidf_vectorizer.fit_transform(sentences)
    kmeans = KMeans(n_clusters=nb_of_clusters)
    kmeans.fit(tfidf_matrix)
    
    kmeans_array = kmeans.fit_predict(tfidf_matrix)
    
    clusters = collections.defaultdict(list)
    for i, label in enumerate(kmeans.labels_):
            clusters[label].append(i)


    return dict(clusters), kmeans_array
clusters, kmeans_predict = cluster_sentences(sentences, nclusters)<jupyter_output><empty_output><jupyter_text>For every line in the fit_predict array, it finds those lines that belong to the same cluster and adds them to a list. Then it is finished it saves the list to a bigger list.<jupyter_code>output = []

for i in range(len(kmeans_predict)):
    temp = [i]
    for j in range(1,len(kmeans_predict)):
        jl = [j]
        if kmeans_predict[i]==kmeans_predict[j] and j not in temp:
            temp += [j]
    output.append(temp)<jupyter_output><empty_output><jupyter_text>Then for lists with more than 5, we remove the last values. For lists with less than 5 we duplicate the second value<jupyter_code>for line in output:
    if len(line) > 5:
        index = output.index(line)
        output[index] = line[0:5]
    if len(line) > 1  and len(line) < 5:
        line += [line[1]]*(5-len(line))
    if len(line) == 1:
        line += [line[0]]*4

for line in output:
    print(line)<jupyter_output>[0, 158, 353, 844, 1305]
[1, 176, 196, 1405, 176]
[2, 320, 499, 980, 1321]
[3, 48, 352, 623, 636]
[4, 997, 1050, 1142, 997]
[5, 336, 749, 930, 954]
[6, 200, 216, 461, 799]
[7, 520, 596, 642, 652]
[8, 562, 562, 562, 562]
[9, 184, 388, 455, 597]
[10, 30, 59, 74, 905]
[11, 79, 89, 112, 146]
[12, 300, 507, 1241, 1292]
[13, 101, 204, 306, 662]
[14, 549, 763, 867, 1048]
[15, 197, 258, 697, 924]
[16, 71, 333, 701, 782]
[17, 266, 273, 330, 555]
[18, 122, 294, 347, 790]
[19, 386, 695, 966, 1327]
[20, 506, 512, 926, 959]
[21, 524, 588, 700, 1271]
[22, 759, 855, 871, 1086]
[23, 574, 872, 1331, 574]
[24, 50, 84, 343, 876]
[25, 540, 540, 540, 540]
[26, 34, 53, 167, 547]
[27, 304, 609, 1098, 1264]
[28, 142, 901, 1012, 1026]
[29, 284, 731, 1248, 1300]
[30, 10, 59, 74, 905]
[31, 174, 257, 497, 174]
[32, 115, 145, 168, 263]
[33, 139, 268, 367, 472]
[34, 26, 53, 167, 547]
[35, 276, 384, 933, 1113]
[36, 337, 839, 858, 1087]
[37, 311, 393, 452, 1053]
[38, 129, 813, 1392, 1461]
[39, 490, 635, 838, 907]
[40[...]<jupyter_text>Import CSV module write each row into the file. Currently using a true comma separator.<jupyter_code>import csv

with open('results.csv', 'w', newline='') as csvfile:
    results = csv.writer(csvfile, delimiter=',')
    for line in output:
        results.writerow(line)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1313364055299539, 'fraction_numerical': 0.16618663594470046, 'mean_word_length': 3.363065326633166, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '371163', 'license': 'MIT', 'n_tokens_mistral': 1791, 'n_tokens_neox': 1296, 'n_words': 431, 'path': 'Old/Cluster algorithm_v2.ipynb', 'repo_name': 'Rick-Mackenbach/DPA-Group-Assignment', 'size': '3472'}