<jupyter_start><jupyter_text>Data Understanding<jupyter_code># Menyiapkan format bahasa dan mengunduh dictionary
locale.setlocale(locale.LC_ALL, '')
nltk.download(['stopwords', 'punkt'])
# Kumpulan stopwords
stopwords_list = set(stopwords.words('english') + ['AT_USER', 'URL'])
stemmer = SnowballStemmer(language='english')
# membaca data dari file CSV
newsgroups_all = fetch_20newsgroups(subset='all', categories=["sci.electronics", "sci.med", "sci.space"], remove=('headers', 'footers', 'quotes'), random_state=42)
print("Total data:", len(newsgroups_all.data))
print("Sampel berita:", newsgroups_all.data[0])<jupyter_output>Total data: 2961
Sampel berita: I live at sea-level, and am called-upon to travel to high-altitude cities
quite frequently, on business.  The cities in question are at 7000 to 9000
feet of altitude.  One of them especially is very polluted...

Often I feel faint the first two or three days.  I feel lightheaded, and
my heart seems to pound a lot more than at sea-level.  Also, it is very
dry in these cities, so I will tend to drink a lot of water, and keep
away from dehydrating drinks, such as those containing caffeine or alcohol.

Thing is, I still have symptoms.  How can I ensure that my short trips there
(no, I don't usually have a week to acclimatize) are as comfortable as possible?
Is there something else that I could do?

A long time ago (possibly two years ago) there was a discussion here about
altitude adjustment.  Has anyone saved the messages?

Many thanks,<jupyter_text>Data Preparation<jupyter_code># Fungsi untuk melakukan pembersihan teks
def preprocess_text(text):
    text = text.lower()
    text = re.sub('((www\.[^\s]+)|(https?://[^\s]+))', 'URL', text)
    text = re.sub('@[^\s]+', 'AT_USER', text)
    text = re.sub(r'#([^\s]+)', r'\1', text)
    text = word_tokenize(text)
    text = [word for word in text if word not in stopwords_list and word.isalpha()]
    text = [stemmer.stem(word) for word in text]
    return ' '.join(text)

texts = Parallel(n_jobs=4, batch_size=128, verbose=10)(delayed(preprocess_text)(text) for text in newsgroups_all.data)
# Membuat korpus kata-kata umum
freqdist = FreqDist()
for sentence in texts:
    for word in word_tokenize(sentence):
        freqdist[word] += 1

# Visualisasi frekuensi kata-kata umum
print(freqdist.most_common(10))
freqdist.plot(20, cumulative=False)<jupyter_output>[('use', 2042), ('would', 1686), ('one', 1666), ('get', 1185), ('space', 1128), ('like', 1115), ('also', 983), ('time', 943), ('know', 872), ('could', 774)]<jupyter_text>Modelling<jupyter_code># Membuat model
pipeline = Pipeline([
    ('tfidf-vectorize', TfidfVectorizer(stop_words='english', max_df=0.5, min_df=2, use_idf=True, max_features=10000)),
    ('lsa-reduce', TruncatedSVD(n_components=3))
])

projections = pipeline.fit_transform(texts)
tfidf_matrix = pipeline.steps[0][1].transform(texts)
dist = 1 - cosine_similarity(tfidf_matrix)
linkage_matrix = ward(dist)
cluster_labels = fcluster(linkage_matrix, 3, criterion='maxclust')

fig, ax = plt.subplots(figsize=(5, 5))
ax = dendrogram(linkage_matrix, orientation="left", labels=cluster_labels, p=3, truncate_mode="level");

plt.tick_params(axis= 'x', which='both', bottom='off', top='off', labelbottom='off')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>df = pd.DataFrame({
    'text': texts,
    'true_label': newsgroups_all.target,
    'pred_label': cluster_labels
})
print("Davies-Bouldin score (true):", davies_bouldin_score(projections, df['true_label']))
print("Davies-Bouldin score (pred):", davies_bouldin_score(projections, df['pred_label']))
print("Silhouette score (true):", silhouette_score(projections, df['true_label']))
print("Silhouette score (pred):", silhouette_score(projections, df['pred_label']))
plt.figure(figsize=(10, 7))
for cluster_i in range(3):
    cluster_i_data = df[df['pred_label'] == cluster_i + 1]

    freqdist = FreqDist()
    for sentence in cluster_i_data['text']:
        for word in word_tokenize(sentence):
            freqdist[word] += 1
    
    print("Cluster:", cluster_i)
    print("Kata populer:", freqdist.most_common(5))

    wc = WordCloud(max_font_size=40).generate_from_frequencies(freqdist)
    ax = plt.subplot(1, 3, cluster_i + 1)
    ax.imshow(wc)
    ax.set_title("Cluster #%d" % cluster_i)

plt.tight_layout()
plt.show()
fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.scatter(projections[:, 0], projections[:, 1], projections[:, 2], c=cluster_labels)
ax.set_xlabel('PC1')
ax.set_ylabel('PC2')
ax.set_zlabel('PC3')
fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.scatter(projections[:, 0], projections[:, 1], projections[:, 2], c=newsgroups_all.target)
ax.set_xlabel('PC1')
ax.set_ylabel('PC2')
ax.set_zlabel('PC3')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15019762845849802, 'fraction_numerical': 0.022259205325566883, 'mean_word_length': 5.937950937950938, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '466524', 'license': 'MIT', 'n_tokens_mistral': 1806, 'n_tokens_neox': 1688, 'n_words': 426, 'path': 'jupyter/clustering_hierarchical.ipynb', 'repo_name': 'fahminlb33/buku-ml-python', 'size': '4807'}