fixed and built generation from topics

.gitignore

@@ -1,3 +1,4 @@
 /__pycache__*
 recursive-exclude * *.py[co]
 /.vscode*
+/prompter*

app.py

@@ -30,15 +30,18 @@ if file1 is not None and file2 is not None:
         topics['insight1'] = [keywords1, concepts1]
         keywords2, concepts2 = insight2.generate_topics()
         topics['insight2'] = [keywords2, concepts2]
-        st.success('Done!')
         
     with st.spinner("Flux capacitor is fluxing..."):
         embedder = utils.load_model()
-        clutered = utils.cluster_based_on_topics(embedder, cleaned_text1, cleaned_text2)
+        clutered = utils.cluster_based_on_topics(embedder, cleaned_text1, cleaned_text2, num_clusters=5)
         print(clutered)
-        st.success("Done!")
         
     with st.spinner("Polishing up"):
         results = utils.generate_insights(topics, file1.name, file2.name, cleaned_text1, cleaned_text2, clutered)
-        st.write(results)
         st.success("Done!")
+        
+    st.title("Insights generated")
+
+    for result in results:
+        with st.expander(result["name"]):
+            st.write(result["description"])

generation.py

@@ -1,3 +0,0 @@
-import openai 
-
-def

insights.prompt

@@ -14,4 +14,14 @@ The more complex concepts in document 2 is : {{complex2}}
 
 The sentences in one of the clusters is : {{sentences}}
 
-From the sentences and topics above, explain the common idea between the documents and write a paragraph about it and give me 3 new concepts that are linked to this idea. 
+From the sentences and topics above, explain the common idea between the documents and write a paragraph about it and give me 3 new concepts that are linked to this idea. 
+You output format should be:
+
+"""
+name: <FILL-CONCEPT-NAME-HERE>
+description: <FILL-CONCEPT-DESCRIPTION-HERE>
+related:
+- <FILL-RELATED-CONCEPT-1>
+- <FILL-RELATED-CONCEPT-2>
+- <FILL-RELATED-CONCEPT-3>
+"""

prompter/insights_33.prompt

@@ -1,21 +1,41 @@
 You are a highly intelligent bot that is tasked with common ideas between documents. The following are two documents that have been topic modelled and have been clustered based on concepts.
 
-The name for document 1 is : AI tutors will be held back by culture - by Henrik Karlsson.md
+The name for document 1 is : Good conversations have lots of doorknobs.md
 
-The name for document 2 is : The Stability of Beliefs.md
+The name for document 2 is : First we shape our social graph; then it shapes us.md
 
-The topics for document 1 is : bull,picasso,education,ai,chilean,bull 1945,the bull,of bull,prize bull,bull to
+The topics for document 1 is : spiderman,singing,musical,chorus,song,singing something,about spiderman,spiderman spiderman,spiderman,spiderman and
 
-The topics for document 2 is : belief,beliefs,philosophy,epistemological,philosophic,science belief,scientific beliefs,beliefs ensconced,beliefs of,certain beliefs
+The topics for document 2 is : chimpanzees,genetically,womb,consciously,upbringings,from chimpanzees,chimpanzees as,chimpanzees in,chimpanzees and,chimpanzees
 
-The more complex concepts in document 1 is : picasso lithographs bull,story bull bruce,bull culture necessary,lithographs bull 1945,bull didn know
+The more complex concepts in document 1 is : singing like spiderman,spiderman sudden pianist,songs spiderman scientific,just songs spiderman,excerpt spiderman boyfriend
 
-The more complex concepts in document 2 is : beliefs michael polanyi,beliefs held scientists,belief science declared,1951 scientific beliefs,michael polanyi essay
+The more complex concepts in document 2 is : chimpanzees born habitat,die chimpanzees born,sets apart chimpanzees,fast die chimpanzees,old children chimpanzees
 
-The sentences in one of the clusters is : # key takeaways --- # transcript ## excerpt gpt-4, khan academy, wolfram alpha - we're seeing progress ai tools learning.
-demo state art ai tutoring capabilities, watch video march 14 salman khan khan academy demonstrates system built top gpt-4. video, khan uses ai model socratic tutor.
-gpt-4 occasionally hallucinates answers true.
-models improving faster anticipated, gpt-4 already scores top 10 percent university exams.
-march 23, nine days khan demo:ed tutoring system, openai partnered wolfram released plugin gives gpt-4 ability things like: way fluidly interacting information, shaping dialogue, immensely powerful.
+The sentences in one of the clusters is : ask remove mask spot really hard, trick kept us afloat called “take-and-take focus,” meaning whoever singing keep going someone jumped take spotlight them, happen quickly often.
+it’s easy remember lonely feels taker refuses cede spotlight you, easy forget lovely feels don’t want spotlight taker lets recline mezzanine fill stage.
+it’s often unclear, stand around waiting someone else take turn invite us take ours.
+we’re standing perimeter empty dance circle, takers martyrs launch middle .
 
-From the sentences and topics above, explain the common idea between the documents and write a paragraph about it and give me 3 new concepts that are linked to this idea. 
+From the sentences and topics above, explain the common idea between the documents and write a paragraph about it and give me 3 new concepts that are linked to this idea. 
+You output format should be:
+
+"""
+name: <FILL-CONCEPT-NAME-HERE>
+description: <FILL-CONCEPT-DESCRIPTION-HERE>
+related:
+- <FILL-RELATED-CONCEPT-1>
+- <FILL-RELATED-CONCEPT-2>
+- <FILL-RELATED-CONCEPT-3>
+"""
+
+The common idea between the documents is the importance of collaboration and teamwork. In the first document, the idea of collaboration is explored in the context of music, with the chorus singing together to create a beautiful song. In the second document, the idea of collaboration is explored in the context of chimpanzees, with the idea that they work together to survive and thrive in their environment. 
+
+The concept of collaboration is an important one, and it is essential for any group of individuals to work together to achieve a common goal.
+
+name: Group Dynamics
+description: Group dynamics is the study of how people interact in groups and how their behavior affects the group as a whole.
+related:
+- Interpersonal Relationships
+- Social Interaction
+- Conflict Resolution

utils.py

@@ -66,7 +66,7 @@ def generate_keywords(kw_model, document: str) -> list:
 				final_topics.append(extraction[0])
 		return final_topics
 
-def cluster_based_on_topics(embedder, text1:str, text2:str):
+def cluster_based_on_topics(embedder, text1:str, text2:str, num_clusters=3):
 	nlp = spacy.load("en_core_web_sm")
 	
 	# Preprocess and tokenize the texts
@@ -87,7 +87,7 @@ def cluster_based_on_topics(embedder, text1:str, text2:str):
 	all_embeddings = all_embeddings /  np.linalg.norm(all_embeddings, axis=1, keepdims=True)
 
 	# Perform agglomerative clustering
-	clustering_model = AgglomerativeClustering(n_clusters=None, distance_threshold=1.5)
+	clustering_model = AgglomerativeClustering(n_clusters=num_clusters)
 	clustering_model.fit(all_embeddings)
 	cluster_assignment = clustering_model.labels_
 
@@ -121,27 +121,34 @@ def generate_insights(topics:dict, name1:str, name2:str, text1:str, text2:str, c
 	
 	for cluster_id, sentences in clusters.items():  
 		
-		PROMPT = PROMPT.replace("{{sentences}}", "\n".join(sentences))
+		print(cluster_id, " ", sentences)
+		final_prompt = PROMPT.replace("{{sentences}}", "\n".join(sentences))
 	
-		with open(f"prompter/insights_{cluster_id}.prompt", "w") as f:
-			f.write(PROMPT)
+		# with open(f"prompter/insights_{cluster_id}.prompt", "w") as f:
+		# 	f.write(final_prompt)
      
 		# Generate insights for each cluster
 		response = openai.Completion.create(
 			model="text-davinci-003",
-			prompt=PROMPT,
-			temperature=0.5,
+			prompt=final_prompt,
+			max_tokens=200,
+			temperature=0.4,
 			top_p=1,
-			max_tokens=1000,
 			frequency_penalty=0.0,
 			presence_penalty=0.0,
 		)
 
 		text = response['choices'][0]['text']
-		with open(f"prompter/insights_{cluster_id}.txt", "a") as f:
-			f.write(text)
+		name_location = text.find("Name:")
+		description_location = text.find("Description:")
+		name_of_insight = text[name_location+6:name_location+6+text[name_location+6:].find("\n")]
+		description = text[:name_location] + text[description_location+13:description_location+13+text[description_location+13:].find("\n")]
+		final_insights.append({"name": name_of_insight, "description": description})
+		
+		# with open(f"prompter/insights_{cluster_id}.prompt", "a") as f:
+		# 	f.write(text)
    
-		final_insights.append(text)
+		# final_insights.append(text)
 
 	return final_insights