<jupyter_start><jupyter_text>Outline the business domain ![knowledge-graph-2.png](images/knowledge-graph-2.png) |**[Overview](Overview)** |**[Prior-steps](Prior-steps)**|**[How-to-use](How-to-use)**|**[Next-steps](Next-steps)**|**[Postscript](Postscript)**|**[Acknowledgements](Acknowledgments)| Overview Step 7 has built a knowledge-graph-from-keywords. Now we use the knowledge graph to understand the business domain covered by the document library we are looking at. In consultation with the portfolio stakeholders, and domain experts, we can modify the knowledge-graph to reflect a sensible way of handling the business domain.  The immediate output is either: - a knowledge graph, which can handle multiple views or facets reflecting different stakeholder understanding  or - tree data structure, cut down from the knowledge graph, that reflects one convergent, dominating knowledge structure suitable for organising portfolio services in this business domain.  InstallationCheck installation has been made, as per the [READme](https://github.com/lawrencerowland/Data-Model-for-Project-Frameworks/blob/master/Project-frameworks-by-using-NLP-with-Python-libraries/README.md)At this point you will need Neo4j (or you can do this in YEd or Gephi).  Prior-stepsSteps 5-7 minimum How-to-use Open Neo4j<jupyter_code>#hide
#Use either Neo4j Desktop, or create a Neo4j sandbox. 
Create a new graph
Load APOC plug in
#Change security settings for APOC. Where you do this is different between v 3.5 and 4
apoc.import.file.enabled=true
apoc.export.file.enabled=true
#See https://neo4j.com/docs/labs/apoc/current/import/graphml<jupyter_output><empty_output><jupyter_text>Import knowledge-graph<jupyter_code>CALL apoc.import.graphml("https://raw.githubusercontent.com/lawrencerowland/Data-Model-for-Project-Frameworks/master/Project-frameworks-by-using-NLP-with-Python-libraries/Interim-results/Keyword-graph.graphml",{storeNodeIds: true})<jupyter_output><empty_output><jupyter_text>Give all keywords a label.<jupyter_code>MATCH (n)
SET n:keyword
RETURN n.name, labels(n) AS labels<jupyter_output><empty_output><jupyter_text>![graph.png](attachment:96795955-a031-4426-87e9-564209d96ea4.png)  Adjust graph with domain experts I look at the diagram and drag nodes around until I see a set of useful relationships to review with stakeholders.- Example- I see an Operating model comprising: Design  Process ,Operations- I see a site which runs the Operating model. - I see a set of 'requirements which are confirmed to be appropriate'   Safety, SecurityThere will be many ways of seeing things. That is feature, not a bug!- Example- Because I could also see how Operations/Operability sit in requirements. The good thing about knowledge graphs is that one can retain whichever relationships are helpful in the mind's eye of one or other of the stakeholders. - Example- But it is also sometimes needful to simplify, and so I go with Operations as being in business model. - The nodes, 'requirements' and 'appropriate' have been useful in seeing this, but now I dont need them as separate nodes. - So I first delete 'requirements' and 'appropriate'<jupyter_code>MATCH (m) WHERE m.id='requir' OR m.id='appropri' DETACH DELETE (m)<jupyter_output><empty_output><jupyter_text>Create new labels that reflect your preferred clusters or groups- Example- I label the categories of requirement as Requirement- So also for Site and Operating modelI can now handle nodes by their appropriate label, colouring and arranging them accordingly.One can also invert relationships.MATCH (f:Foo)-[rel:FOOBAR {a:1}]->(b:Bar)CALL apoc.refactor.invert(rel)yield input, outputRETURN input, output<jupyter_code>MATCH (m) WHERE m.id='safe' OR m.id='secur' SET m:requirement RETURN (m)
MATCH (m) WHERE m.id='oper'OR m.id='site'OR m.id='process' OR m.id='design' SET m:keyword:Opmodel RETURN (m)
MATCH (m) WHERE m.id='site ' SET m:site RETURN (m)
MATCH (m:keyword) REMOVE m:keyword RETURN (m)<jupyter_output><empty_output><jupyter_text>The schema of the graph looks like this (CALL db.schema()) ![schema.png](attachment:21df4d4a-27e9-4379-9ea1-03b47733b8c0.png) The full knowledge graph looks like this: ![final.png](attachment:189a3836-fcbd-452b-9339-bad4c770120d.png)  Tidy up the graph1. Remove any self-self relationships2. Make the relationships all flow on in one direction3. add any implied relationships, as per stakeholder discussionsFor .3. in the example, stakeholders agree that the dominant view is of - Operating Model drives requirements - Requirements drive changes to the site So we add, some relationships from Op model into requirements, and requirements into site. These replace any relationships that jump straight from requirements to site, and we remove these. i.e. We are looking for a schema as per this figure.  ![graph-schema-2.png](attachment:bccc5f7c-0f2f-4811-81a5-c4aa9460ca59.png)<jupyter_code>#hide
# 1. Remove self-self relations
MATCH (m)-[r]-(m) DELETE r

# 2. Reverse directions where needed 

MATCH (a)-[rel]->(b) WHERE a.id='secur' AND b.id='oper'
CALL apoc.refactor.invert(rel)
yield input, output
RETURN input, output

#similar for the following (need to add the bottom 3 lines each time)
MATCH (a)-[rel]->(b) WHERE a.id='site' AND b.id='secur'
MATCH (a)-[rel]->(b) WHERE a.id='site' AND b.id='process'

# And deleted some unncessary reverse relationships
MATCH (a)-[rel]->(b) WHERE a.id='site' AND b.id='oper' DELETE rel

3. Add any implied relationships
MATCH (a:Opmodel), (b:requirement) WHERE (a.id='design' OR a.id='process') AND (b.id='safe' OR b.id='secur')
CREATE (a)-[r:RELATED]->(b)
RETURN type(r)

MATCH (a), (b) WHERE (a.id='safe') AND (b.id='site') CREATE (a)-[r:RELATED]->(b) RETURN type(r)

MATCH (a)-[r]-(b) WHERE (a.id='design') AND (b.id='site') DELETE r
MATCH (a)-[r]-(b) WHERE (a.id='process') AND (b.id='site') DELETE r<jupyter_output><empty_output><jupyter_text>We arrive at this streamlined knowledge graph, which meets the simplified schema we showed in the the last figure above.  ![knowledge-graph-2.png](attachment:6c32349c-dca0-4267-ad08-d3bde124a1e1.png)  Export graphTo view it in YEd, one can either:- go via the YWorks NEo4j explorer app-or open the graphml file in YEd, and use the properties mapper to map across the label. -or go to gephi (for this, there is an extra parameter in the export API)  hide![YEd-properties.png](attachment:b4b77d50-8660-40a3-9b9b-a6839c380f12.png)<jupyter_code>CALL apoc.export.graphml.all("Keyword-graph-2.graphml.graphml", {})
#hide
#This will be saved in the Import folder within Neo4j Graph settings
# It needs to be manually moved to the 'Interim-results folder'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10687481236865806, 'fraction_numerical': 0.023566496547583308, 'mean_word_length': 5.5068359375, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '102170', 'license': 'MIT', 'n_tokens_mistral': 2185, 'n_tokens_neox': 2034, 'n_words': 744, 'path': 'Project-frameworks-by-using-NLP-with-Python-libraries/Jupyter-notebooks/Step-8-Outline-the-business-domain.ipynb', 'repo_name': 'lawrencerowland/Automation-of-construction-contract-review', 'size': '6662'}