<jupyter_start><jupyter_text>Project 3: Graph Databases/Neo4j**Implementation Report**Big Data Management Systems (Associate Prof. <NAME>)\<NAME>, 8160192 (<EMAIL>)  1. IntroductionNeo4j is a graph database management system, representing data as nodes and edges (relationships). Additionally, each node/edge may be characterized by zero or more properties, secondary data relevant to them. As relationships between nodes are given explicit importance (in comparison to relational databases) higher performance is achieved when data is modeled and stored using a network structure.In this report I demonstrate proficiency in using Neo4j by importing the ["Social Network: MOOC User Action Dataset"](https://snap.stanford.edu/data/act-mooc.html) into a graph database and retrieving data using [Cypher](https://en.wikipedia.org/wiki/Cypher_(query_language)), a query language designed around the specific characteristics of graph databases.  2. Installing Neo4j and retrieving datasetThis assignment was carried out on a personal Fedora 33 machine.[Neo4j Community Edition 4.2.6](https://neo4j.com/download-center/community) was downloaded from the website [as a tarball](https://neo4j.com/docs/operations-manual/current/installation/linux/tarball/) and extracted to my home directory in order to make configuration and cleanup easier. I then started the server and set a custom password from the browser interface:    tar -xf neo4j-community-4.2.6-unix.tar.gz    cd neo4j-community-4.2.6/    ./bin/neo4j console    The dataset tarball was downloaded and extracted in the project directory:    wget https://snap.stanford.edu/data/act-mooc.tar.gz    tar -xf act-mooc.tar.gz  3. Importing data into Neo4jThe dataset was imported into Neo4j programatically using a Python script and the [Neo4j Python Driver 4.2](https://neo4j.com/docs/api/python-driver/current/), which should be available in your environment for this script to execute successfully (as well as the dataset itself, check the previous section for downloading instructions). Make sure before execution that the "neo4j" database is wiped of all data (`MATCH (n) DETACH DELETE (n)`). The script parses each line of the three files, building two sets and a dictionary with all the data to be imported to Neo4j in advance. This allows using only CREATE clauses in Cypher, as the specification of the assignment suggested. An unguided personal implementation would make use of MERGE and SET clauses for nodes/relationships and properties respectively, with each piece of data being imported as the files are being iterated upon.<jupyter_code>from neo4j import GraphDatabase

# storage for data
users = set()
targets = set()
actions = {}

# iterate upon each file, extract data and store function
def import_data():
    file = open('act-mooc/mooc_actions.tsv', 'r')
    next(file) # skip column names
    for line in file:
        line = line.strip() # Remove newline character
        data = line.split('\t')

        users.add(data[1])
        targets.add(data[2])
        actions[data[0]] = { 'timestamp' : data[3],
                             'user' : data[1],
                             'target' : data[2] }

    file = open('act-mooc/mooc_action_features.tsv', 'r')
    next(file) # skip column names
    for line in file:
        line = line.strip() # Remove newline character
        data = line.split('\t')

        actions[data[0]].update( { 'feature-0' : data[1],
                                   'feature-1' : data[2],
                                   'feature-2' : data[3],
                                   'feature-3' : data[4] } )

    file = open('act-mooc/mooc_action_labels.tsv', 'r')
    next(file) # skip column names
    for line in file:
        line = line.strip() # Remove newline character
        data = line.split('\t')

        actions[data[0]]['label'] = data[1]

# create node and relationship functions
def create_user_node(tx, userid):
    tx.run("CREATE (n:user{userid:$userid})", userid = userid)
    
def create_target_node(tx, targetid):
    tx.run("CREATE (n:target{targetid:$targetid})", targetid = targetid)
    
def create_action(tx, userid, targetid, actionid, timestamp, feature0, feature1, feature2, feature3, label):
    tx.run("MATCH (u:user) WHERE u.userid = $userid "
           "MATCH (t:target) WHERE t.targetid = $ targetid "
           "CREATE (u)-[r:action {actionid: $actionid, timestamp:$timestamp, feature0:$feature0, feature1:$feature1, feature2:$feature2, feature3:$feature3, label:$label}]->(t)",
           userid = userid, targetid = targetid, actionid = actionid, timestamp = timestamp,
           feature0 = feature0, feature1 = feature1, feature2 = feature2,
           feature3 = feature3, label = label)

import_data()

# initialize neo4j driver
# enter your own server url-username-password combo
neo4j = GraphDatabase.driver('neo4j://localhost:7687', auth=("neo4j", "neo5j"))

with neo4j.session() as session:
    for userid in users:
        session.write_transaction(create_user_node, userid)
        
    for targetid in targets:
        session.write_transaction(create_target_node, targetid)
        
    for action in actions:
        userid = actions[action]['user']
        targetid = actions[action]['target']
        timestamp = actions[action]['timestamp']
        feature0 = actions[action]['feature-0']
        feature1 = actions[action]['feature-1']
        feature2 = actions[action]['feature-2']
        feature3 = actions[action]['feature-3']
        # Not all actions have labels (some have duplicates in the dataset)
        # If so use -1 to signify lack
        try:
            label = actions[action]['label']
        except KeyError:
            label = -1
        session.write_transaction(create_action, userid, targetid, action, timestamp, feature0, feature1, feature2, feature3, label)
        
neo4j.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10105120379789759, 'fraction_numerical': 0.017463546965072907, 'mean_word_length': 3.3793615441722347, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '194290', 'license': 'MIT', 'n_tokens_mistral': 1685, 'n_tokens_neox': 1614, 'n_words': 571, 'path': 'neo4j/Neo4j_ImplementationReport.ipynb', 'repo_name': 'tplessas/bdms_assignments', 'size': '5935'}