<commit_before>= Neo4j Doc Manager
Neo4j Team <<EMAIL>>
v1.0, 2015-08
:toc:
:toclevels: 6

:sectnums:

== Our Goal

=== TL;DR

What's Neo4j Doc Manager? It is a tool that enables you to migrate existing documents into MongoDB to a Neo4j graph structure. You just run it in background and the information that is in MongoDB will be imported to a graph.

=== A little longer explanation

Neo4j is an OLTP graph database which excels at querying data relationships, which is a weakness of other NoSQL and SQL solutions.  We created the Neo4j Doc Manager for the Mongo Connector to allow MongoDB developers to store JSON data in Mongo while querying the relationship between the data using Neo4j.  We call this polyglot persistence - using the database best suited for the type of data and querying your application requires.

MongoDB stores data as JSON documents, while Neo4j stores data as graphs.  In order to enable graph-based querying of Mongo data, we need to determine how to map between these two different data structures.  Our initial goal was to implement a default mapping plan, covering the most well-known cases. We wanted to follow convention instead of requiring configuration. We collected some generic Mongo Document structures based on community feedback and structured the mapping based on them. It's important to note that we are opened for further suggestions and improvements.

image::resources/images/doc_to_graph.png[]

This project is based upon the link:https://github.com/10gen-labs/mongo-connector[Mongo Connector]. It provides a simple protocol to transfer data from MongoDB to another database. While Mongo publishes Doc Manager implementations for other databases, they did not provide an implementation for Neo4j.  You can grab more detailed information about **mongo-connector** into the official link:https://github.com/10gen-labs/mongo-connector/wiki[Project Wiki].

The Mongo Connector requires creating a MongoDB link:http://docs.mongodb.org/manual/tutorial/deploy-replica-set/[replica set]. An **OplogThread** thread then will listen to all CUD actions occurring in MongoDB. The **mongo-connector** provides an interface to collect the events caught by the OplogThread. The communication interface is implemented into a structure called **DocManager**, which can properly receive and handle Mongo documents and information about the database and its collections.

By extending the **DocManager** class, we have created a Configuration that interacts with Neo4j. Some methods are required to be implemented to keep Mongo Connector protocol consistent. The next section describes in detail how **neo4j_doc_manager** handles each of these methods.

Detailed documentation for the *DocManager* superclass and its protocol can be found link:https://github.com/10gen-labs/mongo-connector/wiki/Writing-Your-Own-DocManager[here]

== About Neo4j DocManager

=== Setup

==== Install neo4j-doc-manager

The preferred method of installation is with the pip package manager:

```
pip install neo4j-doc-manager --pre
```

===== Alternate installation

If you experience any issues with pip you may use the following installation method:

First of all, install link:https://github.com/10gen-labs/mongo-connector[mongo-connector]. This can be easily done with the following command:

```
pip install mongo-connector
```

Now install **neo4j_doc_manager** by cloning this repository and setting the `PYTHONPATH` to it's local directory:

```
git clone https://github.com/neo4j-contrib/neo4j_doc_manager.git
cd neo4j_doc_manager
export PYTHONPATH=.
```

You also must install **py2neo**:

```
pip install py2neo
```

==== Start Neo4j and Mongo

Ensure that you have a Neo4j instance up and running.

If you have authentication enabled for Neo4j, be sure to set **NEO4J_AUTH** environment variable, containing your user and password. 

```
export NEO4J_AUTH=<user>:<password>
```

Ensure that mongo is running a *replica set*. To initiate a replica set start mongo with:

```
mongod --replSet myDevReplSet
```

Then open link:http://docs.mongodb.org/master/tutorial/getting-started-with-the-mongo-shell[mongo-shell] and run:

```
rs.initiate()
```

Please refer to link:https://github.com/10gen-labs/mongo-connector/wiki/FAQ[Mongo Connector FAQ] for more information.


==== Start the Neo4j Doc Manager service

To start the service, run the following command:

```
mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager
```

* **-m** provides Mongo endpoint
* **-t** provides Neo4j endpoint. Be sure to specify the protocol (http).
* **-d** specifies Neo4j Doc Manager.


=== Methods

==== Constructor

By invoking **Neo4j Doc Manager** initialisation command with proper parameters ( `mongo-connector -m [mongo_url] -t [neo4j_server_url] -d neo4j_doc_manager` ), the **Neo4jDocManager** constructor is called.

Constructor receives the following arguments:
[source, python]
----
(self, url, auto_commit_interval=DEFAULT_COMMIT_INTERVAL,
                 unique_key='_id', chunk_size=DEFAULT_MAX_BULK, **kwargs)

----

**url** corresponds the address where a Neo4j server instance is running.

**unique_key** corresponds to the identifier refers to the unique key that is being used in Mongo. Default value is *_id* .

===== Authentication

If you have authentication enabled for Neo4j, be sure to set **NEO4J_AUTH** environment variable, containing your user and password. 

```
export NEO4J_AUTH=<user>:<password>
```

If authentication is not enabled on Neo4j, no action is required. To disable authentication on Neo4j, go to Neo4j install directory, and then edit __conf/neo4j-server.properties__ :

```
dbms.security.auth_enabled=false
```

==== Upsert

Upsert consists into the method that creates new nodes and relationships given a Mongo Document. The method signature is described as below:
[source, python]
----
upsert(self, doc, namespace, timestamp):
----

Basically we **translate every element of a collection into a new node**. Since the elements can be composite, we have adopted some patterns to properly convert each document into a group of nodes and relationships:

* Each new node will be receive *Document* Label
* Also the document type (the referred collection from the incoming document) will also be inserted as a node label
* Document id will be propagated to the node. That means node will have the same '_id' that Mongo Document has.
* If the document contains the elements below, they will recursively be transformed into new nodes as well
  ** a nested document
  ** an array of documents
* All the other types of data into the document will be translated into node properties.

In terms of relationships, every time we find composite documents, we will stablish a relationship between the root document and the nested document.

To clarify our scenario, let's imagine an empty MongoDB instance.
Let's also consider an empty instance of Neo4j.

===== Simple case

We then run the following statement into *mongo*, to insert a talk into a collection of talks:

[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----

This will insert the following document into Mongo:
[source, json]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

This will be reflected as follows into Neo4j:

image::resources/images/neograph1.png[]

Check the detailed generated graph:


image::resources/images/graph1.png[]

**Created nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.

The node chain is preserved. For example, imagine that you insert the following document in MongoDB:

[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.", "conference": { "city": "London" } }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----


[source, json]
----
{
  "_id" : ObjectId("560dd583cf74773fae3fd001"),
  "session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.",
    "conference" : {
      "city" : "London"
    }
  },
  "topics" : [
    "keynote",
    "spring"
  ],
  "room" : "Auditorium",
  "timeslot" : "Wed 29th, 09:30-10:30",
  "speaker" : {
    "name" : "<NAME>",
    "bio" : "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter" : "https://twitter.com/springjuergen",
    "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

In Neo4j, we will have:

image::resources/images/graph2.png[]


**Created nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.
* _Document:conference_ - a Node that is nested to _session_.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.
* A relationship that connects *session* and *conference* nodes, called **session_conference**.

===== Case containing a JSON Array

Now let's insert the following data. Note the nested JSON array represented by **tracks**:
[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "tracks": [{ "main":"Java" }, { "second":"Languages" }], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----

[source, javascript]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "tracks": [{ "main":"Java" }, { "second":"Languages" }],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

The above document will be translated into Neo4j as follows:

image::resources/images/graph3.png[]


**Created nodes:**

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).

* _Document:tracks0_ - A node that represents the first JSON of **tracks** array [at index 0]. It contains the propagated **talks** id, plus the properties of the nested document.
* _Document:tracks1_ - A node that represents the second JSON of **tracks** array [at index 1]. It contains the propagated **talks** id, plus the properties of the nested document.

* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.
* A relationship that connects *talks* and the first element of *tracks* array (_tracks0_), called *talks_tracks0*
* A relationship that connects *talks* and the second element of *tracks* array (_tracks1_), called *talks_tracks1*

===== Case containing Mongo documents joined by an _id reference

Let's imagine now an explicit *_id* reference between two documents, such as:

[source]
----
db.places.insert({"_id": "32434ab234324", "name": "The cool place", "url": "cool.example.net" })
----

[source, javascript]
----
{ 
  "_id": "32434ab234324", 
  "name": "The cool place", 
  "url": "cool.example.net" 
}
----

[source]
----
db.people.insert({ "name": "Michael", "places_id": "32434ab234324", "url": "neo4j.com/Michael" })
----

[source, javascript]
----
{
  "name": "Michael", 
  "places_id": "32434ab234324", 
  "url": "neo4j.com/Michael" 
}
----

Note that two documents were inserted, and *people* references *place* explicitly by *id*. __Neo4j Doc Manager__ will map every field that ends with ** \_id ** into an explicit relationship. First, we run a *MERGE* to see if the respective node exists. In the above example, we insert a _place_, and then a _people_. When inserting the _people_ type, the connector will identify an explicit _id_ relationship, through __places\_id__ , and will try to find the respective node. If it does exist (and it should), a relationship between the two nodes will be created.

image::resources/images/graph4.png[]


**Created nodes:**

* _Document:places_ - Simple root node, with the properties _name_ and _url_ and an *_id*.
* _Document:people_ - Another node, that comes from an different _upsert_ method call. It creates another simple node, with the properties _name_ and _url_.

**Created Relationships:**

* A relationship that connects *people* and *places* nodes is created due to the property *places_id* on _people_ node. It is called **people_places**.

==== Bulk Upsert

If you already have data inserted on your MongoDB, the first time you run _Neo4j DocManager_ the **bulk_upsert** method will be called. It acts as a normal **upsert**, described in previous section, but all the database actions will be batched in a single transaction.

This will avoid a massive commit into Neo4j if the current Mongo database already has several documents. This will also avoid inconsistencies on an initial import.

Keep in mind that batch_upsert tends to have more performance if you are import a huge amount of data. The key to have this method called is the absence of a file called __oplog.timestamp__. If this file is not present, the document import will happen via __bulk_upsert__.

This can be useful if you call a __mongoimport__ commmand that will bring up a huge amount of data. For this scenario, you could manually remove the __oplog.timestamp__, which is automatically created the first time you call **mongo-connector** command. This file usually lives on the root of your **neo4j-doc-manager** Python Package project. 

Of course you do not have to remove the file. **bulk_upsert** is nor mandatory, but it can help you to achieve a better performance on situations where you have many documents to bring to Neo4j.

__bulk_upsert__ has a maximum chunk size of 1000 transactions. That means any transaction block on Neo4j will have more than 1000 nested statements.

==== Update

Update consists into the method that will update information into a document, by modifying an existing property or adding a new one; to a single document or multiple ones. The behaviour varies according to the instruction passed to Mongo.

===== $set

*$set* clause updates a single document. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section, and now we want to update the *room*, which is *Auditorium*, to *Auditorium2*. We have to run the following instruction:

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "room": "Auditorium2"} })
----

This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update.

**Updated Nodes**

* The node with _room: "Auditorium"_ now will have the property _room_ with the value of _"Auditorium2"_.

Compare both graphs:

*Before the update*

image::resources/images/graph1.png[]


*After the update*

image::resources/images/graph5.png[]


Let's assume we have inserted another talk in Mongo:

[source]
----
db.talks.insert(  { "session": { "title": "First steps with React", "abstract": "A little about React and how helpful it can be to your projects." }, "topics":  ["keynote", "javascript"], "room": "Auditorium2", "timeslot": "Wed 29th, 10:30-11:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/react_developer", "picture": "http://www.reactiospeakers.org/wp-content/uploads/2015/09/peter-220x220.jpeg" } } );
----

image::resources/images/graph6.png[]


Note that both talks should be held at _Auditorium2_. If we run the following command:

[source]
----
db.talks.update({ "room": "Auditorium2"}, { $set: { "room": "Auditorium"} })
----

*Only the first document found by Mongo will be updated*, as shown on the image below.

image::resources/images/graph7.png[]

If we want to change all documents, we must use _multi_ parameter, described in the following section.

Many properties can be changed with a single _update_ clause. For example, if we run

[source]
----
db.talks.update({ "room": "Auditorium2"}, { $set: { "room": "Auditorium", "timeslot": "Wed 29th, 10:00-11:30" } })
----

We will have both properties, _room_ and _timeslot_, updated into the graph.

image::resources/images/graph8.png[]


===== $unset

*$unset* clause updates a single document by removing a property on a document. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section, and now we want to remove the __timeslot__ property for the talk that has its __room__ as **Auditorium**. We have to run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { $unset: { timeslot:""  } });
----

Compare both graphs:

*Before the update*

image::resources/images/graph8.png[]


*After the update*

image::resources/images/graph9.png[]


This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update, removing the property (notice on the node on the left side of the image).

**Updated Nodes by removing a property**

* The node with _room: "Auditorium"_ now will have the property _timeslot_ removed from it.

*Only the first document found by Mongo will be updated and have __timeslot__ property removed*. If we want to change all documents, we must use _multi_ parameter, described in the following section.

Many properties can be changed with a single _update_ clause. For example, if we run

[source]
----
db.talks.update({ "room": "Auditorium"}, { $unset: { "room": "", "timeslot": "" } })
----

We will have both properties, _room_ and _timeslot_, removed of the node into the graph.

image::resources/images/graph10.png[]


__$unset__ can also remove connected nodes and relationships. Assuming our default __talks__ example:

image::resources/images/graph1.png[]


If we run:

[source]
----
db.talks.update({ room: "Auditorium" }, { $unset: { session:""  } });
----

In Neo4j it will cause a removal of the node with the label __session__ for the room with the property __Auditorium__ and also the removal os the relationship connecting __talks__ and __session__.

image::resources/images/graph11.png[]


===== Updating without $set or $unset (document replacement)

It is also possible to update a document by specifying the entire change desired on it. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section.

image::resources/images/graph1.png[]


Now we want to update the document to select the one whose __room__ will be __Auditorium__ and clear all the root data and have only a property called __level__, which value will be __intermediate__. We have to run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { level: "Intermediate"  } );
----


This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update.

image::resources/images/graph12.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed and only __level__ property will be created and will remain. So we will have d:Documents:talks with its **_id** and a **level**.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed. We will end up with a single node, without any relationship.

Attention: Calling an update clause without __$set__ or __$unset__ will lead to property overriding, not concatenating with the existing ones.

It is also possible to run an update clause that contains a nested document as an argument. Imagine our default __talks__ example:

image::resources/images/graph1.png[]

Then we run:

[source]
----
db.talks.update({ room: "Auditorium" },  { conference: { name: "GraphConnect", city: "London" }   });
----

This instruction will remove all the properties from the __talks__ node (but it will still being the root node). A new node, with the label __conference__, will be created. Also, a relationship between __talks__ and __conference__ will be made:

image::resources/images/graph13.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed. So we will have d:Documents:talks with its **_id** only, with any remaining property. All the connected nodes (__session__ and __speaker__) and its properties will be removed.
* A new node, **Document::conference**, will be created, with the properties __name__ and __city__.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed from the original __talks__ node. A new relationship between __talks__ and __conference__ will be made.

We can also run a composite update clause where we create a new node and also update the root node:

[source]
----
db.talks.update({ room: "Auditorium" },  { conference: { name: "GraphConnect", city: "London" }, level: "intermediate"   });
----

This instruction will remove all the properties from the __talks__ node (but it will still being the root node). It will also create a **level** property on __talks__, with **intermediate** value. A new node, with the label __conference__, will be created. Also, a relationship between __talks__ and __conference__ will be made:

image::resources/images/graph14.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed. So we will have d:Documents:talks with its **_id** and a new property, **level**. All the connected nodes (__session__ and __speaker__) and its properties will be removed.
* A new node, **Document::conference**, will be created, with the properties __name__ and __city__.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed from the original __talks__ node. A new relationship between __talks__ and __conference__ will be made.

===== multi

We can update all the documents that match to a following criteria. Following the example above, to update all document _rooms_ to _Auditorium_, we should run:

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "room": "Auditorium2"} }, { multi: true } )
----

_multi: true_ will update all documents that match the specified clauses. This behaviour will also be reflected into Neo4j - all Nodes will be updated. So, if before the clause we had:

Before the update:

image::resources/images/graph15.png[]

After the update:

image::resources/images/graph16.png[]

**Nodes**

* Two nodes with *room* setted to *Auditorium2*

After running the update clause with _multi_ parameter, we end up with:

**Updated Nodes**

* The two nodes now have _room_ setted for _Auditorium_.

===== Inserting new properties

Update clauses also can be used for inserting new properties into documents. This will impact into a new property for a node. Let's assume the _talks_ previously inserted. Let's set a _level_ property for all the talks that will happen into _Auditorium_ room, pointing that they require an _intermediate_ level. Before running the update clause, we have the following into Neo4j graph:

* Two nodes labelled as *Document:talks* without a _level_ property.

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "level": "intermediate"} }, { multi: true })
----

After running the update clause, we have:

image::resources/images/graph17.png[]


* The same two nodes labelled as *Document:talks*, now with a _level_ property, containing _"intermediate"_ as its value.



===== Creating new documents by an update action

Let's assume the graph below:

image::resources/images/graph1.png[]


If the update clause does not match any document, by default a new document is not created. However, if you pass the parameter _{upsert: true}_, a new document is created. For example, assume we run the following clause:


[source]
----
db.talks.update({ "room": "Auditorium4"}, { $set: { "session": { "title": "Introduction to Neo4j", "abstract": "First steps with Neo4j, basic configuration and data modelling." }, "topics":  ["keynote", "databases"], "room": "Auditorium4", "timeslot": "Wed 29th, 13:30-14:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/neo4j" } } })
----

At the moment we do not have any document that matches with _room_ _Auditorium4_. If we do not specify anything, nothing is done to Mongo or Neo4j and we end up with a graph identical to the initial one:

image::resources/images/graph1.png[]


However, if we specify the _upsert_ as a _true_ parameter,

[source]
----
db.talks.update({ "room": "Auditorium4"}, { $set: { "session": { "title": "Introduction to Neo4j", "abstract": "First steps with Neo4j, basic configuration and data modelling." }, "topics":  ["keynote", "databases"], "room": "Auditorium4", "timeslot": "Wed 29th, 13:30-14:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/neo4j" } } }, {upsert: true})
----

A new document will be inserted into Mongo and a new group of nodes and relationships will be inserted into Neo4j. SO, after running the above query, we will have:

image::resources/images/graph18.png[]


**Update nodes**

* None

**Inserted nodes**

* _Document:talks_ - a new node is created, with _room_ setted for _Auditorium4_ and timeslot as _Wed 29th, 13:30-14:30_.
* _Document:session_ - Node created from Nested Document.
* _Document:speaker_ - also nested Document.

===== Creating new nodes by an update action

We can also invoke an update action that contains a nested Document. For example, imagine that we have the following document in Mongo, that we have been using in the past examples:

[source, javascript]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "tracks": [{ "main":"Java" }, { "second":"Languages" }],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

At Neo4j, we have:

image::resources/images/graph1.png[]


**Nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.


And then we run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { $set: { conference: { name: "GraphConnect", city: "London" }  } });
----

This will cause the following update in Mongo:

[source, javascript]
----
{
  "session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics" : [
    "keynote",
    "spring"
  ],
  "room" : "Auditorium",
  "timeslot" : "Wed 29th, 09:30-10:30",
  "speaker" : {
    "name" : "<NAME>",
    "bio" : "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter" : "https://twitter.com/springjuergen",
    "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  },
  "conference" : {
    "name" : "GraphConnect",
    "city" : "London"
  }
}
----

Note that the nested document __conference__ has been inserted. This will be translated as a new node and a new relationship into Neo4j:

image::resources/images/graph19.png[]


**Created by update action Nodes:** 

* _Document:conference_ - Simple node with the properties __name__ and __city__.

**Created by update action Relationchips:** 

* A relationship that connects *talks* and *conference* nodes, called **talks_conference**

==== Delete

It is possible to remove documents from MongoDB my calling ```db.[your_collection].remove()``` method. If you want to remove all the documents from **talks** collection, for example, you should call

```
db.talks.remove({})
```
So let's imagine that we had **two** nodes on talks, previously inserted. Each node has relationships and connected nodes:

image::resources/images/graph15.png[]


* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.
* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.


 By calling ```db.talks.remove({})```, we will remove all **talks** and their relationships and connected nodes. We end up with the removal of all elements listed above.

===== Removing relationships

When a node will be removed, the nodes created from nested documents will also be removed. Also, all the relationships between these nodes will be deleted, to avoid orphans.

===== Removing nodes with clauses

It is also possible to specify a document parameter that refers to the document that we want to remove. For example, we can run:

```
db.talks.remove( { room : "Auditorium" }, 1 )
```

This will remove a single document with **room** marked as **Auditorium**.

Before the update:

image::resources/images/graph15.png[]


After the update:

image::resources/images/graph20.png[]


The translation will be held the same way for Neo4j - The corresponding **Document::talks** node will be removed with all his nested information.


=== Customising the data to be imported

It is possible to specify which collections should be imported to Neo4j from MongoDB.

When invoking __mongo-connector__ command it is possible to pass **-n** as an argument and list the collections to be imported following the format
```
db_name.collection.name
```

For example, imagine that we switched to a database called __test__ in Mongo:
```
use test<commit_msg>Use absolute URIs for included images<commit_after>= Neo4j Doc Manager
:toc:
:toclevels: 6
:sectnums:

toc::[]

== Our Goal

=== TL;DR

What's Neo4j Doc Manager? It is a tool that enables you to migrate existing documents into MongoDB to a Neo4j graph structure. You just run it in background and the information that is in MongoDB will be imported to a graph.

=== A little longer explanation

Neo4j is an OLTP graph database which excels at querying data relationships, which is a weakness of other NoSQL and SQL solutions.  We created the Neo4j Doc Manager for the Mongo Connector to allow MongoDB developers to store JSON data in Mongo while querying the relationship between the data using Neo4j.  We call this polyglot persistence - using the database best suited for the type of data and querying your application requires.

MongoDB stores data as JSON documents, while Neo4j stores data as graphs.  In order to enable graph-based querying of Mongo data, we need to determine how to map between these two different data structures.  Our initial goal was to implement a default mapping plan, covering the most well-known cases. We wanted to follow convention instead of requiring configuration. We collected some generic Mongo Document structures based on community feedback and structured the mapping based on them. It's important to note that we are opened for further suggestions and improvements.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/doc_to_graph.png[]

This project is based upon the link:https://github.com/10gen-labs/mongo-connector[Mongo Connector]. It provides a simple protocol to transfer data from MongoDB to another database. While Mongo publishes Doc Manager implementations for other databases, they did not provide an implementation for Neo4j.  You can grab more detailed information about **mongo-connector** into the official link:https://github.com/10gen-labs/mongo-connector/wiki[Project Wiki].

The Mongo Connector requires creating a MongoDB link:http://docs.mongodb.org/manual/tutorial/deploy-replica-set/[replica set]. An **OplogThread** thread then will listen to all CUD actions occurring in MongoDB. The **mongo-connector** provides an interface to collect the events caught by the OplogThread. The communication interface is implemented into a structure called **DocManager**, which can properly receive and handle Mongo documents and information about the database and its collections.

By extending the **DocManager** class, we have created a Configuration that interacts with Neo4j. Some methods are required to be implemented to keep Mongo Connector protocol consistent. The next section describes in detail how **neo4j_doc_manager** handles each of these methods.

Detailed documentation for the *DocManager* superclass and its protocol can be found link:https://github.com/10gen-labs/mongo-connector/wiki/Writing-Your-Own-DocManager[here]

== About Neo4j DocManager

=== Setup

==== Install neo4j-doc-manager

The preferred method of installation is with the pip package manager:

```
pip install neo4j-doc-manager --pre
```

===== Alternate installation

If you experience any issues with pip you may use the following installation method:

First of all, install link:https://github.com/10gen-labs/mongo-connector[mongo-connector]. This can be easily done with the following command:

```
pip install mongo-connector
```

Now install **neo4j_doc_manager** by cloning this repository and setting the `PYTHONPATH` to it's local directory:

```
git clone https://github.com/neo4j-contrib/neo4j_doc_manager.git
cd neo4j_doc_manager
export PYTHONPATH=.
```

You also must install **py2neo**:

```
pip install py2neo
```

==== Start Neo4j and Mongo

Ensure that you have a Neo4j instance up and running.

If you have authentication enabled for Neo4j, be sure to set **NEO4J_AUTH** environment variable, containing your user and password. 

```
export NEO4J_AUTH=<user>:<password>
```

Ensure that mongo is running a *replica set*. To initiate a replica set start mongo with:

```
mongod --replSet myDevReplSet
```

Then open link:http://docs.mongodb.org/master/tutorial/getting-started-with-the-mongo-shell[mongo-shell] and run:

```
rs.initiate()
```

Please refer to link:https://github.com/10gen-labs/mongo-connector/wiki/FAQ[Mongo Connector FAQ] for more information.


==== Start the Neo4j Doc Manager service

To start the service, run the following command:

```
mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager
```

* **-m** provides Mongo endpoint
* **-t** provides Neo4j endpoint. Be sure to specify the protocol (http).
* **-d** specifies Neo4j Doc Manager.


=== Methods

==== Constructor

By invoking **Neo4j Doc Manager** initialisation command with proper parameters ( `mongo-connector -m [mongo_url] -t [neo4j_server_url] -d neo4j_doc_manager` ), the **Neo4jDocManager** constructor is called.

Constructor receives the following arguments:
[source, python]
----
(self, url, auto_commit_interval=DEFAULT_COMMIT_INTERVAL,
                 unique_key='_id', chunk_size=DEFAULT_MAX_BULK, **kwargs)

----

**url** corresponds the address where a Neo4j server instance is running.

**unique_key** corresponds to the identifier refers to the unique key that is being used in Mongo. Default value is *_id* .

===== Authentication

If you have authentication enabled for Neo4j, be sure to set **NEO4J_AUTH** environment variable, containing your user and password. 

```
export NEO4J_AUTH=<user>:<password>
```

If authentication is not enabled on Neo4j, no action is required. To disable authentication on Neo4j, go to Neo4j install directory, and then edit __conf/neo4j-server.properties__ :

```
dbms.security.auth_enabled=false
```

==== Upsert

Upsert consists into the method that creates new nodes and relationships given a Mongo Document. The method signature is described as below:
[source, python]
----
upsert(self, doc, namespace, timestamp):
----

Basically we **translate every element of a collection into a new node**. Since the elements can be composite, we have adopted some patterns to properly convert each document into a group of nodes and relationships:

* Each new node will be receive *Document* Label
* Also the document type (the referred collection from the incoming document) will also be inserted as a node label
* Document id will be propagated to the node. That means node will have the same '_id' that Mongo Document has.
* If the document contains the elements below, they will recursively be transformed into new nodes as well
  ** a nested document
  ** an array of documents
* All the other types of data into the document will be translated into node properties.

In terms of relationships, every time we find composite documents, we will stablish a relationship between the root document and the nested document.

To clarify our scenario, let's imagine an empty MongoDB instance.
Let's also consider an empty instance of Neo4j.

===== Simple case

We then run the following statement into *mongo*, to insert a talk into a collection of talks:

[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----

This will insert the following document into Mongo:
[source, json]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

This will be reflected as follows into Neo4j:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/neograph1.png[]

Check the detailed generated graph:


image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]

**Created nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.

The node chain is preserved. For example, imagine that you insert the following document in MongoDB:

[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.", "conference": { "city": "London" } }, "topics":  ["keynote", "spring"], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----


[source, json]
----
{
  "_id" : ObjectId("560dd583cf74773fae3fd001"),
  "session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.",
    "conference" : {
      "city" : "London"
    }
  },
  "topics" : [
    "keynote",
    "spring"
  ],
  "room" : "Auditorium",
  "timeslot" : "Wed 29th, 09:30-10:30",
  "speaker" : {
    "name" : "<NAME>",
    "bio" : "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter" : "https://twitter.com/springjuergen",
    "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

In Neo4j, we will have:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph2.png[]


**Created nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.
* _Document:conference_ - a Node that is nested to _session_.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.
* A relationship that connects *session* and *conference* nodes, called **session_conference**.

===== Case containing a JSON Array

Now let's insert the following data. Note the nested JSON array represented by **tracks**:
[source]
----
db.talks.insert(  { "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics":  ["keynote", "spring"], "tracks": [{ "main":"Java" }, { "second":"Languages" }], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "<NAME>", "bio": "<NAME> is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" } } );
----

[source, javascript]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "tracks": [{ "main":"Java" }, { "second":"Languages" }],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

The above document will be translated into Neo4j as follows:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph3.png[]


**Created nodes:**

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).

* _Document:tracks0_ - A node that represents the first JSON of **tracks** array [at index 0]. It contains the propagated **talks** id, plus the properties of the nested document.
* _Document:tracks1_ - A node that represents the second JSON of **tracks** array [at index 1]. It contains the propagated **talks** id, plus the properties of the nested document.

* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Created Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.
* A relationship that connects *talks* and the first element of *tracks* array (_tracks0_), called *talks_tracks0*
* A relationship that connects *talks* and the second element of *tracks* array (_tracks1_), called *talks_tracks1*

===== Case containing Mongo documents joined by an _id reference

Let's imagine now an explicit *_id* reference between two documents, such as:

[source]
----
db.places.insert({"_id": "32434ab234324", "name": "The cool place", "url": "cool.example.net" })
----

[source, javascript]
----
{ 
  "_id": "32434ab234324", 
  "name": "The cool place", 
  "url": "cool.example.net" 
}
----

[source]
----
db.people.insert({ "name": "Michael", "places_id": "32434ab234324", "url": "neo4j.com/Michael" })
----

[source, javascript]
----
{
  "name": "Michael", 
  "places_id": "32434ab234324", 
  "url": "neo4j.com/Michael" 
}
----

Note that two documents were inserted, and *people* references *place* explicitly by *id*. __Neo4j Doc Manager__ will map every field that ends with ** \_id ** into an explicit relationship. First, we run a *MERGE* to see if the respective node exists. In the above example, we insert a _place_, and then a _people_. When inserting the _people_ type, the connector will identify an explicit _id_ relationship, through __places\_id__ , and will try to find the respective node. If it does exist (and it should), a relationship between the two nodes will be created.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph4.png[]


**Created nodes:**

* _Document:places_ - Simple root node, with the properties _name_ and _url_ and an *_id*.
* _Document:people_ - Another node, that comes from an different _upsert_ method call. It creates another simple node, with the properties _name_ and _url_.

**Created Relationships:**

* A relationship that connects *people* and *places* nodes is created due to the property *places_id* on _people_ node. It is called **people_places**.

==== Bulk Upsert

If you already have data inserted on your MongoDB, the first time you run _Neo4j DocManager_ the **bulk_upsert** method will be called. It acts as a normal **upsert**, described in previous section, but all the database actions will be batched in a single transaction.

This will avoid a massive commit into Neo4j if the current Mongo database already has several documents. This will also avoid inconsistencies on an initial import.

Keep in mind that batch_upsert tends to have more performance if you are import a huge amount of data. The key to have this method called is the absence of a file called __oplog.timestamp__. If this file is not present, the document import will happen via __bulk_upsert__.

This can be useful if you call a __mongoimport__ commmand that will bring up a huge amount of data. For this scenario, you could manually remove the __oplog.timestamp__, which is automatically created the first time you call **mongo-connector** command. This file usually lives on the root of your **neo4j-doc-manager** Python Package project. 

Of course you do not have to remove the file. **bulk_upsert** is nor mandatory, but it can help you to achieve a better performance on situations where you have many documents to bring to Neo4j.

__bulk_upsert__ has a maximum chunk size of 1000 transactions. That means any transaction block on Neo4j will have more than 1000 nested statements.

==== Update

Update consists into the method that will update information into a document, by modifying an existing property or adding a new one; to a single document or multiple ones. The behaviour varies according to the instruction passed to Mongo.

===== $set

*$set* clause updates a single document. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section, and now we want to update the *room*, which is *Auditorium*, to *Auditorium2*. We have to run the following instruction:

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "room": "Auditorium2"} })
----

This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update.

**Updated Nodes**

* The node with _room: "Auditorium"_ now will have the property _room_ with the value of _"Auditorium2"_.

Compare both graphs:

*Before the update*

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


*After the update*

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph5.png[]


Let's assume we have inserted another talk in Mongo:

[source]
----
db.talks.insert(  { "session": { "title": "First steps with React", "abstract": "A little about React and how helpful it can be to your projects." }, "topics":  ["keynote", "javascript"], "room": "Auditorium2", "timeslot": "Wed 29th, 10:30-11:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/react_developer", "picture": "http://www.reactiospeakers.org/wp-content/uploads/2015/09/peter-220x220.jpeg" } } );
----

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph6.png[]


Note that both talks should be held at _Auditorium2_. If we run the following command:

[source]
----
db.talks.update({ "room": "Auditorium2"}, { $set: { "room": "Auditorium"} })
----

*Only the first document found by Mongo will be updated*, as shown on the image below.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph7.png[]

If we want to change all documents, we must use _multi_ parameter, described in the following section.

Many properties can be changed with a single _update_ clause. For example, if we run

[source]
----
db.talks.update({ "room": "Auditorium2"}, { $set: { "room": "Auditorium", "timeslot": "Wed 29th, 10:00-11:30" } })
----

We will have both properties, _room_ and _timeslot_, updated into the graph.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph8.png[]


===== $unset

*$unset* clause updates a single document by removing a property on a document. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section, and now we want to remove the __timeslot__ property for the talk that has its __room__ as **Auditorium**. We have to run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { $unset: { timeslot:""  } });
----

Compare both graphs:

*Before the update*

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph8.png[]


*After the update*

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph9.png[]


This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update, removing the property (notice on the node on the left side of the image).

**Updated Nodes by removing a property**

* The node with _room: "Auditorium"_ now will have the property _timeslot_ removed from it.

*Only the first document found by Mongo will be updated and have __timeslot__ property removed*. If we want to change all documents, we must use _multi_ parameter, described in the following section.

Many properties can be changed with a single _update_ clause. For example, if we run

[source]
----
db.talks.update({ "room": "Auditorium"}, { $unset: { "room": "", "timeslot": "" } })
----

We will have both properties, _room_ and _timeslot_, removed of the node into the graph.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph10.png[]


__$unset__ can also remove connected nodes and relationships. Assuming our default __talks__ example:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


If we run:

[source]
----
db.talks.update({ room: "Auditorium" }, { $unset: { session:""  } });
----

In Neo4j it will cause a removal of the node with the label __session__ for the room with the property __Auditorium__ and also the removal os the relationship connecting __talks__ and __session__.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph11.png[]


===== Updating without $set or $unset (document replacement)

It is also possible to update a document by specifying the entire change desired on it. For example, imagine we have inserted the _talks_ previously described into _Upsert_ section.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


Now we want to update the document to select the one whose __room__ will be __Auditorium__ and clear all the root data and have only a property called __level__, which value will be __intermediate__. We have to run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { level: "Intermediate"  } );
----


This instruction will get the first document in Mongo that matches with the specified criteria and generate an update method call into _Neo4j Doc Manager_. Considering we have a document previously inserted into Mongo by the Upsert example, we will have a single update.

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph12.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed and only __level__ property will be created and will remain. So we will have d:Documents:talks with its **_id** and a **level**.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed. We will end up with a single node, without any relationship.

Attention: Calling an update clause without __$set__ or __$unset__ will lead to property overriding, not concatenating with the existing ones.

It is also possible to run an update clause that contains a nested document as an argument. Imagine our default __talks__ example:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]

Then we run:

[source]
----
db.talks.update({ room: "Auditorium" },  { conference: { name: "GraphConnect", city: "London" }   });
----

This instruction will remove all the properties from the __talks__ node (but it will still being the root node). A new node, with the label __conference__, will be created. Also, a relationship between __talks__ and __conference__ will be made:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph13.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed. So we will have d:Documents:talks with its **_id** only, with any remaining property. All the connected nodes (__session__ and __speaker__) and its properties will be removed.
* A new node, **Document::conference**, will be created, with the properties __name__ and __city__.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed from the original __talks__ node. A new relationship between __talks__ and __conference__ will be made.

We can also run a composite update clause where we create a new node and also update the root node:

[source]
----
db.talks.update({ room: "Auditorium" },  { conference: { name: "GraphConnect", city: "London" }, level: "intermediate"   });
----

This instruction will remove all the properties from the __talks__ node (but it will still being the root node). It will also create a **level** property on __talks__, with **intermediate** value. A new node, with the label __conference__, will be created. Also, a relationship between __talks__ and __conference__ will be made:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph14.png[]


**Updated Nodes**

* The node with _room: "Auditorium"_ now will have all it's properties removed. So we will have d:Documents:talks with its **_id** and a new property, **level**. All the connected nodes (__session__ and __speaker__) and its properties will be removed.
* A new node, **Document::conference**, will be created, with the properties __name__ and __city__.

**Updated Relationships**

* By running the previous statement, all the connected nodes and relationships will be removed from the original __talks__ node. A new relationship between __talks__ and __conference__ will be made.

===== multi

We can update all the documents that match to a following criteria. Following the example above, to update all document _rooms_ to _Auditorium_, we should run:

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "room": "Auditorium2"} }, { multi: true } )
----

_multi: true_ will update all documents that match the specified clauses. This behaviour will also be reflected into Neo4j - all Nodes will be updated. So, if before the clause we had:

Before the update:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph15.png[]

After the update:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph16.png[]

**Nodes**

* Two nodes with *room* setted to *Auditorium2*

After running the update clause with _multi_ parameter, we end up with:

**Updated Nodes**

* The two nodes now have _room_ setted for _Auditorium_.

===== Inserting new properties

Update clauses also can be used for inserting new properties into documents. This will impact into a new property for a node. Let's assume the _talks_ previously inserted. Let's set a _level_ property for all the talks that will happen into _Auditorium_ room, pointing that they require an _intermediate_ level. Before running the update clause, we have the following into Neo4j graph:

* Two nodes labelled as *Document:talks* without a _level_ property.

[source]
----
db.talks.update({ "room": "Auditorium"}, { $set: { "level": "intermediate"} }, { multi: true })
----

After running the update clause, we have:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph17.png[]


* The same two nodes labelled as *Document:talks*, now with a _level_ property, containing _"intermediate"_ as its value.



===== Creating new documents by an update action

Let's assume the graph below:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


If the update clause does not match any document, by default a new document is not created. However, if you pass the parameter _{upsert: true}_, a new document is created. For example, assume we run the following clause:


[source]
----
db.talks.update({ "room": "Auditorium4"}, { $set: { "session": { "title": "Introduction to Neo4j", "abstract": "First steps with Neo4j, basic configuration and data modelling." }, "topics":  ["keynote", "databases"], "room": "Auditorium4", "timeslot": "Wed 29th, 13:30-14:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/neo4j" } } })
----

At the moment we do not have any document that matches with _room_ _Auditorium4_. If we do not specify anything, nothing is done to Mongo or Neo4j and we end up with a graph identical to the initial one:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


However, if we specify the _upsert_ as a _true_ parameter,

[source]
----
db.talks.update({ "room": "Auditorium4"}, { $set: { "session": { "title": "Introduction to Neo4j", "abstract": "First steps with Neo4j, basic configuration and data modelling." }, "topics":  ["keynote", "databases"], "room": "Auditorium4", "timeslot": "Wed 29th, 13:30-14:30", "speaker": { "name": "<NAME>", "bio": "Senior Developer.", "twitter": "https://twitter.com/neo4j" } } }, {upsert: true})
----

A new document will be inserted into Mongo and a new group of nodes and relationships will be inserted into Neo4j. SO, after running the above query, we will have:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph18.png[]


**Update nodes**

* None

**Inserted nodes**

* _Document:talks_ - a new node is created, with _room_ setted for _Auditorium4_ and timeslot as _Wed 29th, 13:30-14:30_.
* _Document:session_ - Node created from Nested Document.
* _Document:speaker_ - also nested Document.

===== Creating new nodes by an update action

We can also invoke an update action that contains a nested Document. For example, imagine that we have the following document in Mongo, that we have been using in the past examples:

[source, javascript]
----
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey",
    "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics":  ["keynote", "spring"], 
  "tracks": [{ "main":"Java" }, { "second":"Languages" }],
  "room": "Auditorium",
  "timeslot": "Wed 29th, 09:30-10:30",
  "speaker": {
    "name": "<NAME>",
    "bio": "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter": "https://twitter.com/springjuergen",
    "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  }
}
----

At Neo4j, we have:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph1.png[]


**Nodes:** 

* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.

**Relationships:**

* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.


And then we run the following instruction:

[source]
----
db.talks.update({ room: "Auditorium" }, { $set: { conference: { name: "GraphConnect", city: "London" }  } });
----

This will cause the following update in Mongo:

[source, javascript]
----
{
  "session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015."
  },
  "topics" : [
    "keynote",
    "spring"
  ],
  "room" : "Auditorium",
  "timeslot" : "Wed 29th, 09:30-10:30",
  "speaker" : {
    "name" : "<NAME>",
    "bio" : "<NAME> is co-founder of the Spring Framework open source project.",
    "twitter" : "https://twitter.com/springjuergen",
    "picture" : "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg"
  },
  "conference" : {
    "name" : "GraphConnect",
    "city" : "London"
  }
}
----

Note that the nested document __conference__ has been inserted. This will be translated as a new node and a new relationship into Neo4j:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph19.png[]


**Created by update action Nodes:** 

* _Document:conference_ - Simple node with the properties __name__ and __city__.

**Created by update action Relationchips:** 

* A relationship that connects *talks* and *conference* nodes, called **talks_conference**

==== Delete

It is possible to remove documents from MongoDB my calling ```db.[your_collection].remove()``` method. If you want to remove all the documents from **talks** collection, for example, you should call

```
db.talks.remove({})
```
So let's imagine that we had **two** nodes on talks, previously inserted. Each node has relationships and connected nodes:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph15.png[]


* _Document:talks_ - talks is the root node, coming from Mongo Document Collection, with an id that also comes from MongoDB. Non nested Documents are converted into regular properties, such as "room", "topics" and "timeslot" (a common String array).
* _Document:session_ - Nested Document. Inner key/values are converted into Node properties. Note that the id incoming from root *talks* collection is propagated to this Node. Also, note that this node is labelled as its direct document key, in this case, *session*.
* _Document:speaker_ - also nested Document.
* A relationship that connects *talks* and *session* nodes, called **talks_session**,
* A relationship that connects *talks* and *speaker* nodes, called **talks_speaker**.


 By calling ```db.talks.remove({})```, we will remove all **talks** and their relationships and connected nodes. We end up with the removal of all elements listed above.

===== Removing relationships

When a node will be removed, the nodes created from nested documents will also be removed. Also, all the relationships between these nodes will be deleted, to avoid orphans.

===== Removing nodes with clauses

It is also possible to specify a document parameter that refers to the document that we want to remove. For example, we can run:

```
db.talks.remove( { room : "Auditorium" }, 1 )
```

This will remove a single document with **room** marked as **Auditorium**.

Before the update:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph15.png[]


After the update:

image::https://raw.githubusercontent.com/neo4j-contrib/neo4j_doc_manager/master/docs/resources/images/graph20.png[]


The translation will be held the same way for Neo4j - The corresponding **Document::talks** node will be removed with all his nested information.


=== Customising the data to be imported

It is possible to specify which collections should be imported to Neo4j from MongoDB.

When invoking __mongo-connector__ command it is possible to pass **-n** as an argument and list the collections to be imported following the format
```
db_name.collection.name
```

For example, imagine that we switched to a database called __test__ in Mongo:
```
use test
{'fraction_non_alphanumeric': 0.10707813883779549, 'fraction_numerical': 0.01579126643504326, 'mean_word_length': 4.546856106408706, 'pattern_counts': {'":': 272, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 66, 'lorem ipsum': 0, 'www.': 18, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1661959', 'n_tokens_mistral': 21891, 'n_tokens_neox': 20429, 'n_words': 9010}