<commit_before>= Load XML

== Load XML Introduction

Many existing (enterprise) applications, endpoints and files use XML as data exchange format.

To make these datastructures available to Cypher, you can use `apoc.load.xml`.
It takes a file or http URL and parses the XML into a map datastructure.

NOTE: in previous releases we've had `apoc.load.xmlSimple`. This is now deprecated and got superseeded by
`apoc.load.xml(url, [xPath], [config], true)`.Simple XML Format

See the following usage-examples for the procedures.

== Example File

"How do you access XML doc attributes in children fields ?"

(Thanks <NAME>)

For example, if my XML file is the example https://msdn.microsoft.com/en-us/library/ms762271%28v=vs.85%29.aspx[book.xml provided by Microsoft].

[source,xml]
----
<?xml version="1.0"?>
<catalog>
   <book id="bk101">
      <author><NAME></author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications
      with XML.</description>
   </book>
   <book id="bk102">
      <author><NAME></author>
      <title>Midnight Rain</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-12-16</publish_date>
      <description>A former architect battles corporate zombies,
...
----

We have the file here, https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml[on GitHub].

== Simple XML Format

In a simpler XML representation, each type of children gets it's own entry within the parent map.
The element-type as key is prefixed with "_" to prevent collisions with attributes.

If there is a single element, then the entry will just have that element as value, not a collection.
If there is more than one element there will be a list of values.

Each child will still have its `_type` field to discern them.

Here is the example file from above loaded with `apoc.load.xmlSimple`

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml", '', {}, true)
----

[source,javascript]
----
{_type: "catalog", _book: [
  {_type: "book", id: "bk101",
    _author: [{_type: "author", _text: "<NAME>"},{_type: author, _text: "<NAME>"}],
    _title: {_type: "title", _text: "XML Developer's Guide"},
    _genre: {_type: "genre", _text: "Computer"},
    _price: {_type: "price", _text: "44.95"},
    _publish_date: {_type: "publish_date", _text: "2000-10-01"},
    _description: {_type: description, _text: An in-depth look at creating applications ....
----

=== Simple XML Examples

.Example 1
[source,cypher]
----
WITH "https://maps.googleapis.com/maps/api/directions/xml?origin=Mertens%20en%20Torfsstraat%2046,%202018%20Antwerpen&destination=Rubensstraat%2010,%202300%20Turnhout&sensor=false&mode=bicycling&alternatives=false&key=<KEY>" AS url
CALL apoc.load.xmlSimple(url) YIELD value
RETURN value._route._leg._distance._value, keys(value), keys(value._route), keys(value._route._leg), keys(value._route._leg._distance._value)
----
image::{img}/apoc.load.xmlSimple.ex1.png[width=800]

.Example 2
[source,cypher]
----
WITH "https://maps.googleapis.com/maps/api/directions/xml?origin=Mertens%20en%20Torfsstraat%2046,%202018%20Antwerpen&destination=Rubensstraat%2010,%202300%20Turnhout&sensor=false&mode=bicycling&alternatives=false&key=<KEY>" AS url
CALL apoc.load.xmlSimple(url) YIELD value
UNWIND keys(value) AS key
RETURN key, apoc.meta.type(value[key]);
----
image::{img}/apoc.load.xmlSimple.ex2.png[width=800]

== xPath

It's possible to define a xPath (optional) to selecting nodes from the XML document.

=== xPath Example

From the Microsoft's book.xml file we can get only the books that have as `genre` Computer

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.1/src/test/resources/books.xml", '/catalog/book[genre=\"Computer\"]') yield value as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['title','price'] | attr._text] as pairs
RETURN id, pairs[0] as title, pairs[1] as price
----

image::{img}/apoc.load.xml.xpath.png[width=800]

In this case we return only `id`, `title` and `prize` but we can return any other elements

We can also return just a single specific element.
For example the `author` of the book with `id = bg102`

[source,cypher]
----
call apoc.load.xml('https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.1/src/test/resources/books.xml', '/catalog/book[@id="bk102"]/author') yield value as result
WITH result._text as author
RETURN author
----

image::{img}/apoc.load.xml.xpath2.png[width=800]



== Load XML and Introspect

Let's just load it and see what it looks like.
It's returned as value map with nested `_type` and `_children` fields, per group of elements.
Attributes are turned into map-entries.
And each element into their own little map with `_type`, attributes and `_children` if applicable.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml")
----

[source,javascript]
----
{_type: catalog, _children: [
  {_type: book, id: bk101, _children: [
    {_type: author, _text: <NAME>}, 
    {_type: title, _text: XML Developer's Guide}, 
    {_type: genre, _text: Computer}, 
    {_type: price, _text: 44.95}, 
    {_type: publish_date, _text: 2000-10-01}, 
    {_type: description, _text: An in-depth look at creating applications ....
----

=== For each book, how do I access book id ?

You can access attributes per element directly.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
RETURN book.id
----

----
╒═══════╕
│book.id│
╞═══════╡
│bk101  │
├───────┤
│bk102  │
----


=== For each book, how do I access book author and title ?

==== Filter into collection

You have to filter over the sub-elements  in the `_childrens` array in this case.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
RETURN book.id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
----

----
╒═══════╤════════════════════════════════════════════════════════════════════════╕
│book.id│pairs                                                                   │
╞═══════╪════════════════════════════════════════════════════════════════════════╡
│bk101  │[[author, <NAME>], [title, XML Developer's Guide]]        │
├───────┼────────────────────────────────────────────────────────────────────────┤
│bk102  │[[author, <NAME>], [title, Midnight Rain]]                          │
----

==== How do I return collection elements?

This is not too nice, we could also just have returned the values and then grabbed them out of the list, but that relies on element-order.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | attr._text] as pairs
RETURN id, pairs[0] as author, pairs[1] as title
----

----
╒═════╤════════════════════╤══════════════════════════════╕
│id   │author              │title                         │
╞═════╪════════════════════╪══════════════════════════════╡
│bk101│<NAME>│XML Developer's Guide         │
├─────┼────────────────────┼──────────────────────────────┤
│bk102│<NAME>          │Midnight Rain                 │
----

== Extracting Datastructures

=== Turn Pairs into Map

So better is to turn them into a map with `apoc.map.fromPairs`

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
CALL apoc.map.fromPairs(pairs) yield value
RETURN id, value
----

----
╒═════╤════════════════════════════════════════════════════════════════════╕
│id   │value                                                               │
╞═════╪════════════════════════════════════════════════════════════════════╡
│bk101│{author: <NAME>, title: XML Developer's Guide}        │
├─────┼────────────────────────────────────────────────────────────────────┤
│bk102│{author: <NAME>, title: Midnight Rain}                          │
├─────┼────────────────────────────────────────────────────────────────────┤
│bk103│{author: <NAME>, title: Maeve Ascendant}                       │
----

==== Return individual Columns

And now we can cleanly access the attributes from the map.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
CALL apoc.map.fromPairs(pairs) yield value
RETURN id, value.author, value.title
----

----
╒═════╤════════════════════╤══════════════════════════════╕
│id   │value.author        │value.title                   │
╞═════╪════════════════════╪══════════════════════════════╡
│bk101│<NAME>│XML Developer's Guide         │
├─────┼────────────────────┼──────────────────────────────┤
│bk102│<NAME>          │Midnight Rain                 │
├─────┼────────────────────┼──────────────────────────────┤
│bk103│<NAME>         │Maeve Ascendant               │
----

== import xml directly

In case you don't want to transform your xml (like you do with `apoc.load.xml/apoc.load.xmlSimple` before you create nodes and relationships and you want to have a 1:1 mapping of xml into the graph you can use `apoc.xml.import`.

=== usage

[source,cypher]
----
CALL apoc.import.xml(<url>, <config>?) YIELD node
----

The procedure will return a node representing the xml document containing nodes/rels underneath mapping to the xml structure. The following mapping rules are applied:

[options="header"]
|===========
| xml | label | properties
| document | XmlDocument | _xmlVersion, _xmlEncoding
| processing instruction | XmlProcessingInstruction | _piData, _piTarget
| Element/Tag | XmlTag | _name
| Attribute | n/a | property in the XmlTag node
| Text | XmlWord | for each word a separate node is created
|===========

The nodes for the xml document are connected:

[options="header"]
|=====
| relationship type | description
| :IS_CHILD_OF | pointing to a nested xml element
| :FIRST_CHILD_OF | pointing to the first child
| :NEXT_SIBLING | pointing to the next xml element on the same nesting level
| :NEXT | produces a linear chain through the full document
| :NEXT_WORD | only produced if config map has `createNextWordRelationships:true`. Connects words in xml to a text flow.
|====

=== example

[source,cypher]<commit_msg>fix xml docs<commit_after>= Load XML

== Load XML Introduction

Many existing (enterprise) applications, endpoints and files use XML as data exchange format.

To make these datastructures available to Cypher, you can use `apoc.load.xml`.
It takes a file or http URL and parses the XML into a map datastructure.

NOTE: in previous releases we've had `apoc.load.xmlSimple`. This is now deprecated and got superseeded by
`apoc.load.xml(url, [xPath], [config], true)`.Simple XML Format

See the following usage-examples for the procedures.

== Example File

"How do you access XML doc attributes in children fields ?"

(Thanks <NAME>)

For example, if my XML file is the example https://msdn.microsoft.com/en-us/library/ms762271%28v=vs.85%29.aspx[book.xml provided by Microsoft].

[source,xml]
----
<?xml version="1.0"?>
<catalog>
   <book id="bk101">
      <author><NAME></author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications
      with XML.</description>
   </book>
   <book id="bk102">
      <author><NAME></author>
      <title>Midnight Rain</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-12-16</publish_date>
      <description>A former architect battles corporate zombies,
...
----

We have the file here, https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml[on GitHub].

== Simple XML Format

In a simpler XML representation, each type of children gets it's own entry within the parent map.
The element-type as key is prefixed with "_" to prevent collisions with attributes.

If there is a single element, then the entry will just have that element as value, not a collection.
If there is more than one element there will be a list of values.

Each child will still have its `_type` field to discern them.

Here is the example file from above loaded with `apoc.load.xmlSimple`

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml", '', {}, true)
----

[source,javascript]
----
{_type: "catalog", _book: [
  {_type: "book", id: "bk101",
    _author: [{_type: "author", _text: "<NAME>"},{_type: author, _text: "<NAME>"}],
    _title: {_type: "title", _text: "XML Developer's Guide"},
    _genre: {_type: "genre", _text: "Computer"},
    _price: {_type: "price", _text: "44.95"},
    _publish_date: {_type: "publish_date", _text: "2000-10-01"},
    _description: {_type: description, _text: An in-depth look at creating applications ....
----

=== Simple XML Examples

.Example 1
[source,cypher]
----
WITH "https://maps.googleapis.com/maps/api/directions/xml?origin=Mertens%20en%20Torfsstraat%2046,%202018%20Antwerpen&destination=Rubensstraat%2010,%202300%20Turnhout&sensor=false&mode=bicycling&alternatives=false&key=<KEY>" AS url
CALL apoc.load.xmlSimple(url) YIELD value
RETURN value._route._leg._distance._value, keys(value), keys(value._route), keys(value._route._leg), keys(value._route._leg._distance._value)
----
image::{img}/apoc.load.xmlSimple.ex1.png[width=800]

.Example 2
[source,cypher]
----
WITH "https://maps.googleapis.com/maps/api/directions/xml?origin=Mertens%20en%20Torfsstraat%2046,%202018%20Antwerpen&destination=Rubensstraat%2010,%202300%20Turnhout&sensor=false&mode=bicycling&alternatives=false&key=<KEY>" AS url
CALL apoc.load.xmlSimple(url) YIELD value
UNWIND keys(value) AS key
RETURN key, apoc.meta.type(value[key]);
----
image::{img}/apoc.load.xmlSimple.ex2.png[width=800]

== xPath

It's possible to define a xPath (optional) to selecting nodes from the XML document.

=== xPath Example

From the Microsoft's book.xml file we can get only the books that have as `genre` Computer

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.1/src/test/resources/books.xml", '/catalog/book[genre=\"Computer\"]') yield value as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['title','price'] | attr._text] as pairs
RETURN id, pairs[0] as title, pairs[1] as price
----

image::{img}/apoc.load.xml.xpath.png[width=800]

In this case we return only `id`, `title` and `prize` but we can return any other elements

We can also return just a single specific element.
For example the `author` of the book with `id = bg102`

[source,cypher]
----
call apoc.load.xml('https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.1/src/test/resources/books.xml', '/catalog/book[@id="bk102"]/author') yield value as result
WITH result._text as author
RETURN author
----

image::{img}/apoc.load.xml.xpath2.png[width=800]



== Load XML and Introspect

Let's just load it and see what it looks like.
It's returned as value map with nested `_type` and `_children` fields, per group of elements.
Attributes are turned into map-entries.
And each element into their own little map with `_type`, attributes and `_children` if applicable.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml")
----

[source,javascript]
----
{_type: catalog, _children: [
  {_type: book, id: bk101, _children: [
    {_type: author, _text: <NAME>}, 
    {_type: title, _text: XML Developer's Guide}, 
    {_type: genre, _text: Computer}, 
    {_type: price, _text: 44.95}, 
    {_type: publish_date, _text: 2000-10-01}, 
    {_type: description, _text: An in-depth look at creating applications ....
----

=== For each book, how do I access book id ?

You can access attributes per element directly.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
RETURN book.id
----

----
╒═══════╕
│book.id│
╞═══════╡
│bk101  │
├───────┤
│bk102  │
----


=== For each book, how do I access book author and title ?

==== Filter into collection

You have to filter over the sub-elements  in the `_childrens` array in this case.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
RETURN book.id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
----

----
╒═══════╤════════════════════════════════════════════════════════════════════════╕
│book.id│pairs                                                                   │
╞═══════╪════════════════════════════════════════════════════════════════════════╡
│bk101  │[[author, <NAME>], [title, XML Developer's Guide]]        │
├───────┼────────────────────────────────────────────────────────────────────────┤
│bk102  │[[author, <NAME>], [title, Midnight Rain]]                          │
----

==== How do I return collection elements?

This is not too nice, we could also just have returned the values and then grabbed them out of the list, but that relies on element-order.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | attr._text] as pairs
RETURN id, pairs[0] as author, pairs[1] as title
----

----
╒═════╤════════════════════╤══════════════════════════════╕
│id   │author              │title                         │
╞═════╪════════════════════╪══════════════════════════════╡
│bk101│<NAME>│XML Developer's Guide         │
├─────┼────────────────────┼──────────────────────────────┤
│bk102│<NAME>          │Midnight Rain                 │
----

== Extracting Datastructures

=== Turn Pairs into Map

So better is to turn them into a map with `apoc.map.fromPairs`

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
CALL apoc.map.fromPairs(pairs) yield value
RETURN id, value
----

----
╒═════╤════════════════════════════════════════════════════════════════════╕
│id   │value                                                               │
╞═════╪════════════════════════════════════════════════════════════════════╡
│bk101│{author: <NAME>, title: XML Developer's Guide}        │
├─────┼────────────────────────────────────────────────────────────────────┤
│bk102│{author: <NAME>, title: Midnight Rain}                          │
├─────┼────────────────────────────────────────────────────────────────────┤
│bk103│{author: <NAME>, title: Maeve Ascendant}                       │
----

==== Return individual Columns

And now we can cleanly access the attributes from the map.

[source,cypher]
----
call apoc.load.xml("https://raw.githubusercontent.com/neo4j-contrib/neo4j-apoc-procedures/3.3/src/test/resources/books.xml") yield value as catalog
UNWIND catalog._children as book
WITH book.id as id, [attr IN book._children WHERE attr._type IN ['author','title'] | [attr._type, attr._text]] as pairs
CALL apoc.map.fromPairs(pairs) yield value
RETURN id, value.author, value.title
----

----
╒═════╤════════════════════╤══════════════════════════════╕
│id   │value.author        │value.title                   │
╞═════╪════════════════════╪══════════════════════════════╡
│bk101│<NAME>│XML Developer's Guide         │
├─────┼────────────────────┼──────────────────────────────┤
│bk102│<NAME>          │Midnight Rain                 │
├─────┼────────────────────┼──────────────────────────────┤
│bk103│<NAME>         │Maeve Ascendant               │
----

== import xml directly

In case you don't want to transform your xml (like you do with `apoc.load.xml/apoc.load.xmlSimple` before you create nodes and relationships and you want to have a 1:1 mapping of xml into the graph you can use `apoc.xml.import`.

=== usage

[source,cypher]
----
CALL apoc.import.xml(<url>, <config>?) YIELD node
----

The procedure will return a node representing the xml document containing nodes/rels underneath mapping to the xml structure. The following mapping rules are applied:

[options="header"]
|===
| xml | label | properties
| document | XmlDocument | _xmlVersion, _xmlEncoding
| processing instruction | XmlProcessingInstruction | _piData, _piTarget
| Element/Tag | XmlTag | _name
| Attribute | n/a | property in the XmlTag node
| Text | XmlWord | for each word a separate node is created
|===

The nodes for the xml document are connected:

[options="header"]
|===
| relationship type | description
| :IS_CHILD_OF | pointing to a nested xml element
| :FIRST_CHILD_OF | pointing to the first child
| :NEXT_SIBLING | pointing to the next xml element on the same nesting level
| :NEXT | produces a linear chain through the full document
| :NEXT_WORD | only produced if config map has `createNextWordRelationships:true`. Connects words in xml to a text flow.
|===

=== example

[source,cypher]
{'fraction_non_alphanumeric': 0.2122270356290337, 'fraction_numerical': 0.020687826010078684, 'mean_word_length': 3.975368374752584, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 2, '>': 99, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 103}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2800673', 'n_tokens_mistral': 8222, 'n_tokens_neox': 7265, 'n_words': 2059}