<commit_before># IBM Data Merge Utility v4.0.0

## What Is IDMU
The IBM Data Merge Utility is an open source Java Utility that provides high performance transformation and enrichment services through a simple template based process. If you can use your word processors Mail Merge feature, then you can use the IBM Data Merge utility. The use of templates makes this transformation tool uniquely well suited to creating Scripts, Code or other complex documents. Optimized performance of small payload transformations makes it well suited to doing JSON or XML microservice transformations. When combined with enrichment services this becomes a powerful tool for the composition of complex microservice structures from a variety of sources. While this utility does not expose a Rest API directly it is designed with that in mind. See [IDMU-REST Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-REST)  

## Related Pages
1. [IDMU-CLI Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-CLI) - Command Line merge tool
1. [IDMU-REST Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-REST) - Rest Interface merge tool
1. [IDMU Users Guide](https://flatballflyer.github.io/IBM-Data-Merge-Utility/USERS_GUIDE)

---
## Overview
A surprisingly large segment of Information Processing can be thought of as consisting of Transformation Services (take this data and transform it into another format) or Enrichment Services (take this data and go get me some more data). The IDMU utilizes a template based approach to these services, in which Merging Templates produces Transformed/Enriched output. The merge process is similar to the familiar Mail Merge feature in most word processors. Potential uses for this tool include:
- Simple transformations of XML to JSON or the other way around
- Enriching a JSON request with additional data from a database
- Enriching a JSON request with additional data from a Rest data source 
- Generating a XML or HTML document with data from a database
- Generating Configurations, Script or Code based on options specified as parameters.  

## Processing Overview
Merge Directives are specified as part of the template. Each Directive is executed in order and the resulting template content is then returned as the output of the Merge. The following directives are used to drive merge processing, and any number of directives can be used.   
1. Enrich - Get some data from an external source and put it in the DataManager
1. Replace - Take some data from the data manager and Replace **Tags** with data values
1. Insert - Insert sub-templates at **Bookmarks** based on some data in the Data Manager. Sub-Templates are full-fledged templates and can contain their own directives.
1. Parse - Get some marked-up data from the DataManager, parse it and place the resulting data structure back in the Data Manager
1. Save - Save the output of the current template to an entry in the Merge Archive. (for archive creation - tar, zip, jar, gzip...)

![alt text](http://flatballflyer.github.io/IBM-Data-Merge-Utility/WebContent/images/overview.png "Logo Title Text 1")

When a template is merged, each directive is executed in order. Simple Transformation templates usually have a **Parse** directive to parse the idmuPayload value and then **Replace** and/or **Insert** directives to put that data into a template. Sub-templates are merge processed when they are inserted at a bookmark. Templates that need additional data from an outside data source such as a Database, File or Rest Service can use the **Enrich** directive to get that data. Use cases that need to generate a collection of files instead of a single output message can use the **Save** directive to create an archive with multiple files in it.  
---

## Template Content
From an IDMU perspective, template content is just a block of text, it could be JSON, XML, HTML, a Bash Script, a Java Class, a NodeJs program, a CSV file.... anything you want to generate. The text in a template can have **Replacement Tags** and **Bookmarks** that identify where in the template data is to be placed, or sub-templates are to be inserted. A template also has other attributes, including a list of **Directives** that are the instructions which drive the merge process. Here is a sample of some JSON Template content:
```
{
	name: "<NAME>",
	address: "<ADDRESS>",
	friends: [<bookmark="friend" group="test" template="friend">]
}
```  
In this template we are using &lt; and &gt; to wrap the **tags** and **bookmarks**. The tags &lt;NAME&gt; and &lt;ADDRESS&gt; will be replaced with data during the merge process (by a **Replace** directive). Sub-Templates for each "friend" will be inserted at the **friend bookmark** by an **Insert** directive. 

## The Data Manager
IDMU has an internal Object Data store called the Data Manager. This object store will be used to store request data and parameters as well as data that is fetched from an external data source (by an **Enrich** directive). This data is transient, and is released from memory after the merge is completed. Most directives will either read data from the data store and take some action, or write data to the data store for use by other directives. Data in the Data Manager is accessed via a "path" style address. For example, given this data structure:
```
{ 
	mydata: { 
		name: "fred", 
		address: "123 Anywhere",
		friends: [
			{ 	name: "allen",
		  		since: "way back" 
		  	},
			{ 	name: "betty", 
		  		since: "last week" 
		  	}
		]
	},<commit_msg>formatting<commit_after># IBM Data Merge Utility v4.0.0

## What Is IDMU
The IBM Data Merge Utility is an open source Java Utility that provides high performance transformation and enrichment services through a simple template based process. If you can use your word processors Mail Merge feature, then you can use the IBM Data Merge utility. The use of templates makes this transformation tool uniquely well suited to creating Scripts, Code or other complex documents. Optimized performance of small payload transformations makes it well suited to doing JSON or XML microservice transformations. When combined with enrichment services this becomes a powerful tool for the composition of complex microservice structures from a variety of sources. While this utility does not expose a Rest API directly it is designed with that in mind. See [IDMU-REST Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-REST)  

## Related Pages
1. [IDMU-CLI Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-CLI) - Command Line merge tool
1. [IDMU-REST Project](https://github.com/FlatBallFlyer/IBM-Data-Merge-Utility-REST) - Rest Interface merge tool
1. [IDMU Users Guide](https://flatballflyer.github.io/IBM-Data-Merge-Utility/USERS_GUIDE)

---
## Overview
A surprisingly large segment of Information Processing can be thought of as consisting of Transformation Services (take this data and transform it into another format) or Enrichment Services (take this data and go get me some more data). The IDMU utilizes a template based approach to these services, in which Merging Templates produces Transformed/Enriched output. The merge process is similar to the familiar Mail Merge feature in most word processors. Potential uses for this tool include:
- Simple transformations of XML to JSON or the other way around
- Enriching a JSON request with additional data from a database
- Enriching a JSON request with additional data from a Rest data source 
- Generating a XML or HTML document with data from a database
- Generating Configurations, Script or Code based on options specified as parameters.  

## Processing Overview
Merge Directives are specified as part of the template. Each Directive is executed in order and the resulting template content is then returned as the output of the Merge. The following directives are used to drive merge processing, and any number of directives can be used.   
1. Enrich - Get some data from an external source and put it in the DataManager
1. Replace - Take some data from the data manager and Replace **Tags** with data values
1. Insert - Insert sub-templates at **Bookmarks** based on some data in the Data Manager. Sub-Templates are full-fledged templates and can contain their own directives.
1. Parse - Get some marked-up data from the DataManager, parse it and place the resulting data structure back in the Data Manager
1. Save - Save the output of the current template to an entry in the Merge Archive. (for archive creation - tar, zip, jar, gzip...)

![alt text](http://flatballflyer.github.io/IBM-Data-Merge-Utility/WebContent/images/overview.png "Logo Title Text 1")

When a template is merged, each directive is executed in order. Simple Transformation templates usually have a **Parse** directive to parse the idmuPayload value and then **Replace** and/or **Insert** directives to put that data into a template. Sub-templates are merge processed when they are inserted at a bookmark. Templates that need additional data from an outside data source such as a Database, File or Rest Service can use the **Enrich** directive to get that data. Use cases that need to generate a collection of files instead of a single output message can use the **Save** directive to create an archive with multiple files in it.  

---

## Template Content
From an IDMU perspective, template content is just a block of text, it could be JSON, XML, HTML, a Bash Script, a Java Class, a NodeJs program, a CSV file.... anything you want to generate. The text in a template can have **Replacement Tags** and **Bookmarks** that identify where in the template data is to be placed, or sub-templates are to be inserted. A template also has other attributes, including a list of **Directives** that are the instructions which drive the merge process. Here is a sample of some JSON Template content:
```
{
	name: "<NAME>",
	address: "<ADDRESS>",
	friends: [<bookmark="friend" group="test" template="friend">]
}
```  
In this template we are using &lt; and &gt; to wrap the **tags** and **bookmarks**. The tags &lt;NAME&gt; and &lt;ADDRESS&gt; will be replaced with data during the merge process (by a **Replace** directive). Sub-Templates for each "friend" will be inserted at the **friend bookmark** by an **Insert** directive. 

## The Data Manager
IDMU has an internal Object Data store called the Data Manager. This object store will be used to store request data and parameters as well as data that is fetched from an external data source (by an **Enrich** directive). This data is transient, and is released from memory after the merge is completed. Most directives will either read data from the data store and take some action, or write data to the data store for use by other directives. Data in the Data Manager is accessed via a "path" style address. For example, given this data structure:
```
{ 
	mydata: { 
		name: "fred", 
		address: "123 Anywhere",
		friends: [
			{ 	name: "allen",
		  		since: "way back" 
		  	},
			{ 	name: "betty", 
		  		since: "last week" 
		  	}
		]
	},
{'fraction_non_alphanumeric': 0.06522131887985547, 'fraction_numerical': 0.0027100271002710027, 'mean_word_length': 4.3174831892411145, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1581038', 'n_tokens_mistral': 2855, 'n_tokens_neox': 2680, 'n_words': 1616}