<commit_before># Domino JNA
Cross-platform access to IBM Notes/Domino C API methods from Java

## Features
The project provides functionality that is not available in the classic Java API of IBM Notes/Domino or that is poorly implemented, for example:

* **view lookups using different key formats** (e.g. strings, numbers, dates, date ranges) and equality/**inequality searches (e.g. find less or greater than a search key)**
* **decodes all available types of view column data types** (string, string list, number, number list, datetime, datetime list) **and row data** (e.g. just note id, UNID, counts, unread flag etc.)
* read view data as **another Notes user**
* extract **read access information for rows in a view** that Domino stores in the view index
* **separation of Notes views and their data into multiple databases** (like programmatically creating private views and keeping them up-to-date incrementally)
* **dynamic filtering of view rows based on a Note id list with paging support (this really rocks!)**
* **reading categorized views** with expanded/collapsed entries and min/max level
* read **design collection** with design elements in a database
* **differential view reads** : on second view lookup, only read rows that have changed
* support for view resorting (changing the collation in C API terms)
* **direct attachment streaming** (IBM's Java API extracts files to temp disk space first to read attachment data)
* basic APIs to read note (document) item values like String/Double/Calendar single and multiple values
* several APIs to write item values and attachments (needs more work)
* **richtext item reading**: convenience method to extract text content, advanced API to read all CD records (IRichTextNavigator)
* **richtext item writing**: create richtext items with text, images, doclinks, by rendering other notes and appending other richtet items
* **richtext item conversion**: multi-level conversion of richtext item content, e.g. to **add/remove file hotspots (file icons with custom image that open file on click) independent from the actual file attachment** or do mail merge with richtext
* **richtext-html conversion** with advanced quality and access to embedded images
* **incremental data synchronization** with external databases or indexers. Generic design, sample implementation for [CQEngine](https://github.com/npgall/cqengine) and [SQLite](https://www.sqlite.org)
* **quick check if a document is editable** by a specified user (without the need to scan through author items)
* **fulltext index creation** with all available options
* supports incremental synchronization of Domino databases by **reading noteid lists of modified and deleted documents** (IBM's Java API does not return ids of deleted docs)
* searching NSF data with formula on the fly (NSFSearch in the C API) with all parameters and return values, e.g. **get summary buffer data for each document matching a formula and compute your own field values like the view indexer does**
* quick reading of files and folders in the Domino data directory or in subdirectories (IBM's DbDirectory is slow for many DBs and does not support subdirectory scanning)
* **run agents bypassing ECL checks**, pass an in-memory document for data exchange readable as `Session.DocumentContext` and redirect Agent output to a Java `Writer`
* read/write replication data (replica id and flags)
* clearing the replication history
* fast noteid / UNID bulk conversion with lookup of "modified in this file" property (part of the note OID - originator id)
* compute @Usernameslist values for any Notes user
* faster **formula execution on documents** with document modified/selected/deleted by formula check
* **SSO token computation** (with tokens also working on Websphere)
* APIs to **get/put/sync IDs with the ID Vault** and to **sign/encrypt/decrypt** documents and attachments
* APIs to **read extended busytime information** like UNID/start/end of busytime entries (not just the freetime search that IBM provides)
* APIs to **read and modify the ECL**

**Please note:**

**The project gives access to some really low level functions of IBM Notes/Domino. Using them in the wrong way or sending unexpected parameter values might crash your application server, so make sure you know what you are doing and test your code on a local machine first!**

One reason for open sourcing all this stuff was to get more hands on it and make it as robust as possible.

## Supported platforms
The code should run in 32 and 64 bit Notes Client and Domino server environments on Windows, Linux and Mac.

It is not expected to run without changes on other platforms, mainly because of little endian / big endian differences or memory alignments, but
we don't currently have access to those platforms anyway.

## Usage
Here is a code snippet to demonstrate how to use the API:

```java
NotesGC.runWithAutoGC(new Callable<Object>() {

	public Object call() throws Exception {
		NotesDatabase dbData = getFakeNamesDb();
				
		//open People view
		NotesCollection colFromDbData = dbData.openCollectionByName("People");

		//read all note ids from the collection
		boolean includeCategoryIds = false;
		LinkedHashSet<Integer> allIds = colFromDbData.getAllIds(includeCategoryIds);
				
		//pick random note ids
		Integer[] allIdsArr = allIds.toArray(new Integer[allIds.size()]);
		Set<Integer> pickedNoteIds = new HashSet<Integer>();
		while (pickedNoteIds.size() < 1000) {
			int randomIndex = (int) (Math.random() * allIdsArr.length);
			int randomNoteId = allIdsArr[randomIndex];
			pickedNoteIds.add(randomNoteId);
		}
				
		//populate the collection's selected list with picked ids
		NotesIDTable selectedList = colFromDbData.getSelectedList();
		selectedList.clear();
		selectedList.addNotes(pickedNoteIds);

		//for remote databases, re-send modified SelectedList
		colFromDbData.updateFilters(EnumSet.of(UpdateCollectionFilters.FILTER_SELECTED));

		//next, traverse selected entries only, starting at position "0" (top of the view)
		String startPos = "0";
		//skip from "0" to the first entry that we are allowed to read
		//(its position could be different from "1" caused by reader fields)
		int entriesToSkip = 1;
		//add all read entries to the result list
		int entriesToReturn = Integer.MAX_VALUE;
		//tell the API how to navigate in the view: from one entry in the selectedList
		//to the next one (in view ordering)
		EnumSet<Navigate> returnNavigator = EnumSet.of(Navigate.NEXT_SELECTED);
		//preload the maximum number of entries, can be useful when implementing
		//filter method in EntriesAsListCallback
		int bufferSize = Integer.MAX_VALUE;
		//tell the API which data we want to read (in this case note ids and column itemname/value map)
		EnumSet<ReadMask> returnData = EnumSet.of(ReadMask.NOTEID, ReadMask.SUMMARY);
		
		List<NotesViewEntryData> selectedEntries = colFromDbData.getAllEntries(startPos, entriesToSkip,
				returnNavigator, Integer.MAX_VALUE,
				returnData, new EntriesAsListCallback(entriesToReturn));
				
		for (NotesViewEntryData currEntry : selectedEntries) {
		
			//check that all entries that we read were from our picked id list
			Assert.assertTrue("Entry read from view is contained in selected list",
				pickedNoteIds.contains(currEntry.getNoteId()));
				
			//read column values with their programmatic name
			String firstName = (String) currEntry.get("firstname");
			String lastName = (String) currEntry.get("lastname");
			//...
		}
		
		//now remove all read ids from pickedNoteIds and make sure that we found everything
		//we were searching for
		for (NotesViewEntryData currEntry : selectedEntries) {
			pickedNoteIds.remove(currEntry.getNoteId());
		}
		
		Assert.assertTrue("All ids from the selected list can be found in the view", pickedNoteIds.isEmpty());
		return null;
	}
});
```

This selected list feature demonstrated above is already the first big surprise, if you only know IBM's Java API for Domino.<commit_msg>Used SUMMARYVALUES instead of SUMMARY in example<commit_after># Domino JNA
Cross-platform access to IBM Notes/Domino C API methods from Java

## Features
The project provides functionality that is not available in the classic Java API of IBM Notes/Domino or that is poorly implemented, for example:

* **view lookups using different key formats** (e.g. strings, numbers, dates, date ranges) and equality/**inequality searches (e.g. find less or greater than a search key)**
* **decodes all available types of view column data types** (string, string list, number, number list, datetime, datetime list) **and row data** (e.g. just note id, UNID, counts, unread flag etc.)
* read view data as **another Notes user**
* extract **read access information for rows in a view** that Domino stores in the view index
* **separation of Notes views and their data into multiple databases** (like programmatically creating private views and keeping them up-to-date incrementally)
* **dynamic filtering of view rows based on a Note id list with paging support (this really rocks!)**
* **reading categorized views** with expanded/collapsed entries and min/max level
* read **design collection** with design elements in a database
* **differential view reads** : on second view lookup, only read rows that have changed
* support for view resorting (changing the collation in C API terms)
* **direct attachment streaming** (IBM's Java API extracts files to temp disk space first to read attachment data)
* basic APIs to read note (document) item values like String/Double/Calendar single and multiple values
* several APIs to write item values and attachments (needs more work)
* **richtext item reading**: convenience method to extract text content, advanced API to read all CD records (IRichTextNavigator)
* **richtext item writing**: create richtext items with text, images, doclinks, by rendering other notes and appending other richtet items
* **richtext item conversion**: multi-level conversion of richtext item content, e.g. to **add/remove file hotspots (file icons with custom image that open file on click) independent from the actual file attachment** or do mail merge with richtext
* **richtext-html conversion** with advanced quality and access to embedded images
* **incremental data synchronization** with external databases or indexers. Generic design, sample implementation for [CQEngine](https://github.com/npgall/cqengine) and [SQLite](https://www.sqlite.org)
* **quick check if a document is editable** by a specified user (without the need to scan through author items)
* **fulltext index creation** with all available options
* supports incremental synchronization of Domino databases by **reading noteid lists of modified and deleted documents** (IBM's Java API does not return ids of deleted docs)
* searching NSF data with formula on the fly (NSFSearch in the C API) with all parameters and return values, e.g. **get summary buffer data for each document matching a formula and compute your own field values like the view indexer does**
* quick reading of files and folders in the Domino data directory or in subdirectories (IBM's DbDirectory is slow for many DBs and does not support subdirectory scanning)
* **run agents bypassing ECL checks**, pass an in-memory document for data exchange readable as `Session.DocumentContext` and redirect Agent output to a Java `Writer`
* read/write replication data (replica id and flags)
* clearing the replication history
* fast noteid / UNID bulk conversion with lookup of "modified in this file" property (part of the note OID - originator id)
* compute @Usernameslist values for any Notes user
* faster **formula execution on documents** with document modified/selected/deleted by formula check
* **SSO token computation** (with tokens also working on Websphere)
* APIs to **get/put/sync IDs with the ID Vault** and to **sign/encrypt/decrypt** documents and attachments
* APIs to **read extended busytime information** like UNID/start/end of busytime entries (not just the freetime search that IBM provides)
* APIs to **read and modify the ECL**

**Please note:**

**The project gives access to some really low level functions of IBM Notes/Domino. Using them in the wrong way or sending unexpected parameter values might crash your application server, so make sure you know what you are doing and test your code on a local machine first!**

One reason for open sourcing all this stuff was to get more hands on it and make it as robust as possible.

## Supported platforms
The code should run in 32 and 64 bit Notes Client and Domino server environments on Windows, Linux and Mac.

It is not expected to run without changes on other platforms, mainly because of little endian / big endian differences or memory alignments, but
we don't currently have access to those platforms anyway.

## Usage
Here is a code snippet to demonstrate how to use the API:

```java
NotesGC.runWithAutoGC(new Callable<Object>() {

	public Object call() throws Exception {
		NotesDatabase dbData = getFakeNamesDb();
				
		//open People view
		NotesCollection colFromDbData = dbData.openCollectionByName("People");

		//read all note ids from the collection
		boolean includeCategoryIds = false;
		LinkedHashSet<Integer> allIds = colFromDbData.getAllIds(includeCategoryIds);
				
		//pick random note ids
		Integer[] allIdsArr = allIds.toArray(new Integer[allIds.size()]);
		Set<Integer> pickedNoteIds = new HashSet<Integer>();
		while (pickedNoteIds.size() < 1000) {
			int randomIndex = (int) (Math.random() * allIdsArr.length);
			int randomNoteId = allIdsArr[randomIndex];
			pickedNoteIds.add(randomNoteId);
		}
				
		//populate the collection's selected list with picked ids
		NotesIDTable selectedList = colFromDbData.getSelectedList();
		selectedList.clear();
		selectedList.addNotes(pickedNoteIds);

		//for remote databases, re-send modified SelectedList
		colFromDbData.updateFilters(EnumSet.of(UpdateCollectionFilters.FILTER_SELECTED));

		//next, traverse selected entries only, starting at position "0" (top of the view)
		String startPos = "0";
		//skip from "0" to the first entry that we are allowed to read
		//(its position could be different from "1" caused by reader fields)
		int entriesToSkip = 1;
		//add all read entries to the result list
		int entriesToReturn = Integer.MAX_VALUE;
		//tell the API how to navigate in the view: from one entry in the selectedList
		//to the next one (in view ordering)
		EnumSet<Navigate> returnNavigator = EnumSet.of(Navigate.NEXT_SELECTED);
		//preload the maximum number of entries, can be useful when implementing
		//filter method in EntriesAsListCallback
		int bufferSize = Integer.MAX_VALUE;
		//tell the API which data we want to read (in this case note ids and column values map)
		EnumSet<ReadMask> returnData = EnumSet.of(ReadMask.NOTEID, ReadMask.SUMMARYVALUES);
		
		List<NotesViewEntryData> selectedEntries = colFromDbData.getAllEntries(startPos, entriesToSkip,
				returnNavigator, Integer.MAX_VALUE,
				returnData, new EntriesAsListCallback(entriesToReturn));
				
		for (NotesViewEntryData currEntry : selectedEntries) {
		
			//check that all entries that we read were from our picked id list
			Assert.assertTrue("Entry read from view is contained in selected list",
				pickedNoteIds.contains(currEntry.getNoteId()));
				
			//read column values with their programmatic name
			String firstName = (String) currEntry.get("firstname");
			String lastName = (String) currEntry.get("lastname");
			//...
		}
		
		//now remove all read ids from pickedNoteIds and make sure that we found everything
		//we were searching for
		for (NotesViewEntryData currEntry : selectedEntries) {
			pickedNoteIds.remove(currEntry.getNoteId());
		}
		
		Assert.assertTrue("All ids from the selected list can be found in the view", pickedNoteIds.isEmpty());
		return null;
	}
});
```

This selected list feature demonstrated above is already the first big surprise, if you only know IBM's Java API for Domino.
{'fraction_non_alphanumeric': 0.07421556961232542, 'fraction_numerical': 0.0016283584893843552, 'mean_word_length': 4.7151037938439515, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 17, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '31795', 'n_tokens_mistral': 4345, 'n_tokens_neox': 4008, 'n_words': 2033}