<commit_before>*We can help you get Chronicle up and running in your organisation, we suggest you invite us in for
consultancy, charged on an ad-hoc basis, we can discuss the best options tailored to your individual
requirements. - [Contact Us](<EMAIL>)*

*Or you may already be using Chronicle and just want some help - [find out more..](http://openhft.net/support/)*

# Chronicle Map

Replicate your Key Value Store across your network, with consistency, durability and performance.
![Chronicle Map](http://openhft.net/wp-content/uploads/2014/07/ChronicleMap_200px.png)

#### Maven Artifact Download
```xml
<dependency>                                   
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version><!--replace with the latest version--></version>
</dependency>
```
Click here to get the [Latest Version Number](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22net.openhft%22%20AND%20a%3A%22chronicle-map%22) 

#### Contents

* [Should I use Chronicle Queue or Chronicle Map](https://github.com/OpenHFT/Chronicle-Map#should-i-use-chronicle-queue-or-chronicle-map)
* [What is the difference between SharedHashMap and Chronicle Map](https://github.com/OpenHFT/Chronicle-Map#what-is-the-difference-between-sharedhashmap-and-chronicle-map)
* [Overview](https://github.com/OpenHFT/Chronicle-Map#overview)
* [JavaDoc](http://openhft.github.io/Chronicle-Map/apidocs)
* [Getting Started Guide](https://github.com/OpenHFT/Chronicle-Map#getting-started)
 *  [Simple Construction](https://github.com/OpenHFT/Chronicle-Map#simple-construction)
 *   [Maven Download](https://github.com/OpenHFT/Chronicle-Map#maven-artifact-download-1)
 *   [Snapshot Download](https://github.com/OpenHFT/Chronicle-Map#maven-snapshot-download)
 *   [Key Value Object Types](https://github.com/OpenHFT/Chronicle-Map#key-value-object-types)
 *   [Off Heap and How to improve performance]()
 *   [Sharing Data Between Two or More Maps](https://github.com/OpenHFT/Chronicle-Map#sharing-data-between-two-or-more-maps)
 *   [Entries](https://github.com/OpenHFT/Chronicle-Map#entries)
 *   [Size of Space Reserved on Disk](https://github.com/OpenHFT/Chronicle-Map#size-of-space-reserved-on-disk)
 *   [Chronicle Map Interface](https://github.com/OpenHFT/Chronicle-Map#chronicle-map-interface)
* [Oversized Entries Support] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#oversized-entries-support)  
* [Serialization](https://github.com/OpenHFT/Chronicle-Map#serialization)
  *   [Simple Types](https://github.com/OpenHFT/Chronicle-Map#simple-types)
  *   [Complex Types](https://github.com/OpenHFT/Chronicle-Map#complex-types)
  *   [Import/Export entries as JSON](https://github.com/OpenHFT/Chronicle-Map#importexport-entries-as-json)
* [Close](https://github.com/OpenHFT/Chronicle-Map#close)
* [TCP / UDP Replication](https://github.com/OpenHFT/Chronicle-Map#tcp--udp-replication)
 * [TCP / UDP Background.](https://github.com/OpenHFT/Chronicle-Map#tcp--udp-background)
 *   [How to setup UDP Replication](https://github.com/OpenHFT/Chronicle-Map#how-to-setup-udp-replication)
 *  [TCP/IP Throttling](https://github.com/OpenHFT/Chronicle-Map#tcpip--throttling)
 *  [Replication How it works](https://github.com/OpenHFT/Chronicle-Map#replication-how-it-works)
 *  [Multiple Processes on the same server with Replication](https://github.com/OpenHFT/Chronicle-Map#multiple-processes-on-the-same-server-with-replication)
 *  [Identifier for Replication](https://github.com/OpenHFT/Chronicle-Map#identifier-for-replication)
 *  [Bootstrapping](https://github.com/OpenHFT/Chronicle-Map#bootstrapping)
 *    [Identifier](https://github.com/OpenHFT/Chronicle-Map#identifier)
 * [Port](https://github.com/OpenHFT/Chronicle-Map#port)
 * [Heart Beat Interval](https://github.com/OpenHFT/Chronicle-Map#heart-beat-interval)
* [Multi Chronicle Maps - Network Distributed](https://github.com/OpenHFT/Chronicle-Map#multiple-chronicle-maps---network-distributed)
* [Stateless Client](https://github.com/OpenHFT/Chronicle-Map#stateless-client)
* [Questions/Answers](https://github.com/OpenHFT/Chronicle-Map#questions-and-answers)

#### Miscellaneous

 * [Known Issues](https://github.com/OpenHFT/Chronicle-Map#known-issues)
 * [Stackoverflow](http://stackoverflow.com/tags/chronicle/info)
 * [Development Tasks - JIRA] (https://higherfrequencytrading.atlassian.net/browse/HCOLL)
 * [Use Case Which include Chronicle Map] (http://openhft.net/products/chronicle-engine/)
 
#### Examples

 * [Hello World - A map which stores data off heap](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--simple-hello-world)
 * [Sharing the map between two ( or more ) processes on the same computer](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--sharing-the-map-on-two--or-more--processes-on-the-same-machine)
 * [Replicating data between process on different servers with TCP/IP Replication](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--replicating-data-between-process-on-different-servers-via-tcp)
 * [Replicating data between process on different servers with UDP] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--replicating-data-between-process-on-different-servers-using-udp)
 *  [Creating a Chronicle Set and adding data to it](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--creating-a-chronicle-set-and-adding-data-to-it)

#### Performance Topics

* [Chronicle Map with Large Data ](https://github.com/OpenHFT/Chronicle-Map#chronicle-map-with-large-data)
* [Lock Contention] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#lock-contention)
* [Better to use small keys](https://github.com/OpenHFT/Chronicle-Map#better-to-use-small-keys)
* [ConcurrentHashMap v ChronicleMap](https://github.com/OpenHFT/Chronicle-Map#concurrenthashmap-v-chroniclemap)

### Overview
Chronicle Map implements the `java.util.concurrent.ConcurrentMap`, however unlike the standard
java map, ChronicleMap is able to share your entries accross processes:

![](http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-diagram_04.jpg)

## When to use
#### When to use HashMap
If you compare `HashMap`, `ConcurrentHashMap` and `ChronicleMap`, most of the maps in your system
are likely to be HashMap.  This is because `HashMap` is lightweight and synchronized HashMap works
well for lightly contended use cases.  By contention we mean, how many threads on average are trying
to use a Map.  One reason you can't have many contended resources, is that you only have so many
CPUs and they can only be accessing so many resources at once (ideally no more than one or two
per thread at a time).

####  When to use ConcurrentHashMap
`ConcurrentHashMap` scales very well when highly contended.  It uses more memory but if you only
have a few of them, this doesn't matter.  They have higher throughput than the other two solutions,
but also it creates the highest garbage.  If garbage pressure is an issue for you, you may want
to consider `ChronicleMap`

One of the main differences between chronicle and ConcurrentHashMap, is how you go about creating
an instance see the getting started guide below for details.

####  When to use Chronicle Map
If you have;
* lots of small key-values
* you want to minimise garbage produced, and medium lived objects.
* you need to share data between JVMs
* you need persistence

#### Should I use Chronicle Queue or Chronicle Map
Chronicle queue is designed to send every update. If your network can't do this something has
to give. You could compress the data but at some point you have to work within the limits of your
hardware or get more hardware. Chronicle Map on the other hand sends the latest value only.
This will naturally drop updates and is a more natural choice for low bandwidth connections.

#### What is the difference between [SharedHashMap](https://github.com/OpenHFT/HugeCollections) and Chronicle Map
SharedHashMap was the old name given to ChronicleMap, Since the last release of SharedHashMap
 we have added a lot of new features to ChronicleMap, most of these are listed in this readme.

## Getting Started

#### Tutorial 1 - Creating an instance of Chronicle Map
[![ScreenShot](http://openhft.net/wp-content/uploads/2014/09/Screen-Shot-2014-10-14-at-17.49.36.png)](http://openhft.net/chronicle-map-video-tutorial-1/)

### Simple Construction

To download the JAR which contains Chronicle Map, we recommend you use maven, which will download it
from [Maven Central](http://search.maven.org), once you have installed maven, all you have to do is
add the following to your projects `pom.xml`:

#### Maven Artifact Download
```xml
<dependency>
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version><!--replace with the latest version--></version>
</dependency>
```
To get the latest version number
[Click Here](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22net.openhft%22%20AND%20a%3A%22chronicle-map%22) 

when you add ( the above dependency ) to your pom maven will usually attempt to download the release artifacts from: 
```
http://repo1.maven.org/maven2/net/openhft/chronicle-map
```

#### Maven Snapshot Download
If you want to try out the latest pre-release code, you can download the snapshot artifact manually from: 
```xml
https://oss.sonatype.org/content/repositories/snapshots/net/openhft/chronicle-map/
```
a better way is to add the following to your setting.xml, to allow maven to download snapshots :

```xml
<repository>
    <id>Snapshot Repository</id>
    <name>Snapshot Repository</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>
```
and define the snapshot version in your pom.xml, for example:
```xml
<dependency>
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version>1.0.1-SNAPSHOT</version>
</dependency>
```

#### Key Value Object Types

Unlike HashMap which will support any heap object, Chronicle Map only works with objects that it 
can store off heap, so the objects have to be  :  (one of the following )

- AutoBoxed primitives - for good performance.
- Strings - for good performance.
- implements Serializable  
- implements Externalizable ( with a public default constructor ) 
- implements our custom interface BytesMarshallable ( with a public default constructor ) - use 
this for best performance.

or value objects that are created through, a directClass interface, for example : 
``` java
      ChronicleMap<String, BondVOInterface> chm = ChronicleMapBuilder
               .of(String.class, BondVOInterface.class)
               .create();

```

Object graphs can also be included as long as the outer object supports Serializable, Externalizable or BytesMarshallable.

#### Java Class Construction

Creating an instance of Chronicle Map is a little more complexed than just calling a constructor.
To create an instance you have to use the ChronicleMapBuilder.

``` java
import net.openhft.chronicle.map.*
.....

try {

    String tmp = System.getProperty("java.io.tmpdir");
    String pathname = tmp + "/shm-test/myfile.dat";

    File file = new File(pathname);

    ChronicleMapBuilder<Integer, CharSequence> builder =
        ChronicleMapBuilder.of(Integer.class, CharSequence.class);
    ConcurrentMap<Integer, CharSequence> map = builder.createPersistedTo(file);
 
} catch (IOException e) {
    e.printStackTrace();
}
```

Chronicle Map stores its data off the java heap, If you wish to share this off-heap memory between
processes on the same server, you must provide a "file", this file must be the same "file" for all
the instances of Chronicle Map on the same server. The name and location of the "file" is entirely
up to you.  For the best performance on many unix systems we recommend using
[tmpfs](http://en.wikipedia.org/wiki/Tmpfs).

### Sharing Data Between Two or More Maps
Since this file is memory mapped, if you were to create another instance of the Chronicle Map,
pointing to the same file, both Chronicle Maps use this file as a common memory store, which they
both read and write into. The good thing about this is the two ( or more instances of the Chronicle Map )
don't have to be running in the same java process. Ideally and for best performance, the two processes
should be running on the same server. Since the file is memory mapped, ( in most cases ) the read
and writes to the file are hitting the disk cache. Allowing the chronicle map to exchange data
between processes by just using memory and in around 40 nanoseconds. 

``` java 
ConcurrentMap<Integer, CharSequence> map1, map2;

// this could could be on one process
map1 = ChronicleMapBuilder.of(Integer.class, CharSequence.class).createPersistedTo(file);

// this could be on the other process
map2 = ChronicleMapBuilder.of(Integer.class, CharSequence.class).createPersistedTo(file);
```
Note: In order to share data between map1 and map2, the file has to point to the same file location
on your server.

### Entries

One of the differences with Chronicle Map against ConcurrentHashMap, is that it can't be resized,
unlike the ConcurrentHashMap, Chronicle Map is not limited to the available on heap memory.
Resizing is a very expensive operation for Hash Maps, as it can stall your application, so as such
we don't do it. When you are building a Chronicle Map you can set the maximum number of entries that
you are ever likely to support, its ok to over exaggerate this number. As the Chronicle Map is not
limited to your available memory, At worst you will end up having a very large file on disk.

You set the maximum number of entries by the builder:

``` java
ConcurrentMap<Integer, CharSequence> map =
    ChronicleMapBuilder.of(Integer.class, CharSequence.class)
    constantValueSizeBySample("a long sample string"),
    .entries(1000) // set the max number of entries here
    .create();
```
In this example above we have set 1000 entries.


We have optimised chronicle, So that you can have situations where you either don't use;

- all the entries you have allowed for.  This works best on Unix where the disk space and memory
used reflect the number of actual entries, not the number you allowed for.

- all the space you allow for each entry.  This helps if you have entries which are multiple cache
lines (128 bytes +), only the lines you touch sit in your CPU cache and if you have multiple pages
(8+ Kbytes) only the pages you touch use memory or disk.  The CPU cache usage matters as it can be
10000x smaller than main memory.

#### Entry size

The size of each entry depends on the type of the Keys and Values, as some types are larger than others. For example, if an entry uses Integers for both the keys and values, both the key and value each take exactly 4 bytes.
There is also some other overhead which is internal to chronicle, such as its internal multi-map, which has an overhead of
* 6 bytes per entry for segment size of < 64k,
* 9 bytes per entry for segment size of < 16m but > 64k
* 12 bytes per entry for segment size of > 16m entries per segment.

Also, if you create a replicated map, there is an additional 10 bytes per entry.

We suggest you don't configure size for constant-sized keys or values, instead you can use the
builder methods .constantKeySizeBySample(sampleKey) and
.constantValueSizeBySample(sampleValue), For common types like Integer we suggest you don't use
these methods, for example ChronicleMap knows that Integer is 4 bytes long, Long is 8, etc.


### Size of space reserved on disk

In linux, if you looked at the size of the 'file', it will report the used entry size so if you
have just added one entry, it will report the size of this entry, but Windows will report
the reserved size, as it reserves the disk space eagerly ( in fact windows also reserves the memory
eagerly as well ) in other words number-of-entries x entry-size. 

so on linux, if your type
``` 
# It shows you the extents. 
ls -l <file>

# It shows you how much is actually used.
du <file>
```

To illustrate this with an example - On Ubuntu we can create a 100 TB chronicle map.  Both `top` and
`ls -l` say the process virtual size / file size is 100 TB, however the resident memory via `du`
says the size is 71 MB after adding 10000 entries. You can see the size actually used with du.

### How Operating Systems differ

As a pure java library, the same chronicle map java byte code can be run on Windows, Linux and Mac OSX.
However these operating systems work with memory mapped files differently, these differences effect how
chronicle is able to map memory to a file, and hence this can impact the total number of entries that you are
able to configure.

- Windows allocates memory and disk eagerly, Windows will fail if more than 4 GB is allocated in a single memory
mapping, ( calculated as 4GB = 2^20 * 4 KB pages). Windows doesn't fail when a memory mapped region is mapped, rather it will fail when it is used up. This limitation doesn't apply to newer or server based versions of Windows. Eager memory allocation means you can't map more than free memory, but it should reduce jitter when you use it. In the future we may support multiple mappings to avoid this limitation, but there is no immediate plan to do so.
- Linux allocates memory and disk lazily. Linux systems see a performance degradation at around 200% of main memory.
- Mac OSX allocates memory lazily and disk eagerly.

Chronicle Map allocates head room which is a waste on Windows (Linux's sparse allocation means the head room has little
impact).

##### For production
- on Windows we recommend you use map sizes of less than 4 GB each, and less than 50% main memory in total.
- on Linux we recommend you use small to large maps of less than double main memory. e.g. if you have a 128 GB server, we recommend you have less than 256 GB of maps on the server.
- on Mac OSX, we have no specific recommendations.

### Chronicle Map Interface
The Chronicle Map interface adds a few methods above an beyond the standard ConcurrentMap,
the ChronicleMapBuilder can also be used to return the ChronicleMap, see the example below :

``` java
ChronicleMap<Integer, CharSequence> map =
    ChronicleMapBuilder.of(Integer.class, CharSequence.class).create();
```
One way to achieve good performance is to focus on unnecessary object creation, as this reduces
the amount of work that has to be carried out by the Garbage Collector. As such ChronicleMap
supports the following methods :

 - [`V getUsing(K key, V value);`](http://openhft.github.io/Chronicle-Map/apidocs/net/openhft/chronicle/map/ChronicleMap.html#getUsing-K-V-)
 - [`V acquireUsing(K key, V value);`](http://openhft.github.io/Chronicle-Map/apidocs/net/openhft/chronicle/map/ChronicleMap.html#acquireUsing-K-V-)
 - [`ReadContext<K, V> getUsingLocked(@NotNull K key, @NotNull V usingValue);`]  (https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance)
 - [`WriteContext<K, V> acquireUsingLocked(@NotNull K key, @NotNull V usingValue);`]    (https://github.com/OpenHFT/Chronicle-Map#acquireusinglocked)

These methods let you provide the object which the data will be written to, but the value use to be mutable. For example

``` java
CharSequence using = new StringBuilder();
CharSequence myResult = map.getUsing("key", using);
// at this point the myString and myResult will both point to the same object
```

The `map.getUsing()` method is similar to `map.get()`, but because Chronicle Map stores its data off
heap, if you were to call get("key"), a new object would be created each time, map.getUsing() works
by reusing the heap memory which was used by the original Object "using". This technique provides
you with better control over your object creation.

Exactly like `map.getUsing()`, `map.acquireUsing()` will give you back a reference to an value 
based on
a key, but unlike `map.getUsing()` if there is not an entry in the map for this key the entry 
will be added and the value return will we the same value which you provided. ( The section below
 covers both `map.getUsing()` and `map.acquireUsing()` in more detail )

#### Off heap storage and how using a Proxy Object can improve performance

Chronicle Map stores its data off heap. There are some distinct advantages in using off heap data storage

 * When the off heap data is backed by a memory mapped file, the entire map and its contents are automatically persisted. So if you have to restart your system, you won’t lose the content of your map.
 * Off heap data structures are not visited by the garbage collector, with on heap maps like HashMap the garbage collector has to scan your entire object graph ( in fact just the live objects ) to remove garbage, if you are able to keep your on heap footprint low, the garbage collector has a lot less work to do, this in turn improves your performance.
 * Your Chronicle map and your data contained within it is serialised so it can be stored in off 
heap memory. When this memory is shared between processes on the same server, 
the chronicle map is able to distribute entries between processes with extremely low overhead, 
as both processes are sharing the same off heap memory space. Since your objects are already stored off
heap ( as a series of bytes ), replicating entries over the network adds relatively low over head.

One of the downsides of an Off Heap Map is that whenever you wish to get a value ( on heap ) from an entry
which is off heap, for example calling :

 ``` java
Value v = get(key)
```

that entry has to be deserialised onto the java heap so that you can use its value just like any other java object. So if you were to call get(key) ten times, 

 ``` java
for(int i=1;i<=10;i++) {
  Value v = get(key)
}
``` 
this would create 10 separate instances of the value. As each time get() is called the map has to
first create a Object to store the result in and then deserialise the value stored in the off 
heap entry. If you want to get the value back on heap so that you can use it like a normal java 
value, There is nothing we can do about the deserialisation as this has to occur every time, 
( since the value may have been changed by another thread ), but we don’t have to create the 
object each time. This is why we create `getUsing(key,using)`. By reducing the number of objects
you create, you reduce the amount of work that the garbage collector has to carry out, 
this in turn may, improve your overall performance. So back to `getUsing(key,using)`, If you wish to reuse
and existing object ( in this case the ‘using’ value ), you can instead call :
 
 ``` java
Value using = new Value();

for(int i=1;i<=10;i++) {
  Value bond = map.getUsing(key,using); // this won’t create a new value each time.
  assert using == bond; // this will always be the same instance and the Vaule type can not be immutable
}
``` 
We can get a further performance improvement if we don't deserialize the whole object, 
but only deserialize only the bytes that we are actually interested in, 

Lets assume that we had the following interface :

``` java
public interface LongValue {
    long getValue();
    void setValue(long value);
}
``` 
It is possible to use chronicle as an off heap proxy that can go directly to the memory location off heap and just get back the long that you require. This is especially use full if the value object has a lot of fields in it like the BondVOInterface
``` java
public interface BondVOInterface {
 ... 

    long getIssueDate();
    void setIssueDate(long issueDate);  /* time in millis */

    long getMaturityDate();
    void setMaturityDate(long maturityDate);  /* time in millis */

    double getCoupon();
    void setCoupon(double coupon);

    String getSymbol();
    void setSymbol(@MaxSize(20) String symbol);
 ... 
}
``` 
If for example you want to just do the following

``` java
BondVOInterface  bond = map.get(key);
bond.getCoupon()

```
lets say that it is only the `coupon` field that we are interested in, then its better not to have to
deserialise the whole object that implements the `BondVOInterface`. The `ChronicleMapBuilder` will look a the types of keys
 and values that you use, If the value type is a simple accessor/mutator interface that is exposing a non nested pojo, which uses simple types
 like `CharSequence` and primitives with corresponding get..() and
 set..() methods, Chronicle is able to generate off heap poxies so the whole object is not desrialized each
  time it is accessed, The off heap poxies are able to read
and write into
the off heap data structures directly, this reduced serialisation can give you a big performance boost.
Below we show you how you can work directly with the off heap entries.

``` java
        ChronicleMap<CharSequence, BondVOInterface> chm = ChronicleMapBuilder
                .of(String.class, BondVOInterface.class)
                .keySize(10)
                .create();
```

notice that the

``` java
.of(CharSequence.class, BondVOInterface.class)
``` 

value class, in our case `BondVOInterface.class` is an `interface` rather than a `class`,  now
like before, we can
use the `getUsing(key,using)` method, but this time we have to create the ‘using’ instance slightly
differently, we have to call the `newDirectReference(..)` method.

``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
``` 

the call to `getUsing(key,value)` is the same as we had in the earlier example

``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
BondVOInterface  bond = map.getUsing(key,using);
assert using == bond; // this will always be the same instance
```

`getUsing(key,using)` won’t create any on heap objects, and it won’t deserialize the ‘bond’ entry from off heap to on
heap, all it does is sets the bond as a proxy to the off heap memory, this proxy object was created by
`DataValueClasses.newDirectReference(BondVOInterface.class)`, it allows us access to the fields of our
entry, directly into the off heap storage.

As above, ideally you would reuse the `using` variable.

 ``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);

for(int i=1;i<=10;i++) {
  Value bond = map.getUsing(key,using); // this won’t create a new value each time.
  double coupon = bond.getCoupon()
  assert using == bond; // this will always be the same instance
}
```

so when you call :

``` java
bond.getCoupon()
```

its only the coupon that gets deserialized.

Just like any other concurrent map, chronicle map uses segment locking, if you wish to obtain a read lock
when calling getUsing(key,using) you can do this :

``` java
BondVOInterface  bond = map.getUsing(key,value);
try (ReadContext<?, BondVOInterface> context = map.getUsingLocked(key,bond)) {
   long issueDate =  bond.getIssueDate();
   String symbol = bond.getSymbol();

	// add your logic here ( the lock will ensure this bond can not be changed by another thread )

} // the lock is released here because close() is called

```

To ensure that you can read the 'issueDate' and 'symbol' can be read atomically, these values
must be read while the segment read lock is in place.

when you call map.getUsingLocked(key,using) we return a ReadContext, the ReadContext extends
AutoCloseable so will automatically unlock the segment when the try block is exited.

If you wish not to use a try block you must manually release the segment lock by calling
``` java
context.close() // the lock will get released when this is called
```

####  acquireUsingLocked()
Just like getUsing(), acquireUsing() will also recycle the value you pass it, the following
code is a pattern that you will often come across, `acquireUsing(key,value)` offers this
functionality i the example, below with a single method call :

 ``` java
V acquireUsing(key,value) {
    Lock l = ...; // the segment lock of the map
    l.lock();
    try {
           V v = map.getUsing(key,value)
           if (v == null) {
               map.put(key,value); // create a new entry
               return value;
           } else
               return v; // return the value of the existing entry
    } finally {
        l.unlock();
    }
}
```

If you are only accessing ChronicleMap from a single thread. If you are not doing replication
and don't care about atomic reads. Then its simpler ( and faster ) to use acquireUsing() otherwise we
recommend you use `acquireUsingLocked(key,value)`

Since the acquireUsing() method can end up creating an entry, the acquireUsingLocked(key,value) method must hold
a segment write lock, this is unlike  getUsing(key,using) which only holds a segment read lock.

``` java
BondVOInterface bond = ... // create your instance
try (WriteContext<?, BondVOInterface> context = map.acquireUsingLocked("one", bond)) {
 assert bond ==  context.value();

 long issueDate =  bond.getIssueDate();
 String symbol = bond.getSymbol();

 if (condition)  // based on your own business logic
    context.removeEntry();
}
```

If after you have read the 'issueDate' and  'symbol' and you wish to remove the entry based on some
business logic, its more efficient to use the 'context' to remove the entry, as the contents is
already aware when the entry is in memory.

## Oversized Entries Support

It is possible for the size of your entry to be 64 times as large as the maximum entry size,
we refer to this type of entry as an oversized entry. Oversized entries are there to cater for the case 
where only a small
percentage of your entries are larger than the others, in this case your large entry will
span across a number of entries. The alternative would be to increase your maximum entry size to be similar
to the size of the largest entry, but this approach is wasteful of memory, especially when most
entries are no where near the max entry size.  

## Serialization

Chronicle Map stores your data into off heap memory, so when you give it a Key or Value, it will
serialise these objects into bytes.

### Simple Types
If you are using simple auto boxed objects based on the primitive types, Chronicle Map will
automatically handle the serialisation for you.  

### Complex Types
For anything other than the standard object, the Objects either have to :
* implement "java.io.Serializable" ( which we don't recommend as this can be slow )
* we also support "java.io.Externalizable", we recommend this over Serializable as its usually faster.
* or for the best performance implement net.openhft.lang.io.serialization.BytesMarshallable,
an example of how to do this can be found at "IntValue$$Native"
* alternatively, you could write a "Custom Marshaller", the custom marshaller can be implemented
for a single type or a number of types.

### Import/Export entries as JSON
![Import/Export](http://openhft.net/wp-content/uploads/2014/09/Export-import_04.jpg)

Chronicle Map supports importing and exporting all the entries into a JSON encoded file.

``` java
void getAll(File toFile) throws IOException;
void putAll(File fromFile) throws IOException;
```

Its only the entries of your map that are exported, not the configuration of your map. So care
must be taken to populate the data in to a map of the correct Key/Value type and with enough
available entries. When importing data :
* entries that are in the map but not in the JSON file will remain untouched.
* entries that are in the map and in the JSON file will be updated.
* entries that are not in the map but are in the JSON file wil added.
* In other words importing data into a Chronicle Map works like `map.putAll(<JSON entries>)`.

When Importing data if you are also writing to the map at the same time, the last update will win.
In other words a write lock is not held for the entire import process.
Importing and exporting the map, is ideal if you wish to:
* Bulk load data from one chronicle map into another.
* migrate data between versions of chronicle map.

WARNING : The current version only supports Chronicle Maps that contained serialized KEYS and
VALUES, future versions will support a binary encoding of objects that are `net.openhft.lang.io
.serialization.BytesMarshallable`.

## Close
Unlike ConcurrentHashMap, chronicle map stores its data off heap, often in a memory mapped file.
Its recommended that you call close() once you have finished working with a Chronicle Map.
``` java
map.close()
```

You only need to close to clean up resources deterministically.  If your program is exiting, 
you don't need to close the collection, as Chronicle never knows when the program might crash, 
so we have designed it so you don't have to close() it.

WARNING : If you call close too early before you have finished working with the map, this can cause<commit_msg>updated import<commit_after>*We can help you get Chronicle up and running in your organisation, we suggest you invite us in for
consultancy, charged on an ad-hoc basis, we can discuss the best options tailored to your individual
requirements. - [Contact Us](<EMAIL>)*

*Or you may already be using Chronicle and just want some help - [find out more..](http://openhft.net/support/)*

# Chronicle Map

Replicate your Key Value Store across your network, with consistency, durability and performance.
![Chronicle Map](http://openhft.net/wp-content/uploads/2014/07/ChronicleMap_200px.png)

#### Maven Artifact Download
```xml
<dependency>                                   
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version><!--replace with the latest version--></version>
</dependency>
```
Click here to get the [Latest Version Number](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22net.openhft%22%20AND%20a%3A%22chronicle-map%22) 

#### Contents

* [Should I use Chronicle Queue or Chronicle Map](https://github.com/OpenHFT/Chronicle-Map#should-i-use-chronicle-queue-or-chronicle-map)
* [What is the difference between SharedHashMap and Chronicle Map](https://github.com/OpenHFT/Chronicle-Map#what-is-the-difference-between-sharedhashmap-and-chronicle-map)
* [Overview](https://github.com/OpenHFT/Chronicle-Map#overview)
* [JavaDoc](http://openhft.github.io/Chronicle-Map/apidocs)
* [Getting Started Guide](https://github.com/OpenHFT/Chronicle-Map#getting-started)
 *  [Simple Construction](https://github.com/OpenHFT/Chronicle-Map#simple-construction)
 *   [Maven Download](https://github.com/OpenHFT/Chronicle-Map#maven-artifact-download-1)
 *   [Snapshot Download](https://github.com/OpenHFT/Chronicle-Map#maven-snapshot-download)
 *   [Key Value Object Types](https://github.com/OpenHFT/Chronicle-Map#key-value-object-types)
 *   [Off Heap and How to improve performance]()
 *   [Sharing Data Between Two or More Maps](https://github.com/OpenHFT/Chronicle-Map#sharing-data-between-two-or-more-maps)
 *   [Entries](https://github.com/OpenHFT/Chronicle-Map#entries)
 *   [Size of Space Reserved on Disk](https://github.com/OpenHFT/Chronicle-Map#size-of-space-reserved-on-disk)
 *   [Chronicle Map Interface](https://github.com/OpenHFT/Chronicle-Map#chronicle-map-interface)
* [Oversized Entries Support] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#oversized-entries-support)  
* [Serialization](https://github.com/OpenHFT/Chronicle-Map#serialization)
  *   [Simple Types](https://github.com/OpenHFT/Chronicle-Map#simple-types)
  *   [Complex Types](https://github.com/OpenHFT/Chronicle-Map#complex-types)
  *   [Import/Export entries as JSON](https://github.com/OpenHFT/Chronicle-Map#importexport-entries-as-json)
* [Close](https://github.com/OpenHFT/Chronicle-Map#close)
* [TCP / UDP Replication](https://github.com/OpenHFT/Chronicle-Map#tcp--udp-replication)
 * [TCP / UDP Background.](https://github.com/OpenHFT/Chronicle-Map#tcp--udp-background)
 *   [How to setup UDP Replication](https://github.com/OpenHFT/Chronicle-Map#how-to-setup-udp-replication)
 *  [TCP/IP Throttling](https://github.com/OpenHFT/Chronicle-Map#tcpip--throttling)
 *  [Replication How it works](https://github.com/OpenHFT/Chronicle-Map#replication-how-it-works)
 *  [Multiple Processes on the same server with Replication](https://github.com/OpenHFT/Chronicle-Map#multiple-processes-on-the-same-server-with-replication)
 *  [Identifier for Replication](https://github.com/OpenHFT/Chronicle-Map#identifier-for-replication)
 *  [Bootstrapping](https://github.com/OpenHFT/Chronicle-Map#bootstrapping)
 *    [Identifier](https://github.com/OpenHFT/Chronicle-Map#identifier)
 * [Port](https://github.com/OpenHFT/Chronicle-Map#port)
 * [Heart Beat Interval](https://github.com/OpenHFT/Chronicle-Map#heart-beat-interval)
* [Multi Chronicle Maps - Network Distributed](https://github.com/OpenHFT/Chronicle-Map#multiple-chronicle-maps---network-distributed)
* [Stateless Client](https://github.com/OpenHFT/Chronicle-Map#stateless-client)
* [Questions/Answers](https://github.com/OpenHFT/Chronicle-Map#questions-and-answers)

#### Miscellaneous

 * [Known Issues](https://github.com/OpenHFT/Chronicle-Map#known-issues)
 * [Stackoverflow](http://stackoverflow.com/tags/chronicle/info)
 * [Development Tasks - JIRA] (https://higherfrequencytrading.atlassian.net/browse/HCOLL)
 * [Use Case Which include Chronicle Map] (http://openhft.net/products/chronicle-engine/)
 
#### Examples

 * [Hello World - A map which stores data off heap](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--simple-hello-world)
 * [Sharing the map between two ( or more ) processes on the same computer](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--sharing-the-map-on-two--or-more--processes-on-the-same-machine)
 * [Replicating data between process on different servers with TCP/IP Replication](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--replicating-data-between-process-on-different-servers-via-tcp)
 * [Replicating data between process on different servers with UDP] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--replicating-data-between-process-on-different-servers-using-udp)
 *  [Creating a Chronicle Set and adding data to it](https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#example--creating-a-chronicle-set-and-adding-data-to-it)

#### Performance Topics

* [Chronicle Map with Large Data ](https://github.com/OpenHFT/Chronicle-Map#chronicle-map-with-large-data)
* [Lock Contention] (https://github.com/OpenHFT/Chronicle-Map/blob/master/README.md#lock-contention)
* [Better to use small keys](https://github.com/OpenHFT/Chronicle-Map#better-to-use-small-keys)
* [ConcurrentHashMap v ChronicleMap](https://github.com/OpenHFT/Chronicle-Map#concurrenthashmap-v-chroniclemap)

### Overview
Chronicle Map implements the `java.util.concurrent.ConcurrentMap`, however unlike the standard
java map, ChronicleMap is able to share your entries accross processes:

![](http://openhft.net/wp-content/uploads/2014/07/Chronicle-Map-diagram_04.jpg)

## When to use
#### When to use HashMap
If you compare `HashMap`, `ConcurrentHashMap` and `ChronicleMap`, most of the maps in your system
are likely to be HashMap.  This is because `HashMap` is lightweight and synchronized HashMap works
well for lightly contended use cases.  By contention we mean, how many threads on average are trying
to use a Map.  One reason you can't have many contended resources, is that you only have so many
CPUs and they can only be accessing so many resources at once (ideally no more than one or two
per thread at a time).

####  When to use ConcurrentHashMap
`ConcurrentHashMap` scales very well when highly contended.  It uses more memory but if you only
have a few of them, this doesn't matter.  They have higher throughput than the other two solutions,
but also it creates the highest garbage.  If garbage pressure is an issue for you, you may want
to consider `ChronicleMap`

One of the main differences between chronicle and ConcurrentHashMap, is how you go about creating
an instance see the getting started guide below for details.

####  When to use Chronicle Map
If you have;
* lots of small key-values
* you want to minimise garbage produced, and medium lived objects.
* you need to share data between JVMs
* you need persistence

#### Should I use Chronicle Queue or Chronicle Map
Chronicle queue is designed to send every update. If your network can't do this something has
to give. You could compress the data but at some point you have to work within the limits of your
hardware or get more hardware. Chronicle Map on the other hand sends the latest value only.
This will naturally drop updates and is a more natural choice for low bandwidth connections.

#### What is the difference between [SharedHashMap](https://github.com/OpenHFT/HugeCollections) and Chronicle Map
SharedHashMap was the old name given to ChronicleMap, Since the last release of SharedHashMap
 we have added a lot of new features to ChronicleMap, most of these are listed in this readme.

## Getting Started

#### Tutorial 1 - Creating an instance of Chronicle Map
[![ScreenShot](http://openhft.net/wp-content/uploads/2014/09/Screen-Shot-2014-10-14-at-17.49.36.png)](http://openhft.net/chronicle-map-video-tutorial-1/)

### Simple Construction

To download the JAR which contains Chronicle Map, we recommend you use maven, which will download it
from [Maven Central](http://search.maven.org), once you have installed maven, all you have to do is
add the following to your projects `pom.xml`:

#### Maven Artifact Download
```xml
<dependency>
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version><!--replace with the latest version--></version>
</dependency>
```
To get the latest version number
[Click Here](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22net.openhft%22%20AND%20a%3A%22chronicle-map%22) 

when you add ( the above dependency ) to your pom maven will usually attempt to download the release artifacts from: 
```
http://repo1.maven.org/maven2/net/openhft/chronicle-map
```

#### Maven Snapshot Download
If you want to try out the latest pre-release code, you can download the snapshot artifact manually from: 
```xml
https://oss.sonatype.org/content/repositories/snapshots/net/openhft/chronicle-map/
```
a better way is to add the following to your setting.xml, to allow maven to download snapshots :

```xml
<repository>
    <id>Snapshot Repository</id>
    <name>Snapshot Repository</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>
```
and define the snapshot version in your pom.xml, for example:
```xml
<dependency>
  <groupId>net.openhft</groupId>
  <artifactId>chronicle-map</artifactId>
  <version>1.0.1-SNAPSHOT</version>
</dependency>
```

#### Key Value Object Types

Unlike HashMap which will support any heap object, Chronicle Map only works with objects that it 
can store off heap, so the objects have to be  :  (one of the following )

- AutoBoxed primitives - for good performance.
- Strings - for good performance.
- implements Serializable  
- implements Externalizable ( with a public default constructor ) 
- implements our custom interface BytesMarshallable ( with a public default constructor ) - use 
this for best performance.

or value objects that are created through, a directClass interface, for example : 
``` java
      ChronicleMap<String, BondVOInterface> chm = ChronicleMapBuilder
               .of(String.class, BondVOInterface.class)
               .create();

```

Object graphs can also be included as long as the outer object supports Serializable, Externalizable or BytesMarshallable.

#### Java Class Construction

Creating an instance of Chronicle Map is a little more complexed than just calling a constructor.
To create an instance you have to use the ChronicleMapBuilder.

``` java
import net.openhft.chronicle.map.*
.....

try {

    String tmp = System.getProperty("java.io.tmpdir");
    String pathname = tmp + "/shm-test/myfile.dat";

    File file = new File(pathname);

    ChronicleMapBuilder<Integer, CharSequence> builder =
        ChronicleMapBuilder.of(Integer.class, CharSequence.class);
    ConcurrentMap<Integer, CharSequence> map = builder.createPersistedTo(file);
 
} catch (IOException e) {
    e.printStackTrace();
}
```

Chronicle Map stores its data off the java heap, If you wish to share this off-heap memory between
processes on the same server, you must provide a "file", this file must be the same "file" for all
the instances of Chronicle Map on the same server. The name and location of the "file" is entirely
up to you.  For the best performance on many unix systems we recommend using
[tmpfs](http://en.wikipedia.org/wiki/Tmpfs).

### Sharing Data Between Two or More Maps
Since this file is memory mapped, if you were to create another instance of the Chronicle Map,
pointing to the same file, both Chronicle Maps use this file as a common memory store, which they
both read and write into. The good thing about this is the two ( or more instances of the Chronicle Map )
don't have to be running in the same java process. Ideally and for best performance, the two processes
should be running on the same server. Since the file is memory mapped, ( in most cases ) the read
and writes to the file are hitting the disk cache. Allowing the chronicle map to exchange data
between processes by just using memory and in around 40 nanoseconds. 

``` java 
ConcurrentMap<Integer, CharSequence> map1, map2;

// this could could be on one process
map1 = ChronicleMapBuilder.of(Integer.class, CharSequence.class).createPersistedTo(file);

// this could be on the other process
map2 = ChronicleMapBuilder.of(Integer.class, CharSequence.class).createPersistedTo(file);
```
Note: In order to share data between map1 and map2, the file has to point to the same file location
on your server.

### Entries

One of the differences with Chronicle Map against ConcurrentHashMap, is that it can't be resized,
unlike the ConcurrentHashMap, Chronicle Map is not limited to the available on heap memory.
Resizing is a very expensive operation for Hash Maps, as it can stall your application, so as such
we don't do it. When you are building a Chronicle Map you can set the maximum number of entries that
you are ever likely to support, its ok to over exaggerate this number. As the Chronicle Map is not
limited to your available memory, At worst you will end up having a very large file on disk.

You set the maximum number of entries by the builder:

``` java
ConcurrentMap<Integer, CharSequence> map =
    ChronicleMapBuilder.of(Integer.class, CharSequence.class)
    constantValueSizeBySample("a long sample string"),
    .entries(1000) // set the max number of entries here
    .create();
```
In this example above we have set 1000 entries.


We have optimised chronicle, So that you can have situations where you either don't use;

- all the entries you have allowed for.  This works best on Unix where the disk space and memory
used reflect the number of actual entries, not the number you allowed for.

- all the space you allow for each entry.  This helps if you have entries which are multiple cache
lines (128 bytes +), only the lines you touch sit in your CPU cache and if you have multiple pages
(8+ Kbytes) only the pages you touch use memory or disk.  The CPU cache usage matters as it can be
10000x smaller than main memory.

#### Entry size

The size of each entry depends on the type of the Keys and Values, as some types are larger than others. For example, if an entry uses Integers for both the keys and values, both the key and value each take exactly 4 bytes.
There is also some other overhead which is internal to chronicle, such as its internal multi-map, which has an overhead of
* 6 bytes per entry for segment size of < 64k,
* 9 bytes per entry for segment size of < 16m but > 64k
* 12 bytes per entry for segment size of > 16m entries per segment.

Also, if you create a replicated map, there is an additional 10 bytes per entry.

We suggest you don't configure size for constant-sized keys or values, instead you can use the
builder methods .constantKeySizeBySample(sampleKey) and
.constantValueSizeBySample(sampleValue), For common types like Integer we suggest you don't use
these methods, for example ChronicleMap knows that Integer is 4 bytes long, Long is 8, etc.


### Size of space reserved on disk

In linux, if you looked at the size of the 'file', it will report the used entry size so if you
have just added one entry, it will report the size of this entry, but Windows will report
the reserved size, as it reserves the disk space eagerly ( in fact windows also reserves the memory
eagerly as well ) in other words number-of-entries x entry-size. 

so on linux, if your type
``` 
# It shows you the extents. 
ls -l <file>

# It shows you how much is actually used.
du <file>
```

To illustrate this with an example - On Ubuntu we can create a 100 TB chronicle map.  Both `top` and
`ls -l` say the process virtual size / file size is 100 TB, however the resident memory via `du`
says the size is 71 MB after adding 10000 entries. You can see the size actually used with du.

### How Operating Systems differ

As a pure java library, the same chronicle map java byte code can be run on Windows, Linux and Mac OSX.
However these operating systems work with memory mapped files differently, these differences effect how
chronicle is able to map memory to a file, and hence this can impact the total number of entries that you are
able to configure.

- Windows allocates memory and disk eagerly, Windows will fail if more than 4 GB is allocated in a single memory
mapping, ( calculated as 4GB = 2^20 * 4 KB pages). Windows doesn't fail when a memory mapped region is mapped, rather it will fail when it is used up. This limitation doesn't apply to newer or server based versions of Windows. Eager memory allocation means you can't map more than free memory, but it should reduce jitter when you use it. In the future we may support multiple mappings to avoid this limitation, but there is no immediate plan to do so.
- Linux allocates memory and disk lazily. Linux systems see a performance degradation at around 200% of main memory.
- Mac OSX allocates memory lazily and disk eagerly.

Chronicle Map allocates head room which is a waste on Windows (Linux's sparse allocation means the head room has little
impact).

##### For production
- on Windows we recommend you use map sizes of less than 4 GB each, and less than 50% main memory in total.
- on Linux we recommend you use small to large maps of less than double main memory. e.g. if you have a 128 GB server, we recommend you have less than 256 GB of maps on the server.
- on Mac OSX, we have no specific recommendations.

### Chronicle Map Interface
The Chronicle Map interface adds a few methods above an beyond the standard ConcurrentMap,
the ChronicleMapBuilder can also be used to return the ChronicleMap, see the example below :

``` java
ChronicleMap<Integer, CharSequence> map =
    ChronicleMapBuilder.of(Integer.class, CharSequence.class).create();
```
One way to achieve good performance is to focus on unnecessary object creation, as this reduces
the amount of work that has to be carried out by the Garbage Collector. As such ChronicleMap
supports the following methods :

 - [`V getUsing(K key, V value);`](http://openhft.github.io/Chronicle-Map/apidocs/net/openhft/chronicle/map/ChronicleMap.html#getUsing-K-V-)
 - [`V acquireUsing(K key, V value);`](http://openhft.github.io/Chronicle-Map/apidocs/net/openhft/chronicle/map/ChronicleMap.html#acquireUsing-K-V-)
 - [`ReadContext<K, V> getUsingLocked(@NotNull K key, @NotNull V usingValue);`]  (https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance)
 - [`WriteContext<K, V> acquireUsingLocked(@NotNull K key, @NotNull V usingValue);`]    (https://github.com/OpenHFT/Chronicle-Map#acquireusinglocked)

These methods let you provide the object which the data will be written to, but the value use to be mutable. For example

``` java
CharSequence using = new StringBuilder();
CharSequence myResult = map.getUsing("key", using);
// at this point the myString and myResult will both point to the same object
```

The `map.getUsing()` method is similar to `map.get()`, but because Chronicle Map stores its data off
heap, if you were to call get("key"), a new object would be created each time, map.getUsing() works
by reusing the heap memory which was used by the original Object "using". This technique provides
you with better control over your object creation.

Exactly like `map.getUsing()`, `map.acquireUsing()` will give you back a reference to an value 
based on
a key, but unlike `map.getUsing()` if there is not an entry in the map for this key the entry 
will be added and the value return will we the same value which you provided. ( The section below
 covers both `map.getUsing()` and `map.acquireUsing()` in more detail )

#### Off heap storage and how using a Proxy Object can improve performance

Chronicle Map stores its data off heap. There are some distinct advantages in using off heap data storage

 * When the off heap data is backed by a memory mapped file, the entire map and its contents are automatically persisted. So if you have to restart your system, you won’t lose the content of your map.
 * Off heap data structures are not visited by the garbage collector, with on heap maps like HashMap the garbage collector has to scan your entire object graph ( in fact just the live objects ) to remove garbage, if you are able to keep your on heap footprint low, the garbage collector has a lot less work to do, this in turn improves your performance.
 * Your Chronicle map and your data contained within it is serialised so it can be stored in off 
heap memory. When this memory is shared between processes on the same server, 
the chronicle map is able to distribute entries between processes with extremely low overhead, 
as both processes are sharing the same off heap memory space. Since your objects are already stored off
heap ( as a series of bytes ), replicating entries over the network adds relatively low over head.

One of the downsides of an Off Heap Map is that whenever you wish to get a value ( on heap ) from an entry
which is off heap, for example calling :

 ``` java
Value v = get(key)
```

that entry has to be deserialised onto the java heap so that you can use its value just like any other java object. So if you were to call get(key) ten times, 

 ``` java
for(int i=1;i<=10;i++) {
  Value v = get(key)
}
``` 
this would create 10 separate instances of the value. As each time get() is called the map has to
first create a Object to store the result in and then deserialise the value stored in the off 
heap entry. If you want to get the value back on heap so that you can use it like a normal java 
value, There is nothing we can do about the deserialisation as this has to occur every time, 
( since the value may have been changed by another thread ), but we don’t have to create the 
object each time. This is why we create `getUsing(key,using)`. By reducing the number of objects
you create, you reduce the amount of work that the garbage collector has to carry out, 
this in turn may, improve your overall performance. So back to `getUsing(key,using)`, If you wish to reuse
and existing object ( in this case the ‘using’ value ), you can instead call :
 
 ``` java
Value using = new Value();

for(int i=1;i<=10;i++) {
  Value bond = map.getUsing(key,using); // this won’t create a new value each time.
  assert using == bond; // this will always be the same instance and the Vaule type can not be immutable
}
``` 
We can get a further performance improvement if we don't deserialize the whole object, 
but only deserialize only the bytes that we are actually interested in, 

Lets assume that we had the following interface :

``` java
public interface LongValue {
    long getValue();
    void setValue(long value);
}
``` 
It is possible to use chronicle as an off heap proxy that can go directly to the memory location off heap and just get back the long that you require. This is especially use full if the value object has a lot of fields in it like the BondVOInterface
``` java
public interface BondVOInterface {
 ... 

    long getIssueDate();
    void setIssueDate(long issueDate);  /* time in millis */

    long getMaturityDate();
    void setMaturityDate(long maturityDate);  /* time in millis */

    double getCoupon();
    void setCoupon(double coupon);

    String getSymbol();
    void setSymbol(@MaxSize(20) String symbol);
 ... 
}
``` 
If for example you want to just do the following

``` java
BondVOInterface  bond = map.get(key);
bond.getCoupon()

```
lets say that it is only the `coupon` field that we are interested in, then its better not to have to
deserialise the whole object that implements the `BondVOInterface`. The `ChronicleMapBuilder` will look a the types of keys
 and values that you use, If the value type is a simple accessor/mutator interface that is exposing a non nested pojo, which uses simple types
 like `CharSequence` and primitives with corresponding get..() and
 set..() methods, Chronicle is able to generate off heap poxies so the whole object is not desrialized each
  time it is accessed, The off heap poxies are able to read
and write into
the off heap data structures directly, this reduced serialisation can give you a big performance boost.
Below we show you how you can work directly with the off heap entries.

``` java
        ChronicleMap<CharSequence, BondVOInterface> chm = ChronicleMapBuilder
                .of(String.class, BondVOInterface.class)
                .keySize(10)
                .create();
```

notice that the

``` java
.of(CharSequence.class, BondVOInterface.class)
``` 

value class, in our case `BondVOInterface.class` is an `interface` rather than a `class`,  now
like before, we can
use the `getUsing(key,using)` method, but this time we have to create the ‘using’ instance slightly
differently, we have to call the `newDirectReference(..)` method.

``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
``` 

the call to `getUsing(key,value)` is the same as we had in the earlier example

``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);
BondVOInterface  bond = map.getUsing(key,using);
assert using == bond; // this will always be the same instance
```

`getUsing(key,using)` won’t create any on heap objects, and it won’t deserialize the ‘bond’ entry from off heap to on
heap, all it does is sets the bond as a proxy to the off heap memory, this proxy object was created by
`DataValueClasses.newDirectReference(BondVOInterface.class)`, it allows us access to the fields of our
entry, directly into the off heap storage.

As above, ideally you would reuse the `using` variable.

 ``` java
BondVOInterface using = DataValueClasses.newDirectReference(BondVOInterface.class);

for(int i=1;i<=10;i++) {
  Value bond = map.getUsing(key,using); // this won’t create a new value each time.
  double coupon = bond.getCoupon()
  assert using == bond; // this will always be the same instance
}
```

so when you call :

``` java
bond.getCoupon()
```

its only the coupon that gets deserialized.

Just like any other concurrent map, chronicle map uses segment locking, if you wish to obtain a read lock
when calling getUsing(key,using) you can do this :

``` java
BondVOInterface  bond = map.getUsing(key,value);
try (ReadContext<?, BondVOInterface> context = map.getUsingLocked(key,bond)) {
   long issueDate =  bond.getIssueDate();
   String symbol = bond.getSymbol();

	// add your logic here ( the lock will ensure this bond can not be changed by another thread )

} // the lock is released here because close() is called

```

To ensure that you can read the 'issueDate' and 'symbol' can be read atomically, these values
must be read while the segment read lock is in place.

when you call map.getUsingLocked(key,using) we return a ReadContext, the ReadContext extends
AutoCloseable so will automatically unlock the segment when the try block is exited.

If you wish not to use a try block you must manually release the segment lock by calling
``` java
context.close() // the lock will get released when this is called
```

####  acquireUsingLocked()
Just like getUsing(), acquireUsing() will also recycle the value you pass it, the following
code is a pattern that you will often come across, `acquireUsing(key,value)` offers this
functionality i the example, below with a single method call :

 ``` java
V acquireUsing(key,value) {
    Lock l = ...; // the segment lock of the map
    l.lock();
    try {
           V v = map.getUsing(key,value)
           if (v == null) {
               map.put(key,value); // create a new entry
               return value;
           } else
               return v; // return the value of the existing entry
    } finally {
        l.unlock();
    }
}
```

If you are only accessing ChronicleMap from a single thread. If you are not doing replication
and don't care about atomic reads. Then its simpler ( and faster ) to use acquireUsing() otherwise we
recommend you use `acquireUsingLocked(key,value)`

Since the acquireUsing() method can end up creating an entry, the acquireUsingLocked(key,value) method must hold
a segment write lock, this is unlike  getUsing(key,using) which only holds a segment read lock.

``` java
BondVOInterface bond = ... // create your instance
try (WriteContext<?, BondVOInterface> context = map.acquireUsingLocked("one", bond)) {
 assert bond ==  context.value();

 long issueDate =  bond.getIssueDate();
 String symbol = bond.getSymbol();

 if (condition)  // based on your own business logic
    context.removeEntry();
}
```

If after you have read the 'issueDate' and  'symbol' and you wish to remove the entry based on some
business logic, its more efficient to use the 'context' to remove the entry, as the contents is
already aware when the entry is in memory.

## Oversized Entries Support

It is possible for the size of your entry to be 64 times as large as the maximum entry size,
we refer to this type of entry as an oversized entry. Oversized entries are there to cater for the case 
where only a small
percentage of your entries are larger than the others, in this case your large entry will
span across a number of entries. The alternative would be to increase your maximum entry size to be similar
to the size of the largest entry, but this approach is wasteful of memory, especially when most
entries are no where near the max entry size.  

## Serialization

Chronicle Map stores your data into off heap memory, so when you give it a Key or Value, it will
serialise these objects into bytes.

### Simple Types
If you are using simple auto boxed objects based on the primitive types, Chronicle Map will
automatically handle the serialisation for you.  

### Complex Types
For anything other than the standard object, the Objects either have to :
* implement "java.io.Serializable" ( which we don't recommend as this can be slow )
* we also support "java.io.Externalizable", we recommend this over Serializable as its usually faster.
* or for the best performance implement net.openhft.lang.io.serialization.BytesMarshallable,
an example of how to do this can be found at "IntValue$$Native"
* alternatively, you could write a "Custom Marshaller", the custom marshaller can be implemented
for a single type or a number of types.

### Import/Export entries as JSON
![Import/Export](http://openhft.net/wp-content/uploads/2014/09/Export-import_04.jpg)

Chronicle Map supports importing and exporting all the entries into a JSON encoded file.

``` java
void getAll(File toFile) throws IOException;
void putAll(File fromFile) throws IOException;
```

Its only the entries of your map that are exported, not the configuration of your map. So care
must be taken to populate the data in to a map of the correct Key/Value type and with enough
available entries. When importing data :

* entries that are in the map but not in the JSON file will remain untouched.
* entries that are in the map and in the JSON file will be updated.
* entries that are not in the map but are in the JSON file will be added.

In other words importing data into a Chronicle Map works like `map.putAll(<JSON entries>)`.

When Importing data if you are also writing to the map at the same time, the last update will win.
In other words a write lock is not held for the entire import process.
Importing and exporting the map, is ideal if you wish to:
* Bulk load data from one chronicle map into another.
* migrate data between versions of chronicle map.

WARNING : The current version only supports Chronicle Maps that contained serialized KEYS and
VALUES, future versions will support a binary encoding of objects that are `net.openhft.lang.io
.serialization.BytesMarshallable`.

## Close
Unlike ConcurrentHashMap, chronicle map stores its data off heap, often in a memory mapped file.
Its recommended that you call close() once you have finished working with a Chronicle Map.
``` java
map.close()
```

You only need to close to clean up resources deterministically.  If your program is exiting, 
you don't need to close the collection, as Chronicle never knows when the program might crash, 
so we have designed it so you don't have to close() it.

WARNING : If you call close too early before you have finished working with the map, this can cause
{'fraction_non_alphanumeric': 0.08279866498011186, 'fraction_numerical': 0.005455903195818157, 'mean_word_length': 4.277304165996461, 'pattern_counts': {'":': 0, '<': 119, '<?xml version=': 0, '>': 113, 'https://': 96, 'lorem ipsum': 0, 'www.': 0, 'xml': 16}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '157358', 'n_tokens_mistral': 18717, 'n_tokens_neox': 17799, 'n_words': 8325}