<commit_before><!---
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

The YARN Timeline Service v.2
========================

<!-- MACRO{toc|fromDepth=0|toDepth=2} -->

## <a name="Overview"></a>Overview

### <a name="Introduction"></a>Introduction

YARN Timeline Service v.2 is the next major iteration of Timeline Server, following v.1 and v.1.5.
V.2 is created to address two major challenges of v.1.

#### Scalability
V.1 is limited to a single instance of writer/reader and storage, and does not scale well beyond
small clusters. V.2 uses a more scalable distributed writer architecture and a scalable backend
storage.

YARN Timeline Service v.2 separates the collection (writes) of data from serving (reads) of data.
It uses distributed collectors, essentially one collector for each YARN application. The readers
are separate instances that are dedicated to serving queries via REST API.

YARN Timeline Service v.2 chooses Apache HBase as the primary backing storage, as Apache HBase
scales well to a large size while maintaining good response times for reads and writes.

#### Usability improvements
In many cases, users are interested in information at the level of "flows" or logical groups of
YARN applications. It is much more common to launch a set or series of YARN applications to
complete a logical application. Timeline Service v.2 supports the notion of flows explicitly. In
addition, it supports aggregating metrics at the flow level.

Also, information such as configuration and metrics is treated and supported as first-class
citizens.

The following diagrams illustrates the relationship between different YARN entities modelling flows.

![Flow Hierarchy](./images/flow_hierarchy.png)

###<a name="Architecture"></a>Architecture

YARN Timeline Service v.2 uses a set of collectors (writers) to write data to the backend storage.
The collectors are distributed and co-located with the application masters to which they are
dedicated. All data that belong to that application are sent to the application level timeline
collectors with the exception of the resource manager timeline collector.

For a given application, the application master can write data for the application to the
co-located timeline collectors (which is an NM auxiliary service in this release). In addition,
node managers of other nodes that are running the containers for the application also write data
to the timeline collector on the node that is running the application master.

The resource manager also maintains its own timeline collector. It emits only YARN-generic
lifecycle events to keep its volume of writes reasonable.

The timeline readers are separate daemons separate from the timeline collectors, and they are
dedicated to serving queries via REST API.

The following diagram illustrates the design at a high level.

![Timeline Service v.2  architecture](./images/timeline_v2.jpg)

### <a name="Current_Status"></a>Current Status and Future Plans

YARN Timeline Service v.2 is currently in alpha ("alpha 1"). It is very much work in progress, and
many things can and will change rapidly. Users must enable Timeline Service v.2 only on a test or
experimental cluster to test the feature.

Most importantly, **security is not enabled**. Do not set up or use Timeline Service v.2 until
security is implemented if security is a requirement.

A complete end-to-end flow of writes and reads is functional, with Apache HBase as the backend.
You should be able to start generating data. When enabled, all YARN-generic events are
published as well as YARN system metrics such as CPU and memory. Furthermore, some applications
including Distributed Shell and MapReduce can write per-framework data to YARN Timeline Service
v.2.

The basic mode of accessing data is via REST. Currently there is no support for command line
access. The REST API comes with a good number of useful and flexible query patterns (see below for
more information).

The collectors (writers) are currently embedded in the node managers as auxiliary services. The
resource manager also has its dedicated in-process collector. The reader is currently a single
instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN
application (i.e. no off-cluster client).

When YARN Timeline Service v.2 is disabled, one can expect no functional or performance impact
on any other existing functionality.

The work to make it truly production-ready continues. Some key items include

* More robust storage fault tolerance
* Security
* Support for off-cluster clients
* More complete and integrated web UI
* Better support for long-running apps
* Offline (time-based periodic) aggregation for flows, users, and queues for reporting and
analysis
* Timeline collectors as separate instances from node managers
* Clustering of the readers
* Migration and compatibility with v.1


## <a name="Deployment"></a>Deployment

### <a name="Configurations"></a>Configurations

New configuration parameters that are introduced with v.2 are marked bold.

#### Basic configuration

| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the `TimelineClient` library used by applications will post entities and events to the Timeline server. Defaults to `false`. |
| `yarn.timeline-service.version` | Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and must bring up the timeline service v.1.5 (and nothing else). On the client side, if the client uses the same version of timeline service, it must succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. Defaults to `1.0f`. |
| **`yarn.timeline-service.writer.class`** | The class for the backend storage writer. Defaults to HBase storage writer. |
| **`yarn.timeline-service.reader.class`** | The class for the backend storage reader. Defaults to HBase storage reader. |
| **`yarn.system-metrics-publisher.enabled`** | The setting that controls whether yarn system metrics is published on the Timeline service or not by RM And NM. Defaults to `false`. |
| **`yarn.timeline-service.schema.prefix`** | The schema prefix for hbase tables. Defaults to "prod.". |

#### Advanced configuration

| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.hostname` | The hostname of the Timeline service web application. Defaults to `0.0.0.0` |
| `yarn.timeline-service.address` | Address for the Timeline server to start the RPC server. Defaults to `${yarn.timeline-service.hostname}:10200`. |
| `yarn.timeline-service.webapp.address` | The http address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8188`. |
| `yarn.timeline-service.webapp.https.address` | The https address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8190`. |
| **`yarn.timeline-service.hbase.configuration.file`** | Optional URL to an hbase-site.xml configuration file to be used to connect to the timeline-service hbase cluster. If empty or not specified, then the HBase configuration will be loaded from the classpath. When specified the values in the specified configuration file will override those from the ones that are present on the classpath. Defaults to `null`. |
| **`yarn.timeline-service.writer.flush-interval-seconds`** | The setting that controls how often the timeline collector flushes the timeline writer. Defaults to `60`. |
| **`yarn.timeline-service.app-collector.linger-period.ms`** | Time period till which the application collector will be alive in NM, after the  application master container finishes. Defaults to `1000` (1 second). |
| **`yarn.timeline-service.timeline-client.number-of-async-entities-to-merge`** | Time line V2 client tries to merge these many number of async entities (if available) and then call the REST ATS V2 API to submit. Defaults to `10`. |
| **`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). This should be set in the HBase cluster. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |

#### Enabling CORS support
To enable cross-origin support (CORS) for the Timeline Service v.2, please set the following configuration parameters:

In yarn-site.xml, set yarn.timeline-service.http-cross-origin.enabled to true.

In core-site.xml, add org.apache.hadoop.security.HttpCrossOriginFilterInitializer to hadoop.http.filter.initializers.

For more configurations used for cross-origin support, refer to [HttpAuthentication](../../hadoop-project-dist/hadoop-common/HttpAuthentication.html#CORS). Please note that yarn.timeline-service.http-cross-origin.enabled, if set to true, overrides hadoop.http.cross-origin.enabled.



### <a name="Enabling_Timeline_Service_v2"></a>Enabling Timeline Service v.2

#### Preparing Apache HBase cluster for storage
The first part is to set up or pick an Apache HBase cluster to use as the storage cluster. The
version of Apache HBase that is supported with Timeline Service v.2 is 1.1.x. The 1.0.x versions
do not work with Timeline Service v.2. The 1.2.x versions have not been tested.

Once you have an Apache HBase cluster ready to use for this purpose, perform the following steps.

First, add the timeline service jar to the HBase classpath in all HBase machines in the cluster. It
is needed for the coprocessor as well as the schema creator. For example,

    cp hadoop-yarn-server-timelineservice-hbase-3.0.0-alpha1-SNAPSHOT.jar /usr/hbase/lib/

Then, enable the coprocessor that handles the aggregation. To enable it, add the following entry in
region servers' `hbase-site.xml` file (generally located in the `conf` directory) as follows:

```
<property>
  <name>hbase.coprocessor.region.classes</name>
  <value>org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor</value>
</property>
```

Restart the region servers and the master to pick up the timeline service jar as well as the config
change. In this version, the coprocessor is loaded statically (i.e. system coprocessor) as opposed
to a dynamically (table coprocessor).

Finally, run the schema creator tool to create the necessary tables:

    bin/hadoop org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create

The `TimelineSchemaCreator` tool supports a few options that may come handy especially when you
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to skip existing tables
and continue to create other tables rather than failing the schema creation. By default, the tables
will have a schema prefix of "prod.". When no option or '-help' ('-h' for short) is provided, the
command usage is printed.

#### Enabling Timeline Service v.2
Following are the basic configurations to start Timeline service v.2:

```
<property>
  <name>yarn.timeline-service.version</name>
  <value>2.0f</value>
</property>

<property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name><commit_msg>YARN-5980. Update documentation for single node hbase deploy. Contributed by <NAME>.<commit_after><!---
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

The YARN Timeline Service v.2
========================

<!-- MACRO{toc|fromDepth=0|toDepth=2} -->

## <a name="Overview"></a>Overview

### <a name="Introduction"></a>Introduction

YARN Timeline Service v.2 is the next major iteration of Timeline Server, following v.1 and v.1.5.
V.2 is created to address two major challenges of v.1.

#### Scalability
V.1 is limited to a single instance of writer/reader and storage, and does not scale well beyond
small clusters. V.2 uses a more scalable distributed writer architecture and a scalable backend
storage.

YARN Timeline Service v.2 separates the collection (writes) of data from serving (reads) of data.
It uses distributed collectors, essentially one collector for each YARN application. The readers
are separate instances that are dedicated to serving queries via REST API.

YARN Timeline Service v.2 chooses Apache HBase as the primary backing storage, as Apache HBase
scales well to a large size while maintaining good response times for reads and writes.

#### Usability improvements
In many cases, users are interested in information at the level of "flows" or logical groups of
YARN applications. It is much more common to launch a set or series of YARN applications to
complete a logical application. Timeline Service v.2 supports the notion of flows explicitly. In
addition, it supports aggregating metrics at the flow level.

Also, information such as configuration and metrics is treated and supported as first-class
citizens.

The following diagrams illustrates the relationship between different YARN entities modelling flows.

![Flow Hierarchy](./images/flow_hierarchy.png)

###<a name="Architecture"></a>Architecture

YARN Timeline Service v.2 uses a set of collectors (writers) to write data to the backend storage.
The collectors are distributed and co-located with the application masters to which they are
dedicated. All data that belong to that application are sent to the application level timeline
collectors with the exception of the resource manager timeline collector.

For a given application, the application master can write data for the application to the
co-located timeline collectors (which is an NM auxiliary service in this release). In addition,
node managers of other nodes that are running the containers for the application also write data
to the timeline collector on the node that is running the application master.

The resource manager also maintains its own timeline collector. It emits only YARN-generic
lifecycle events to keep its volume of writes reasonable.

The timeline readers are separate daemons separate from the timeline collectors, and they are
dedicated to serving queries via REST API.

The following diagram illustrates the design at a high level.

![Timeline Service v.2  architecture](./images/timeline_v2.jpg)

### <a name="Current_Status"></a>Current Status and Future Plans

YARN Timeline Service v.2 is currently in alpha ("alpha 1"). It is very much work in progress, and
many things can and will change rapidly. Users must enable Timeline Service v.2 only on a test or
experimental cluster to test the feature.

Most importantly, **security is not enabled**. Do not set up or use Timeline Service v.2 until
security is implemented if security is a requirement.

A complete end-to-end flow of writes and reads is functional, with Apache HBase as the backend.
You should be able to start generating data. When enabled, all YARN-generic events are
published as well as YARN system metrics such as CPU and memory. Furthermore, some applications
including Distributed Shell and MapReduce can write per-framework data to YARN Timeline Service
v.2.

The basic mode of accessing data is via REST. Currently there is no support for command line
access. The REST API comes with a good number of useful and flexible query patterns (see below for
more information).

The collectors (writers) are currently embedded in the node managers as auxiliary services. The
resource manager also has its dedicated in-process collector. The reader is currently a single
instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN
application (i.e. no off-cluster client).

When YARN Timeline Service v.2 is disabled, one can expect no functional or performance impact
on any other existing functionality.

The work to make it truly production-ready continues. Some key items include

* More robust storage fault tolerance
* Security
* Support for off-cluster clients
* More complete and integrated web UI
* Better support for long-running apps
* Offline (time-based periodic) aggregation for flows, users, and queues for reporting and
analysis
* Timeline collectors as separate instances from node managers
* Clustering of the readers
* Migration and compatibility with v.1


## <a name="Deployment"></a>Deployment

### <a name="Configurations"></a>Configurations

New configuration parameters that are introduced with v.2 are marked bold.

#### Basic configuration

| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the `TimelineClient` library used by applications will post entities and events to the Timeline server. Defaults to `false`. |
| `yarn.timeline-service.version` | Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and must bring up the timeline service v.1.5 (and nothing else). On the client side, if the client uses the same version of timeline service, it must succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. Defaults to `1.0f`. |
| **`yarn.timeline-service.writer.class`** | The class for the backend storage writer. Defaults to HBase storage writer. |
| **`yarn.timeline-service.reader.class`** | The class for the backend storage reader. Defaults to HBase storage reader. |
| **`yarn.system-metrics-publisher.enabled`** | The setting that controls whether yarn system metrics is published on the Timeline service or not by RM And NM. Defaults to `false`. |
| **`yarn.timeline-service.schema.prefix`** | The schema prefix for hbase tables. Defaults to "prod.". |

#### Advanced configuration

| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.hostname` | The hostname of the Timeline service web application. Defaults to `0.0.0.0` |
| `yarn.timeline-service.address` | Address for the Timeline server to start the RPC server. Defaults to `${yarn.timeline-service.hostname}:10200`. |
| `yarn.timeline-service.webapp.address` | The http address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8188`. |
| `yarn.timeline-service.webapp.https.address` | The https address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8190`. |
| **`yarn.timeline-service.hbase.configuration.file`** | Optional URL to an hbase-site.xml configuration file to be used to connect to the timeline-service hbase cluster. If empty or not specified, then the HBase configuration will be loaded from the classpath. When specified the values in the specified configuration file will override those from the ones that are present on the classpath. Defaults to `null`. |
| **`yarn.timeline-service.writer.flush-interval-seconds`** | The setting that controls how often the timeline collector flushes the timeline writer. Defaults to `60`. |
| **`yarn.timeline-service.app-collector.linger-period.ms`** | Time period till which the application collector will be alive in NM, after the  application master container finishes. Defaults to `1000` (1 second). |
| **`yarn.timeline-service.timeline-client.number-of-async-entities-to-merge`** | Time line V2 client tries to merge these many number of async entities (if available) and then call the REST ATS V2 API to submit. Defaults to `10`. |
| **`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). This should be set in the HBase cluster. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |

#### Enabling CORS support
To enable cross-origin support (CORS) for the Timeline Service v.2, please set the following configuration parameters:

In yarn-site.xml, set yarn.timeline-service.http-cross-origin.enabled to true.

In core-site.xml, add org.apache.hadoop.security.HttpCrossOriginFilterInitializer to hadoop.http.filter.initializers.

For more configurations used for cross-origin support, refer to [HttpAuthentication](../../hadoop-project-dist/hadoop-common/HttpAuthentication.html#CORS). Please note that yarn.timeline-service.http-cross-origin.enabled, if set to true, overrides hadoop.http.cross-origin.enabled.



### <a name="Enabling_Timeline_Service_v2"></a>Enabling Timeline Service v.2

#### Preparing Apache HBase cluster for storage
There are a few steps to be done for preparing the storage for Timeline Service v.2:

Step 1) [Set up the HBase cluster](#Set_up_the_HBase_cluster)

Step 2) [Enable the coprocessor](#Enable_the_coprocessor)

Step 3) [Create the schema for Timeline Service v.2](#Create_schema)

Each step is explained in more detail below.

##### <a name="Set_up_the_HBase_cluster"> </a>Step 1) Set up the HBase cluster
The first part is to set up or pick an Apache HBase cluster to use as the storage cluster. The
version of Apache HBase that is supported with Timeline Service v.2 is 1.2.4. The 1.0.x versions
do not work with Timeline Service v.2. Later versions of HBase have not been tested with
Timeline Service.

HBase has different deployment modes. Refer to the HBase book for understanding them and pick a
mode that is suitable for your setup.
(http://hbase.apache.org/book.html#standalone_dist)

##### Simple deployment for HBase
If you are intent on a simple deploy profile for the Apache HBase cluster
where the data loading is light but the data needs to persist across node
comings and goings, you could consider the "Standalone HBase over HDFS" deploy mode.

This is a useful variation on the standalone HBase setup and has all HBase daemons running inside
one JVM but rather than persisting to the local filesystem, it persists to an HDFS instance.
Writing to HDFS where data is replicated ensures that data is persisted across node
comings and goings. To configure this standalone variant, edit your `hbase-site.xml` setting
the `hbase.rootdir` to point at a directory in your HDFS instance but then set
`hbase.cluster.distributed` to false. For example:

```
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://namenode.example.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>false</value>
  </property>
</configuration>
```

For more details on this mode, refer to
http://hbase.apache.org/book.html#standalone.over.hdfs .

Once you have an Apache HBase cluster ready to use, perform the following steps.

##### <a name="Enable_the_coprocessor"> </a>Step 2) Enable the coprocessor

Step 2.1) Add the timeline service jar to the HBase classpath in all HBase machines in the cluster. It
is needed for the coprocessor as well as the schema creator. For example,

    cp hadoop-yarn-server-timelineservice-hbase-3.0.0-alpha1-SNAPSHOT.jar /usr/hbase/lib/

Step 2.2) Enable the coprocessor that handles the aggregation. To enable it, add the following entry in
region servers' `hbase-site.xml` file (generally located in the `conf` directory) as follows:

```
<property>
  <name>hbase.coprocessor.region.classes</name>
  <value>org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor</value>
</property>
```

Step 2.3) Restart the region servers and the master to pick up the timeline service jar as well
as the config change. In this version, the coprocessor is loaded statically
(i.e. system coprocessor) as opposed to a dynamically (table coprocessor).

##### <a name="Create_schema"> </a>Step 3) Create the timeline service schema
Finally, run the schema creator tool to create the necessary tables:

    bin/hadoop org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create

The `TimelineSchemaCreator` tool supports a few options that may come handy especially when you
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to skip existing tables
and continue to create other tables rather than failing the schema creation. By default, the tables
will have a schema prefix of "prod.". When no option or '-help' ('-h' for short) is provided, the
command usage is printed.

#### Enabling Timeline Service v.2
Following are the basic configurations to start Timeline service v.2:

```
<property>
  <name>yarn.timeline-service.version</name>
  <value>2.0f</value>
</property>

<property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
{'fraction_non_alphanumeric': 0.07710120719763115, 'fraction_numerical': 0.007858173259433604, 'mean_word_length': 4.6168443496801705, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 98, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 9}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4944553', 'n_tokens_mistral': 7086, 'n_tokens_neox': 6796, 'n_words': 3388}