<commit_before>[[install-config-cluster-metrics]]
= Enabling Cluster Metrics
{product-author}
{product-version}
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:
:prewrap!:

toc::[]

== Overview

The
xref:../architecture/infrastructure_components/kubernetes_infrastructure.adoc#kubelet[kubelet]
exposes metrics that can be collected and stored in back-ends by
link:https://github.com/GoogleCloudPlatform/heapster[Heapster].

As an {product-title} administrator, you can view a cluster's metrics from all
containers and components in one user interface.  These metrics are also
used by xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscalers]
in order to determine when and how to scale.

This topic describes using
link:https://github.com/hawkular/hawkular-metrics[Hawkular Metrics] as a metrics
engine which stores the data persistently in a
link:http://cassandra.apache.org/[Cassandra] database. When this is configured,
CPU, memory and network-based metrics are viewable from the {product-title} web console
and are available for use by xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal
pod autoscalers].

Heapster retrieves a list of all nodes from the master server, then contacts
each node individually through the `/stats` endpoint. From there, Heapster
scrapes the metrics for CPU, memory and network usage, then exports them into Hawkular
Metrics.

Browsing individual pods in the web console displays separate sparkline charts
for memory and CPU. The time range displayed is selectable, and these charts
automatically update every 30 seconds. If there are multiple containers on the
pod, then you can select a specific container to display its metrics.

If xref:../admin_guide/limits.adoc#admin-guide-limits[resource limits] are defined for your
project, then you can also see a donut chart for each pod. The donut chart
displays usage against the resource limit. For example: `145 Available of 200
MiB`, with the donut chart showing `55 MiB Used`.

ifdef::openshift-origin[]
For more information about the metrics integration, please refer to the
link:https://github.com/openshift/origin-metrics[Origin Metrics] GitHub project.
endif::[]

[[cluster-metrics-before-you-begin]]
== Before You Begin

ifdef::openshift-origin[]
[WARNING]
====
If your {product-title} installation was originally performed on a version
previous to v1.0.8, even if it has since been updated to a newer version, follow
the instructions for node certificates outlined in xref:upgrading/manual_upgrades.adoc#manual-updating-master-and-node-certificates[Updating
Master and Node Certificates]. If the node certificate does not contain the IP
address of the node, then Heapster will fail to retrieve any metrics.
====
endif::[]

The components for cluster metrics must be deployed to the *openshift-infra*
project. This allows xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod
autoscalers] to discover the Heapster service and use it to retrieve metrics
that can be used for autoscaling.

All of the following commands in this topic must be executed under the
*openshift-infra* project. To switch to the *openshift-infra* project:

----
$ oc project openshift-infra
----

To enable cluster metrics, you must next configure the following:

- xref:../install_config/cluster_metrics.adoc#metrics-service-accounts[Service Accounts]
- xref:../install_config/cluster_metrics.adoc#metrics-data-storage[Metrics Data Storage]
- xref:../install_config/cluster_metrics.adoc#metrics-deployer[Metrics Deployer]

[[metrics-service-accounts]]
== Service Accounts

You must configure xref:../admin_guide/service_accounts.adoc#admin-guide-service-accounts[service accounts]
for:

* xref:../install_config/cluster_metrics.adoc#metrics-deployer-service-account[Metrics Deployer]
* xref:../install_config/cluster_metrics.adoc#heapster-service-account[Heapster]

[[metrics-deployer-service-account]]
=== Metrics Deployer Service Account

The xref:metrics-deployer[Metrics Deployer] will be discussed in a later step,
but you must first set up a service account for it:

. Create a *metrics-deployer* service account:
+
----
$ oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-deployer
secrets:
- name: metrics-deployer
API
----

. Before it can deploy components, the *metrics-deployer* service account must
also be granted the `edit` permission for the *openshift-infra* project:
+
----
$ oadm policy add-role-to-user \
    edit system:serviceaccount:openshift-infra:metrics-deployer
----

[[heapster-service-account]]
=== Heapster Service Account

The Heapster component requires access to the master server to list all
available nodes and access the `/stats` endpoint for each node. Before it can do
this, the Heapster service account requires the `cluster-reader` permission:

----
$ oadm policy add-cluster-role-to-user \
    cluster-reader system:serviceaccount:openshift-infra:heapster
----

[NOTE]
====
The Heapster service account is created automatically during the
xref:../install_config/cluster_metrics.adoc#deploying-the-metrics-components[Deploying
the Metrics Components] step.
====

[[metrics-data-storage]]
== Metrics Data Storage

You can store the metrics data to either
xref:../architecture/additional_concepts/storage.adoc#architecture-additional-concepts-storage[persistent storage] or to
a temporary xref:../dev_guide/volumes.adoc#dev-guide-volumes[pod volume].

[[metrics-persistent-storage]]
=== Persistent Storage

Running {product-title} cluster metrics with persistent storage means that
your metrics will be stored to a
xref:../architecture/additional_concepts/storage.adoc#persistent-volumes[persistent
volume] and be able to survive a pod being restarted or recreated. This is
ideal if you require your metrics data to be guarded from data loss.

The size of the persisted volume can be specified with the `CASSANDRA_PV_SIZE`
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[template
parameter]. By default it is set to 10 GB, which may or may not be sufficient
for the size of the cluster you are using. If you require more space, for
instance 100 GB, you could specify it with something like this:

----
$ oc process -f metrics-deployer.yaml -v \
    HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com,CASSANDRA_PV_SIZE=100Gi \
    | oc create -f -
----

The size requirement of the Cassandra storage is dependent on the cluster size.
It is the administrator's responsibility to ensure that the size requirements
are sufficient for their setup and to monitor usage to ensure that the disk does
not become full.

[WARNING]
====
Data loss will result if the Cassandra persisted volume runs out of sufficient space.
====

For cluster metrics to work with persistent storage, ensure that the persistent
volume has the *ReadWriteOnce* access mode. If not, the persistent volume claim
will not be able to find the persistent volume, and Cassandra will fail to
start.

To use persistent storage with the metric components, ensure that a
xref:../architecture/additional_concepts/storage.adoc#persistent-volumes[persistent
volume] of sufficient size is available. The creation of
xref:../architecture/additional_concepts/storage.adoc#persistent-volume-claims[persistent
volume claims] is handled by the
xref:../install_config/cluster_metrics.adoc#metrics-deployer[Metrics Deployer].

If you would like to use xref:../install_config/persistent_storage/dynamically_provisioning_pvs.adoc#install-config-persistent-storage-dynamically-provisioning-pvs[dynamically provisioned
persistent volumes] you will need to set the `*DYNAMICALLY_PROVISION_STORAGE*`
xref:../install_config/cluster_metrics.adoc#modifying-the-deployer-template[template
option] to `true` for the Metrics Deployer.

[[metrics-non-persistent-storage]]
=== Non-Persistent Storage

Running {product-title} cluster metrics with non-persistent storage means that
any stored metrics will be deleted when the pod is deleted. While it is much
easier to run cluster metrics with non-persistent data, running with
non-persistent data does come with the risk of permanent data loss. However,
metrics can still survive a container being restarted.

In order to use non-persistent storage, you must set the
`*USE_PERSISTENT_STORAGE*`
xref:../install_config/cluster_metrics.adoc#modifying-the-deployer-template[template
option] to `false` for the Metrics Deployer.

[[metrics-deployer]]
== Metrics Deployer

The Metrics Deployer deploys and configures all of the metrics components. You
can configure it by passing in information from
xref:../dev_guide/secrets.adoc#dev-guide-secrets[secrets] and by passing
parameters to the Metrics Deployer's
xref:../architecture/core_concepts/templates.adoc#architecture-core-concepts-templates[template].

[[metrics-deployer-using-secrets]]
=== Using Secrets

The Metrics Deployer will auto-generate self-signed certificates for use between its
components and will generate a 
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encrypting route] to expose
the Hawkular Metrics service. This route is what allows the web console to access the Hawkular Metrics
service.

In order for the browser running the web console to trust the connection through this route,
it must trust the route's certificate. This can be accomplished by 
xref:metrics-using-secrets-byo-certs[providing your own certificates] signed by a trusted
Certificate Authority. The Metric Deployer's secret allows you to pass your own certificates
which it will then use when creating the route.

If you do not wish to provide your own certificates, the router's default certificate will
be used instead. 

[[metrics-using-secrets-byo-certs]]
==== Providing Your Own Certificates

To provide your own certificate which will be used by the 
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encrypting route],
you can pass these values as 
xref:../dev_guide/secrets.adoc#dev-guide-secrets[secrets] to the Metrics Deployer.

The `hawkular-metrics.pem` value needs to contain the certificate in its *_.pem_*
format. You may also need to provide the certificate for the Certificate Authority
which signed this *_pem_* file via the `hawkular-metrics-ca.cert` secret.

====
----
$ oc secrets new metrics-deployer \
    hawkular-metrics.pem=/home/openshift/metrics/hm.pem \
    hawkular-metrics-ca.cert=/home/openshift/metrics/hm-ca.cert
----
====

When these secrets are provided, the deployer uses these values to specify the
`key`, `certificate` and `caCertificate` values for the re-encrypting route it generated.

For more information, please see the 
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encryption
route documentation].


[[metrics-using-secrets-autogenerated]]
==== Using the Router's Default Certificate

If the `hawkular-metrics.pem` value is not specified, the re-encrypting route will
use the router's default certificate, which may not be trusted by browsers.

A secret named *metrics-deployer* will still be required in this situation. This
can be considered a "dummy" secret since the secret it specifies is not actually used
by the component.

To create a "dummy" secret that does not specify a certificate value:

----
$ oc secrets new metrics-deployer nothing=/dev/null
----

[[deployer-secret-options]]
==== Deployer Secret Options

The following table contains more advanced configuration options, detailing all
the secrets which can be used by the deployer:

[cols="2,4",options="header"]
|===

|Secret Name |Description

|*_hawkular-metrics.pem_*
|The *_pem_* file to use for the Hawkular Metrics certificate used for the re-encrypting route. This
certificate *must* contain the hostname used by the route (eg HAWKULAR_METRICS_HOSTNAME). The default
router's certificate will be used for the route if this option is unspecified.

|*_hawkular-metrics-ca.cert_*
|The certificate for the CA used to sign the *_hawkular-metrics.pem_*. This is optional if the *_hawkular-metrics.pem_*
does not contain the CA certificate directly.

|*_heapster.cert_*
|The certificate for Heapster to use. This is auto-generated if unspecified.

|*_heapster.key_*
|The key to use with the Heapster certificate. This is ignored if
*_heapster.cert_* is not specified

|*_heapster_client_ca.cert_*
|The certificate that generates *_heapster.cert_*. This is required if
*_heapster.cert_* is specified.  Otherwise, the main CA for the {product-title}
installation is used. In order for
xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscaling] to function
properly, this should not be overridden.

|*_heapster_allowed_users_*
|A file containing a comma-separated list of CN to accept from certificates
signed with the specified CA. By default, this is set to allow the
{product-title} service proxy to connect.  If you override this, make sure to
add `system:master-proxy` to the list in order to allow
xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscaling] to function
properly.

|===

[[modifying-the-deployer-template]]
=== Modifying the Deployer Template

The {product-title}  installer uses a
xref:../architecture/core_concepts/templates.adoc#architecture-core-concepts-templates[template] to deploy the
metrics components. The default template can be found at the following path:

ifdef::openshift-origin[]
====
----
/usr/share/openshift/examples/infrastructure-templates/origin/metrics-deployer.yaml
----
====

[NOTE]
====
Depending on your installation method, the template may not be present in your
{product-title} installation. If so, the template can be found at the following GitHub
location:

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/origin/metrics-deployer.yaml
====

endif::[]
ifdef::openshift-enterprise[]
====
----
/usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml
----
====
endif::[]

In case you need to make any changes to this file, copy it to another directory
with the file name *_metrics-deployer.yaml_* and refer to the new location when
using it in the following sections.

[[deployer-template-parameters]]
==== Deployer Template Parameters

The deployer template parameter options and their defaults are listed in the
default *_metrics-deployer.yaml_* file. If required, you can override these
values when creating the Metrics Deployer.

.Template Parameters
[options="header"]
|===

|Parameter |Description

|`*METRIC_DURATION*`
|The number of days metrics should be stored.

|`*CASSANDRA_PV_SIZE*`
|The persistent volume size for each of the Cassandra nodes.

|`*CASSANDRA_NODES*`
|The number of initial Cassandra nodes to deploy

|`*USE_PERSISTENT_STORAGE*`
|Set to *true* for persistent storage; set to *false* to use non-persistent storage.

|`*DYNAMICALLY_PROVISION_STORAGE*`
|Set to *true* to allow for xref:../install_config/persistent_storage/dynamically_provisioning_pvs.adoc#install-config-persistent-storage-dynamically-provisioning-pvs[dynamically provisioned storage].

|`*REDEPLOY*`
|If set to *true*, the deployer will try to delete all the existing components before trying to redeploy.

|`*HAWKULAR_METRICS_HOSTNAME*`
|External host name where clients can reach Hawkular Metrics.

|`*MASTER_URL*`
|Internal URL for the master, for authentication retrieval.

|`*IMAGE_VERSION*`
|Specify version for metrics components. For example, for *openshift/origin-metrics-deployer:latest*, set version to *latest*.

|`*IMAGE_PREFIX*`
|Specify prefix for metrics components. For example, for *openshift/origin-metrics-deployer:latest*, set prefix to *openshift/origin-*.

|`*MODE*`
|Can be set to: *preflight* to perform validation before a deployment; *deploy* to perform an initial deployment; *refresh* to delete and redeploy all components but to keep persisted data and routes; *redeploy* to delete and redeploy everything (losing all data in the process); *validate* to re-run validations after a deployment.

|`*IGNORE_PREFLIGHT*`
|Can be set to *true* to disable the preflight checks. This will allow the deployer to continue even if the preflight check has failed.

|`*USER_WRITE_ACCESS*`
|If user accounts should be able to write metrics. Defaults to 'false' so that only Heapster can write metrics and not individual users. It is recommended to disable user write access, if enabled any user will be able to write metrics to the system which can affect performance and can cause Cassandra disk usage to unpredictably increase.

|===

The only required parameter is `*HAWKULAR_METRICS_HOSTNAME*`. This value is
required when creating the deployer because it specifies the hostname for the
Hawkular Metrics xref:../architecture/core_concepts/routes.adoc#architecture-core-concepts-routes[route]. This
value should correspond to a fully qualified domain name. You will need to know
the value of `*HAWKULAR_METRICS_HOSTNAME*` when
xref:../install_config/cluster_metrics.adoc#configuring-openshift-metrics[configuring
the console] for metrics access.

If you are using
xref:../install_config/cluster_metrics.adoc#metrics-persistent-storage[persistent
storage] with Cassandra, it is the administrator's responsibility to set a
sufficient disk size for the cluster using the `*CASSANDRA_PV_SIZE*` parameter.
It is also the administrator's responsibility to monitor disk usage to make sure
that it does not become full.

[WARNING]
====
Data loss will result if the Cassandra persisted volume runs out of sufficient space.
====

All of the other parameters are optional and allow for greater customization.
For instance, if you have a custom install in which the Kubernetes master is not
available under *_https://kubernetes.default.svc:443_* you can specify the value
to use instead with the `*MASTER_URL*` parameter. To deploy a specific version
of the metrics components, use the `*IMAGE_VERSION*` parameter.

[WARNING]
====
It is highly recommended to not use *latest* for the *IMAGE_VERSION*. The *latest*
version corresponds to the very latest version available and can cause issues if it brings in a 
newer version not meant to function on the version of {product-title} you are currently running.
====

[[deploying-the-metrics-components]]
== Deploying the Metric Components

Because deploying and configuring all the metric components is handled by the
Metrics Deployer, you can simply deploy everything in one step.

The following examples show you how to deploy metrics with and without
persistent storage using the default template parameters. Optionally, you can
specify any of the
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[template
parameters] when calling these commands.

.Deploying with Persistent Storage
====
The following command sets the Hawkular Metrics route to use
*hawkular-metrics.example.com* and is deployed using persistent storage.

You must have a persistent volume of sufficient size available.

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com
----
====

.Deploying without Persistent Storage
====
The following command sets the Hawkular Metrics route to use
*hawkular-metrics.example.com* and deploy without persistent storage.

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -p USE_PERSISTENT_STORAGE=false
----
====

[WARNING]
====
Because this is being deployed without persistent storage, metric data loss
can occur.
====

[[metrics-deployer-validations]]
=== Metrics Deployer Validations

The metrics deployer does some validation both before and after deployment. If
the pre-flight validation fails, the environment for deployment is considered
unsuitable and the deployment is aborted. However, you can add
`*IGNORE_PREFLIGHT=true*` to the deployer parameters if you believe the
validation has erred.

If post-deployment validation fails, the deployer finishes in an *Error* state,
which indicates that you should check the deployer logs for issues that may
require addressing.  For example, the validation may detect that the external
*hawkular-metrics* route is not actually in use, because the route was already
created somewhere else. The validation output at the end of a deployment should
explain as clearly as possible any issues it finds and what you can do to
address them.

Once you have addressed deployment validation issues, you can re-run just the
validation by running the deployer again with the `*MODE=validate*` parameter
added, for example:

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -p MODE=validate
----

[[configuring-openshift-metrics]]
== Configuring {product-title}

The {product-title} web console uses the data coming from the Hawkular Metrics
service to display its graphs. The URL for accessing the Hawkular Metrics
service must be configured via the `*metricsPublicURL*` option in the
xref:../install_config/master_node_configuration.adoc#master-configuration-files[master
configuration file] (*_/etc/origin/master/master-config.yaml_*). This URL
corresponds to the route created with the `*HAWKULAR_METRICS_HOSTNAME*` template
parameter during the
xref:../install_config/cluster_metrics.adoc#deploying-the-metrics-components[deployment]
of the metrics components.

[NOTE]
====
You must be able to resolve the `*HAWKULAR_METRICS_HOSTNAME*` from the browser
accessing the console.
====

For example, if your `*HAWKULAR_METRICS_HOSTNAME*` corresponds to
`hawkular-metrics.example.com`, then you must make the following change in the
*_master-config.yaml_* file:

====
[source,yaml,]
----
  assetConfig:
    ...
    metricsPublicURL: "https://hawkular-metrics.example.com/hawkular/metrics"
----
====

Once you have updated and saved the *_master-config.yaml_* file, you must
restart your {product-title} instance.

When your {product-title} server is back up and running, metrics will be
displayed on the pod overview pages.

[CAUTION]
====
If you are using self-signed certificates, remember that the Hawkular Metrics
service is hosted under a different host name and uses different certificates
than the console. You may need to explicitly open a browser tab to the value
specified in `*metricsPublicURL*` and accept that certificate.

To avoid this issue, use certificates which are configured to be acceptable by
your browser.
====

[[cluster-metrics-accessing-hawkular-metrics-directly]]
== Accessing Hawkular Metrics Directly

To access and manage metrics more directly, use the Hawkular Metrics API.

[NOTE]
====
When accessing Hawkular Metrics via the API, you will only be able to perform
reads. Writing metrics has been disabled by default. If you want for individual
users to also be able to write metrics, you will need to set the
`*USER_WRITE_ACCESS*`
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[deployer template parameter]
to *true*.

However, it is recommended to use the default configuration and only have
metrics enter the system via Heapster. If write access is enabled, any user
will be able to write metrics to the system, which can affect performance and
cause Cassandra disk usage to unpredictably increase.
====

The link:http://www.hawkular.org/docs/rest/rest-metrics.html[Hawkular Metrics
documentation] covers how to use the API, but there are a few differences when
dealing with the version of Hawkular Metrics configured for use on
{product-title}:

[[cluster-metrics-openshift-projects-and-hawkular-tenants]]
=== {product-title} Projects and Hawkular Tenants

Hawkular Metrics is a multi-tenanted application. It is configured so that a
project in {product-title} corresponds to a tenant in Hawkular Metrics.

As such, when accessing metrics for a project named *MyProject* you will need
to set the
link:http://www.hawkular.org/docs/rest/rest-metrics.html#_tenant_header[*Hawkular-Tenant*]
header to *MyProject*.

There is also a special tenant named *_system* which contains system level
metrics. This will require either a *cluster-reader* or *cluster-admin* level
privileges to access.

=== Authorization

The Hawkular Metrics service will authenticate the user against {product-title}
to determine if the user has access to the project it is trying to access.

Hawkular Metrics accepts a bearer token from the client and verifies that token
with the {product-title} server using a *SubjectAccessReview*. If the user has
proper read privileges for the project, they are allowed to read the metrics
for that project. For the *_system* tenant, the user requesting to read from
this tenant must have *cluster-reader* permission.

When accessing the Hawkular Metrics API, you will need to pass a bearer token in
the *Authorization* header.

ifdef::openshift-origin[]
== Accessing Heapster Directly

Heapster has been configured to be only accessible via the
xref:../rest_api/kubernetes_v1.adoc#proxy-get-requests-to-service[API proxy].
Accessing it will required either a cluster-reader or cluster-admin privileges.

For example, to access the Heapster *validate* page, you need to access it
using something similar to:

----
$ curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXX" \
       -X GET https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/validate
----

For more information about Heapster and how to access its APIs, please refer the
link:https://github.com/kubernetes/heapster/[Heapster] project.

endif::[]

[[metrics-cleanup]]
== Cleanup

You can remove everything deloyed by the metrics deployer by performing the
following steps:

----
$ oc delete all,sa,templates,secrets,pvc --selector="metrics-infra"
----

To remove the deployer components, perform the following steps:

----
$ oc delete sa,secret metrics-deployer
----
<commit_msg>Bug 1368881, added information about the diagnostic for metrics in the Metrics Deployer Validations section<commit_after>[[install-config-cluster-metrics]]
= Enabling Cluster Metrics
{product-author}
{product-version}
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:
:prewrap!:

toc::[]

== Overview

The
xref:../architecture/infrastructure_components/kubernetes_infrastructure.adoc#kubelet[kubelet]
exposes metrics that can be collected and stored in back-ends by
link:https://github.com/GoogleCloudPlatform/heapster[Heapster].

As an {product-title} administrator, you can view a cluster's metrics from all
containers and components in one user interface.  These metrics are also
used by xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscalers]
in order to determine when and how to scale.

This topic describes using
link:https://github.com/hawkular/hawkular-metrics[Hawkular Metrics] as a metrics
engine which stores the data persistently in a
link:http://cassandra.apache.org/[Cassandra] database. When this is configured,
CPU, memory and network-based metrics are viewable from the {product-title} web console
and are available for use by xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal
pod autoscalers].

Heapster retrieves a list of all nodes from the master server, then contacts
each node individually through the `/stats` endpoint. From there, Heapster
scrapes the metrics for CPU, memory and network usage, then exports them into Hawkular
Metrics.

Browsing individual pods in the web console displays separate sparkline charts
for memory and CPU. The time range displayed is selectable, and these charts
automatically update every 30 seconds. If there are multiple containers on the
pod, then you can select a specific container to display its metrics.

If xref:../admin_guide/limits.adoc#admin-guide-limits[resource limits] are defined for your
project, then you can also see a donut chart for each pod. The donut chart
displays usage against the resource limit. For example: `145 Available of 200
MiB`, with the donut chart showing `55 MiB Used`.

ifdef::openshift-origin[]
For more information about the metrics integration, please refer to the
link:https://github.com/openshift/origin-metrics[Origin Metrics] GitHub project.
endif::[]

[[cluster-metrics-before-you-begin]]
== Before You Begin

ifdef::openshift-origin[]
[WARNING]
====
If your {product-title} installation was originally performed on a version
previous to v1.0.8, even if it has since been updated to a newer version, follow
the instructions for node certificates outlined in xref:upgrading/manual_upgrades.adoc#manual-updating-master-and-node-certificates[Updating
Master and Node Certificates]. If the node certificate does not contain the IP
address of the node, then Heapster will fail to retrieve any metrics.
====
endif::[]

The components for cluster metrics must be deployed to the *openshift-infra*
project. This allows xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod
autoscalers] to discover the Heapster service and use it to retrieve metrics
that can be used for autoscaling.

All of the following commands in this topic must be executed under the
*openshift-infra* project. To switch to the *openshift-infra* project:

----
$ oc project openshift-infra
----

To enable cluster metrics, you must next configure the following:

- xref:../install_config/cluster_metrics.adoc#metrics-service-accounts[Service Accounts]
- xref:../install_config/cluster_metrics.adoc#metrics-data-storage[Metrics Data Storage]
- xref:../install_config/cluster_metrics.adoc#metrics-deployer[Metrics Deployer]

[[metrics-service-accounts]]
== Service Accounts

You must configure xref:../admin_guide/service_accounts.adoc#admin-guide-service-accounts[service accounts]
for:

* xref:../install_config/cluster_metrics.adoc#metrics-deployer-service-account[Metrics Deployer]
* xref:../install_config/cluster_metrics.adoc#heapster-service-account[Heapster]

[[metrics-deployer-service-account]]
=== Metrics Deployer Service Account

The xref:metrics-deployer[Metrics Deployer] will be discussed in a later step,
but you must first set up a service account for it:

. Create a *metrics-deployer* service account:
+
----
$ oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-deployer
secrets:
- name: metrics-deployer
API
----

. Before it can deploy components, the *metrics-deployer* service account must
also be granted the `edit` permission for the *openshift-infra* project:
+
----
$ oadm policy add-role-to-user \
    edit system:serviceaccount:openshift-infra:metrics-deployer
----

[[heapster-service-account]]
=== Heapster Service Account

The Heapster component requires access to the master server to list all
available nodes and access the `/stats` endpoint for each node. Before it can do
this, the Heapster service account requires the `cluster-reader` permission:

----
$ oadm policy add-cluster-role-to-user \
    cluster-reader system:serviceaccount:openshift-infra:heapster
----

[NOTE]
====
The Heapster service account is created automatically during the
xref:../install_config/cluster_metrics.adoc#deploying-the-metrics-components[Deploying
the Metrics Components] step.
====

[[metrics-data-storage]]
== Metrics Data Storage

You can store the metrics data to either
xref:../architecture/additional_concepts/storage.adoc#architecture-additional-concepts-storage[persistent storage] or to
a temporary xref:../dev_guide/volumes.adoc#dev-guide-volumes[pod volume].

[[metrics-persistent-storage]]
=== Persistent Storage

Running {product-title} cluster metrics with persistent storage means that
your metrics will be stored to a
xref:../architecture/additional_concepts/storage.adoc#persistent-volumes[persistent
volume] and be able to survive a pod being restarted or recreated. This is
ideal if you require your metrics data to be guarded from data loss.

The size of the persisted volume can be specified with the `CASSANDRA_PV_SIZE`
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[template
parameter]. By default it is set to 10 GB, which may or may not be sufficient
for the size of the cluster you are using. If you require more space, for
instance 100 GB, you could specify it with something like this:

----
$ oc process -f metrics-deployer.yaml -v \
    HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com,CASSANDRA_PV_SIZE=100Gi \
    | oc create -f -
----

The size requirement of the Cassandra storage is dependent on the cluster size.
It is the administrator's responsibility to ensure that the size requirements
are sufficient for their setup and to monitor usage to ensure that the disk does
not become full.

[WARNING]
====
Data loss will result if the Cassandra persisted volume runs out of sufficient space.
====

For cluster metrics to work with persistent storage, ensure that the persistent
volume has the *ReadWriteOnce* access mode. If not, the persistent volume claim
will not be able to find the persistent volume, and Cassandra will fail to
start.

To use persistent storage with the metric components, ensure that a
xref:../architecture/additional_concepts/storage.adoc#persistent-volumes[persistent
volume] of sufficient size is available. The creation of
xref:../architecture/additional_concepts/storage.adoc#persistent-volume-claims[persistent
volume claims] is handled by the
xref:../install_config/cluster_metrics.adoc#metrics-deployer[Metrics Deployer].

If you would like to use xref:../install_config/persistent_storage/dynamically_provisioning_pvs.adoc#install-config-persistent-storage-dynamically-provisioning-pvs[dynamically provisioned
persistent volumes] you will need to set the `*DYNAMICALLY_PROVISION_STORAGE*`
xref:../install_config/cluster_metrics.adoc#modifying-the-deployer-template[template
option] to `true` for the Metrics Deployer.

[[metrics-non-persistent-storage]]
=== Non-Persistent Storage

Running {product-title} cluster metrics with non-persistent storage means that
any stored metrics will be deleted when the pod is deleted. While it is much
easier to run cluster metrics with non-persistent data, running with
non-persistent data does come with the risk of permanent data loss. However,
metrics can still survive a container being restarted.

In order to use non-persistent storage, you must set the
`*USE_PERSISTENT_STORAGE*`
xref:../install_config/cluster_metrics.adoc#modifying-the-deployer-template[template
option] to `false` for the Metrics Deployer.

[[metrics-deployer]]
== Metrics Deployer

The Metrics Deployer deploys and configures all of the metrics components. You
can configure it by passing in information from
xref:../dev_guide/secrets.adoc#dev-guide-secrets[secrets] and by passing
parameters to the Metrics Deployer's
xref:../architecture/core_concepts/templates.adoc#architecture-core-concepts-templates[template].

[[metrics-deployer-using-secrets]]
=== Using Secrets

The Metrics Deployer will auto-generate self-signed certificates for use between its
components and will generate a
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encrypting route] to expose
the Hawkular Metrics service. This route is what allows the web console to access the Hawkular Metrics
service.

In order for the browser running the web console to trust the connection through this route,
it must trust the route's certificate. This can be accomplished by
xref:metrics-using-secrets-byo-certs[providing your own certificates] signed by a trusted
Certificate Authority. The Metric Deployer's secret allows you to pass your own certificates
which it will then use when creating the route.

If you do not wish to provide your own certificates, the router's default certificate will
be used instead.

[[metrics-using-secrets-byo-certs]]
==== Providing Your Own Certificates

To provide your own certificate which will be used by the
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encrypting route],
you can pass these values as
xref:../dev_guide/secrets.adoc#dev-guide-secrets[secrets] to the Metrics Deployer.

The `hawkular-metrics.pem` value needs to contain the certificate in its *_.pem_*
format. You may also need to provide the certificate for the Certificate Authority
which signed this *_pem_* file via the `hawkular-metrics-ca.cert` secret.

====
----
$ oc secrets new metrics-deployer \
    hawkular-metrics.pem=/home/openshift/metrics/hm.pem \
    hawkular-metrics-ca.cert=/home/openshift/metrics/hm-ca.cert
----
====

When these secrets are provided, the deployer uses these values to specify the
`key`, `certificate` and `caCertificate` values for the re-encrypting route it generated.

For more information, please see the
xref:../architecture/core_concepts/routes.adoc#secured-routes[re-encryption
route documentation].


[[metrics-using-secrets-autogenerated]]
==== Using the Router's Default Certificate

If the `hawkular-metrics.pem` value is not specified, the re-encrypting route will
use the router's default certificate, which may not be trusted by browsers.

A secret named *metrics-deployer* will still be required in this situation. This
can be considered a "dummy" secret since the secret it specifies is not actually used
by the component.

To create a "dummy" secret that does not specify a certificate value:

----
$ oc secrets new metrics-deployer nothing=/dev/null
----

[[deployer-secret-options]]
==== Deployer Secret Options

The following table contains more advanced configuration options, detailing all
the secrets which can be used by the deployer:

[cols="2,4",options="header"]
|===

|Secret Name |Description

|*_hawkular-metrics.pem_*
|The *_pem_* file to use for the Hawkular Metrics certificate used for the re-encrypting route. This
certificate *must* contain the hostname used by the route (eg HAWKULAR_METRICS_HOSTNAME). The default
router's certificate will be used for the route if this option is unspecified.

|*_hawkular-metrics-ca.cert_*
|The certificate for the CA used to sign the *_hawkular-metrics.pem_*. This is optional if the *_hawkular-metrics.pem_*
does not contain the CA certificate directly.

|*_heapster.cert_*
|The certificate for Heapster to use. This is auto-generated if unspecified.

|*_heapster.key_*
|The key to use with the Heapster certificate. This is ignored if
*_heapster.cert_* is not specified

|*_heapster_client_ca.cert_*
|The certificate that generates *_heapster.cert_*. This is required if
*_heapster.cert_* is specified.  Otherwise, the main CA for the {product-title}
installation is used. In order for
xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscaling] to function
properly, this should not be overridden.

|*_heapster_allowed_users_*
|A file containing a comma-separated list of CN to accept from certificates
signed with the specified CA. By default, this is set to allow the
{product-title} service proxy to connect.  If you override this, make sure to
add `system:master-proxy` to the list in order to allow
xref:../dev_guide/pod_autoscaling.adoc#dev-guide-pod-autoscaling[horizontal pod autoscaling] to function
properly.

|===

[[modifying-the-deployer-template]]
=== Modifying the Deployer Template

The {product-title}  installer uses a
xref:../architecture/core_concepts/templates.adoc#architecture-core-concepts-templates[template] to deploy the
metrics components. The default template can be found at the following path:

ifdef::openshift-origin[]
====
----
/usr/share/openshift/examples/infrastructure-templates/origin/metrics-deployer.yaml
----
====

[NOTE]
====
Depending on your installation method, the template may not be present in your
{product-title} installation. If so, the template can be found at the following GitHub
location:

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/origin/metrics-deployer.yaml
====

endif::[]
ifdef::openshift-enterprise[]
====
----
/usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml
----
====
endif::[]

In case you need to make any changes to this file, copy it to another directory
with the file name *_metrics-deployer.yaml_* and refer to the new location when
using it in the following sections.

[[deployer-template-parameters]]
==== Deployer Template Parameters

The deployer template parameter options and their defaults are listed in the
default *_metrics-deployer.yaml_* file. If required, you can override these
values when creating the Metrics Deployer.

.Template Parameters
[options="header"]
|===

|Parameter |Description

|`*METRIC_DURATION*`
|The number of days metrics should be stored.

|`*CASSANDRA_PV_SIZE*`
|The persistent volume size for each of the Cassandra nodes.

|`*CASSANDRA_NODES*`
|The number of initial Cassandra nodes to deploy

|`*USE_PERSISTENT_STORAGE*`
|Set to *true* for persistent storage; set to *false* to use non-persistent storage.

|`*DYNAMICALLY_PROVISION_STORAGE*`
|Set to *true* to allow for xref:../install_config/persistent_storage/dynamically_provisioning_pvs.adoc#install-config-persistent-storage-dynamically-provisioning-pvs[dynamically provisioned storage].

|`*REDEPLOY*`
|If set to *true*, the deployer will try to delete all the existing components before trying to redeploy.

|`*HAWKULAR_METRICS_HOSTNAME*`
|External host name where clients can reach Hawkular Metrics.

|`*MASTER_URL*`
|Internal URL for the master, for authentication retrieval.

|`*IMAGE_VERSION*`
|Specify version for metrics components. For example, for *openshift/origin-metrics-deployer:latest*, set version to *latest*.

|`*IMAGE_PREFIX*`
|Specify prefix for metrics components. For example, for *openshift/origin-metrics-deployer:latest*, set prefix to *openshift/origin-*.

|`*MODE*`
|Can be set to: *preflight* to perform validation before a deployment; *deploy* to perform an initial deployment; *refresh* to delete and redeploy all components but to keep persisted data and routes; *redeploy* to delete and redeploy everything (losing all data in the process); *validate* to re-run validations after a deployment.

|`*IGNORE_PREFLIGHT*`
|Can be set to *true* to disable the preflight checks. This will allow the deployer to continue even if the preflight check has failed.

|`*USER_WRITE_ACCESS*`
|If user accounts should be able to write metrics. Defaults to 'false' so that only Heapster can write metrics and not individual users. It is recommended to disable user write access, if enabled any user will be able to write metrics to the system which can affect performance and can cause Cassandra disk usage to unpredictably increase.

|===

The only required parameter is `*HAWKULAR_METRICS_HOSTNAME*`. This value is
required when creating the deployer because it specifies the hostname for the
Hawkular Metrics xref:../architecture/core_concepts/routes.adoc#architecture-core-concepts-routes[route]. This
value should correspond to a fully qualified domain name. You will need to know
the value of `*HAWKULAR_METRICS_HOSTNAME*` when
xref:../install_config/cluster_metrics.adoc#configuring-openshift-metrics[configuring
the console] for metrics access.

If you are using
xref:../install_config/cluster_metrics.adoc#metrics-persistent-storage[persistent
storage] with Cassandra, it is the administrator's responsibility to set a
sufficient disk size for the cluster using the `*CASSANDRA_PV_SIZE*` parameter.
It is also the administrator's responsibility to monitor disk usage to make sure
that it does not become full.

[WARNING]
====
Data loss will result if the Cassandra persisted volume runs out of sufficient space.
====

All of the other parameters are optional and allow for greater customization.
For instance, if you have a custom install in which the Kubernetes master is not
available under *_https://kubernetes.default.svc:443_* you can specify the value
to use instead with the `*MASTER_URL*` parameter. To deploy a specific version
of the metrics components, use the `*IMAGE_VERSION*` parameter.

[WARNING]
====
It is highly recommended to not use *latest* for the *IMAGE_VERSION*. The *latest*
version corresponds to the very latest version available and can cause issues if it brings in a
newer version not meant to function on the version of {product-title} you are currently running.
====

[[deploying-the-metrics-components]]
== Deploying the Metric Components

Because deploying and configuring all the metric components is handled by the
Metrics Deployer, you can simply deploy everything in one step.

The following examples show you how to deploy metrics with and without
persistent storage using the default template parameters. Optionally, you can
specify any of the
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[template
parameters] when calling these commands.

.Deploying with Persistent Storage
====
The following command sets the Hawkular Metrics route to use
*hawkular-metrics.example.com* and is deployed using persistent storage.

You must have a persistent volume of sufficient size available.

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com
----
====

.Deploying without Persistent Storage
====
The following command sets the Hawkular Metrics route to use
*hawkular-metrics.example.com* and deploy without persistent storage.

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -p USE_PERSISTENT_STORAGE=false
----
====

[WARNING]
====
Because this is being deployed without persistent storage, metric data loss
can occur.
====

[[metrics-deployer-validations]]
=== Metrics Deployer Validations

The metrics deployer does some validation both before and after deployment. If
the pre-flight validation fails, the environment for deployment is considered
unsuitable and the deployment is aborted. However, you can add
`*IGNORE_PREFLIGHT=true*` to the deployer parameters if you believe the
validation has erred.

If post-deployment validation fails, the deployer finishes in an *Error* state,
which indicates that you should check the deployer logs for issues that may
require addressing.  For example, the validation may detect that the external
*hawkular-metrics* route is not actually in use, because the route was already
created somewhere else. The validation output at the end of a deployment should
explain as clearly as possible any issues it finds and what you can do to
address them.

Once you have addressed deployment validation issues, you can re-run just the
validation by running the deployer again with the `*MODE=validate*` parameter
added, for example:

----
$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -p MODE=validate
----

There is also a diagnostic for metrics:

----
$ oadm diagnostics MetricsApiProxy
----

[[configuring-openshift-metrics]]
== Configuring {product-title}

The {product-title} web console uses the data coming from the Hawkular Metrics
service to display its graphs. The URL for accessing the Hawkular Metrics
service must be configured via the `*metricsPublicURL*` option in the
xref:../install_config/master_node_configuration.adoc#master-configuration-files[master
configuration file] (*_/etc/origin/master/master-config.yaml_*). This URL
corresponds to the route created with the `*HAWKULAR_METRICS_HOSTNAME*` template
parameter during the
xref:../install_config/cluster_metrics.adoc#deploying-the-metrics-components[deployment]
of the metrics components.

[NOTE]
====
You must be able to resolve the `*HAWKULAR_METRICS_HOSTNAME*` from the browser
accessing the console.
====

For example, if your `*HAWKULAR_METRICS_HOSTNAME*` corresponds to
`hawkular-metrics.example.com`, then you must make the following change in the
*_master-config.yaml_* file:

====
[source,yaml,]
----
  assetConfig:
    ...
    metricsPublicURL: "https://hawkular-metrics.example.com/hawkular/metrics"
----
====

Once you have updated and saved the *_master-config.yaml_* file, you must
restart your {product-title} instance.

When your {product-title} server is back up and running, metrics will be
displayed on the pod overview pages.

[CAUTION]
====
If you are using self-signed certificates, remember that the Hawkular Metrics
service is hosted under a different host name and uses different certificates
than the console. You may need to explicitly open a browser tab to the value
specified in `*metricsPublicURL*` and accept that certificate.

To avoid this issue, use certificates which are configured to be acceptable by
your browser.
====

[[cluster-metrics-accessing-hawkular-metrics-directly]]
== Accessing Hawkular Metrics Directly

To access and manage metrics more directly, use the Hawkular Metrics API.

[NOTE]
====
When accessing Hawkular Metrics via the API, you will only be able to perform
reads. Writing metrics has been disabled by default. If you want for individual
users to also be able to write metrics, you will need to set the
`*USER_WRITE_ACCESS*`
xref:../install_config/cluster_metrics.adoc#deployer-template-parameters[deployer template parameter]
to *true*.

However, it is recommended to use the default configuration and only have
metrics enter the system via Heapster. If write access is enabled, any user
will be able to write metrics to the system, which can affect performance and
cause Cassandra disk usage to unpredictably increase.
====

The link:http://www.hawkular.org/docs/rest/rest-metrics.html[Hawkular Metrics
documentation] covers how to use the API, but there are a few differences when
dealing with the version of Hawkular Metrics configured for use on
{product-title}:

[[cluster-metrics-openshift-projects-and-hawkular-tenants]]
=== {product-title} Projects and Hawkular Tenants

Hawkular Metrics is a multi-tenanted application. It is configured so that a
project in {product-title} corresponds to a tenant in Hawkular Metrics.

As such, when accessing metrics for a project named *MyProject* you will need
to set the
link:http://www.hawkular.org/docs/rest/rest-metrics.html#_tenant_header[*Hawkular-Tenant*]
header to *MyProject*.

There is also a special tenant named *_system* which contains system level
metrics. This will require either a *cluster-reader* or *cluster-admin* level
privileges to access.

=== Authorization

The Hawkular Metrics service will authenticate the user against {product-title}
to determine if the user has access to the project it is trying to access.

Hawkular Metrics accepts a bearer token from the client and verifies that token
with the {product-title} server using a *SubjectAccessReview*. If the user has
proper read privileges for the project, they are allowed to read the metrics
for that project. For the *_system* tenant, the user requesting to read from
this tenant must have *cluster-reader* permission.

When accessing the Hawkular Metrics API, you will need to pass a bearer token in
the *Authorization* header.

ifdef::openshift-origin[]
== Accessing Heapster Directly

Heapster has been configured to be only accessible via the
xref:../rest_api/kubernetes_v1.adoc#proxy-get-requests-to-service[API proxy].
Accessing it will required either a cluster-reader or cluster-admin privileges.

For example, to access the Heapster *validate* page, you need to access it
using something similar to:

----
$ curl -H "Authorization: Bearer XXX<KEY>" \
       -X GET https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/validate
----

For more information about Heapster and how to access its APIs, please refer the
link:https://github.com/kubernetes/heapster/[Heapster] project.

endif::[]

[[metrics-cleanup]]
== Cleanup

You can remove everything deloyed by the metrics deployer by performing the
following steps:

----
$ oc delete all,sa,templates,secrets,pvc --selector="metrics-infra"
----

To remove the deployer components, perform the following steps:

----
$ oc delete sa,secret metrics-deployer
----

{'fraction_non_alphanumeric': 0.08249860611770135, 'fraction_numerical': 0.0013265914290658105, 'mean_word_length': 5.067187682258252, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 4, 'https://': 16, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1767825', 'n_tokens_mistral': 14989, 'n_tokens_neox': 14237, 'n_words': 5888}