<commit_before>////
/**
* @@@ START COPYRIGHT @@@
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements.  See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership.  The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License.  You may obtain a copy of the License at
*
*   http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied.  See the License for the
* specific language governing permissions and limitations
* under the License.
*
* @@@ END COPYRIGHT @@@
*/
////

[[introduction]]
= Introduction

{project-name} is a Hadoop add-on service that provides transactional SQL on top of HBase. Typically, you
use {project-name} as the database for applications that require Online Transaction Processing (OLTP),
Operational Data Store (ODS), and/or strong reporting capabilities. You access {project-name} using
standard JDBC and ODBC APIs.

You may choose whether to add {project-name} to an existing Hadoop environment or to create a standalone
Hadoop environment specifically for Hadoop.

This guide assumes that a Hadoop environment exists upon which your provisioning {project-name}. Refer to
<<requirements-hadoop-software,Hadoop Software>> for information about what Hadoop software is required
{project-name}.

[[introduction-security-considerations]]
== Security Considerations

The following users and principals need be considered for {project-name}:

* *Provisioning User*: A Linux-level user that performs the {project-name} provisioning tasks. This user ID
requires `sudo` access and passwordless ssh among the nodes where {project-name} is installed. In addition,
this user ID requires access to Hadoop distribution, HDFS, and HBase administrative users to change
respective environment's configuration settings per {project-name} requirements. Refer to
<<requirements-trafodion-provisioning-user,{project-name} Provisioning User>> for more information
about the requirements and usage associated with this user ID.

* *Runtime User*: A Linux-level user under which the {project-name} software runs, default name is `trafodion`. This user ID must be registered
as a user in the Hadoop Distributed File System (HDFS) to store and access objects in HDFS, HBase, and Hive.
In addition, this user ID requires passwordless access among the nodes where {project-name} is installed.
Refer to <<requirements-trafodion-runtime-user,{project-name} Runtime User>> for more information about this user ID.

* *{project-name} Database Users*: {project-name} users are managed by {project-name} security features (grant, revoke, etc.),
which can be integrated with LDAP if so desired. These users are referred to as *database users* and
do not have direct access to the operating system. Refer to <<enable-security-ldap,LDAP>> for 
details on enabling LDAP for authenticating database users. 
Refer to {docs-url}/sql_reference/index.html#register_user_statement[Register User],
{docs-url}/sql_reference/index.html#grant_statement[Grant], and other SQL statements
in the {docs-url}/sql_reference/index.html[{project-name} SQL Reference Manual] for
more information about managing {project-name} Database Users.
 +
 +
If your environment has been provisioned with Kerberos, then the following additional information is required. 

* *KDC admin principal*: {project-name} requires administrator access to Kerberos to create principals 
and keytabs for the `trafodion` user, and to look-up principal names for HDFS and HBase keytabs.  Refer to 
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this principal.

* *HBase keytab location*: {project-name} requires administrator access to HBase to grant required privileges to the `trafodion` user.  Refer to
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this keytab.

* *HDFS keytab location*: {project-name} requires administrator access to HDFS to create directories that store files needed to perform SQL requests 
such as data loads and backups.  Refer to
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this keytab.
 +
 +
If your environment is using LDAP for authentication, then the following additional information is required.

* *LDAP username for database root access*:  When {project-name} is installed, it creates a predefined database user referred to as the DB\__ROOT user.  
In order to connect to the database as database root, there must be a mapping between the database user DB__ROOT and an LDAP user. Refer to  
<<enable-security-ldap,LDAP>> for more information about this option.

* *LDAP search user name*: {project-name} optionally requests an LDAP username and password in order to perform LDAP operations 
such as LDAP search.  Refer to
<<enable-security-ldap,LDAP>> for more information about this option.

[[introduction-provisioning-options]]
== Provisioning Options

{project-name} includes two options for installation: a plug-in integration with Apache Ambari and command-line installation scripts.

The Ambari integration provides support for Hortonworks Hadoop distributions, while the command-line {project-name} Installer
supports Cloudera and  Hortonworks Hadoop distributions, and for select vanilla Hadoop installations.

The {project-name} Installer supports Linux distributions SUSE and RedHat/CentOS. There are, however, some differences.
Prerequisite software packages are not installed automatically on SUSE.

The {project-name} Installer automates many of the tasks required to install/upgrade {project-name}, from downloading and
installing required software packages and making required configuration changes to your Hadoop environment via creating
the {project-name} runtime user ID to installing and starting {project-name}. It is, therefore,  highly recommend that
you use the {project-name} Installer for initial installation and upgrades of {project-name}. These steps are referred to as
"Script-Based Provisioning" in this guide. Refer to <<introduction-trafodion-installer, {project-name} Installer>> that provides
usage information.

The command-line installer has been replaced for the 2.1.0 release. Written in python, it replaces the legacy bash-script installer.
The bash command-line installer is deprecated as of 2.1.0, but is still provided, just in case you experience any problems with
the new installer. If so, please report those problems to the project team, since the legacy installer will soon be obsoleted.

[[introduction-provisioning-activities]]
== Provisioning Activities

{project-name} provisioning is divided into the following main activities:

* *<<requirements,Requirements>>*: Activities and documentation required to install the {project-name} software.
These activities include tasks such as understanding hardware and operating system requirements,
Hadoop requirements, what software packages that need to be downloaded, configuration settings that need to be changed,
and user ID requirements.

* *<<prepare,Prepare>>*: Activities to prepare the operating system and the Hadoop ecosystem to run
{project-name}. These activities include tasks such as installing required software packages, configure
the {project-name} Installation User, gather information about the Hadoop environment, and the modify configuration
for different Hadoop services.

* *<<install,Install>>*: Activities related to installing the {project-name} software. These activities
include tasks such as unpacking the {project-name} tar files, creating the {project-name} Runtime User,
creating {project-name} HDFS directories, installing the {project-name} software, and enabling security features.

* *<<upgrade,Upgrade>>*: Activities related to the upgrading the {project-name} software. These activities
include tasks such as shutting down {project-name} and installing a new version of the {project-name} software.
The upgrade tasks vary depending on the differences between the current and new release of
{project-name}. For example, an upgrade may or may not include an upgrade of the {project-name} metadata.

* *<<activate,Activate>>*: Activities related to starting the {project-name} software. These actives
include basic management tasks such as starting and checking the status of the {project-name} components and performing basic smoke tests.

* *<<remove,Remove>>*: Activities related to removing {project-name} from your Hadoop cluster.

* *<<enable-security,Enable Security>>*: Activities related to enabling security features on an already installed
{project-name} installation.  These activities include tasks such as adding Kerberos principals and keytabs,
and setting up the LDAP configuration files. *Only support in bash installer for now*

[[introduction-provisioning-master-node]]
== Provisioning Master Node
All provisioning tasks are performed from a single node in the cluster, which can be any node
as long as it has access to the Hadoop environment you're adding {project-name} to.
This node is referred to as the "*Provisioning Master Node*" in this guide.

The {project-name} Provisioning User must have access to all other nodes from the Provisioning
Master Node in order to perform provisioning tasks on the cluster.

[[introduction-trafodion-installer]]
== {project-name} Installer

The {project-name} Installer is a set of scripts automates most of the tasks requires to install/upgrade {project-name}.
You download the {project-name} Installer tar file from the {project-name} {download-url}[download] page.
Next, you unpack the tar file.

*Example*
<commit_msg>[TRAFODION-2644] Also remove doc references to bash installer<commit_after>////
/**
* @@@ START COPYRIGHT @@@
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements.  See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership.  The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License.  You may obtain a copy of the License at
*
*   http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied.  See the License for the
* specific language governing permissions and limitations
* under the License.
*
* @@@ END COPYRIGHT @@@
*/
////

[[introduction]]
= Introduction

{project-name} is a Hadoop add-on service that provides transactional SQL on top of HBase. Typically, you
use {project-name} as the database for applications that require Online Transaction Processing (OLTP),
Operational Data Store (ODS), and/or strong reporting capabilities. You access {project-name} using
standard JDBC and ODBC APIs.

You may choose whether to add {project-name} to an existing Hadoop environment or to create a standalone
Hadoop environment specifically for Hadoop.

This guide assumes that a Hadoop environment exists upon which your provisioning {project-name}. Refer to
<<requirements-hadoop-software,Hadoop Software>> for information about what Hadoop software is required
{project-name}.

[[introduction-security-considerations]]
== Security Considerations

The following users and principals need be considered for {project-name}:

* *Provisioning User*: A Linux-level user that performs the {project-name} provisioning tasks. This user ID
requires `sudo` access and passwordless ssh among the nodes where {project-name} is installed. In addition,
this user ID requires access to Hadoop distribution, HDFS, and HBase administrative users to change
respective environment's configuration settings per {project-name} requirements. Refer to
<<requirements-trafodion-provisioning-user,{project-name} Provisioning User>> for more information
about the requirements and usage associated with this user ID.

* *Runtime User*: A Linux-level user under which the {project-name} software runs, default name is `trafodion`. This user ID must be registered
as a user in the Hadoop Distributed File System (HDFS) to store and access objects in HDFS, HBase, and Hive.
In addition, this user ID requires passwordless access among the nodes where {project-name} is installed.
Refer to <<requirements-trafodion-runtime-user,{project-name} Runtime User>> for more information about this user ID.

* *{project-name} Database Users*: {project-name} users are managed by {project-name} security features (grant, revoke, etc.),
which can be integrated with LDAP if so desired. These users are referred to as *database users* and
do not have direct access to the operating system. Refer to <<enable-security-ldap,LDAP>> for 
details on enabling LDAP for authenticating database users. 
Refer to {docs-url}/sql_reference/index.html#register_user_statement[Register User],
{docs-url}/sql_reference/index.html#grant_statement[Grant], and other SQL statements
in the {docs-url}/sql_reference/index.html[{project-name} SQL Reference Manual] for
more information about managing {project-name} Database Users.
 +
 +
If your environment has been provisioned with Kerberos, then the following additional information is required. 

* *KDC admin principal*: {project-name} requires administrator access to Kerberos to create principals 
and keytabs for the `trafodion` user, and to look-up principal names for HDFS and HBase keytabs.  Refer to 
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this principal.

* *HBase keytab location*: {project-name} requires administrator access to HBase to grant required privileges to the `trafodion` user.  Refer to
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this keytab.

* *HDFS keytab location*: {project-name} requires administrator access to HDFS to create directories that store files needed to perform SQL requests 
such as data loads and backups.  Refer to
<<enable-security-kerberos,Kerberos>> for more information about the requirements and usage associated with this keytab.
 +
 +
If your environment is using LDAP for authentication, then the following additional information is required.

* *LDAP username for database root access*:  When {project-name} is installed, it creates a predefined database user referred to as the DB\__ROOT user.  
In order to connect to the database as database root, there must be a mapping between the database user DB__ROOT and an LDAP user. Refer to  
<<enable-security-ldap,LDAP>> for more information about this option.

* *LDAP search user name*: {project-name} optionally requests an LDAP username and password in order to perform LDAP operations 
such as LDAP search.  Refer to
<<enable-security-ldap,LDAP>> for more information about this option.

[[introduction-provisioning-options]]
== Provisioning Options

{project-name} includes two options for installation: a plug-in integration with Apache Ambari and command-line installation scripts.

The Ambari integration provides support for Hortonworks Hadoop distributions, while the command-line {project-name} Installer
supports Cloudera and  Hortonworks Hadoop distributions, and for select vanilla Hadoop installations.

The {project-name} Installer supports Linux distributions SUSE and RedHat/CentOS. There are, however, some differences.
Prerequisite software packages are not installed automatically on SUSE.

The {project-name} Installer automates many of the tasks required to install/upgrade {project-name}, from downloading and
installing required software packages and making required configuration changes to your Hadoop environment via creating
the {project-name} runtime user ID to installing and starting {project-name}. It is, therefore,  highly recommend that
you use the {project-name} Installer for initial installation and upgrades of {project-name}. These steps are referred to as
"Script-Based Provisioning" in this guide. Refer to <<introduction-trafodion-installer, {project-name} Installer>> that provides
usage information.


[[introduction-provisioning-activities]]
== Provisioning Activities

{project-name} provisioning is divided into the following main activities:

* *<<requirements,Requirements>>*: Activities and documentation required to install the {project-name} software.
These activities include tasks such as understanding hardware and operating system requirements,
Hadoop requirements, what software packages that need to be downloaded, configuration settings that need to be changed,
and user ID requirements.

* *<<prepare,Prepare>>*: Activities to prepare the operating system and the Hadoop ecosystem to run
{project-name}. These activities include tasks such as installing required software packages, configure
the {project-name} Installation User, gather information about the Hadoop environment, and the modify configuration
for different Hadoop services.

* *<<install,Install>>*: Activities related to installing the {project-name} software. These activities
include tasks such as unpacking the {project-name} tar files, creating the {project-name} Runtime User,
creating {project-name} HDFS directories, installing the {project-name} software, and enabling security features.

* *<<upgrade,Upgrade>>*: Activities related to the upgrading the {project-name} software. These activities
include tasks such as shutting down {project-name} and installing a new version of the {project-name} software.
The upgrade tasks vary depending on the differences between the current and new release of
{project-name}. For example, an upgrade may or may not include an upgrade of the {project-name} metadata.

* *<<activate,Activate>>*: Activities related to starting the {project-name} software. These actives
include basic management tasks such as starting and checking the status of the {project-name} components and performing basic smoke tests.

* *<<remove,Remove>>*: Activities related to removing {project-name} from your Hadoop cluster.

[[introduction-provisioning-master-node]]
== Provisioning Master Node
All provisioning tasks are performed from a single node in the cluster, which can be any node
as long as it has access to the Hadoop environment you're adding {project-name} to.
This node is referred to as the "*Provisioning Master Node*" in this guide.

The {project-name} Provisioning User must have access to all other nodes from the Provisioning
Master Node in order to perform provisioning tasks on the cluster.

[[introduction-trafodion-installer]]
== {project-name} Installer

The {project-name} Installer is a set of scripts automates most of the tasks requires to install/upgrade {project-name}.
You download the {project-name} Installer tar file from the {project-name} {download-url}[download] page.
Next, you unpack the tar file.

*Example*

{'fraction_non_alphanumeric': 0.06756057154690412, 'fraction_numerical': 0.0009318699523710913, 'mean_word_length': 5.281951219512195, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '980924', 'n_tokens_mistral': 4819, 'n_tokens_neox': 4638, 'n_words': 2429}