<commit_before>// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

== Administration

=== Hardware

Because we are running essentially two or three systems simultaneously layered
across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
at least one core and 2 - 4 GB each.

One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.

It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
each, but in this case it is recommended to only run up to two processes per
machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
not all three. The constraint here is having enough available heap space for all the
processes on a machine.

=== Network

Accumulo communicates via remote procedure calls over TCP/IP for both passing
data and control messages. In addition, Accumulo uses HDFS clients to
communicate with HDFS. To achieve good ingest and query performance, sufficient
network bandwidth must be available between any two machines.

In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
use the following default ports. Please make sure that they are open, or change
their value in conf/accumulo-site.xml.

.Accumulo default ports
[width="75%",cols=">,^2,^2"]
[options="header"]
|====
|Port | Description | Property Name
|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
|9997 | Tablet Server | tserver.port.client
|9999 | Master Server | master.port.client
|12234 | Accumulo Tracer | trace.port.client
|42424 | Accumulo Proxy Server | n/a
|50091 | Accumulo GC | gc.port.client
|50095 | Accumulo HTTP monitor | monitor.port.client
|10001 | Master Replication service | master.replication.coordinator.port
|10002 | TabletServer Replication service | replication.receipt.service.port
|====

In addition, the user can provide +0+ and an ephemeral port will be chosen instead. This
ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
use +0+ instead of an explicit value, should, in most cases, work around any issues of
running multiple distinct Accumulo instances (or any other process which tries to use the
same default ports) on the same hardware.

=== Installation
Choose a directory for the Accumulo installation. This directory will be referenced
by the environment variable +$ACCUMULO_HOME+. Run the following:

  $ tar xzf accumulo-1.6.0-bin.tar.gz    # unpack to subdirectory
  $ mv accumulo-1.6.0 $ACCUMULO_HOME # move to desired location

Repeat this step at each machine within the cluster. Usually all machines have the
same +$ACCUMULO_HOME+.

=== Dependencies
Accumulo requires HDFS and ZooKeeper to be configured and running
before starting. Password-less SSH should be configured between at least the
Accumulo master and TabletServer machines. It is also a good idea to run Network
Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
sync, which can cause problems with automatically timestamped data.

=== Configuration

Accumulo is configured by editing several Shell and XML files found in
+$ACCUMULO_HOME/conf+. The structure closely resembles Hadoop's configuration
files.

==== Edit conf/accumulo-env.sh<commit_msg>ACCUMULO-2954 Add docs about logging configuration files<commit_after>// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

== Administration

=== Hardware

Because we are running essentially two or three systems simultaneously layered
across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to
consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have
at least one core and 2 - 4 GB each.

One core running HDFS can typically keep 2 to 4 disks busy, so each machine may
typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.

It is possible to do with less than this, such as with 1u servers with 2 cores and 4GB
each, but in this case it is recommended to only run up to two processes per
machine -- i.e. DataNode and TabletServer or DataNode and MapReduce worker but
not all three. The constraint here is having enough available heap space for all the
processes on a machine.

=== Network

Accumulo communicates via remote procedure calls over TCP/IP for both passing
data and control messages. In addition, Accumulo uses HDFS clients to
communicate with HDFS. To achieve good ingest and query performance, sufficient
network bandwidth must be available between any two machines.

In addition to needing access to ports associated with HDFS and ZooKeeper, Accumulo will
use the following default ports. Please make sure that they are open, or change
their value in conf/accumulo-site.xml.

.Accumulo default ports
[width="75%",cols=">,^2,^2"]
[options="header"]
|====
|Port | Description | Property Name
|4445 | Shutdown Port (Accumulo MiniCluster) | n/a
|4560 | Accumulo monitor (for centralized log display) | monitor.port.log4j
|9997 | Tablet Server | tserver.port.client
|9999 | Master Server | master.port.client
|12234 | Accumulo Tracer | trace.port.client
|42424 | Accumulo Proxy Server | n/a
|50091 | Accumulo GC | gc.port.client
|50095 | Accumulo HTTP monitor | monitor.port.client
|10001 | Master Replication service | master.replication.coordinator.port
|10002 | TabletServer Replication service | replication.receipt.service.port
|====

In addition, the user can provide +0+ and an ephemeral port will be chosen instead. This
ephemeral port is likely to be unique and not already bound. Thus, configuring ports to
use +0+ instead of an explicit value, should, in most cases, work around any issues of
running multiple distinct Accumulo instances (or any other process which tries to use the
same default ports) on the same hardware.

=== Installation
Choose a directory for the Accumulo installation. This directory will be referenced
by the environment variable +$ACCUMULO_HOME+. Run the following:

  $ tar xzf accumulo-1.6.0-bin.tar.gz    # unpack to subdirectory
  $ mv accumulo-1.6.0 $ACCUMULO_HOME # move to desired location

Repeat this step at each machine within the cluster. Usually all machines have the
same +$ACCUMULO_HOME+.

=== Dependencies
Accumulo requires HDFS and ZooKeeper to be configured and running
before starting. Password-less SSH should be configured between at least the
Accumulo master and TabletServer machines. It is also a good idea to run Network
Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of
sync, which can cause problems with automatically timestamped data.

=== Configuration

Accumulo is configured by editing several Shell and XML files found in
+$ACCUMULO_HOME/conf+. The structure closely resembles Hadoop's configuration
files.

Logging is primarily controlled using the log4j configuration files,
+generic_logger.xml+ and +monitor_logger.xml+ (or their corresponding
+.properties+ version if the +.xml+ version is missing). The generic logger is
used for most server types, and is typically configured to send logs to the
monitor, as well as log files. The monitor logger is used by the monitor, and
is typically configured to log only errors the monitor itself generates,
rather than all the logs that it receives from other server types.

==== Edit conf/accumulo-env.sh
{'fraction_non_alphanumeric': 0.059365290723479805, 'fraction_numerical': 0.01875277407900577, 'mean_word_length': 4.38089552238806, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4223716', 'n_tokens_mistral': 2500, 'n_tokens_neox': 2306, 'n_words': 1290}