<commit_before><?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:id="performance"
         xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xmlns:svg="http://www.w3.org/2000/svg"
         xmlns:m="http://www.w3.org/1998/Math/MathML"
         xmlns:html="http://www.w3.org/1999/xhtml"
         xmlns:db="http://docbook.org/ns/docbook">
  <title>Performance Tuning</title>

  <para>Start with the <link
  xlink:href="http://wiki.apache.org/hadoop/PerformanceTuning">wiki
  Performance Tuning</link> page. It has a general discussion of the main
  factors involved; RAM, compression, JVM settings, etc. Afterward, come back
  here for more pointers.</para>

  <note xml:id="rpc.logging"><title>Enabling RPC-level logging</title>
  <para>Enabling the RPC-level logging on a RegionServer can often given
      insight on timings at the server.  Once enabled, the amount of log
      spewed is voluminous.  It is not recommended that you leave this
      logging on for more than short bursts of time.  To enable RPC-level
      logging, browse to the RegionServer UI and click on 
      <emphasis>Log Level</emphasis>.  Set the log level to <varname>DEBUG</varname> for the package
      <classname>org.apache.hadoop.ipc</classname> (Thats right, for
      hadoop.ipc, NOT, hbase.ipc).  Then tail the RegionServers log.
      Analyze.</para>
  <para>To disable, set the logging level back to <varname>INFO</varname> level.
  </para>
  </note>

  <section xml:id="jvm">
    <title>Java</title>

    <section xml:id="gc">
      <title>The Garbage Collector and HBase</title>

      <section xml:id="gcpause">
        <title>Long GC pauses</title>

        <para>In his presentation, <link
        xlink:href="http://www.slideshare.net/cloudera/hbase-hug-presentation">Avoiding
        Full GCs with MemStore-Local Allocation Buffers</link>, <NAME>
        describes two cases of stop-the-world garbage collections common in
        HBase, especially during loading; CMS failure modes and old generation
        heap fragmentation brought. To address the first, start the CMS
        earlier than default by adding
        <code>-XX:CMSInitiatingOccupancyFraction</code> and setting it down
        from defaults. Start at 60 or 70 percent (The lower you bring down the
        threshold, the more GCing is done, the more CPU used). To address the
        second fragmentation issue, Todd added an experimental facility that
        must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in
        0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code>
        to true in your <classname>Configuration</classname>. See the cited
        slides for background and detail.</para>
      </section>
    </section>
  </section>

  <section xml:id="perf.configurations">
    <title>Configurations</title>

    <para>See <xref linkend="recommended_configurations" />.</para>

    <section xml:id="perf.number.of.regions">
      <title>Number of Regions</title>

      <para>The number of regions for an HBase table is driven by the <xref
              linkend="bigger.regions" />. Also, see the architecture
          section on <xref linkend="arch.regions.size" /></para>
    </section>

    <section xml:id="perf.compactions.and.splits">
      <title>Managing Compactions</title>

      <para>For larger systems, managing <link
      linkend="disable.splitting">compactions and splits</link> may be
      something you want to consider.</para>
    </section>

    <section xml:id="perf.compression">
      <title>Compression</title>
      <para>Production systems should use compression such as <xref linkend="lzo" /> compression with their column family
      definitions.</para>
    </section>

    <section xml:id="perf.handlers">
        <title><varname>hbase.regionserver.handler.count</varname></title>
        <para>This setting is in essence sets how many requests are
            concurrently being processed inside the RegionServer at any
            one time.  If set too high, then throughput may suffer as
            the concurrent requests contend; if set too low, requests will
            be stuck waiting to get into the machine.  You can get a
            sense of whether you have too little or too many handlers by
            <xref linkend="rpc.logging" />
            on an individual RegionServer then tailing its logs.</para>
    </section>

  </section>

  <section xml:id="perf.number.of.cfs">
    <title>Number of Column Families</title>

    <para>See <xref linkend="number.of.cfs" />.</para>
  </section>

  <section xml:id="perf.one.region">
    <title>Data Clumping</title>

    <para>If all your data is being written to one region, then re-read the
    section on processing <link linkend="timeseries">timeseries</link>
    data.</para>
  </section>

  <section xml:id="perf.batch.loading">
    <title>Batch Loading</title>
    <para>Use the bulk load tool if you can.  See
        <link xlink:href="http://hbase.apache.org/bulk-loads.html">Bulk Loads</link>.
        Otherwise, pay attention to the below.
    </para>

  <section xml:id="precreate.regions">
  <title>
  Table Creation: Pre-Creating Regions
  </title><commit_msg>HBASE-3825  performance.xml - adding a few common configuration changes in the 'config' sub-section<commit_after><?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:id="performance"
         xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xmlns:svg="http://www.w3.org/2000/svg"
         xmlns:m="http://www.w3.org/1998/Math/MathML"
         xmlns:html="http://www.w3.org/1999/xhtml"
         xmlns:db="http://docbook.org/ns/docbook">
  <title>Performance Tuning</title>

  <para>Start with the <link
  xlink:href="http://wiki.apache.org/hadoop/PerformanceTuning">wiki
  Performance Tuning</link> page. It has a general discussion of the main
  factors involved; RAM, compression, JVM settings, etc. Afterward, come back
  here for more pointers.</para>

  <note xml:id="rpc.logging"><title>Enabling RPC-level logging</title>
  <para>Enabling the RPC-level logging on a RegionServer can often given
      insight on timings at the server.  Once enabled, the amount of log
      spewed is voluminous.  It is not recommended that you leave this
      logging on for more than short bursts of time.  To enable RPC-level
      logging, browse to the RegionServer UI and click on 
      <emphasis>Log Level</emphasis>.  Set the log level to <varname>DEBUG</varname> for the package
      <classname>org.apache.hadoop.ipc</classname> (Thats right, for
      hadoop.ipc, NOT, hbase.ipc).  Then tail the RegionServers log.
      Analyze.</para>
  <para>To disable, set the logging level back to <varname>INFO</varname> level.
  </para>
  </note>

  <section xml:id="jvm">
    <title>Java</title>

    <section xml:id="gc">
      <title>The Garbage Collector and HBase</title>

      <section xml:id="gcpause">
        <title>Long GC pauses</title>

        <para>In his presentation, <link
        xlink:href="http://www.slideshare.net/cloudera/hbase-hug-presentation">Avoiding
        Full GCs with MemStore-Local Allocation Buffers</link>, <NAME>con
        describes two cases of stop-the-world garbage collections common in
        HBase, especially during loading; CMS failure modes and old generation
        heap fragmentation brought. To address the first, start the CMS
        earlier than default by adding
        <code>-XX:CMSInitiatingOccupancyFraction</code> and setting it down
        from defaults. Start at 60 or 70 percent (The lower you bring down the
        threshold, the more GCing is done, the more CPU used). To address the
        second fragmentation issue, Todd added an experimental facility that
        must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in
        0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code>
        to true in your <classname>Configuration</classname>. See the cited
        slides for background and detail.</para>
      </section>
    </section>
  </section>

  <section xml:id="perf.configurations">
    <title>Configurations</title>

    <para>See <xref linkend="recommended_configurations" />.</para>

    <section xml:id="perf.number.of.regions">
      <title>Number of Regions</title>

      <para>The number of regions for an HBase table is driven by the <xref
              linkend="bigger.regions" />. Also, see the architecture
          section on <xref linkend="arch.regions.size" /></para>
    </section>

    <section xml:id="perf.compactions.and.splits">
      <title>Managing Compactions</title>

      <para>For larger systems, managing <link
      linkend="disable.splitting">compactions and splits</link> may be
      something you want to consider.</para>
    </section>

    <section xml:id="perf.compression">
      <title>Compression</title>
      <para>Production systems should use compression such as <xref linkend="lzo" /> compression with their column family
      definitions.</para>
    </section>

    <section xml:id="perf.handlers">
        <title><varname>hbase.regionserver.handler.count</varname></title>
        <para>See <xref linkend="hbase.regionserver.handler.count"/>. 
            This setting in essence sets how many requests are
            concurrently being processed inside the RegionServer at any
            one time.  If set too high, then throughput may suffer as
            the concurrent requests contend; if set too low, requests will
            be stuck waiting to get into the machine.  You can get a
            sense of whether you have too little or too many handlers by
            <xref linkend="rpc.logging" />
            on an individual RegionServer then tailing its logs (Queued requests
            consume memory).</para>
    </section>
    <section xml:id="perf.hfile.block.cache.size">
        <title><varname>hfile.block.cache.size</varname></title>
        <para>See <xref linkend="hfile.block.cache.size"/>. 
        A memory setting for the RegionServer process.
        </para>
    </section>    
    <section xml:id="perf.rs.memstore.upperlimit">
        <title><varname>hbase.regionserver.global.memstore.upperLimit</varname></title>
        <para>See <xref linkend="hbase.regionserver.global.memstore.upperLimit"/>.  
        This memory setting is often adjusted for the RegionServer process depending on needs.
        </para>
    </section>    
    <section xml:id="perf.rs.memstore.lowerlimit">
        <title><varname>hbase.regionserver.global.memstore.lowerLimit</varname></title>
        <para>See <xref linkend="hbase.regionserver.global.memstore.lowerLimit"/>.  
        This memory setting is often adjusted for the RegionServer process depending on needs.
        </para>
    </section>
    <section xml:id="perf.hstore.blockingstorefiles">
        <title><varname>hbase.hstore.blockingStoreFiles</varname></title>
        <para>See <xref linkend="hbase.hstore.blockingStoreFiles"/>.  
        If there is blocking in the RegionServer logs, increasing this can help.
        </para>
    </section>
    <section xml:id="perf.hregion.memstore.block.multiplier">
        <title><varname>hbase.hregion.memstore.block.multiplier</varname></title>
        <para>See <xref linkend="hbase.hregion.memstore.block.multiplier"/>.  
        If there is enough RAM, increasing this can help.  
        </para>
    </section>

  </section>

  <section xml:id="perf.number.of.cfs">
    <title>Number of Column Families</title>

    <para>See <xref linkend="number.of.cfs" />.</para>
  </section>

  <section xml:id="perf.one.region">
    <title>Data Clumping</title>

    <para>If all your data is being written to one region, then re-read the
    section on processing <link linkend="timeseries">timeseries</link>
    data.</para>
  </section>

  <section xml:id="perf.batch.loading">
    <title>Batch Loading</title>
    <para>Use the bulk load tool if you can.  See
        <link xlink:href="http://hbase.apache.org/bulk-loads.html">Bulk Loads</link>.
        Otherwise, pay attention to the below.
    </para>

  <section xml:id="precreate.regions">
  <title>
  Table Creation: Pre-Creating Regions
  </title>
{'fraction_non_alphanumeric': 0.11527284468840814, 'fraction_numerical': 0.006780755569906361, 'mean_word_length': 3.027633289986996, 'pattern_counts': {'":': 0, '<': 271, '<?xml version=': 2, '>': 271, 'https://': 0, 'lorem ipsum': 0, 'www.': 12, 'xml': 50}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7361013', 'n_tokens_mistral': 3774, 'n_tokens_neox': 3616, 'n_words': 1096}