<commit_before>=================
Iceberg connector
=================

.. raw:: html

  <img src="../_static/img/iceberg.png" class="connector-logo">

Apache Iceberg is an open table format for huge analytic datasets.
The Iceberg connector allows querying data stored in
files written in Iceberg format, as defined in the
`Iceberg Table Spec <https://iceberg.apache.org/spec/>`_. It supports Apache
Iceberg table spec version 1 and 2.

The Iceberg table state is maintained in metadata files. All changes to table state
create a new metadata file and replace the old metadata with an atomic swap.
The table metadata file tracks the table schema, partitioning config,
custom properties, and snapshots of the table contents.

Iceberg data files can be stored in either Parquet or ORC format, as
determined by the ``format`` property in the table definition.  The
table ``format`` defaults to ``ORC``.

Iceberg is designed to improve on the known scalability limitations of Hive, which stores
table metadata in a metastore that is backed by a relational database such as MySQL.  It tracks
partition locations in the metastore, but not individual data files.  Trino queries
using the :doc:`/connector/hive` must first call the metastore to get partition locations,
then call the underlying filesystem to list all data files inside each partition,
and then read metadata from each data file.

Since Iceberg stores the paths to data files in the metadata files, it
only consults the underlying file system for files that must be read.

Requirements
------------

To use Iceberg, you need:

* Network access from the Trino coordinator and workers to the distributed
  object storage.
* Access to a Hive metastore service (HMS) or AWS Glue.
* Network access from the Trino coordinator to the HMS. Hive
  metastore access with the Thrift protocol defaults to using port 9083.

Configuration
-------------

The connector supports two Iceberg catalog types, you may use either a Hive
metastore service (HMS) or AWS Glue. The catalog type is determined by the
``iceberg.catalog.type`` property, it can be set to either ``HIVE_METASTORE``
or ``GLUE``.

Hive metastore catalog
^^^^^^^^^^^^^^^^^^^^^^

The Hive metastore catalog is the default implementation.
When using it, the Iceberg connector supports the same metastore
configuration properties as the Hive connector. At a minimum,
``hive.metastore.uri`` must be configured, see
:ref:`Thrift metastore configuration<hive-thrift-metastore>`.

.. code-block:: text

    connector.name=iceberg
    hive.metastore.uri=thrift://localhost:9083

Glue catalog
^^^^^^^^^^^^

When using the Glue catalog, the Iceberg connector supports the same
configuration properties as the Hive connector's Glue setup. See
:ref:`AWS Glue metastore configuration<hive-glue-metastore>`.

.. code-block:: text

    connector.name=iceberg
    iceberg.catalog.type=glue


General configuration
^^^^^^^^^^^^^^^^^^^^^

These configuration properties are independent of which catalog implementation
is used.

.. list-table:: Iceberg general configuration properties
  :widths: 30, 58, 12
  :header-rows: 1

  * - Property name
    - Description
    - Default
  * - ``iceberg.file-format``
    - Define the data storage file format for Iceberg tables.
      Possible values are

      * ``PARQUET``
      * ``ORC``
    - ``ORC``
  * - ``iceberg.compression-codec``
    - The compression codec to be used when writing files.
      Possible values are

      * ``NONE``
      * ``SNAPPY``
      * ``LZ4``
      * ``ZSTD``
      * ``GZIP``
    - ``ZSTD``
  * - ``iceberg.max-partitions-per-writer``
    - Maximum number of partitions handled per writer.
    - 100
  * - ``iceberg.target-max-file-size``
    - Target maximum size of written files; the actual size may be larger
    - ``1GB``
  * - ``iceberg.unique-table-location``
    - Use randomized, unique table locations
    - ``false``
  * - ``iceberg.delete-schema-locations-fallback``
    - Whether schema locations should be deleted when Trino can't determine whether they contain external files.
    - ``false``
  * - ``iceberg.minimum-assigned-split-weight``
    - A decimal value in the range (0, 1] used as a minimum for weights assigned to each split. A low value may improve performance
      on tables with small files. A higher value may improve performance for queries with highly skewed aggregations or joins.
    - 0.05
  * - ``iceberg.table-statistics-enabled``
    - Enables :doc:`/optimizer/statistics`. The equivalent
      :doc:`catalog session property </sql/set-session>`
      is ``statistics_enabled`` for session specific use.
      Set to ``false`` to disable statistics. Disabling statistics
      means that :doc:`/optimizer/cost-based-optimizations` can
      not make smart decisions about the query plan.
    - ``true``

ORC format configuration
^^^^^^^^^^^^^^^^^^^^^^^^

The following properties are used to configure the read and write operations
with ORC files performed by the Iceberg connector.
<commit_msg>Add docs for missing Iceberg properties<commit_after>=================
Iceberg connector
=================

.. raw:: html

  <img src="../_static/img/iceberg.png" class="connector-logo">

Apache Iceberg is an open table format for huge analytic datasets.
The Iceberg connector allows querying data stored in
files written in Iceberg format, as defined in the
`Iceberg Table Spec <https://iceberg.apache.org/spec/>`_. It supports Apache
Iceberg table spec version 1 and 2.

The Iceberg table state is maintained in metadata files. All changes to table state
create a new metadata file and replace the old metadata with an atomic swap.
The table metadata file tracks the table schema, partitioning config,
custom properties, and snapshots of the table contents.

Iceberg data files can be stored in either Parquet or ORC format, as
determined by the ``format`` property in the table definition.  The
table ``format`` defaults to ``ORC``.

Iceberg is designed to improve on the known scalability limitations of Hive, which stores
table metadata in a metastore that is backed by a relational database such as MySQL.  It tracks
partition locations in the metastore, but not individual data files.  Trino queries
using the :doc:`/connector/hive` must first call the metastore to get partition locations,
then call the underlying filesystem to list all data files inside each partition,
and then read metadata from each data file.

Since Iceberg stores the paths to data files in the metadata files, it
only consults the underlying file system for files that must be read.

Requirements
------------

To use Iceberg, you need:

* Network access from the Trino coordinator and workers to the distributed
  object storage.
* Access to a Hive metastore service (HMS) or AWS Glue.
* Network access from the Trino coordinator to the HMS. Hive
  metastore access with the Thrift protocol defaults to using port 9083.

Configuration
-------------

The connector supports two Iceberg catalog types, you may use either a Hive
metastore service (HMS) or AWS Glue. The catalog type is determined by the
``iceberg.catalog.type`` property, it can be set to either ``HIVE_METASTORE``
or ``GLUE``.

Hive metastore catalog
^^^^^^^^^^^^^^^^^^^^^^

The Hive metastore catalog is the default implementation.
When using it, the Iceberg connector supports the same metastore
configuration properties as the Hive connector. At a minimum,
``hive.metastore.uri`` must be configured, see
:ref:`Thrift metastore configuration<hive-thrift-metastore>`.

.. code-block:: text

    connector.name=iceberg
    hive.metastore.uri=thrift://localhost:9083

Glue catalog
^^^^^^^^^^^^

When using the Glue catalog, the Iceberg connector supports the same
configuration properties as the Hive connector's Glue setup. See
:ref:`AWS Glue metastore configuration<hive-glue-metastore>`.

.. code-block:: text

    connector.name=iceberg
    iceberg.catalog.type=glue


General configuration
^^^^^^^^^^^^^^^^^^^^^

These configuration properties are independent of which catalog implementation
is used.

.. list-table:: Iceberg general configuration properties
  :widths: 30, 58, 12
  :header-rows: 1

  * - Property name
    - Description
    - Default
  * - ``iceberg.file-format``
    - Define the data storage file format for Iceberg tables.
      Possible values are

      * ``PARQUET``
      * ``ORC``
    - ``ORC``
  * - ``iceberg.compression-codec``
    - The compression codec to be used when writing files.
      Possible values are

      * ``NONE``
      * ``SNAPPY``
      * ``LZ4``
      * ``ZSTD``
      * ``GZIP``
    - ``ZSTD``
  * - ``iceberg.use-file-size-from-metadata``
    - Read file sizes from metadata instead of file system.
    - ``true``
  * - ``iceberg.max-partitions-per-writer``
    - Maximum number of partitions handled per writer.
    - 100
  * - ``iceberg.target-max-file-size``
    - Target maximum size of written files; the actual size may be larger.
    - ``1GB``
  * - ``iceberg.unique-table-location``
    - Use randomized, unique table locations.
    - ``false``
  * - ``iceberg.dynamic-filtering.wait-timeout``
    - Maximum duration to wait for completion of dynamic filters during split generation.
    - ``0s``
  * - ``iceberg.delete-schema-locations-fallback``
    - Whether schema locations should be deleted when Trino can't determine whether they contain external files.
    - ``false``
  * - ``iceberg.minimum-assigned-split-weight``
    - A decimal value in the range (0, 1] used as a minimum for weights assigned to each split. A low value may improve performance
      on tables with small files. A higher value may improve performance for queries with highly skewed aggregations or joins.
    - 0.05
  * - ``iceberg.table-statistics-enabled``
    - Enables :doc:`/optimizer/statistics`. The equivalent
      :doc:`catalog session property </sql/set-session>`
      is ``statistics_enabled`` for session specific use.
      Set to ``false`` to disable statistics. Disabling statistics
      means that :doc:`/optimizer/cost-based-optimizations` can
      not make smart decisions about the query plan.
    - ``true``
  * - ``iceberg.projection-pushdown-enabled``
    - Enable :doc:`projection pushdown </optimizer/pushdown>`
    - ``true``
  * - ``iceberg.hive-catalog-name``
    - Catalog to redirect to when a Hive table is referenced.
    -

ORC format configuration
^^^^^^^^^^^^^^^^^^^^^^^^

The following properties are used to configure the read and write operations
with ORC files performed by the Iceberg connector.

{'fraction_non_alphanumeric': 0.10444360045575389, 'fraction_numerical': 0.005222180022787694, 'mean_word_length': 3.6606194690265488, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '357849', 'n_tokens_mistral': 2936, 'n_tokens_neox': 2784, 'n_words': 1245}