<commit_before>=================
Iceberg connector
=================

Overview
--------

Apache Iceberg is an open table format for huge analytic datasets.
The Iceberg connector allows querying data stored in
files written in Iceberg format, as defined in the
`Iceberg Table Spec <https://iceberg.apache.org/spec/>`_. It supports Apache
Iceberg table spec version 1.

The Iceberg table state is maintained in metadata files. All changes to table state
create a new metadata file and replace the old metadata with an atomic swap.
The table metadata file tracks the table schema, partitioning config,
custom properties, and snapshots of the table contents.

Iceberg data files can be stored in either Parquet or ORC format, as
determined by the ``format`` property in the table definition.  The
table ``format`` defaults to ``ORC``.

Iceberg is designed to improve on the known scalability limitations of Hive, which stores
table metadata in a metastore that is backed by a relational database such as MySQL.  It tracks
partition locations in the metastore, but not individual data files.  Trino queries
using the :doc:`/connector/hive` must first call the metastore to get partition locations,
then call the underlying filesystem to list all data files inside each partition,
and then read metadata from each data file.

Since Iceberg stores the paths to data files in the metadata files, it
only consults the underlying file system for files that must be read.

Requirements
------------

To use Iceberg, you need:

* Network access from the Trino coordinator and workers to the distributed
  object storage.
* Access to a Hive metastore service (HMS).
* Network access from the Trino coordinator to the HMS. Hive
  metastore access with the Thrift protocol defaults to using port 9083.

Configuration
-------------

Iceberg supports the same metastore configuration properties as the Hive connector.
At a minimum, ``hive.metastore.uri`` must be configured:

.. code-block:: text

    connector.name=iceberg
    hive.metastore.uri=thrift://localhost:9083

.. list-table:: Iceberg configuration properties
  :widths: 35, 80, 5
  :header-rows: 1

  * - Property name
    - Description
    - Default
  * - ``iceberg.file-format``
    - Define the data storage file format for Iceberg tables.
      Possible values are

      * ``PARQUET``
      * ``ORC``
    - ``ORC``
  * - ``iceberg.compression-codec``
    - The compression codec to be used when writing files.
      Possible values are

      * ``NONE``
      * ``SNAPPY``
      * ``LZ4``
      * ``ZSTD``
      * ``GZIP``
    - ``GZIP``
  * - ``iceberg.max-partitions-per-writer``
    - Maximum number of partitions handled per writer.
    - 100

SQL support
-----------

This connector provides read access and write access to data and metadata in
Iceberg. In addition to the :ref:`globally available <sql-globally-available>`
and :ref:`read operation <sql-read-operations>` statements, the connector
supports the following features:

* :doc:`/sql/insert`
* :doc:`/sql/delete`, see also :ref:`iceberg-delete`
* :ref:`sql-schema-table-management`, see also :ref:`iceberg-tables`
* :ref:`sql-materialized-views-management`

.. _iceberg-tables:<commit_msg>Add views to SQL support<commit_after>=================
Iceberg connector
=================

Overview
--------

Apache Iceberg is an open table format for huge analytic datasets.
The Iceberg connector allows querying data stored in
files written in Iceberg format, as defined in the
`Iceberg Table Spec <https://iceberg.apache.org/spec/>`_. It supports Apache
Iceberg table spec version 1.

The Iceberg table state is maintained in metadata files. All changes to table state
create a new metadata file and replace the old metadata with an atomic swap.
The table metadata file tracks the table schema, partitioning config,
custom properties, and snapshots of the table contents.

Iceberg data files can be stored in either Parquet or ORC format, as
determined by the ``format`` property in the table definition.  The
table ``format`` defaults to ``ORC``.

Iceberg is designed to improve on the known scalability limitations of Hive, which stores
table metadata in a metastore that is backed by a relational database such as MySQL.  It tracks
partition locations in the metastore, but not individual data files.  Trino queries
using the :doc:`/connector/hive` must first call the metastore to get partition locations,
then call the underlying filesystem to list all data files inside each partition,
and then read metadata from each data file.

Since Iceberg stores the paths to data files in the metadata files, it
only consults the underlying file system for files that must be read.

Requirements
------------

To use Iceberg, you need:

* Network access from the Trino coordinator and workers to the distributed
  object storage.
* Access to a Hive metastore service (HMS).
* Network access from the Trino coordinator to the HMS. Hive
  metastore access with the Thrift protocol defaults to using port 9083.

Configuration
-------------

Iceberg supports the same metastore configuration properties as the Hive connector.
At a minimum, ``hive.metastore.uri`` must be configured:

.. code-block:: text

    connector.name=iceberg
    hive.metastore.uri=thrift://localhost:9083

.. list-table:: Iceberg configuration properties
  :widths: 35, 80, 5
  :header-rows: 1

  * - Property name
    - Description
    - Default
  * - ``iceberg.file-format``
    - Define the data storage file format for Iceberg tables.
      Possible values are

      * ``PARQUET``
      * ``ORC``
    - ``ORC``
  * - ``iceberg.compression-codec``
    - The compression codec to be used when writing files.
      Possible values are

      * ``NONE``
      * ``SNAPPY``
      * ``LZ4``
      * ``ZSTD``
      * ``GZIP``
    - ``GZIP``
  * - ``iceberg.max-partitions-per-writer``
    - Maximum number of partitions handled per writer.
    - 100

SQL support
-----------

This connector provides read access and write access to data and metadata in
Iceberg. In addition to the :ref:`globally available <sql-globally-available>`
and :ref:`read operation <sql-read-operations>` statements, the connector
supports the following features:

* :doc:`/sql/insert`
* :doc:`/sql/delete`, see also :ref:`iceberg-delete`
* :ref:`sql-schema-table-management`, see also :ref:`iceberg-tables`
* :ref:`sql-materialized-views-management`
* :ref:`sql-views-management`

.. _iceberg-tables:
{'fraction_non_alphanumeric': 0.10181311018131102, 'fraction_numerical': 0.005888733922206725, 'mean_word_length': 3.6532083633741887, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5205420', 'n_tokens_mistral': 1814, 'n_tokens_neox': 1707, 'n_words': 788}