<commit_before>[[ingest-statistics]]
=== Ingest Statistics and Time Dimension Configuration

The available plugins for vector support adjustments to their configuration via the command line.
The system property 'SIMPLE_FEATURE_CONFIG_FILE' may be assigned to the name of a locally accessible JSON file defining
the configuration.

==== Example
[source]
----
geowave -DSIMPLE_FEATURE_CONFIG_FILE=myconfigfile.json -localingest
----

Configuration consists of several parts:
[arabic]
. Selecting temporal attributes for a temporal index.
. Assigning to each attribute the type of statistics to be captured within the Statistics Store
. Determining which attributes should be indexed in a secondary index.
. Determining which attribute contains visibility information for other attributes
. Setting the names of the indices to update in WFS-T transactions via the GeoServer plug-in.

The JSON file is made up of configurations.   Each configuration is defined by a class name and a set of attributes.
Configurations are grouped by the Simple Feature Type name.

==== Temporal Configuration

There are three attributes for the temporal configuration:
[arabic]
. timeName
. startRangeName
. endRangeName

These attributes are associated with the name of a simple feature type attribute that references a time value.  To index by a single time attribute,
set `_timeName_` to the name of the single attribute.  To index by a range, set both `_startRangeName_` and `_endRangeName_` to the
names of the simple feature type attributes that define start and end time values.

==== Statistics Configuration

Each simple feature type attribute may have several assigned statistics.  Bounding box and range statistics are automatically captured for geometry and temporal attributes.

[frame="topbot",options="header"]
|===

|Attribute Type |Statistic Name|Statistic Configuration Attributes (with default values)|Statistic Class
.3+|Numeric
|Fixed Bin Histogram|minValue=-&#8734;,maxValue=&#8734;,bins=32|mil.nga.giat.geowave.adapter.vector.stats.FeatureFixedBinNumericStatistics$FeatureFixedBinConfig
|Dynamic Histogram||mil.nga.giat.geowave.adapter.vector.stats.FeatureNumericHistogramStatistics$FeatureNumericHistogramConfig
|Numeric Range||mil.nga.giat.geowave.adapter.vector.stats.FeatureNumericRangeStatistics$FeatureNumericRangeConfig
.2+|String
|Count Min Sketch|errorFactor=0.001,probabilityOfCorrectness=0.98|mil.nga.giat.geowave.adapter.vector.stats.FeatureCountMinSketchStatistics$FeatureCountMinSketchConfig
|Hyper Log Log|precision=16|mil.nga.giat.geowave.adapter.vector.stats.FeatureHyperLogLogStatistics$FeatureHyperLogLogConfig

===|

==== Visibility Configuration

Visibility configuration has two attributes: the visibility manager class and the visibility attribute name.

A Visibility manager extends mil.nga.giat.geowave.core.store.data.visibility.VisibilityManagement.  An instance of this class
interprets the contents of a visibility attribute, within a simple feature, to determine the visibility constraints of
the other attributes in that simple feature.  The default visibility management class is mil.nga.giat.geowave.adapter.vector.plugin.visibility.JsonDefinitionColumnVisibilityManagement.

==== Secondary Index Configuration

Secondary Index Configurations is made up of one of three classes:
. mil.nga.giat.geowave.adapter.vector.index.NumericSecondaryIndexConfiguration
. mil.nga.giat.geowave.adapter.vector.index.TemporalSecondaryIndexConfiguration
. mil.nga.giat.geowave.adapter.vector.index.TextSecondaryIndexConfiguration

Each of this configurations maintains a set of simple feature attribute names to index in a secondary index.

==== Primary Index Identifiers

The class mil.nga.giat.geowave.adapter.vector.index.SimpleFeaturePrimaryIndexConfiguration is used to maintain the configuration of
primary indices used for adding or updating simple features via the GeoServer plug-in (FeatureWriter).

|===

==== Example
[source]
----
{<commit_msg>Fixes table issues in ingeststats<commit_after>[[ingest-statistics]]
=== Ingest Statistics and Time Dimension Configuration

The available plugins for vector support adjustments to their configuration via the command line.
The system property 'SIMPLE_FEATURE_CONFIG_FILE' may be assigned to the name of a locally accessible JSON file defining
the configuration.

==== Example
[source]
----
geowave -DSIMPLE_FEATURE_CONFIG_FILE=myconfigfile.json -localingest
----

Configuration consists of several parts:
[arabic]
. Selecting temporal attributes for a temporal index.
. Assigning to each attribute the type of statistics to be captured within the Statistics Store
. Determining which attributes should be indexed in a secondary index.
. Determining which attribute contains visibility information for other attributes
. Setting the names of the indices to update in WFS-T transactions via the GeoServer plug-in.

The JSON file is made up of configurations.   Each configuration is defined by a class name and a set of attributes.
Configurations are grouped by the Simple Feature Type name.

==== Temporal Configuration

There are three attributes for the temporal configuration:
[arabic]
. timeName
. startRangeName
. endRangeName

These attributes are associated with the name of a simple feature type attribute that references a time value.  To index by a single time attribute,
set `_timeName_` to the name of the single attribute.  To index by a range, set both `_startRangeName_` and `_endRangeName_` to the
names of the simple feature type attributes that define start and end time values.

==== Statistics Configuration

Each simple feature type attribute may have several assigned statistics.  Bounding box and range statistics are automatically captured for geometry and temporal attributes.

[frame="topbot",options="header"]
|===
|Attribute Type |Statistic Name|Statistic Configuration Attributes (with default values)|Statistic Class
.3+|Numeric
|Fixed Bin Histogram|minValue=-&#8734;,maxValue=&#8734;,bins=32|mil.nga.giat.geowave.adapter.vector.stats.FeatureFixedBinNumericStatistics$FeatureFixedBinConfig
|Dynamic Histogram||mil.nga.giat.geowave.adapter.vector.stats.FeatureNumericHistogramStatistics$FeatureNumericHistogramConfig
|Numeric Range||mil.nga.giat.geowave.adapter.vector.stats.FeatureNumericRangeStatistics$FeatureNumericRangeConfig
.2+|String
|Count Min Sketch|errorFactor=0.001,probabilityOfCorrectness=0.98|mil.nga.giat.geowave.adapter.vector.stats.FeatureCountMinSketchStatistics$FeatureCountMinSketchConfig
|Hyper Log Log|precision=16|mil.nga.giat.geowave.adapter.vector.stats.FeatureHyperLogLogStatistics$FeatureHyperLogLogConfig
|===

==== Visibility Configuration

Visibility configuration has two attributes: the visibility manager class and the visibility attribute name.

A Visibility manager extends mil.nga.giat.geowave.core.store.data.visibility.VisibilityManagement.  An instance of this class
interprets the contents of a visibility attribute, within a simple feature, to determine the visibility constraints of
the other attributes in that simple feature.  The default visibility management class is mil.nga.giat.geowave.adapter.vector.plugin.visibility.JsonDefinitionColumnVisibilityManagement.

==== Secondary Index Configuration

Secondary Index Configurations is made up of one of three classes:
. mil.nga.giat.geowave.adapter.vector.index.NumericSecondaryIndexConfiguration
. mil.nga.giat.geowave.adapter.vector.index.TemporalSecondaryIndexConfiguration
. mil.nga.giat.geowave.adapter.vector.index.TextSecondaryIndexConfiguration

Each of this configurations maintains a set of simple feature attribute names to index in a secondary index.

==== Primary Index Identifiers

The class mil.nga.giat.geowave.adapter.vector.index.SimpleFeaturePrimaryIndexConfiguration is used to maintain the configuration of
primary indices used for adding or updating simple features via the GeoServer plug-in (FeatureWriter).

==== Example
[source]
----
{
{'fraction_non_alphanumeric': 0.0660970092988188, 'fraction_numerical': 0.005277707966825836, 'mean_word_length': 5.975460122699387, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1343968', 'n_tokens_mistral': 2062, 'n_tokens_neox': 1985, 'n_words': 770}