<commit_before># Loki Storage

Loki needs to store two different types of data: **chunks** and **indexes**.

Loki receives logs in separate streams, where each stream is uniquely identified
by its tenant ID and its set of labels. As log entries from a stream arrive,
they are GZipped as "chunks" and saved in the chunks store. See [chunk
format](#chunk-format) for how chunks are stored internally.

The **index** stores each stream's label set and links them to the individual
chunks.

Refer to Loki's [configuration](../../configuration/README.md) for details on
how to configure the storage and the index.

For more information:

1. [Table Manager](table-manager.md)
2. [Retention](retention.md)

## Supported Stores

The following are supported for the index:

* [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
* [Google Bigtable](https://cloud.google.com/bigtable)
* [Apache Cassandra](https://cassandra.apache.org)
* [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki)

The following are supported for the chunks:

* [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
* [Google Bigtable](https://cloud.google.com/bigtable)
* [Apache Cassandra](https://cassandra.apache.org)
* [Amazon S3](https://aws.amazon.com/s3)
* [Google Cloud Storage](https://cloud.google.com/storage/)
* Filesystem (doesn't work when clustering Loki)

## Cloud Storage Permissions

### S3

When using S3 as object storage, the following permissions are needed:

* `s3:ListBucket`
* `s3:PutObject`
* `s3:GetObject`

Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`

### DynamoDB

When using DynamoDB for the index, the following permissions are needed:

* `dynamodb:BatchGetItem`
* `dynamodb:BatchWriteItem`
* `dynamodb:DeleteItem`
* `dynamodb:DescribeTable`
* `dynamodb:GetItem`
* `dynamodb:ListTagsOfResource`
* `dynamodb:PutItem`
* `dynamodb:Query`
* `dynamodb:TagResource`
* `dynamodb:UntagResource`
* `dynamodb:UpdateItem`
* `dynamodb:UpdateTable`

Resources: `arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*`

#### AutoScaling 

If you enable autoscaling from table manager, the following permissions are needed:

##### Deletion

* `dynamodb:DeleteTable`

Resources: `arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*`

##### Application Autoscaling

* `dynamodb:ListTables`
* `application-autoscaling:DescribeScalableTargets`
* `application-autoscaling:DescribeScalingPolicies`
* `application-autoscaling:RegisterScalableTarget`
* `application-autoscaling:PutScalingPolicy`

Resources: `*`

##### IAM

* `iam:GetRole`
* `iam:PassRole`
<commit_msg>documentation: iam requirement for autoscaling (#1350)<commit_after># Loki Storage

Loki needs to store two different types of data: **chunks** and **indexes**.

Loki receives logs in separate streams, where each stream is uniquely identified
by its tenant ID and its set of labels. As log entries from a stream arrive,
they are GZipped as "chunks" and saved in the chunks store. See [chunk
format](#chunk-format) for how chunks are stored internally.

The **index** stores each stream's label set and links them to the individual
chunks.

Refer to Loki's [configuration](../../configuration/README.md) for details on
how to configure the storage and the index.

For more information:

1. [Table Manager](table-manager.md)
2. [Retention](retention.md)

## Supported Stores

The following are supported for the index:

* [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
* [Google Bigtable](https://cloud.google.com/bigtable)
* [Apache Cassandra](https://cassandra.apache.org)
* [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki)

The following are supported for the chunks:

* [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
* [Google Bigtable](https://cloud.google.com/bigtable)
* [Apache Cassandra](https://cassandra.apache.org)
* [Amazon S3](https://aws.amazon.com/s3)
* [Google Cloud Storage](https://cloud.google.com/storage/)
* Filesystem (doesn't work when clustering Loki)

## Cloud Storage Permissions

### S3

When using S3 as object storage, the following permissions are needed:

* `s3:ListBucket`
* `s3:PutObject`
* `s3:GetObject`

Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`

### DynamoDB

When using DynamoDB for the index, the following permissions are needed:

* `dynamodb:BatchGetItem`
* `dynamodb:BatchWriteItem`
* `dynamodb:DeleteItem`
* `dynamodb:DescribeTable`
* `dynamodb:GetItem`
* `dynamodb:ListTagsOfResource`
* `dynamodb:PutItem`
* `dynamodb:Query`
* `dynamodb:TagResource`
* `dynamodb:UntagResource`
* `dynamodb:UpdateItem`
* `dynamodb:UpdateTable`

Resources: `arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*`

#### AutoScaling 

If you enable autoscaling from table manager, the following permissions are needed:

##### Deletion

* `dynamodb:DeleteTable`

Resources: `arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*`

##### Application Autoscaling

* `dynamodb:ListTables`
* `application-autoscaling:DescribeScalableTargets`
* `application-autoscaling:DescribeScalingPolicies`
* `application-autoscaling:RegisterScalableTarget`
* `application-autoscaling:DeregisterScalableTarget`
* `application-autoscaling:PutScalingPolicy`
* `application-autoscaling:DeleteScalingPolicy`

Resources: `*`

##### IAM

* `iam:GetRole`
* `iam:PassRole`

{'fraction_non_alphanumeric': 0.13571961222967935, 'fraction_numerical': 0.004847129008202834, 'mean_word_length': 6.25, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5916357', 'n_tokens_mistral': 1864, 'n_tokens_neox': 1758, 'n_words': 447}