<commit_before>Linux Operating System Configuration
====================================

File Systems
------------

We recommend to **not** use BTRFS on linux, it's known to not work
well in conjunction with ArangoDB.  We experienced that ArangoDB
facing latency issues on accessing its database files on BTRFS
partitions.  In conjunction with BTRFS and AUFS we also saw data loss
on restart.


Virtual Memory Page Sizes
--------------------------

By default, ArangoDB uses Jemalloc as the memory allocator. Jemalloc does a good
job of reducing virtual memory fragmentation, especially for long-running
processes. Unfortunately, some OS configurations can interfere with Jemalloc's
ability to function properly. Specifically, Linux's "transparent hugepages",
Windows' "large pages" and other similar features sometimes prevent Jemalloc
from returning unused memory to the operating system and result in unnecessarily
high memory use. Therefore, we recommend disabling these features when using
Jemalloc with ArangoDB. Please consult your operating system's documentation for
how to do this.

Execute

    sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/enabled"
    sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/defrag"

before executing `arangod`.

Swap Space
----------

It is recommended to assign swap space for a server that is running arangod.
Configuring swap space can prevent the operating system's OOM killer from
killing ArangoDB too eagerly on Linux.

### Over-Commit Memory

For the MMFiles storage engine, execute

    sudo bash -c "echo 0 >/proc/sys/vm/overcommit_memory"

before executing `arangod`.

For the RocksDB storage engine, execute
    
    sudo bash -c "echo 2 >/proc/sys/vm/overcommit_memory"

before starting. 

From [www.kernel.org](https://www.kernel.org/doc/Documentation/sysctl/vm.txt):

- When this flag is 0, the kernel attempts to estimate the amount
  of free memory left when userspace requests more memory.

- When this flag is 1, the kernel pretends there is always enough
  memory until it actually runs out.

- When this flag is 2, the kernel uses a "never overcommit"
  policy that attempts to prevent any overcommit of memory.


Note that then using an `overcommit_memory` setting of 2, this will by default allow 
processes to use all swap space but only half of the available RAM. This can be changed 
by adjusting the value of `overcommit_ratio` as well.

From [www.kernel.org](https://www.kernel.org/doc/Documentation/sysctl/vm.txt):

- When overcommit_memory is set to 2, the committed address
  space is not permitted to exceed swap plus this percentage
  of physical RAM. 

### Zone Reclaim

Execute

    sudo bash -c "echo 0 >/proc/sys/vm/zone_reclaim_mode"

before executing `arangod`.

From [www.kernel.org](https://www.kernel.org/doc/Documentation/sysctl/vm.txt):<commit_msg>Doc - Recommend overcommit_memory 0 or 1 on Linux (planning#3151) (#7258)<commit_after>Linux Operating System Configuration
====================================

File Systems
------------

We recommend to **not** use BTRFS on linux, it's known to not work
well in conjunction with ArangoDB.  We experienced that ArangoDB
facing latency issues on accessing its database files on BTRFS
partitions.  In conjunction with BTRFS and AUFS we also saw data loss
on restart.


Virtual Memory Page Sizes
--------------------------

By default, ArangoDB uses Jemalloc as the memory allocator. Jemalloc does a good
job of reducing virtual memory fragmentation, especially for long-running
processes. Unfortunately, some OS configurations can interfere with Jemalloc's
ability to function properly. Specifically, Linux's "transparent hugepages",
Windows' "large pages" and other similar features sometimes prevent Jemalloc
from returning unused memory to the operating system and result in unnecessarily
high memory use. Therefore, we recommend disabling these features when using
Jemalloc with ArangoDB. Please consult your operating system's documentation for
how to do this.

Execute

    sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/enabled"
    sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/defrag"

before executing `arangod`.

Swap Space
----------

It is recommended to assign swap space for a server that is running arangod.
Configuring swap space can prevent the operating system's OOM killer from
killing ArangoDB too eagerly on Linux.

### Over-Commit Memory

The recommended kernel setting for `overcommit_memory` for both MMFiles and
RocksDB storage engine is 0 or 1. The kernel default is 0.

You can set it as follows before executing `arangod`:

    sudo bash -c "echo 0 >/proc/sys/vm/overcommit_memory"

From [www.kernel.org](https://www.kernel.org/doc/Documentation/sysctl/vm.txt):

- When this flag is 0, the kernel attempts to estimate the amount
  of free memory left when userspace requests more memory.

- When this flag is 1, the kernel pretends there is always enough
  memory until it actually runs out.

- When this flag is 2, the kernel uses a "never overcommit"
  policy that attempts to prevent any overcommit of memory.

### Zone Reclaim

Execute

    sudo bash -c "echo 0 >/proc/sys/vm/zone_reclaim_mode"

before executing `arangod`.

From [www.kernel.org](https://www.kernel.org/doc/Documentation/sysctl/vm.txt):
{'fraction_non_alphanumeric': 0.08978444236176195, 'fraction_numerical': 0.004873477038425492, 'mean_word_length': 4.029217719132894, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 12, 'https://': 5, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4939053', 'n_tokens_mistral': 1515, 'n_tokens_neox': 1407, 'n_words': 673}