<commit_before>TCollector
==========

`tcollector <https://github.com/OpenTSDB/tcollector/>`_ is a client-side
process that gathers data from local collectors and pushes the data to
OpenTSDB. You run it on all your hosts, and it does the work of sending each
host's data to the TSD.

OpenTSDB is designed to make it easy to collect and write data to it.
It has a simple protocol, simple enough for even a shell script to start
sending data. However, to do so reliably and consistently is a bit harder.
What do you do when your TSD server is down? How do you make sure your
collectors stay running?  This is where tcollector comes in.

Tcollector does several things for you:

* Runs all of your data collectors and gathers their data
* Does all of the connection management work of sending data to the TSD
* You don't have to embed all of this code in every collector you write
* Does de-duplication of repeated values
* Handles all of the wire protocol work for you, as well as future enhancements

Deduplication
^^^^^^^^^^^^^

Typically you want to gather data about everything in your system.
This generates a lot of datapoints, the majority of which don't
change very often over time (if ever).  However, you want fine-grained
resolution when they do change.  Tcollector remembers the last value
and timestamp that was sent for all of the time series for all of
the collectors it manages.  If the value doesn't change between sample
intervals, it suppresses sending that datapoint.  Once the value does change
(or 10 minutes have passed), it sends the last suppressed value and timestamp,
plus the current value and timestamp.  In this way all of your graphs and
such are correct.  Deduplication typically reduces the number of datapoints
TSD needs to collect by a large fraction.  This reduces network load and
storage in the backend.  A future OpenTSDB release however will improve on
the storage format by using RLE (among other things), making it essentially
free to store repeated values.

Collecting lots of metrics with tcollector
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Collectors in tcollector can be written in any language.  They just need to
be executable and output the data to stdout.  Tcollector will handle the rest.
The collectors are placed in the ``collectors`` directory.  Tcollector
iterates over every directory named with a number in that directory and runs all
the collectors in each directory.  If you name the directory ``60``,
then tcollector will try to run every collector in that directory every 60
seconds. Use the directory ``0`` for any collectors that are long-lived
and run continuously. Tcollector will read their output and respawn them if they
die. Generally you want to write long-lived collectors since that has less
overhead. OpenTSDB is designed to have lots of datapoints for each metric (for
most metrics we send datapoints every 15 seconds).<commit_msg>Clarifying documentation on folder based run intervals<commit_after>TCollector
==========

`tcollector <https://github.com/OpenTSDB/tcollector/>`_ is a client-side
process that gathers data from local collectors and pushes the data to
OpenTSDB. You run it on all your hosts, and it does the work of sending each
host's data to the TSD.

OpenTSDB is designed to make it easy to collect and write data to it.
It has a simple protocol, simple enough for even a shell script to start
sending data. However, to do so reliably and consistently is a bit harder.
What do you do when your TSD server is down? How do you make sure your
collectors stay running?  This is where tcollector comes in.

Tcollector does several things for you:

* Runs all of your data collectors and gathers their data
* Does all of the connection management work of sending data to the TSD
* You don't have to embed all of this code in every collector you write
* Does de-duplication of repeated values
* Handles all of the wire protocol work for you, as well as future enhancements

Deduplication
^^^^^^^^^^^^^

Typically you want to gather data about everything in your system.
This generates a lot of datapoints, the majority of which don't
change very often over time (if ever).  However, you want fine-grained
resolution when they do change.  Tcollector remembers the last value
and timestamp that was sent for all of the time series for all of
the collectors it manages.  If the value doesn't change between sample
intervals, it suppresses sending that datapoint.  Once the value does change
(or 10 minutes have passed), it sends the last suppressed value and timestamp,
plus the current value and timestamp.  In this way all of your graphs and
such are correct.  Deduplication typically reduces the number of datapoints
TSD needs to collect by a large fraction.  This reduces network load and
storage in the backend.  A future OpenTSDB release however will improve on
the storage format by using RLE (among other things), making it essentially
free to store repeated values.

Collecting lots of metrics with tcollector
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Collectors in tcollector can be written in any language.  They just need to
be executable and output the data to stdout.  Tcollector will handle the rest.
The collectors are placed in the ``collectors`` directory.  Tcollector
iterates over every directory named with a number in that directory and runs all
the collectors in each directory.  If you name the directory ``60``,
then tcollector will try to run every collector in that directory every 60
seconds. The shortest supported interval is 15 seconds, you should use a
long-running collector in the 0 folder for intervals shorter than 15 seconds.
TCollector will sleep for 15 seconds after each time it runs the collectors
so intervals of 15 seconds are the only actually supported intervals. For example,
this would allow you to run a collector every 15, 30, 45, 60, 75, or 90 seconds,
but not 80 or 55 seconds. Use the directory ``0`` for any collectors that are long-lived
and run continuously. Tcollector will read their output and respawn them if they
die. Generally you want to write long-lived collectors since that has less
overhead. OpenTSDB is designed to have lots of datapoints for each metric (for
most metrics we send datapoints every 15 seconds).
{'fraction_non_alphanumeric': 0.05079060852898898, 'fraction_numerical': 0.00686791247404568, 'mean_word_length': 4.507475813544415, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7581604', 'n_tokens_mistral': 1614, 'n_tokens_neox': 1526, 'n_words': 1001}