<commit_before>[[troubleshooting]]
== Troubleshooting

Unfortunately, sometimes things do not go as expected and your {eh} job execution might go awry: incorrect data might be read or written, the job might take significantly longer than expected or you might face some exception.
This section tries to provide help and tips for doing your own diagnostics, identifying the problem and hopefully fixing it.

[float]
=== Isolate the issue

When encountering a problem, do your best to isolate it. This can be quite tricky and many times, it is the hardest part so take your time with it. Take baby steps and try to eliminate unnecessary code or settings in small chunks until you end up with a small, tiny example that exposes your problem.

[float]
=== Use a speedy, local environment

A lot of Hadoop jobs are batch in nature which means they take a long time to execute. To track down the issue faster, use whatever means possible to speed-up the feedback loop: use a small/tiny dataset (no need to load millions of records, some dozens will do) and use a local/pseudo-distributed Hadoop cluster alongside an Elasticsearch node running on your development machine.

[float]
=== Check your settings

Double check your settings and use constants or replicate configurations wherever possible. It is easy to make typos so try to reduce manual configuration by using properties files or constant interfaces/classes.
If you are not sure what a setting is doing, remove it or change its value and see whether it affects your job output.

[float]
=== Verify the input and output

Take a close eye at your input and output; this is typically easier to do with Elasticsearch (the service out-lives the job/script, is real-time and can be accessed right away in a flexible meaner, including the command-line). If your data is not persisted (either in Hadoop or Elasticsearch), consider doing that temporarily to validate each step of your work-flow.

[float]
=== Monitor

While logging helps with bugs and errors, for runtime behavior we strongly recommend doing proper monitoring of your Hadoop and {es} cluster. Both are outside the scope of this chapter however there are several popular, free solutions out there that are worth investigating. For {es}, we recommend http://www.elasticsearch.org/overview/marvel/[Marvel], a free monitoring tool (for development) created by the team behind {es}.
Monitoring gives insight into how the cluster is actually behaving and helps you correlate behavior.
If a monitoring solution is not possible, use the metrics provided by Hadoop, {es} and {eh} to evaluate the runtime behavior.

[float]
=== Increase logging

Logging gives you a lot of insight into what is going on. Both Hadoop and {es} have extensive logging mechanisms as <<logging, does>> {eh} however use that judiciously: too much logging can hide the actual issue so again, do it in small increments.

[float]
=== Measure, do not assume

When encountering a performance issue, do some benchmarking first, in as much isolation as possible. Do not simply assume a certain component is slow; make sure/prove it actually is. Otherwise, more often than not, one might find herself `fixing' the wrong problem (and typically creating a new one).

[float]
=== Find a baseline

Indexing performance depend _heavily_ on the type of data being targeted and its mapping. Same goes for searching but add the query definition to the mix. As mentioned before, experiment and measure the various parts of your dataset to find the sweet-spot of your environment before importing/searching big amounts of data.

[[help]]
=== Getting help

If you need help, there is a strong, helpful community of users on the https://groups.google.com/forum/?fromgroups#!forum/elasticsearch[mailing list] and http://www.elasticsearch.org/community/[IRC] who are happy to help out.

If something is not working, there are two possibilities:

- there is a bug
- you are doing something wrong

Whichever it is, a *clear* description of the problem will help other users to help you. The most complete your report is, the quickest you will receive help from users!

[float]
=== What information is useful?
- OS & JVM version
- Hadoop version / distribution
- if using a certain library (Cascading, Hive, Pig), the version used
- {eh} version
- the job or script that is causing the issue
- Hadoop cluster size<commit_msg>[DOC] minor updates<commit_after>[[troubleshooting]]
== Troubleshooting

Unfortunately, sometimes things do not go as expected and your {eh} job execution might go awry: incorrect data might be read or written, the job might take significantly longer than expected or you might face some exception.
This section tries to provide help and tips for doing your own diagnostics, identifying the problem and hopefully fixing it.

[float]
=== Isolate the issue

When encountering a problem, do your best to isolate it. This can be quite tricky and many times, it is the hardest part so take your time with it. Take baby steps and try to eliminate unnecessary code or settings in small chunks until you end up with a small, tiny example that exposes your problem.

[float]
=== Use a speedy, local environment

A lot of Hadoop jobs are batch in nature which means they take a long time to execute. To track down the issue faster, use whatever means possible to speed-up the feedback loop: use a small/tiny dataset (no need to load millions of records, some dozens will do) and use a local/pseudo-distributed Hadoop cluster alongside an Elasticsearch node running on your development machine.

[float]
=== Check your settings

Double check your settings and use constants or replicate configurations wherever possible. It is easy to make typos so try to reduce manual configuration by using properties files or constant interfaces/classes.
If you are not sure what a setting is doing, remove it or change its value and see whether it affects your job output.

[float]
=== Verify the input and output

Take a close eye at your input and output; this is typically easier to do with Elasticsearch (the service out-lives the job/script, is real-time and can be accessed right away in a flexible meaner, including the command-line). If your data is not persisted (either in Hadoop or Elasticsearch), consider doing that temporarily to validate each step of your work-flow.

[float]
=== Monitor

While logging helps with bugs and errors, for runtime behavior we strongly recommend doing proper monitoring of your Hadoop and {es} cluster. Both are outside the scope of this chapter however there are several popular, free solutions out there that are worth investigating. For {es}, we recommend http://www.elasticsearch.org/overview/marvel/[Marvel], a free monitoring tool (for development) created by the team behind {es}.
Monitoring gives insight into how the cluster is actually behaving and helps you correlate behavior.
If a monitoring solution is not possible, use the metrics provided by Hadoop, {es} and {eh} to evaluate the runtime behavior.

[float]
=== Increase logging

Logging gives you a lot of insight into what is going on. Hadoop, Spark and {es} have extensive logging mechanisms as <<logging, does>> {eh} however use that judiciously: too much logging can hide the actual issue so again, do it in small increments.

[float]
=== Measure, do not assume

When encountering a performance issue, do some benchmarking first, in as much isolation as possible. Do not simply assume a certain component is slow; make sure/prove it actually is. Otherwise, more often than not, one might find herself `fixing' the wrong problem (and typically creating a new one).

[float]
=== Find a baseline

Indexing performance depend _heavily_ on the type of data being targeted and its mapping. Same goes for searching but add the query definition to the mix. As mentioned before, experiment and measure the various parts of your dataset to find the sweet-spot of your environment before importing/searching big amounts of data.

[[help]]
=== Getting help

If you need help, there is a strong, helpful community of users on the https://groups.google.com/forum/?fromgroups#!forum/elasticsearch[mailing list] and http://www.elasticsearch.org/community/[IRC] who are happy to help out.

If something is not working, there are two possibilities:

- there is a bug
- you are doing something wrong

Whichever it is, a *clear* description of the problem will help other users to help you. The most complete your report is, the quickest you will receive help from users!

[float]
=== What information is useful?
- OS & JVM version
- Hadoop / Spark version / distribution
- if using a certain library (Cascading, Hive, Pig), the version used
- {eh} version
- the job or script that is causing the issue
- Hadoop / Spark cluster size
{'fraction_non_alphanumeric': 0.051582782965156004, 'fraction_numerical': 0.0, 'mean_word_length': 4.548325963360708, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5995853', 'n_tokens_mistral': 2116, 'n_tokens_neox': 2044, 'n_words': 1353}