<commit_before>hadoop-log-getter
=================

A simple tool to show all logs of a Hadoop job<commit_msg>Some documentation<commit_after>hadoop-log-getter
=================

A simple tool to show all logs of a Hadoop job

* Takes the job id
* Asks the YARN History Server REST-API for the log URLs of all job attempts
* Scrapes the log pages
* Strips the HTML surrounding the log output
* Returns everything on one page

Prerequisites
-------------
* Hadoop >= 2.0, tested with 2.0.6-alpha
* History Server installed and configured
* Maven for building

Usage
-----
~~~~ 
mvn package
java -jar target/log-getter-1.0-SNAPSHOT.jar
~~~~
* Find out job id (e.g. job_1234567890123_1234)
* Call http://localhost:8080/<job id> (e.g. http://localhost:8080/job_1234567890123_1234)


Limitations
-----------
* Works for already finished jobs only
* Returns all logs in a single request, could be very slow

{'fraction_non_alphanumeric': 0.14317425083240842, 'fraction_numerical': 0.05438401775804662, 'mean_word_length': 3.722513089005236, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1129333', 'n_tokens_mistral': 313, 'n_tokens_neox': 268, 'n_words': 107}