<commit_before># Hadoop Examples
This repository contains some example Hadoop jobs to demonstrate how to test
and package them. In the words of <NAME>:

> If a feature doesn't have a test, it doesn't exist

## CodeTokenizer
CodeTokenizer reads input source code, tokenizes it, and reports the counts of
all the tokens.

## MoabLicenseInfo
MoabLicenseInfo can scan Moab (an HPC scheduler from Adaptive Computing) logs
to determine software license usage statistics.

## Hadoop Version
This code is tested against a CDH 5.3.2 cluster.<commit_msg>Add notice about repo move<commit_after># Hadoop Examples

**This repository has been moved to https://bitbucket.org/umarcts/hadoop-examples**
{'fraction_non_alphanumeric': 0.061224489795918366, 'fraction_numerical': 0.004373177842565598, 'mean_word_length': 5.133928571428571, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2796219', 'n_tokens_mistral': 192, 'n_tokens_neox': 180, 'n_words': 91}