<commit_before>#yii-word-frequency

A class for accumulating tokens from various text sources, filtering them, and counting thier frequency.
This is useful for generating Tag Clouds, determining the top keywords in a set of texts as well as
determining the frequency of specified tokens. It is a nice little tool which brings many array processing 
operations into one convenient class.

##Installation
Place the contents of the 'yii-word-frequency' extension (extracted from the zip file 
or obtained via Github) into the extensions folder extensions/yii-word-frequency 

Add "ext.yii-word-frequency" to the import array in config/main.php
	return array(
	'basePath'=>dirname(__FILE__).DIRECTORY_SEPARATOR.'..',
	
	// preloading 'log' component
	'preload'=>array('log'),

	// autoloading model and component classes
	'import'=>array(
		'application.models.*',
		'application.components.*',
		'ext.yii-word-frequency.*', 
	),
<commit_msg>fix readme again<commit_after>#yii-word-frequency

A class for accumulating tokens from various text sources, filtering them, and counting thier frequency.
This is useful for generating Tag Clouds, determining the top keywords in a set of texts as well as
determining the frequency of specified tokens. It is a nice little tool which brings many array processing 
operations into one convenient class.

##Installation
Place the contents of the 'yii-word-frequency' extension (extracted from the zip file 
or obtained via Github) into the extensions folder extensions/yii-word-frequency 

Add "ext.yii-word-frequency" to the import array in config/main.php
	// preloading 'log' component
	'preload'=>array('log'),

	// autoloading model and component classes
	'import'=>array(
		'application.models.*',
		'application.components.*',
		'ext.yii-word-frequency.*', 
	),

{'fraction_non_alphanumeric': 0.08753462603878116, 'fraction_numerical': 0.0, 'mean_word_length': 4.788461538461538, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4679599', 'n_tokens_mistral': 501, 'n_tokens_neox': 480, 'n_words': 205}