<commit_before>= Teeth
Teeth is a library for fast parsing of log files such as Apache access and error logs.  It uses C extensions generated by flex.  You need to have flex installed for it to run.

= Example
 require "teeth"
 
 access_log = %q{myhost.localdomain:80 172.16.115.1 - - [13/Dec/2008:19:26:11 -0500] "GET /favicon.ico HTTP/1.1" 404 241 "http://172.16.115.130/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"}
 access_log.tokenize_apache_logs
	=>  {:absolute_url=>["http://172.16.115.130/"], :http_method=>["GET"], :browser_string=>["Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"], :message=>"myhost.localdomain:80 172.16.115.1 - - [13/Dec/2008:19:26:11 -0500] \"GET /favicon.ico HTTP/1.1\" 404 241 \"http://172.16.115.130/\" \"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1\"", :relative_url=>["/favicon.ico"], :http_version=>["HTTP/1.1"], :host=>["myhost.localdomain:80"], :http_response=>["404"], :ipv4_addr=>["172.16.115.1"], :id=>"1BEA4E96164C11DEAF6A0017F22FF48F", :word=>["241"], :apache_access_datetime=>["13/Dec/2008:19:26:11 -0500"]}
	
= Formats
Right now, only Apache logs are supported; however, teeth doesn't depend on tokens appearing in a certain order, so it might be able to parse your Thin, Mongrel, or other web server logs.

Support for other web servers as well as other types of servers (e.g., SMTP, etc.) and generic syslog logs is planned for the future.

= Shortcomings and Known Issues
In addition to the lack of support for formats other than Apache error and access log described above, teeth currently: 
* Does not keep strings of description text intact.  Instead all tokens that it sees which do not fall into a predefined catagory (like HTTP verb, response codes, URIs, etc.) are lumped together in one array.  This is also planned to be fixed in a later release
* Does not convert datetimes to Ruby Time objects
* Does not use context or knowledge of the log format to its advantage.  If you had a request which returned 404 bytes, for example, teeth would add 404 to the http_response list in its output.

= Performance
On my laptop, a white MacBook 2.0 GHz Intel Core Duo, teeth can process more than 30k lines of Apache access logs per second.  So it's pretty fast.  If modified to not create a UUID or keep the full message, this can be increased to around 45k lines/sec.  One could potentially do pretty well on the wide_finder2[http://www.tbray.org/ongoing/When/200x/2008/05/01/Wide-Finder-2]...<commit_msg>Add note about 1.9 support to README<commit_after>= Teeth
Teeth is a library for fast parsing of log files such as Apache access and error logs.  It uses C extensions generated by flex.  You need to have flex installed for it to run.

= Example
 require "teeth"
 
 access_log = %q{myhost.localdomain:80 172.16.115.1 - - [13/Dec/2008:19:26:11 -0500] "GET /favicon.ico HTTP/1.1" 404 241 "http://172.16.115.130/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"}
 access_log.tokenize_apache_logs
	=>  {:absolute_url=>["http://172.16.115.130/"], :http_method=>["GET"], :browser_string=>["Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"], :message=>"myhost.localdomain:80 172.16.115.1 - - [13/Dec/2008:19:26:11 -0500] \"GET /favicon.ico HTTP/1.1\" 404 241 \"http://172.16.115.130/\" \"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1\"", :relative_url=>["/favicon.ico"], :http_version=>["HTTP/1.1"], :host=>["myhost.localdomain:80"], :http_response=>["404"], :ipv4_addr=>["172.16.115.1"], :id=>"1BEA4E96164C11DEAF6A0017F22FF48F", :word=>["241"], :apache_access_datetime=>["13/Dec/2008:19:26:11 -0500"]}
	
= Formats
Right now, only Apache logs are supported; however, teeth doesn't depend on tokens appearing in a certain order, so it might be able to parse your Thin, Mongrel, or other web server logs.

Support for other web servers as well as other types of servers (e.g., SMTP, etc.) and generic syslog logs is planned for the future.

= Ruby 1.9
Ruby 1.9 is supported via the ruby1.9 branch[http://github.com/danielsdeleo/teeth/tree/ruby1.9].

= Shortcomings and Known Issues
In addition to the lack of support for formats other than Apache error and access log described above, teeth currently: 
* Does not keep strings of description text intact.  Instead all tokens that it sees which do not fall into a predefined catagory (like HTTP verb, response codes, URIs, etc.) are lumped together in one array.  This is also planned to be fixed in a later release
* Does not convert datetimes to Ruby Time objects
* Does not use context or knowledge of the log format to its advantage.  If you had a request which returned 404 bytes, for example, teeth would add 404 to the http_response list in its output.

= Performance
On my laptop, a white MacBook 2.0 GHz Intel Core Duo, teeth can process more than 30k lines of Apache access logs per second.  So it's pretty fast.  If modified to not create a UUID or keep the full message, this can be increased to around 45k lines/sec.  One could potentially do pretty well on the wide_finder2[http://www.tbray.org/ongoing/When/200x/2008/05/01/Wide-Finder-2]...
{'fraction_non_alphanumeric': 0.11984815618221258, 'fraction_numerical': 0.09146782357194505, 'mean_word_length': 4.269523809523809, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 12, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1715001', 'n_tokens_mistral': 2131, 'n_tokens_neox': 1809, 'n_words': 707}