<commit_before>= YAJL C Bindings for Ruby (work in progress)

This gem (although not in gem form just yet) is a C binding to the excellent YAJL JSON parsing library.

You can read more info at the projects website http://lloydforge.org/projects/yajl or check out it's codes at http://github.com/lloyd/yajl.

== Example of use

First, you're probably gonna want to include it:

 include 'yajl'

Then maybe parse some JSON from:

a File IO

 json_contents = File.new('test.json', 'r')
 hash = Yajl::Native.parse(json)
 
or maybe a StringIO

 json_contents = StringIO.new
 hash = Yajl::Native.parse(json)

or maybe STDIN

 cat someJsonFile.json | ruby -ryajl -e "puts Yajl::Native.parse(STDIN).inspect"


Or lets say you didn't have access to the IO object that contained JSON data, but instead
only had access to chunks of it at a time. No problem!

(Assume we're in an EventMachine::Connection instance)

 def object_parsed(obj)
    puts "Sometimes one pays most for the things one gets for nothing. - <NAME>"
    puts obj.inspect
  end
  
 def connection_completed
   # once a full JSON object has been parsed from the stream
   # object_parsed will be called, and passed the constructed object
   Yajl::Chunked.on_parse_complete = method(:object_parsed)
 end
 
 def receive_data(data)
   # continue passing chunks
   Yajl::Chunked.parse_some(data)
   
   # Or as an alias, you could have done:
   # Yajl::Chunked << data
 end

There are a lot more possibilities, some of which I'm going to write other gems/plugins for.

Some ideas are:
* parsing logs in JSON format
* an ActiveSupport patch (http://github.com/technoweenie/rails/commit/f9abcc93a3660fcf46172c4cd8a2f36ba5a5510d)
* Rack middleware (ideally the JSON body could be handed to the parser while it's still being received)
* use with ohai
* JSON API clients
* etc...

== How to install

First, Yajl uses CMake to build itself (yes, the author realizes this isn't the norm for open source and is willing and ready to accept patches, fork away kids!) so you'll need to grab it first from http://www.cmake.org.

After you've got that, grab the latest version (I suggest at least 1.0.4 as it contains fixes for Unicode parsing) of Yajl itself from the Githubs at http://github.com/lloyd/yajl.

After you have that installed, you should be able to install it like any other gem hosted here like so:

(more instructions here: http://gems.github.com)

 sudo gem install brianmario-yajl-ruby

== Benchmarks

I'll update this readme with some actual data soon.

After I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.

But on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.

The main benefit of this library is in it's memory usage.

Since it's able to parse the stream in chunks, it's memory requirements are very, very low.

Again, I'll post some actual data on this; but in my testing here's what my ruby executable was using to parse a 10MB JSON file 100 times:

 JSON.parse ~60MB
 
 Yajl::Native.parse ~30MB

== Status

This library is still being refined and should by no means be used in production.
That being said, I'd like to invite you to clone/fork and test the crap out of it! ;)

== TODO

While parsing is working for the most part, there are still some edge cases I need to tackle.
Here's a list of things I need to get done to improve the awesomeness of this library.

* Write up a test suite 
* Fix known parsing issues with certain JSON strings (Yajl itself works fine, it's the logic in this library that needs a little tweaking)
* Write more docs
* Change API so it can accept a String or an IO

== Special Thanks

I've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!

* <NAME> (http://github.com/lloyd) - for writing Yajl!!
* <NAME> (http://github.com/besquared) - for peer-pressuring me into getting back into C; it worked ;)
* <NAME> (http://github.com/cypriss) - C hacking help
* <NAME> (http://github.com/rtomsmith) - C hacking help
* Rick (http://github.com/technoweenie) - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)
* The entire Github Crew - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.<commit_msg>minor tweak to special thanks section, removed TODO as I think all of the listed items have been completed<commit_after>= YAJL C Bindings for Ruby (work in progress)

This gem (although not in gem form just yet) is a C binding to the excellent YAJL JSON parsing library.

You can read more info at the projects website http://lloydforge.org/projects/yajl or check out it's codes at http://github.com/lloyd/yajl.

== Example of use

First, you're probably gonna want to include it:

 include 'yajl'

Then maybe parse some JSON from:

a File IO

 json_contents = File.new('test.json', 'r')
 hash = Yajl::Native.parse(json)
 
or maybe a StringIO

 json_contents = StringIO.new
 hash = Yajl::Native.parse(json)

or maybe STDIN

 cat someJsonFile.json | ruby -ryajl -e "puts Yajl::Native.parse(STDIN).inspect"


Or lets say you didn't have access to the IO object that contained JSON data, but instead
only had access to chunks of it at a time. No problem!

(Assume we're in an EventMachine::Connection instance)

 def object_parsed(obj)
    puts "Sometimes one pays most for the things one gets for nothing. - <NAME>"
    puts obj.inspect
  end
  
 def connection_completed
   # once a full JSON object has been parsed from the stream
   # object_parsed will be called, and passed the constructed object
   Yajl::Chunked.on_parse_complete = method(:object_parsed)
 end
 
 def receive_data(data)
   # continue passing chunks
   Yajl::Chunked.parse_some(data)
   
   # Or as an alias, you could have done:
   # Yajl::Chunked << data
 end

There are a lot more possibilities, some of which I'm going to write other gems/plugins for.

Some ideas are:
* parsing logs in JSON format
* an ActiveSupport patch (http://github.com/technoweenie/rails/commit/f9abcc93a3660fcf46172c4cd8a2f36ba5a5510d)
* Rack middleware (ideally the JSON body could be handed to the parser while it's still being received)
* use with ohai
* JSON API clients
* etc...

== How to install

First, Yajl uses CMake to build itself (yes, the author realizes this isn't the norm for open source and is willing and ready to accept patches, fork away kids!) so you'll need to grab it first from http://www.cmake.org.

After you've got that, grab the latest version (I suggest at least 1.0.4 as it contains fixes for Unicode parsing) of Yajl itself from the Githubs at http://github.com/lloyd/yajl.

After you have that installed, you should be able to install it like any other gem hosted here like so:

(more instructions here: http://gems.github.com)

 sudo gem install brianmario-yajl-ruby

== Benchmarks

I'll update this readme with some actual data soon.

After I finished implementation - this library performs close to the same as the current JSON.parse (C gem) does on small/medium files.

But on larger files, and higher amounts of iteration, this library was around 2x faster than JSON.parse.

The main benefit of this library is in it's memory usage.

Since it's able to parse the stream in chunks, it's memory requirements are very, very low.

Again, I'll post some actual data on this; but in my testing here's what my ruby executable was using to parse a 10MB JSON file 100 times:

 JSON.parse ~60MB
 
 Yajl::Native.parse ~30MB

== Status

This library is still being refined and should by no means be used in production.
That being said, I'd like to invite you to clone/fork and test the crap out of it! ;)

== Special Thanks

I've had a lot of inspiration, and a lot of help. Thanks to everyone who's been a part of this and those to come!

* <NAME> (http://github.com/lloyd) - for writing Yajl!!
* <NAME> (http://github.com/besquared) - for peer-pressuring me into getting back into C; it worked ;) Also tons of support over IM
* <NAME> (http://github.com/cypriss) - pointer-hacking help
* <NAME> (http://github.com/rtomsmith) - pointer-hacking help
* Rick (http://github.com/technoweenie) - for making an ActiveSupport patch with support for this library and teasing me that it might go into Rails 3. You sure lit a fire under my ass and I got a ton of work done because of it! :)
* The entire Github Crew - my inspiration, time spent writing this, finding Yajl, So many-MANY other things wouldn't have been possible without this awesome service. I owe you guys some whiskey at Kilowatt.
{'fraction_non_alphanumeric': 0.06977535738597686, 'fraction_numerical': 0.008168822328114363, 'mean_word_length': 3.930089485458613, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 2, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2163503', 'n_tokens_mistral': 2680, 'n_tokens_neox': 2510, 'n_words': 1301}