<commit_before>= Tarantula

== DESCRIPTION

Tarantula is a big fuzzy spider. It crawls your Rails application, fuzzing data to see what breaks.

== Dependencies

htmlentities
hpricot
facets >= 2.4.3

== Usage

   #!sh
   rake tarantula:setup

Creates a Rails integration test that looks like this, filling in your own auth params. You will probably want to include all fixtures.

   require 'relevance/tarantula'

   # in your test
   def test_with_login
       post '/sessions/create', :password => '<PASSWORD>'
       assert_response :redirect
       assert_redirected_to '/'
       follow_redirect!
       tarantula_crawl(self)
   end

If you want to set custom options, you can get access to the crawler and set properties before running it. For example, this would turn on HTMLTidy.

   def test_with_login
       post '/sessions/create', :password => '<PASSWORD>'
       assert_response :redirect
       assert_redirected_to '/'
       follow_redirect!
       t = tarantula_crawler(self)
       t.handlers << Relevance::Tarantula::TidyHandler.new
       t.crawl '/'
   end

Assuming your project is at /work/project/:

         #!sh
         cd /work/project
         rake tarantula:test

== Verbose Mode

If you run the test you will get a report in tmp/tarantula. You can also set VERBOSE=true to see more detail as the test runs.

For more options see the test suite.

== Allowed Errors

If, for example, a 404 is an appropriate response for some URLs, you can
tell Tarantula to allow 404s for URLs matching a regexp:

     t = tarantula_crawler(self)
     t.allow_404_for %r{/users/\d+/}

== Bugs/Requests

Please submit your bug reports, patches or feature requests as a ticket under the component "tarantula" on our Trac instance here: http://opensource.thinkrelevance.com/.  You'll have to create an account (Sorry!  Otherwise we'd get way too much spam).

== License and Copyright

Copyright (c) 2008 Relevance, Inc., released under the MIT license
<commit_msg>improve readme, add install notes and proper license<commit_after>= Tarantula

== DESCRIPTION

Tarantula is a big fuzzy spider. It crawls your Rails application, fuzzing data to see what breaks.

== Dependencies

htmlentities
hpricot
facets >= 2.4.3
actionpack
activesupport

== Usage

   #!sh
   rake tarantula:setup

Creates a Rails integration test that looks like this, filling in your own auth params. You will probably want to include all fixtures.

   require 'relevance/tarantula'

   # in your test
   def test_with_login
       post '/sessions/create', :password => '<PASSWORD>'
       assert_response :redirect
       assert_redirected_to '/'
       follow_redirect!
       tarantula_crawl(self)
   end

If you want to set custom options, you can get access to the crawler and set properties before running it. For example, this would turn on HTMLTidy.

   def test_with_login
       post '/sessions/create', :password => '<PASSWORD>'
       assert_response :redirect
       assert_redirected_to '/'
       follow_redirect!
       t = tarantula_crawler(self)
       t.handlers << Relevance::Tarantula::TidyHandler.new
       t.crawl '/'
   end

Assuming your project is at /work/project/:

         #!sh
         cd /work/project
         rake tarantula:test

== Verbose Mode

If you run the test you will get a report in tmp/tarantula. You can also set VERBOSE=true to see more detail as the test runs.

For more options see the test suite.

== Allowed Errors

If, for example, a 404 is an appropriate response for some URLs, you can
tell Tarantula to allow 404s for URLs matching a regexp:

     t = tarantula_crawler(self)
     t.allow_404_for %r{/users/\d+/}

== Install

The latest and greatest gem will always be available from Github:

    gem install relevance-tarantula --source http://gems.github.com

You can also grab it from Rubyforge, where we will push stable releases but may not be as bleeding edge as the Github gem.

		gem install tarantula

== Bugs/Requests

Please submit your bug reports, patches or feature requests as a ticket under the component "tarantula" on our Trac instance here: 

http://opensource.thinkrelevance.com/

You'll have to create an account (Sorry!  Otherwise we'd get way too much spam).

== License and Copyright

(The MIT License)

Copyright (c) 2008 Relevance, Inc. - http://thinkrelevance.com

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
{'fraction_non_alphanumeric': 0.06317757009345794, 'fraction_numerical': 0.005981308411214953, 'mean_word_length': 3.5347457627118644, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2911217', 'n_tokens_mistral': 1584, 'n_tokens_neox': 1473, 'n_words': 688}