<commit_before>= Tarantula

== DESCRIPTION

Tarantula is a big fuzzy spider. It crawls your Rails 3 application, fuzzing data to see what breaks.

== Usage

=== Installation

The latest and greatest version is always available on GitHub. (See the rakefile for dependencies, or
just let RubyGems handle it).  Add it to your Gemfile as normal:

   gem "tarantula"

=== Crawling Your App

Use the included rake task to create a Rails integration test that will allow Tarantula to crawl your
app.

   #!sh
   rake tarantula:setup

Take a moment to familiarize yourself with the generated test. If parts of your application require
login, update the test to make sure Tarantula can access those parts of your app.

   require "relevance/tarantula"
   require "test_helper"
   
   class TarantulaTest < ActionController::IntegrationTest
     # Load enough test data to ensure that there's a link to every page in your
     # application. Doing so allows Tarantula to follow those links and crawl 
     # every page.  For many applications, you can load a decent data set by
     # loading all fixtures.
     fixtures :all
   
     def test_tarantula
       # If your application requires users to log in before accessing certain 
       # pages, uncomment the lines below and update them to allow this test to
       # log in to your application.  Doing so allows Tarantula to crawl the 
       # pages that are only accessible to logged-in users.
       # 
       #   post '/session', :login => 'quentin', :password => '<PASSWORD>'
       #   follow_redirect!
       
       tarantula_crawl(self)
     end
   end

If you want to set custom options, you can get access to the crawler and set properties before running
it. For example, this would turn on HTMLTidy.

   def test_tarantula
     post '/session', :login => 'kilgore', :password => '<PASSWORD>'
     assert_response :redirect
     assert_redirected_to '/'
     follow_redirect!

     t = tarantula_crawler(self)
     t.handlers << Relevance::Tarantula::TidyHandler.new
     t.crawl '/'
   end

Now it's time to turn Tarantula loose on your app. Assuming your project is at /work/project/:

   #!sh
   cd /work/project
   rake tarantula:test

== Verbose Mode

If you run the test using the steps shown above, Tarantula will produce a report in tmp/tarantula. You
can also set VERBOSE=true to see more detail as the test runs.

For more options, please see the test suite.

== Allowed Errors

If, for example, a 404 is an appropriate response for some URLs, you can tell Tarantula to allow 404s
for URLs matching a given regex:

     t = tarantula_crawler(self)
     t.allow_404_for %r{/users/\d+/}

== Testing for Common Attacks

You can specify the attack strings that Tarantula throws at your application.<commit_msg>tweak readme<commit_after>= Tarantula

== DESCRIPTION

Tarantula is a big fuzzy spider. It crawls your Rails 3 application, fuzzing data to see what breaks.

{<img src="https://secure.travis-ci.org/relevance/tarantula.png"
/>}[http://travis-ci.org/relevance/tarantula]

== Usage

=== Installation

The latest and greatest version is always available on GitHub. (See the rakefile for dependencies, or
just let RubyGems handle it).  Add it to your Gemfile as normal:

   gem "tarantula"

=== Crawling Your App

Use the included rake task to create a Rails integration test that will allow Tarantula to crawl your
app.

   $ rake tarantula:setup

Take a moment to familiarize yourself with the generated test. If parts of your application require
login, update the test to make sure Tarantula can access those parts of your app.

   require "relevance/tarantula"
   require "test_helper"
   
   class TarantulaTest < ActionController::IntegrationTest
     # Load enough test data to ensure that there's a link to every page in your
     # application. Doing so allows Tarantula to follow those links and crawl 
     # every page.  For many applications, you can load a decent data set by
     # loading all fixtures.
     fixtures :all
   
     def test_tarantula
       # If your application requires users to log in before accessing certain 
       # pages, uncomment the lines below and update them to allow this test to
       # log in to your application.  Doing so allows Tarantula to crawl the 
       # pages that are only accessible to logged-in users.
       # 
       #   post '/session', :login => 'quentin', :password => '<PASSWORD>'
       #   follow_redirect!
       
       tarantula_crawl(self)
     end
   end

If you want to set custom options, you can get access to the crawler and set properties before running
it. For example, this would turn on HTMLTidy.

   def test_tarantula
     post '/session', :login => 'kilgore', :password => '<PASSWORD>'
     assert_response :redirect
     assert_redirected_to '/'
     follow_redirect!

     t = tarantula_crawler(self)
     t.handlers << Relevance::Tarantula::TidyHandler.new
     t.crawl '/'
   end

Now it's time to turn Tarantula loose on your app. Assuming your project is at /work/project/:

   $ cd /work/project
   $ rake tarantula:test

== Verbose Mode

If you run the test using the steps shown above, Tarantula will produce a report in tmp/tarantula. You
can also set VERBOSE=true to see more detail as the test runs.

For more options, please see the test suite.

== Allowed Errors

If, for example, a 404 is an appropriate response for some URLs, you can tell Tarantula to allow 404s
for URLs matching a given regex:

     t = tarantula_crawler(self)
     t.allow_404_for %r{/users/\d+/}

== Testing for Common Attacks

You can specify the attack strings that Tarantula throws at your application.
{'fraction_non_alphanumeric': 0.06215689746768196, 'fraction_numerical': 0.003541703559412077, 'mean_word_length': 3.24024024024024, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1939027', 'n_tokens_mistral': 1668, 'n_tokens_neox': 1573, 'n_words': 739}