<commit_before># The Extensible API for Reconnaissance (EAR) Project

## About

The EAR project is an application designed to make it easy to discover interesting facts about organizations, users, computers, and networks on the web, using OSINT techniques. 

The core concepts of the EAR project are the object model (implemented and database-backed with Active Record) and the tasks (implemented as ruby scripts) to modify objects. EAR tasks are simple to create, infinitely extensible, and have just enough structure to extend the EAR framework in useful ways.

EAR keeps track of newly generated data by maintaining the relationship between objects and tasks for you. For example, if you add a host object, and run a 'geolocate_host' task, you'll find that the physical address generated by the task is now a child of that host. You can programmatically query and inspect these relationships.

## News

12/16/2011 - The EAR Project has been updated to Rails 3!
02/25/2012 - The EAR Project has a stubbed out web UI, and is on its way to v1.0

## Setup

EAR has a number of external (mainly gem) dependencies. We use bundler to manage dependencies.  

To set up EAR, run 'bundle install' in the root of the EAR directory.

The EAR is currently tested and working on:
* OS X 10.5.x+
* Ubuntu Linux 9.10+  

## Getting Started with the EAR

You'll need to configure a database before you can use the app. Do this in ear/config/database.yml. A sample has been provided. - SQLite3 is fine for small databases. Use Postgres (homebrew makes this easy on OSX) or MySQL for anything that's not a test. 

Once you have a database configuration, run the following:

    $ rake db:create && rake db:migrate

Once you have a database, simply run `$ util/console.rb` - this will give you access to a pry shell from which you can create objects and run tasks. 

Creating a host object & running tasks: 

    ear> h = Host.create(:ip_address => "8.8.8.8")
    ear> h.run_task("dns_reverse_lookup")
    ear> h.run_task("geolocate_host")
    ear> h.children

You can also access the interface on the web, by running '`$ rails s' in the root of the EAR directory. This will set up a server on :3000 - Note that this is extremely alpha, and is currently not supported. 

## Utils

The EAR ships with a few utilities which you may find of use:

 - util/sniff.rb: This utility sets up a packet sniffer on the local machine, which automatically creates Host objects within the EAR's database. These host objects are then available to you within the EAR. This is a fun & easy way to find out more information about the systems your host is communicating with.

Check out the utils/ directory for more utils.
<commit_msg>srsly, lists<commit_after># The Extensible API for Reconnaissance (EAR) Project

## About

The EAR project is an application designed to make it easy to discover interesting facts about organizations, users, computers, and networks on the web, using OSINT techniques. 

The core concepts of the EAR project are the object model (implemented and database-backed with Active Record) and the tasks (implemented as ruby scripts) to modify objects. EAR tasks are simple to create, infinitely extensible, and have just enough structure to extend the EAR framework in useful ways.

EAR keeps track of newly generated data by maintaining the relationship between objects and tasks for you. For example, if you add a host object, and run a 'geolocate_host' task, you'll find that the physical address generated by the task is now a child of that host. You can programmatically query and inspect these relationships.

## News

12/16/2011 - The EAR Project has been updated to Rails 3!
02/25/2012 - The EAR Project has a stubbed out web UI, and is on its way to v1.0

## Setup

EAR has a number of external (mainly gem) dependencies. We use bundler to manage dependencies.  

To set up EAR, run 'bundle install' in the root of the EAR directory.

The EAR is currently tested and working on:

* OS X 10.5.x+
* Ubuntu Linux 9.10+  

## Getting Started with the EAR

You'll need to configure a database before you can use the app. Do this in ear/config/database.yml. A sample has been provided. - SQLite3 is fine for small databases. Use Postgres (homebrew makes this easy on OSX) or MySQL for anything that's not a test. 

Once you have a database configuration, run the following:

    $ rake db:create && rake db:migrate

Once you have a database, simply run `$ util/console.rb` - this will give you access to a pry shell from which you can create objects and run tasks. 

Creating a host object & running tasks: 

    ear> h = Host.create(:ip_address => "8.8.8.8")
    ear> h.run_task("dns_reverse_lookup")
    ear> h.run_task("geolocate_host")
    ear> h.children

You can also access the interface on the web, by running '`$ rails s' in the root of the EAR directory. This will set up a server on :3000 - Note that this is extremely alpha, and is currently not supported. 

## Utils

The EAR ships with a few utilities which you may find of use:

 - util/sniff.rb: This utility sets up a packet sniffer on the local machine, which automatically creates Host objects within the EAR's database. These host objects are then available to you within the EAR. This is a fun & easy way to find out more information about the systems your host is communicating with.

Check out the utils/ directory for more utils.

{'fraction_non_alphanumeric': 0.05353834753612449, 'fraction_numerical': 0.012597258243793997, 'mean_word_length': 3.8948322756119675, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '178124', 'n_tokens_mistral': 1502, 'n_tokens_neox': 1421, 'n_words': 853}