<commit_before># Robots
[![Build Status](https://travis-ci.org/runtimerevolution/robots.png?branch=master)](https://travis-ci.org/runtimerevolution/robots)

Normally in Rails there is no good way to integrate a robots.txt per environment.
There is just one robots.txt for staging and production environment and we usually
want to have one robots.txt for staging and another for production.

# Getting Started with Robots

### Robots Gem make your robots.txt files for you.

Robots is a DSL purposely made to generate robots.txt files for different environments.

## Installation
```sh
gem 'robots'
```

## How to Use Robots?
For more information about how to build a correct robots.txt page check the [RFC Robots specification](http://www.robotstxt.org/norobots-rfc.txt)

DSL Robots is easy to learn. You can split different allows and disallows by environment context.
First of all you have to write what kind of links want allow to crawl and those who doesn't.
To start using Robots you have to create a rounting entry inside of routes.rb pointing to RobotsController:
```ruby
  get '/robots' => 'robots#robots', :as => :robots
```
And then create the controller itself:

```ruby
RobotsController < ApplicationController

  respond_to :txt

  def robots
  end

end
```
Afterwards create your robots file using Robots DSL:
<commit_msg>correct install explanation of robots<commit_after># Robots
[![Build Status](https://travis-ci.org/runtimerevolution/robots.png?branch=master)](https://travis-ci.org/runtimerevolution/robots)

Normally in Rails there is no good way to integrate a robots.txt per environment.
There is just one robots.txt for staging and production environment and we usually
want to have one robots.txt for staging and another for production.

# Getting Started with Robots

### Robots Gem make your robots.txt files for you.

Robots is a DSL purposely made to generate robots.txt files for different environments.

## Installation
```sh
gem 'robots', :git => 'https://github.com/runtimerevolution/robots.git'
```

## How to Use Robots?
For more information about how to build a correct robots.txt page check the [RFC Robots specification](http://www.robotstxt.org/norobots-rfc.txt)

DSL Robots is easy to learn. You can split different allows and disallows by environment context.
First of all you have to write what kind of links want allow to crawl and those who doesn't.
To start using Robots you have to create a rounting entry inside of routes.rb pointing to RobotsController:
```ruby
  get '/robots' => 'robots#robots', :as => :robots
```
And then create the controller itself:

```ruby
RobotsController < ApplicationController

  respond_to :txt

  def robots
  end

end
```
Afterwards create your robots file using Robots DSL:

{'fraction_non_alphanumeric': 0.08021778584392014, 'fraction_numerical': 0.0, 'mean_word_length': 4.670781893004115, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 8, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1705958', 'n_tokens_mistral': 825, 'n_tokens_neox': 759, 'n_words': 350}