<commit_before># Intel

:monkey_face: Search analytics made easy

[See it in action](http://intel-demo.herokuapp.com/)

- view searches in real-time
- track conversions week over week
- monitor the performance of top searches

:cupid: An amazing companion to [Searchkick](https://github.com/ankane/searchkick)

Works with Rails 3.1+ and any search engine, including Elasticsearch, Sphinx, and Solr

:tangerine: Battle-tested at [Instacart](https://www.instacart.com)

## Get Started

Add this line to your application’s Gemfile:

```ruby
gem "intel"
```

And run the generator. This creates a migration to store searches.

```sh
rails generate intel:install
rake db:migrate
```

Next, add the dashboard to your `config/routes.rb`.

```ruby
mount Intel::Engine, at: "admin/intel"
```

Be sure to protect the endpoint in production - see the [Authentication](#authentication) section for ways to do this.

### Track Searches

Track searches by creating a record in the database.

```ruby
Intel::Search.create(
  search_type: "Item", # typically the model name
  query: "apple",
  results_count: 12
)
```

With [Searchkick](https://github.com/ankane/searchkick), you can use the `track` option to do this automatically.

```ruby
Item.search "apple", track: true
```

If you want to track more attributes, add them to the `intel_searches` table.  Then, pass the values to the `track` option.

```ruby
Item.search "apple", track: {user_id: 1, source: "web"}
```

It’s that easy.

### Track Conversions

First, choose a conversion metric. At Instacart, an item added to the cart from the search results page counts as a conversion.

Next, when a user searches, keep track of the search id. With Searchkick, you can get the id with `@results.search.id`.

When a user converts, find the record and call `convert`.

```ruby
search = Intel::Search.find params[:id]
search.convert
```

Better yet, record the model that converted.

```ruby
item = Item.find params[:item_id]
search.convert(item)
```

### Authentication

Don’t forget to protect the dashboard in production.

#### Basic Authentication

Set the following variables in your environment or an initializer.

```ruby
ENV["INTEL_USERNAME"] = "andrew"
ENV["INTEL_PASSWORD"] = "<PASSWORD>"
```

#### Devise

```ruby
authenticate :user, lambda{|user| user.admin? } do
  mount Intel::Engine, at: "admin/intel"
end
```

### Customize

#### Time Zone

To change the time zone, create an initializer `config/initializers/intel.rb` with:

```ruby
Intel.time_zone = "Pacific Time (US & Canada)" # defaults to Time.zone
```

#### Top Searches

Change the number of top searches shown with:

```ruby
Intel.top_searches = 500 # defaults to 100
```

#### Live Conversions

Show the conversion name in the live stream.

```ruby
Intel.conversion_name = proc{|model| model.name }
```

## TODO

- customize views
- group similar queries
- track pagination, facets, sorting, etc
<commit_msg>Added todo<commit_after># Intel

:monkey_face: Search analytics made easy

[See it in action](http://intel-demo.herokuapp.com/)

- view searches in real-time
- track conversions week over week
- monitor the performance of top searches

:cupid: An amazing companion to [Searchkick](https://github.com/ankane/searchkick)

Works with Rails 3.1+ and any search engine, including Elasticsearch, Sphinx, and Solr

:tangerine: Battle-tested at [Instacart](https://www.instacart.com)

## Get Started

Add this line to your application’s Gemfile:

```ruby
gem "intel"
```

And run the generator. This creates a migration to store searches.

```sh
rails generate intel:install
rake db:migrate
```

Next, add the dashboard to your `config/routes.rb`.

```ruby
mount Intel::Engine, at: "admin/intel"
```

Be sure to protect the endpoint in production - see the [Authentication](#authentication) section for ways to do this.

### Track Searches

Track searches by creating a record in the database.

```ruby
Intel::Search.create(
  search_type: "Item", # typically the model name
  query: "apple",
  results_count: 12
)
```

With [Searchkick](https://github.com/ankane/searchkick), you can use the `track` option to do this automatically.

```ruby
Item.search "apple", track: true
```

If you want to track more attributes, add them to the `intel_searches` table.  Then, pass the values to the `track` option.

```ruby
Item.search "apple", track: {user_id: 1, source: "web"}
```

It’s that easy.

### Track Conversions

First, choose a conversion metric. At Instacart, an item added to the cart from the search results page counts as a conversion.

Next, when a user searches, keep track of the search id. With Searchkick, you can get the id with `@results.search.id`.

When a user converts, find the record and call `convert`.

```ruby
search = Intel::Search.find params[:id]
search.convert
```

Better yet, record the model that converted.

```ruby
item = Item.find params[:item_id]
search.convert(item)
```

### Authentication

Don’t forget to protect the dashboard in production.

#### Basic Authentication

Set the following variables in your environment or an initializer.

```ruby
ENV["INTEL_USERNAME"] = "andrew"
ENV["INTEL_PASSWORD"] = "<PASSWORD>"
```

#### Devise

```ruby
authenticate :user, lambda{|user| user.admin? } do
  mount Intel::Engine, at: "admin/intel"
end
```

### Customize

#### Time Zone

To change the time zone, create an initializer `config/initializers/intel.rb` with:

```ruby
Intel.time_zone = "Pacific Time (US & Canada)" # defaults to Time.zone
```

#### Top Searches

Change the number of top searches shown with:

```ruby
Intel.top_searches = 500 # defaults to 100
```

#### Live Conversions

Show the conversion name in the live stream.

```ruby
Intel.conversion_name = proc{|model| model.name }
```

## TODO

- customize views
- analytics for individual queries
- group similar queries
- track pagination, facets, sorting, etc

{'fraction_non_alphanumeric': 0.12036091249574396, 'fraction_numerical': 0.003745318352059925, 'mean_word_length': 4.574003795066414, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '390703', 'n_tokens_mistral': 1910, 'n_tokens_neox': 1755, 'n_words': 707}