<commit_before># [ipfs-search](http://ipfs-search.com)
[![Build Status](https://travis-ci.org/ipfs-search/ipfs-search.svg?branch=travis)](https://travis-ci.org/ipfs-search/ipfs-search)
[![Maintainability](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/maintainability)](https://codeclimate.com/github/ipfs-search/ipfs-search/maintainability)
[![Test Coverage](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/test_coverage)](https://codeclimate.com/github/ipfs-search/ipfs-search/test_coverage)
[![GoDoc](https://godoc.org/github.com/ipfs-search/ipfs-search?status.svg)](https://godoc.org/github.com/ipfs-search/ipfs-search)

Search engine for the [Interplanetary Filesystem](https://ipfs.io). Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using [ipfs-tika](https://github.com/dokterbob/ipfs-tika), searching is done using ElasticSearch 5, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

## Docs
A preliminary start at providing a minimal amount of documentation can be found in the [docs](docs/) folder.

## Contributors wanted
Building a search engine like this takes a considerable amount of resources (money _and_ TLC).
If you are able to help out with either of them, mail us at <EMAIL> or find us at #ipfssearch on Freenode (or #ipfs-search:chat.weho.st on Matrix).

## Roadmap
For discussing and suggesting features, look at the [project planning](https://github.com/ipfs-search/ipfs-search/projects).

## Requirements

* Go 1.11
* Elasticsearch 5.x
* RabbitMQ / AMQP server
* NodeJS 9.x

## Configuration
Configuration can be done using a YAML configuration file, see [`example_config.yml`](example_config.yml).

The following configuration options can be overridden by environment variables:
* `IPFS_TIKA_URL`
* `IPFS_API_URL`
* `ELASTICSEARCH_URL`
* `AMQP_URL`

or by using environment variables.

## Building<commit_msg>Link to related repo’s in README.<commit_after># [ipfs-search](http://ipfs-search.com)
[![Build Status](https://travis-ci.org/ipfs-search/ipfs-search.svg?branch=travis)](https://travis-ci.org/ipfs-search/ipfs-search)
[![Maintainability](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/maintainability)](https://codeclimate.com/github/ipfs-search/ipfs-search/maintainability)
[![Test Coverage](https://api.codeclimate.com/v1/badges/1c25261992991d72137c/test_coverage)](https://codeclimate.com/github/ipfs-search/ipfs-search/test_coverage)
[![GoDoc](https://godoc.org/github.com/ipfs-search/ipfs-search?status.svg)](https://godoc.org/github.com/ipfs-search/ipfs-search)

Search engine for the [Interplanetary Filesystem](https://ipfs.io). Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using [ipfs-tika](https://github.com/dokterbob/ipfs-tika), searching is done using ElasticSearch 5, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

## Docs
A preliminary start at providing a minimal amount of documentation can be found in the [docs](docs/) folder.

## Related repo's
* [frontend](https://github.com/ipfs-search/ipfs-search-frontend)
* [metadata API](https://github.com/ipfs-search/ipfs-metadata-api)
* [search API](https://github.com/ipfs-search/ipfs-search-api)

## Contributors wanted
Building a search engine like this takes a considerable amount of resources (money _and_ TLC).
If you are able to help out with either of them, mail us at <EMAIL> or find us at #ipfssearch on Freenode (or #ipfs-search:chat.weho.st on Matrix).

## Roadmap
For discussing and suggesting features, look at the [project planning](https://github.com/ipfs-search/ipfs-search/projects).

## Requirements

* Go 1.11
* Elasticsearch 5.x
* RabbitMQ / AMQP server
* NodeJS 9.x

## Configuration
Configuration can be done using a YAML configuration file, see [`example_config.yml`](example_config.yml).

The following configuration options can be overridden by environment variables:
* `IPFS_TIKA_URL`
* `IPFS_API_URL`
* `ELASTICSEARCH_URL`
* `AMQP_URL`

or by using environment variables.

## Building
{'fraction_non_alphanumeric': 0.12568829303327747, 'fraction_numerical': 0.020110126885324396, 'mean_word_length': 5.32072617246596, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 25, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6041121', 'n_tokens_mistral': 1445, 'n_tokens_neox': 1353, 'n_words': 351}