<commit_before>go-web-crawler
==============

A toy web crawler written in Go for a "Dev Day" over the course of about 12 hours.

Requirements
============

3rd party dependencies have not been vendored yet. The following packages are
required to build. You should be able to "go get" these:

Gorp: github.com/coopernurse/gorp

Sqlite3: "github.com/mattn/go-sqlite3"

go-html-transform: "code.google.com/p/go-html-transform"

go.net/html: "code.google.com/p/go.net/html"

robotstxt-go: "github.com/temoto/robotstxt-go"

Known Issues
============

* There are no known issues at this time

Usage
=====

Build the project:

    go build

Basic usage with defaults and provided start URL:

    ./go-web-crawler -start http://www.startpage.com

Use -help to get a list of commands:

    ./go-web-crawler -help

License
=======

[MIT License] (LICENSE)<commit_msg>Fixed README<commit_after>go-web-crawler
==============

A toy web crawler written in Go for a "Dev Day" over the course of about 12 hours.

Requirements
============

3rd party dependencies have not been vendored yet. The following packages are
required to build. You should be able to "go get" these:

[Gorp] (https://github.com/coopernurse/gorp)

[Sqlite3] (https://github.com/mattn/go-sqlite3)

[go-html-transform] (https://code.google.com/p/go-html-transform)

[go.net/html] (https://code.google.com/p/go.net/html)

[robotstxt-go] (https://github.com/temoto/robotstxt-go)

Usage
=====

Build the project:

    go build

Basic usage with defaults and provided start URL:

    ./go-web-crawler -start http://www.startpage.com

Use -help to get a list of commands:

    ./go-web-crawler -help

Known Issues
============

* There are no known issues at this time

License
=======

[MIT License] (LICENSE)
{'fraction_non_alphanumeric': 0.15759637188208617, 'fraction_numerical': 0.005668934240362812, 'mean_word_length': 3.796195652173913, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1131937', 'n_tokens_mistral': 619, 'n_tokens_neox': 594, 'n_words': 188}