<commit_before>scraperwiki-export
==================

A tool to retrieve code and data from scraperwiki classic.<commit_msg>Updated readme with some useful-ish info<commit_after># scraperwiki-export

This tool provides an easy way for you to retrieve your data and code from ScraperWiki Classic.

The quickest way to run it, assuming you have the [go](http://golang.org/doc/install) compiler installed is to do:

  * git clone <EMAIL>:rossjones/scraperwiki-export.git
  * cd scraperwiki-export
  * export GOPATH=\`pwd\`
  * go run *.go --user YOUR_USERNAME --output ./data 
  * Go make a cup of tea.
  
  
If you don't have the go compiler installed, you'll need to wait for a binary.


## Still to-do

 * Progress info
 * Fetch a couple at a time.
 * See if SW supports range for re-starts
 * Implement fetching of a single item
 * Implement fetching a substring match of the name
 * Better error checking
 * See if we can get private scrapers
{'fraction_non_alphanumeric': 0.08898305084745763, 'fraction_numerical': 0.0, 'mean_word_length': 3.748743718592965, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1706365', 'n_tokens_mistral': 283, 'n_tokens_neox': 273, 'n_words': 126}