<commit_before># Pickaxe
---
An easy to use SQL based web scraper language. If you know SQL and a little about CSS selectors and want to capture data from the web, this is the tool for you.
## Downloads
---
Found [here](https://github.com/bitsummation/pickaxe/releases). It requires .NET framework 4.0. The Pickaxe-Console.zip is the command line version without the editor.
## Quickstart
---
Pickaxe uses SQL statements combined with CSS selectors to pick out text from a web page. Download the tool from above and unzip the contents and double click on **Pickaxe.Studio.exe**. Below are some simple examples.
#### Example 1
Capture the commit information from this page.
```sql
select
	case pick '.icon span.octicon-file-text' take text
		when null then 'Folder'
		else 'File'
	end, --folder/file
	pick '.content a' take text, --name
	pick '.message a' take text, --comment
	pick '.age span' take text --date
from download page 'https://github.com/bitsummation/pickaxe'
where nodes = 'table.files tr.js-navigation-item'
```
#### Example 2
What's your ip address?
```sql
select
	pick '#section_left div:nth-child(2) a' take text
from download page 'http://whatismyipaddress.com/'
```
#### In Memory Buffer
Store results in memory.
```sql
create buffer results(type string, folder string, message string, changeDate string)<commit_msg>readme changes<commit_after># Pickaxe
---
An easy to use SQL based web scraper language. If you know SQL and a little about CSS selectors and want to capture data from the web, this is the tool for you.
## Downloads
---
Found [here](https://github.com/bitsummation/pickaxe/releases). It requires .NET framework 4.0. The Pickaxe-Console.zip is the command line version without the editor.
## Quickstart
---
Pickaxe uses SQL statements combined with CSS selectors to pick out text from a web page. Download the tool from above and unzip the contents and double click on **Pickaxe.Studio.exe**. Below are some example snippets. A full runnable example that scrapes a forum I host is found [here](https://raw.githubusercontent.com/bitsummation/pickaxe/master/Examples/vtss.s).
#### Example 1
Capture the commit information from this page.
```sql
select
	case pick '.icon span.octicon-file-text' take text
		when null then 'Folder'
		else 'File'
	end, --folder/file
	pick '.content a' take text, --name
	pick '.message a' take text, --comment
	pick '.age span' take text --date
from download page 'https://github.com/bitsummation/pickaxe'
where nodes = 'table.files tr.js-navigation-item'
```
#### Example 2
What's your ip address?
```sql
select
	pick '#section_left div:nth-child(2) a' take text
from download page 'http://whatismyipaddress.com/'
```
#### In Memory Buffer
Store results in memory.
```sql
create buffer results(type string, folder string, message string, changeDate string)
{'fraction_non_alphanumeric': 0.10709219858156029, 'fraction_numerical': 0.0035460992907801418, 'mean_word_length': 4.224074074074074, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1608580', 'n_tokens_mistral': 837, 'n_tokens_neox': 799, 'n_words': 365}