<commit_before># Crison

# About

A data language for mining data out of websites and webservices (maybe others in the future).

# Requirements

* JVM Java 7
* phantomjs.binary.path set to 2.1.1 installation of phantomjs or later

# Usage

* Build a sample crison data file with a `.csn` extension (see examples below)
* Place crison data file in a directory of your choice relative to this project
* Edit the value of PHANTOM_PATH in the run.sh file
* In the shell in the root of this project execute one of the following
  * `./run.sh input-dir output-dir`  (csn files are in input-dir)
  * `./run.sh input-dir`             (if no second param output-dir is input-dir)
* Configure the width of the driver window with a CRISON_WIDTH env var (default is 1024)
* Configure the height of the driver window with a CRISON_HEIGHT env var (default is 800)

# Examples

## Move to url

```
{:url! "http://www.example.com"}
```


## Click on a link or button

Where the structural markup is good (has an ID):
```
{:click! "magicButton"}
```
Where the structural markup is not so good (has no ID) we need to try to pin
down the element in question by a hopefully unique sequence of attributes:
```
{:click! {:href "http://www.example.com" :class "some-class"}
```


## Fill in forms

### Good structural markup

Simple forms with decent structural markup and have multi fields filled:

```
{:fill!
 [{"field-1" "Thatcham"}
  {"field-2" "Camberley"}]}
```

Fields can be cleared first:

```
{:fill!
 [{"field-1" clear}
  {"field-1" "Thatcham"}
  {"field-2" clear}
  {"field-2" "Camberley"}]
 :screenshot "search"}
```

### Bad structural markup

Where structural markup is not so good on forms we can select over a sequence of
attributes to try to guarantee uniqueness.  Oh and we can combine filling with
submitting (the last item in the array indicates the submit element):

```
{:fill-submit!
 [{:text! "MySearchTerm1" :tag :input :placeholder "Enter your search term 1"}
  {:text! "MySearchTerm2" :tag :input :placeholder "Enter your search term 2"}
  {:value "Search" :class "mypage-search-submit"}]
 :screenshot "main-search"}
```

Once again fields can be cleared first:

```
{:fill-submit!
 [{:clear! "" :tag :input :placeholder "Enter your search term 1"}
  {:text! "York" :tag :input :placeholder "Enter your search term 1"}
  {:value "Search" :class "mypage-search-submit"}]}
```


## Screenshots

Screenshots can be taken on any state changing directives:

```
{:url! "http://www.example.com" :screenshot "arrived-on-homepage"}
{:id "logo"}
{:tag :h2, :text "Home"}
{:click! "magicButton" :screenshot "pressed-the-magic-button"}
```


## Pause after changing the state of the app

It is possible to configure a delay before screenshotting or moving on in the
app after a click or a form submission:

```
{:fill-submit!
 [{:text! "setThisText" :tag :input :placeholder "Enter your text"}
  {:value "Search" :class "page-submit"}]
 :pause 2000 :screenshot "form-submitted"}
```<commit_msg>Add CSS selectors to get elements example<commit_after># Crison

# About

A data language for mining data out of websites and webservices (maybe others in the future).

# Requirements

* JVM Java 7
* phantomjs.binary.path set to 2.1.1 installation of phantomjs or later

# Usage

* Build a sample crison data file with a `.csn` extension (see examples below)
* Place crison data file in a directory of your choice relative to this project
* Edit the value of PHANTOM_PATH in the run.sh file
* In the shell in the root of this project execute one of the following
  * `./run.sh input-dir output-dir`  (csn files are in input-dir)
  * `./run.sh input-dir`             (if no second param output-dir is input-dir)
* Configure the width of the driver window with a CRISON_WIDTH env var (default is 1024)
* Configure the height of the driver window with a CRISON_HEIGHT env var (default is 800)

# Examples

## Move to url

```
{:url! "http://www.example.com"}
```


## Click on a link or button

Where the structural markup is good (has an ID):
```
{:click! "magicButton"}
```
Where the structural markup is not so good (has no ID) we need to try to pin
down the element in question by a hopefully unique sequence of attributes:
```
{:click! {:href "http://www.example.com" :class "some-class"}
```


## Fill in forms

### Good structural markup

Simple forms with decent structural markup and have multi fields filled:

```
{:fill!
 [{"field-1" "Thatcham"}
  {"field-2" "Camberley"}]}
```

Fields can be cleared first:

```
{:fill!
 [{"field-1" clear}
  {"field-1" "Thatcham"}
  {"field-2" clear}
  {"field-2" "Camberley"}]
 :screenshot "search"}
```

### Bad structural markup

Where structural markup is not so good on forms we can select over a sequence of
attributes to try to guarantee uniqueness.  Oh and we can combine filling with
submitting (the last item in the array indicates the submit element):

```
{:fill-submit!
 [{:text! "MySearchTerm1" :tag :input :placeholder "Enter your search term 1"}
  {:text! "MySearchTerm2" :tag :input :placeholder "Enter your search term 2"}
  {:value "Search" :class "mypage-search-submit"}]
 :screenshot "main-search"}
```

Once again fields can be cleared first:

```
{:fill-submit!
 [{:clear! "" :tag :input :placeholder "Enter your search term 1"}
  {:text! "York" :tag :input :placeholder "Enter your search term 1"}
  {:value "Search" :class "mypage-search-submit"}]}
```

## CSS Selectors to get elements

```
[
 {:url! "http://localhost/tricky.html" :pause 200 :screenshot "tricky"}
 {:css "div.goober h1"}
 {:css "div.fail h1"}
]
```

## Screenshots

Screenshots can be taken on any state changing directives:

```
{:url! "http://www.example.com" :screenshot "arrived-on-homepage"}
{:id "logo"}
{:tag :h2, :text "Home"}
{:click! "magicButton" :screenshot "pressed-the-magic-button"}
```


## Pause after changing the state of the app

It is possible to configure a delay before screenshotting or moving on in the
app after a click or a form submission:

```
{:fill-submit!
 [{:text! "setThisText" :tag :input :placeholder "Enter your text"}
  {:value "Search" :class "page-submit"}]
 :pause 2000 :screenshot "form-submitted"}
```
{'fraction_non_alphanumeric': 0.12982171799027553, 'fraction_numerical': 0.009886547811993517, 'mean_word_length': 4.269854824935952, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '550095', 'n_tokens_mistral': 2059, 'n_tokens_neox': 1919, 'n_words': 817}