<commit_before># Schema Wizard

[Detailed User Guide](docs/readme-ext/detailed.md)

[Developer Documentation](docs/readme-ext/contribute.md)

[Known Issues And Security](docs/readme-ext/known-issues.md)
## Version

3.0.1

## About
Schema Wizard is an automation driven, human verified data modeling tool.
* Provide data samples
* Verify automated analysis
* Merge similar fields 
* Export your schema

## Installation

Requirements - 
* Linux environment
* AppArmor
* Docker 1.11
* Docker Compose 1.7.1

Install [docker-compose][docker-compose] and clone this repository.  Then, from the project root directory run:

    sudo apparmor_parser -r -W sw-script-profile
    sudo docker-compose up

We recommend you start with the [example files](docs/example-data) in this project.  When you start up Schema Wizard, a guided tour will help you through your first schema creation.  And remember, always check the ![Bus](/docs/readme-ext/blue-tour-bus.jpg "Tour Bus") and ![Help](/docs/readme-ext/blueQuestionMark_whiteCalloutBg.jpg "Help Button") for guidance!
    
## Use

In addition to automated data sample processing, Schema Wizard has an advanced feature called the [Interpretation Engine](docs/readme-ext/detailed.md#interpretation-engine).  When you are ready to get into the more advanced features of Schema Wizard, head over to the [Detailed Documentation](docs/readme-ext/detailed.md) for more information. 

## Supported Formats
Schema Wizard uses a best effort detection and parsing model.  For any file Schema Wizard receives, it will attempt to extract fields as either numbers or strings.

Basic Formats:
* CSV (text/csv) - Comma Separated Values
* JSON (application/json) - JavaScript Object Notation
* CEF (application/cef) - Common Events Format
* XML (application/xml) - **well formed** Extensible Markup Lanuage
 
Application Formats:
* PDF (application/pdf) - content must be one of the "Basic Formats"
* MS Word (application/vnd.openxmlformats-officedocument.wordprocessingml.document) - content must be one of the "Basic Formats" 

Compressed Formats:
* ZIP (application/zip) - archive of any of the above formats

If Schema Wizard finds a content type that could contain other content types (e.g. a zip of CSV's or a PDF containing XML), it will recursively extract embedded content until it finds numeric or string fields.  For more information on Schema Wizard's parsing strategy, see [Parsing Details](docs/readme-ext/detailed.md#data-samples).

## Supported Browsers
![Chrome](/docs/readme-ext/chrome-icon.jpg "Chrome 51") ![Firefox](/docs/readme-ext/firefox-icon.jpg "Firefox 47") ![Opera](/docs/readme-ext/opera-icon.jpg "Opera 38") ![Internet Explorer](/docs/readme-ext/ie-icon.jpg "IE 11+") ![Safari](/docs/readme-ext/safari-icon.jpg "Safari 9.1")<commit_msg>Grammatical fixes<commit_after># Schema Wizard

[Detailed User Guide](docs/readme-ext/detailed.md)

[Developer Documentation](docs/readme-ext/contribute.md)

[Known Issues And Security](docs/readme-ext/known-issues.md)
## Version

3.0.1

## About
Schema Wizard is an automation driven, human verified data modeling tool.
* Provide data samples
* Verify automated analysis
* Merge similar fields 
* Export your schema

## Installation

Requirements - 
* Linux environment
* AppArmor
* Docker 1.11
* Docker Compose 1.7.1

Install [docker-compose][docker-compose] and clone this repository.  Then, from the project root directory run:

    sudo apparmor_parser -r -W sw-script-profile
    sudo docker-compose up

We recommend you start with the [example files](docs/example-data) in this project.  When you start up Schema Wizard, a guided tour will help you through your first schema creation.  And remember, always check the ![Bus](/docs/readme-ext/blue-tour-bus.jpg "Tour Bus") and ![Help](/docs/readme-ext/blueQuestionMark_whiteCalloutBg.jpg "Help Button") for guidance!
    
## Use

In addition to automated data sample processing, Schema Wizard has an advanced feature called the [Interpretation Engine](docs/readme-ext/detailed.md#interpretation-engine).  When you are ready to get into the more advanced features of Schema Wizard, head over to the [Detailed Documentation](docs/readme-ext/detailed.md) for more information. 

## Supported Formats
Schema Wizard uses a best effort detection and parsing model.  For any file Schema Wizard receives, it will attempt to extract fields as either numbers or strings.

Basic Formats:
* CSV (text/csv) - Comma Separated Values
* JSON (application/json) - JavaScript Object Notation
* CEF (application/cef) - Common Events Format
* XML (application/xml) - **well-formed** Extensible Markup Language
 
Application Formats:
* PDF (application/pdf) - content must be one of the "Basic Formats"
* MS Word (application/vnd.openxmlformats-officedocument.wordprocessingml.document) - content must be one of the "Basic Formats" 

Compressed Formats:
* ZIP (application/zip) - archive of any of the above formats

If Schema Wizard finds a content type that could contain other content types (e.g. a zip of CSV's or a PDF containing XML), it will recursively extract embedded content until it finds numeric or string fields.  For more information on Schema Wizard's parsing strategy, see [Parsing Details](docs/readme-ext/detailed.md#data-samples).

## Supported Browsers
![Chrome](/docs/readme-ext/chrome-icon.jpg "Chrome 51") ![Firefox](/docs/readme-ext/firefox-icon.jpg "Firefox 47") ![Opera](/docs/readme-ext/opera-icon.jpg "Opera 38") ![Internet Explorer](/docs/readme-ext/ie-icon.jpg "IE 11+") ![Safari](/docs/readme-ext/safari-icon.jpg "Safari 9.1")
{'fraction_non_alphanumeric': 0.09697078329449722, 'fraction_numerical': 0.006811256497580211, 'mean_word_length': 4.705521472392638, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '848128', 'n_tokens_mistral': 1721, 'n_tokens_neox': 1603, 'n_words': 607}