<commit_before>DuckDuckGo ZeroClickInfo FatHeads
=================================

About
-----

See https://github.com/duckduckgo/duckduckgo/wiki for a general overview on contributing to DuckDuckGo.

This repository is for contributing static, keyword based content to 0-click, e.g. getting a perl function reference when you search for perl split. 


Contributing
------------

This repository is organized by type of content, each with its own directory. Some of those projects are in use on the live system, and some are still in development.

Inside each directory are a couple of different files for specific cases. 

* project/fetch.xx

This script is called to fetch the data. 

* project/parse.xx

This is the script used to parse the data once it has been fetched.

* project/data.url

Please upload datafiles somewhere (off-repository) and then store the URL to them here. It could be to a .zip if there is a whole directory needed.

.xx can be .pl or .py or .js depending on what language you use.

Output Formats
--------------

Please name the output file project.tsv (tab delimited) but do not store the data file(s) in the repository (as noted above).

The output format from parse.xx depends on the type of content. In any case, it should be a tab delimited file, with one line per entry. Usually there is no need for newline characters, but if there is a need for some reason, escape them with a backslash like \\n.

For programming reference, the fields are:

* my $page = $line[0] || ''; -- REQURIED: this is the name of the function.
* my $namespace = $line[1] || ''; -- Usually blank unless for something like JavaScript
* my $url = $line[2] || ''; -- REQUIRED: this is the target URL for more information.
* my $description = $line[3] || ''; -- SOME COMBO OF THESE IS REQUIRED.
* my $synopsis = $line[4] || ''; -- Look at https://duckduckgo.com/?q=perl+split
* my $details = $line[5] || ''; -- The part in grey is the $synopsis and the stuff below is the $description
* my $type = $line[6] || ''; -- usually blank
* my $lang = $line[7] || ''; -- usually blank<commit_msg>Edited README.md via GitHub<commit_after>DuckDuckGo ZeroClickInfo FatHeads
=================================

About
-----

See https://github.com/duckduckgo/duckduckgo/wiki for a general overview on contributing to DuckDuckGo.

This repository is for contributing static, keyword based content to 0-click, e.g. getting a perl function reference when you search for perl split. 


Contributing
------------

This repository is organized by type of content, each with its own directory. Some of those projects are in use on the live system, and some are still in development.

Inside each directory are a couple of different files for specific cases. 

* project/fetch.sh

This shell script is called to fetch the data. 

* project/parse.xx

This is the script used to parse the data once it has been fetched. .xx can be .pl or .py or .js depending on what language you use.

* project/parse.sh

This shell script is called to run the parser. 

* project/data.url

Please upload datafiles somewhere (off-repository) and then store the URL to them here. It could be to a .zip if there is a whole directory needed.

Output Formats
--------------

Please name the output file project.tsv (tab delimited) but do not store the data file(s) in the repository (as noted above).

The output format from parse.xx depends on the type of content. In any case, it should be a tab delimited file, with one line per entry. Usually there is no need for newline characters, but if there is a need for some reason, escape them with a backslash like \\n.

For programming reference, the fields are:

* my $page = $line[0] || ''; -- REQURIED: this is the name of the function.
* my $namespace = $line[1] || ''; -- Usually blank unless for something like JavaScript
* my $url = $line[2] || ''; -- REQUIRED: this is the target URL for more information.
* my $description = $line[3] || ''; -- SOME COMBO OF THESE IS REQUIRED.
* my $synopsis = $line[4] || ''; -- Look at https://duckduckgo.com/?q=perl+split
* my $details = $line[5] || ''; -- The part in grey is the $synopsis and the stuff below is the $description
* my $type = $line[6] || ''; -- usually blank
* my $lang = $line[7] || ''; -- usually blank
{'fraction_non_alphanumeric': 0.11664329125759701, 'fraction_numerical': 0.004207573632538569, 'mean_word_length': 3.5424628450106157, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1606864', 'n_tokens_mistral': 1211, 'n_tokens_neox': 1176, 'n_words': 611}