<commit_before>Collective Knowledge Infrastructure - enabling open,
collaborative and reproducible experimentation, 
knowledge sharing and predictive analytics.

License
=======
* Permissive 3-clause BSD license 
(see LICENSE.txt file for more details).

Requirements
============
* has been validated on Linux, Windows, Android (partially), Mac OS X
* only python >= 2.6 (3.0+ is natively supported)
* (OpenME API to C,C++,Fortran,PHP,Java)

Introduction
============

We have developed Collective Knowledge Framework and Repository (CK)
primarily to solve numerous problems we faced during past 20 years
of our research on building self-optimizing computer systems
(combining performance autotuning, machine learning and run-time adaptation):

* numerous and ever changing hardware
* black-box compilers
* ad-hoc tools with evolving and incompatible interfaces
* multiple heterogeneous and possibly proprietary data formats
* raising number of design and optimization choices
* raising amount of experimental data to process
* ad-hoc, outdated and non-representative benchmarks with limited data sets
* lack of unified and reproducible mechanisms for exchange of experimental results

Eventually, we created CK as an open-source and light-weight SDK
(~200K python code) to help our colleagues preserve, organize,
describe, cross-link and share their code and data (benchmarks,
data sets, scripts, libraries, tools, experimental results,
models, tables, graphs, articles, etc) as reusable Python-based
components with a very simple API (just one one function with
JSON in and JSON out) and JSON-based schema-free meta
description, as shown in the following examples:
* https://github.com/ctuning/ctuning-programs 
* https://github.com/ctuning/ctuning-datasets-min
* https://github.com/ctuning/ck-analytics
* https://github.com/ctuning/ck-autotuning

CK also helps to abstract access to ever-changing tools and
hardware via wrappers with the same JSON API while protecting
experimental setups from low-level interface/data changes
(wrappers handle such changes as well as setting up environment
for multiple versions of a given tool or library).

All components has DOI-style UID and can be transparently indexed and 
searched via third-party Hadoop-based ElasticSearch, shared via GIT, 
and connected together into experimental pipelines (workflows) 
just as LEGO(TM) to quickly prototype various research ideas, 
crowdsource experiments, reproduce and improve past techniques.

Furthermore, CK can simplify connection of unified experiments 
to powerful predictive analytics tools such as scikit-learn <commit_msg>fixing readme<commit_after>Collective Knowledge Infrastructure - enabling open,
collaborative and reproducible experimentation, 
knowledge sharing and predictive analytics.

License
=======
* Permissive 3-clause BSD license 
(see LICENSE.txt file for more details).

Requirements
============
* has been validated on Linux, Windows, Android (partially), Mac OS X
* only python >= 2.6 (3.0+ is natively supported)
* (OpenME API to C,C++,Fortran,PHP,Java)

Introduction
============

We have developed Collective Knowledge Framework and Repository (CK)
primarily to solve numerous problems we faced during past 20 years
of our research on building self-optimizing computer systems
(combining performance autotuning, machine learning and run-time adaptation):

* numerous and ever changing hardware
* black-box compilers
* ad-hoc tools with evolving and incompatible interfaces
* multiple heterogeneous and possibly proprietary data formats
* raising number of design and optimization choices
* raising amount of experimental data to process
* ad-hoc, outdated and non-representative benchmarks with limited data sets
* lack of unified and reproducible mechanisms for exchange of experimental results

Eventually, we created CK as an open-source and light-weight SDK
(~200K python code) to help our colleagues preserve, organize,
describe, cross-link and share their code and data (benchmarks,
data sets, scripts, libraries, tools, experimental results,
models, tables, graphs, articles, etc) as reusable Python-based
components with a very simple API (just one one function with
JSON in and JSON out) and JSON-based schema-free meta
description, as shown in the following examples:
* https://github.com/ctuning/ctuning-programs 
* https://github.com/ctuning/ctuning-datasets-min
* https://github.com/ctuning/ck-analytics
* https://github.com/ctuning/ck-autotuning

CK also helps to abstract access to ever-changing (and possibly
proprietary) tools and hardware via wrappers with the same JSON
API while protecting experimental setups from low-level
interface/data changes (wrappers handle such changes as well
as setting up environment for multiple versions of a given tool
or library).

All components has DOI-style UID and can be transparently indexed and 
searched via third-party Hadoop-based ElasticSearch, shared via GIT, 
and connected together into experimental pipelines (workflows) 
just as LEGO(TM) to quickly prototype various research ideas, 
crowdsource experiments, reproduce and improve past techniques.

Furthermore, CK can simplify connection of unified experiments 
to powerful predictive analytics tools such as scikit-learn 
{'fraction_non_alphanumeric': 0.06503137478608101, 'fraction_numerical': 0.0038030043734550295, 'mean_word_length': 5.406820950060902, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3034443', 'n_tokens_mistral': 1349, 'n_tokens_neox': 1247, 'n_words': 638}