<commit_before>CK - enabling and simplifying collaborative and reproducible research, 
experimentation, knowledge management and predictive analytics.

Note: the framework is stable but we are now working on
"getting started" guide, documentation and demos:
* https://github.com/ctuning/ck/wiki

License
=======
* Permissive simplified 3-clause BSD license (see LICENSE.txt).

Introduction
============
Collective Knowledge (CK) is a light-weight, portable, modular
and python-based framework, repository, web service and SDK 
to organize, describe, cross-link and share user code, data,
experimental setups and meta information as unified,
cross-linked and reusable components with JSON API 
via standard Git services (such as GITHUB or BitBucket).

Note, that unlike existing centralized web-based services 
that force users to upload all their code and data before
being processed, CK allows you to organize your artifacts 
on your own machine and share them only if needed via
existing private or public repositories or P2P in a 
decentralized manner. 

CK helps researchers assemble experimental setups
(aka workflows or pipelines) from their own or shared components 
just as LEGO(TM) to quickly prototype ideas while automating, 
preserving, distributing, crowdsourcing and reproducing experiments.

CK can also complement existing technology such as Docker 
and VM images as a higher-level light weight container
technology with simple, extensible and unified API.

Furthermore, CK simplifies connection of unified experiments 
to powerful predictive analytics tools such as scikit-learn 
and R (statistical analysis, data mining,  machine learning)
to automate and speed up exploration of multi-dimensional
experimental choices, analysis of results and decision making.

Finally, unified and integrated repository of knowledge allows 
community to systematize unstructured code and data while 
collaboratively validating experimental results, improving models, 
finding missing features and so on.

For example, we use this framework primarily to systematize, 
open and crowdsource our research and experimentation to design
faster, smaller, more power efficient and reliable software and
hardware (by combining autotuning, run-time adaptation, machine
learning and public repository of knowledge):

* [PDF, part I] https://hal.inria.fr/hal-01054763
* [PDF, part II] http://arxiv.org/abs/1506.06256
* [Android app] https://play.google.com/store/apps/details?id=com.collective_mind.node

Further information including getting started guide, 
real usage scenarios, and developer guide is available here:
* https://github.com/ctuning/ck/wiki

Copyright
=========
This software is being developed by the non-profit 
cTuning foundation and its volunteers.

(C)opyright 2014-2015 <NAME>, 
cTuning foundation and contributors

Author
======
<NAME>, http://fursin.net/research

Installation
============
We made a special effort to make CK installation 
as simple as possible - it requires only standard 
Python > 2.6 or 3.x.

Note: since user packages may require extra functionality
such as scipy and scikit-learn, we have been 
successfully using Anaconda python distribution
which includes all scientific packages required 
for predictive analytics in CK.

You can obtain the latest CK version from GITHUB:
 > git clone https://github.com/ctuning/ck.git ck

Note: we plan to add CK to standard Python distributions 
(Linux and Windows) and Debian - if you would like to help, 
please get in touch.

Now, you can simply add CK bin directory to your OS PATH 
environment variable and you are ready to go.

Now you should be able to run CK from command line:
 > ck

If you want to use CK as a standard python module or from 
IPython/IPython notebook, just execute the following command
from the CK root directory (you will need sudo on Linux):
 > python setup.py install

If you have IPython installed, you can then check CK installation 
as following:

 > ipython

 > import ck.kernel as ck

 > ck.test()

 > ck.access('list module')

 > ck.access('load kernel default')

Now, you can use CK to pull various shared repositories and run 
experimental workflows (such as autotuning programs, building
predictive models of performance, analyzing data sets, installing
missing tools, ...).

You can find further details, practical examples, motivation and 
specification in this wiki:

* https://github.com/ctuning/ck/wiki

Questions/comments/discussions?
===============================
Please, use our mailing lists:
* General topics of knowledge preservation, sharing and reuse
  as well as collaborative and reproducible R&D: http://groups.google.com/group/collective-mind
* Software and hardware performance/energy/size/reliability  
  (auto/crowd)-tuning: http://groups.google.com/group/ctuning-discussions

Publications
============
Concepts has been described in the following publications:

* http://arxiv.org/abs/1506.06256
* http://hal.inria.fr/hal-01054763
* http://arxiv.org/abs/1406.4020
* https://hal.inria.fr/inria-00436029

If you found CK useful and/or interesting, you are welcome 
to reference some of the above publications in your reports.

Fun
===
* CK GIT commits as a video: http://cknowledge.org/soft/commits/ck-visualization.mp4

Extra description
=================
CK is intended to enable open and collaborative
research and experimentation particularly in computer
engineering. It can help users preserve, organize, describe,
cross-link and share their code, data and experimental results
as reusable Python-based components with a very simple API (one
function with JSON in and JSON out) and JSON-based schema-free
meta description.

All components can be transparently indexed and searched via
third-party Hadoop-based ElasticSearch, shared via GIT, and
connected together into pipelines (workflows) just as LEGO(R)
to quickly prototype various research ideas, crowdsource
experiments, reproduce past results, perform statistical analysis
and apply predictive analytics (as described
in http://hal.inria.fr/hal-01054763).

CK can be invoked in a unified way via CMD, web service or from
other programs and tools written in practically any language
including C, C++, Fortran, PHP and Java using OpenME event-based
plugin framework developed separately
(https://github.com/ctuning/openme).

CK has been tested with Python 2.x and 3.x on various platforms
including Ubuntu, OpenSUSE, CentOS, Android (partial
support through web services and OpenME) and Windows 7. 

CK is extensively used in the following projects:

* building public repository of knowledge for computer engineering
(http://cknowledge.org/repo , http://c-mind.org/repo)

* implementing our universal multi-dimensional, multi-objective 
plugin-based autotuning approach combined with crowdsourcing, predictive
analytics and run-time adaptation. We support OpenCL, CUDA,
OpenMP, MPI, compiler and any other tuning for performance,
energy, size, reliability, cost and any other metrics across small
kernels/codelets and large applications.
(http://hal.inria.fr/hal-01054763)

* crowdsourcing automatic compiler optimization heuristic tuning
and benchmarking of new architectures across shared resources
such as mobile phones, tablets, cloud services, etc.
(https://play.google.com/store/apps/details?id=com.collective_mind.node)

* supporting artifact evaluation initiatives for major conferences
and journals where all artifacts are shared along with publications
to be validated by the community (http://ctuning.org/event/ae-ppopp2016 ,
http://arxiv.org/abs/1406.4020 )

* enabling open, collaborative and reproducible research and experimentation 
with interactive publications focusing on computer engineering 
(http://cknowledge.org/reproducibility)

Acknowledgments
===============
This project has been partially funded by EU FP7 609491 TETRACOM
project. We would like to thank all volunteers for their valuable
feedback and support.
<commit_msg>more Readme cleaning (simplifying) ...<commit_after>CK - simplifying collaborative and reproducible experimentation,
knowledge management and predictive analytics.

Note: the framework is stable but we are now working on
"getting started" guide, documentation and demos:
* https://github.com/ctuning/ck/wiki

License
=======
* Permissive simplified 3-clause BSD license (see LICENSE.txt).

Introduction
============

Collective Knowledge (CK) is a light-weight, portable, modular
and python-based framework, repository, web service and SDK 
to organize, describe, cross-link and share user code, data,
experimental setups and meta information as unified,
cross-linked and reusable components with JSON API 
via standard Git services (such as GITHUB or BitBucket).

Note, that unlike existing centralized web-based services 
that force users to upload all their code and data before
being processed, CK allows you to organize your artifacts 
and always keep track of them on your own machine 
while sharing them only if needed as P2P or via existing
private and public repositories.

CK helps non-specialists and professionals assemble experimental 
setups (aka workflows or pipelines) from their own or shared components 
just as LEGO(TM) to quickly prototype ideas while automating, 
preserving, distributing, crowdsourcing and reproducing experiments.

CK can also complement existing technology such as Docker 
and VM images as a higher-level light weight container
technology with simple, extensible and unified API.

Furthermore, CK simplifies connection of unified experiments 
to powerful predictive analytics tools such as scikit-learn 
and R (statistical analysis, data mining,  machine learning)
to automate and speed up exploration of multi-dimensional
experimental choices, analysis of results and decision making.

Finally, unified and integrated repository of knowledge allows 
community to systematize unstructured code and data while 
collaboratively validating experimental results, improving models, 
finding missing features and so on.

For example, we use this framework primarily to systematize, 
open and crowdsource our research and experimentation to design
faster, smaller, more power efficient and reliable software and
hardware (by combining autotuning, run-time adaptation, machine
learning and public repository of knowledge):

* [PDF, part I] https://hal.inria.fr/hal-01054763
* [PDF, part II] http://arxiv.org/abs/1506.06256
* [Android app] https://play.google.com/store/apps/details?id=com.collective_mind.node

Further information including getting started guide, 
real usage scenarios, and developer guide is available here:
* https://github.com/ctuning/ck/wiki

Copyright
=========
This software is being developed by the non-profit 
cTuning foundation and its volunteers.

(C)opyright 2014-2015 <NAME>, 
cTuning foundation and contributors

Author
======
<NAME>, http://fursin.net/research

Installation
============
We made a special effort to make CK installation 
as simple as possible - it requires only standard 
Python > 2.6 or 3.x.

Note: since user packages may require extra functionality
such as scipy and scikit-learn, we have been 
successfully using Anaconda python distribution
which includes all scientific packages required 
for predictive analytics in CK.

You can obtain the latest CK version from GITHUB:
 > git clone https://github.com/ctuning/ck.git ck

Note: we plan to add CK to standard Python distributions 
(Linux and Windows) and Debian - if you would like to help, 
please get in touch.

Now, you can simply add CK bin directory to your OS PATH 
environment variable and you are ready to go.

Now you should be able to run CK from command line:
 > ck

If you want to use CK as a standard python module or from 
IPython/IPython notebook, just execute the following command
from the CK root directory (you will need sudo on Linux):
 > python setup.py install

If you have IPython installed, you can then check CK installation 
as following:

 > ipython

 > import ck.kernel as ck

 > ck.test()

 > ck.access('list module')

 > ck.access('load kernel default')

Now, you can use CK to pull various shared repositories and run 
experimental workflows (such as autotuning programs, building
predictive models of performance, analyzing data sets, installing
missing tools, ...).

You can find further details, practical examples, motivation and 
specification in this wiki:

* https://github.com/ctuning/ck/wiki

Questions/comments/discussions?
===============================
Please, use our mailing lists:
* General topics of knowledge preservation, sharing and reuse
  as well as collaborative and reproducible R&D: http://groups.google.com/group/collective-mind
* Software and hardware performance/energy/size/reliability  
  (auto/crowd)-tuning: http://groups.google.com/group/ctuning-discussions

Publications
============
Concepts has been described in the following publications:

* http://arxiv.org/abs/1506.06256
* http://hal.inria.fr/hal-01054763
* http://arxiv.org/abs/1406.4020
* https://hal.inria.fr/inria-00436029

If you found CK useful and/or interesting, you are welcome 
to reference some of the above publications in your reports.

Fun
===
* CK GIT commits as a video: http://cknowledge.org/soft/commits/ck-visualization.mp4

Extra description
=================
CK is intended to enable open and collaborative
research and experimentation particularly in computer
engineering. It can help users preserve, organize, describe,
cross-link and share their code, data and experimental results
as reusable Python-based components with a very simple API (one
function with JSON in and JSON out) and JSON-based schema-free
meta description.

All components can be transparently indexed and searched via
third-party Hadoop-based ElasticSearch, shared via GIT, and
connected together into pipelines (workflows) just as LEGO(R)
to quickly prototype various research ideas, crowdsource
experiments, reproduce past results, perform statistical analysis
and apply predictive analytics (as described
in http://hal.inria.fr/hal-01054763).

CK can be invoked in a unified way via CMD, web service or from
other programs and tools written in practically any language
including C, C++, Fortran, PHP and Java using OpenME event-based
plugin framework developed separately
(https://github.com/ctuning/openme).

CK has been tested with Python 2.x and 3.x on various platforms
including Ubuntu, OpenSUSE, CentOS, Android (partial
support through web services and OpenME) and Windows 7. 

CK is extensively used in the following projects:

* building public repository of knowledge for computer engineering
(http://cknowledge.org/repo , http://c-mind.org/repo)

* implementing our universal multi-dimensional, multi-objective 
plugin-based autotuning approach combined with crowdsourcing, predictive
analytics and run-time adaptation. We support OpenCL, CUDA,
OpenMP, MPI, compiler and any other tuning for performance,
energy, size, reliability, cost and any other metrics across small
kernels/codelets and large applications.
(http://hal.inria.fr/hal-01054763)

* crowdsourcing automatic compiler optimization heuristic tuning
and benchmarking of new architectures across shared resources
such as mobile phones, tablets, cloud services, etc.
(https://play.google.com/store/apps/details?id=com.collective_mind.node)

* supporting artifact evaluation initiatives for major conferences
and journals where all artifacts are shared along with publications
to be validated by the community (http://ctuning.org/event/ae-ppopp2016 ,
http://arxiv.org/abs/1406.4020 )

* enabling open, collaborative and reproducible research and experimentation 
with interactive publications focusing on computer engineering 
(http://cknowledge.org/reproducibility)

Acknowledgments
===============
This project has been partially funded by EU FP7 609491 TETRACOM
project. We would like to thank all volunteers for their valuable
feedback and support.

{'fraction_non_alphanumeric': 0.07088251483089719, 'fraction_numerical': 0.012476301143660938, 'mean_word_length': 4.3022049286640724, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 25, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6897909', 'n_tokens_mistral': 4621, 'n_tokens_neox': 4088, 'n_words': 1934}