<commit_before>Collective Knowledge repository for evaluating and optimising performance of TensorFlow
=======================================================================================

[![logo](https://github.com/ctuning/ck-guide-images/blob/master/logo-powered-by-ck.png)](http://cKnowledge.org)
[![logo](https://github.com/ctuning/ck-guide-images/blob/master/logo-validated-by-the-community-simple.png)](http://cTuning.org)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

# Introduction

We need to have easily customizable TensorFlow builds 
via JSON API to be able to plug it to our framework 
for collaborative benchmarking and optimization of workloads 
across diverse inputs and hardware provided by volunteers 
(see [cKnowledge.org/ai](http://cKnowledge.org/ai), 
[live repo](http://cKnowledge.org/repo)
and papers [1](https://arxiv.org/abs/1506.06256), 
[2](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:maZDTaKrznsC), 
[3](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:LkGwnXOMwfcC) 
for more details).

![](http://cKnowledge.org/images/ai-cloud-resize.png)

# Contributors

* <NAME>
* <NAME>, [dividiti](http://dividiti.com) / [cTuning foundation](http://ctuning.org)
* <NAME>, [dividiti](http://dividiti.com)

# License
* [BSD](https://github.com/dividiti/ck-caffe/blob/master/LICENSE) (3 clause)

# Status
Under development.

# Installing CK-TensorFlow dependencies

## Installing CK

### Ubuntu
```
$ sudo pip install ck
```

### Windows
```
$ pip install ck
```

## Installing CK-TensorFlow with all its dependencies
```
$ ck pull repo:ck-tensorflow
```

## Installing TensorFlow dependencies on Ubuntu

### Java
```
$ sudo apt install openjdk-8-jdk-headless
```
**NB:** Installation fails with `openjdk-9-jdk-headless` (cf. [this](https://github.com/bazelbuild/bazel/issues/1456)).

### CUDA, cuDNN (GPU version only)

If you want to use the GPU, please install CUDA toolkit >= v7.0 and cuDNN >= v2.
If you want to use the GPU and pip, please install CUDA toolkit >= v8.0 and cuDNN >= v5.

- Check if you have a [CUDA-enabled GPU](https://developer.nvidia.com/cuda-gpus).

- [Download and install CUDA](https://developer.nvidia.com/cuda-downloads).

- [Download and install cuDNN](https://developer.nvidia.com/rdp/cudnn-download) (requires registration).

- [More detailed instructions](https://www.tensorflow.org/versions/r0.10/get_started/os_setup.html#optional-install-cuda-gpus-on-linux).

### For installation via pip

For python 2.7, install:
```
$ sudo apt-get install  \
    python-dev \
    python-pip \
    python-setuptools

$ sudo pip install wheel
```

For python 3.5, install:

```
$ sudo apt-get install  \
    python3-dev \
    python3-pip \
    python3-setuptools
$ sudo pip3 install wheel
```

### Python
```
$ sudo pip install \
    protobuf \
    easydict \
    joblib
```<commit_msg>README.md: document deps (Python image, opencv).<commit_after>Collective Knowledge repository for evaluating and optimising performance of TensorFlow
=======================================================================================

[![logo](https://github.com/ctuning/ck-guide-images/blob/master/logo-powered-by-ck.png)](http://cKnowledge.org)
[![logo](https://github.com/ctuning/ck-guide-images/blob/master/logo-validated-by-the-community-simple.png)](http://cTuning.org)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

# Introduction

We need to have easily customizable TensorFlow builds 
via JSON API to be able to plug it to our framework 
for collaborative benchmarking and optimization of workloads 
across diverse inputs and hardware provided by volunteers 
(see [cKnowledge.org/ai](http://cKnowledge.org/ai), 
[live repo](http://cKnowledge.org/repo)
and papers [1](https://arxiv.org/abs/1506.06256), 
[2](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:maZDTaKrznsC), 
[3](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IwcnpkwAAAAJ&citation_for_view=IwcnpkwAAAAJ:LkGwnXOMwfcC) 
for more details).

![](http://cKnowledge.org/images/ai-cloud-resize.png)

# Contributors

* <NAME>
* <NAME>, [dividiti](http://dividiti.com) / [cTuning foundation](http://ctuning.org)
* <NAME>, [dividiti](http://dividiti.com)

# License
* [BSD](https://github.com/dividiti/ck-caffe/blob/master/LICENSE) (3 clause)

# Status
Under development.

# Installing CK-TensorFlow dependencies

## Installing CK

### Ubuntu
```
$ sudo pip install ck
```

### Windows
```
$ pip install ck
```

## Installing CK-TensorFlow with all its dependencies
```
$ ck pull repo:ck-tensorflow
```

## Installing TensorFlow dependencies on Ubuntu

### Java
```
$ sudo apt install openjdk-8-jdk-headless
```
**NB:** Installation fails with `openjdk-9-jdk-headless` (cf. [this](https://github.com/bazelbuild/bazel/issues/1456)).

### CUDA, cuDNN (GPU version only)

If you want to use the GPU, please install CUDA toolkit >= v7.0 and cuDNN >= v2.
If you want to use the GPU and pip, please install CUDA toolkit >= v8.0 and cuDNN >= v5.

- Check if you have a [CUDA-enabled GPU](https://developer.nvidia.com/cuda-gpus).

- [Download and install CUDA](https://developer.nvidia.com/cuda-downloads).

- [Download and install cuDNN](https://developer.nvidia.com/rdp/cudnn-download) (requires registration).

- [More detailed instructions](https://www.tensorflow.org/versions/r0.10/get_started/os_setup.html#optional-install-cuda-gpus-on-linux).

### For installation via pip

For python 2.7, install:
```
$ sudo apt-get install  \
    python-dev \
    python-pip \
    python-setuptools

$ sudo pip install wheel
```

For python 3.5, install:

```
$ sudo apt-get install  \
    python3-dev \
    python3-pip \
    python3-setuptools
$ sudo pip3 install wheel
```

### Python
```
$ sudo pip install \
    protobuf \
    easydict \
    joblib \
    image
```

```
$ sudo apt install \
    python-opencv
```
{'fraction_non_alphanumeric': 0.17760492626802787, 'fraction_numerical': 0.012963863231242911, 'mean_word_length': 4.580470162748644, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 17, 'https://': 26, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2690360', 'n_tokens_mistral': 2262, 'n_tokens_neox': 2078, 'n_words': 436}