<jupyter_start><jupyter_text>[cknowledge.org/ai](http://cknowledge.org/ai): Crowdsourcing benchmarking and optimisation of AI A suite of open-source tools for [collecting knowledge on optimising AI](http://bit.ly/hipeac49-ckdl):* [Android app](https://play.google.com/store/apps/details?id=openscience.crowdsource.video.experiments&hl=en_GB)* [Desktop app](https://github.com/dividiti/ck-crowdsource-dnn-optimization)* [CK-Caffe](https://github.com/dividiti/ck-caffe)* [CK-Caffe2](https://github.com/ctuning/ck-caffe2)* [CK-TensorFlow](https://github.com/ctuning/ck-tensorflow)* [CK-TensorRT](https://github.com/ctuning/ck-tensorrt)* etc.  [PUBLIC] Benchmarking Caffe with OpenBLAS<jupyter_code>platform_id = ''<jupyter_output><empty_output><jupyter_text>Table of Contents 1. [See the code](code) [for developers]1. [Get the data](data) [for developers]1. [See the tables](tables)  1. [All data](df_all)  1. [All execution time data](df_time)  1. [Min execution time per batch](df_min_time_per_batch)  1. [Min execution time per image](df_min_time_per_image)  1. [Max images per second](df_max_images_per_second)  Data wrangling code **NB:** Please ignore this section if you are not interested in re-running or modifying this notebook.  Includes  Standard<jupyter_code>import os
import sys
import json
import re<jupyter_output><empty_output><jupyter_text>Scientific If some of the scientific packages are missing, please install them using:``` pip install jupyter pandas numpy matplotlib```<jupyter_code>import IPython as ip
import pandas as pd
import numpy as np
import matplotlib as mp
print ('IPython version: %s' % ip.__version__)
print ('Pandas version: %s' % pd.__version__)
print ('NumPy version: %s' % np.__version__)
print ('Matplotlib version: %s' % mp.__version__)
from IPython.display import Image
from IPython.core.display import HTML

from IPython.display import display
def display_in_full(df):
    pd.options.display.max_columns = len(df.columns)
    pd.options.display.max_rows = len(df.index)
    display(df)
import matplotlib.pyplot as plt; plt.style.use('classic')
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
%matplotlib inline
default_title = 'Caffe w/ OpenBLAS'
default_ylabel = 'Execution time (ms)'
default_colormap = cm.autumn
default_fontsize = 16
default_figsize = [16, 8]<jupyter_output><empty_output><jupyter_text>Collective Knowledge If CK is not installed, please install it using:``` pip install ck```<jupyter_code>import ck.kernel as ck
print ('CK version: %s' % ck.__version__)<jupyter_output><empty_output><jupyter_text>Access the experimental data<jupyter_code>def get_experimental_results(repo_uoa, tags, time_ms='time_fw_ms'):
    module_uoa = 'experiment'
    r = ck.access({'action':'search', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'tags':tags})
    if r['return']>0:
        print ("Error: %s" % r['error'])
        exit(1)
    experiments = r['lst']
    
    dfs = []
    for experiment in experiments:
        data_uoa = experiment['data_uoa']
        r = ck.access({'action':'list_points', 'repo_uoa':repo_uoa, 'module_uoa':module_uoa, 'data_uoa':data_uoa})
        if r['return']>0:
            print ("Error: %s" % r['error'])
            exit(1)

        # Get (lib_tag, model_tag) from a list of tags that should be available in r['dict']['tags'].
        # Tags include 2 of the 3 irrelevant tags, a model tag and a lib tag.
        # NB: Since it's easier to list all model tags than all lib tags, the latter list is not expicitly specified.
        tags = r['dict']['tags']
        print(tags)
        irrelevant_tags = [ 'explore-batch-size-openblas-threads', 'caffe-time', platform_id ]
        model_tags = [ 'bvlc-alexnet', 'bvlc-googlenet', 'deepscale-squeezenet-1.1', 'deepscale-squeezenet-1.0' ]
        lib_model_tags = [ tag for tag in tags if tag not in irrelevant_tags ]
        model_tags = [ tag for tag in lib_model_tags if tag in model_tags ]
        lib_tags = [ tag for tag in lib_model_tags if tag not in model_tags ]
        if len(lib_tags)==1 and len(model_tags)==1:
            (lib, model) = (lib_tags[0], model_tags[0])
        else:
            continue
        
        for point in r['points']:
            with open(os.path.join(r['path'], 'ckp-%s.0001.json' % point)) as point_file:
                point_data_raw = json.load(point_file)
            characteristics_list = point_data_raw['characteristics_list']
            num_repetitions = len(characteristics_list)
            platform = point_data_raw['features']['platform']['platform']['model']
            batch_size = np.int64(point_data_raw['choices']['env'].get('CK_CAFFE_BATCH_SIZE', -1))
            num_threads = np.int64(point_data_raw['choices']['env'].get('OPENBLAS_NUM_THREADS', -1))
            # Obtain column data.
            data = [
                {
                    # features
                    'platform' : platform,
                    # choices
                    'lib' : lib,
                    'model' : model,
                    'batch_size' : batch_size,
                    'num_threads' : num_threads,
                    # statistical repetition
                    'repetition_id': repetition_id,
                    # runtime characteristics
                    'time (ms)'   : characteristics['run'].get(time_ms, +1e9), # "positive infinity"
                    'per layer info' : characteristics['run'].get('per_layer_info', []),
                    'success?'    : characteristics['run'].get('run_success', 'n/a')
                }
                for (repetition_id, characteristics) in zip(range(num_repetitions), characteristics_list) 
            ]
            # Deal with missing column data (resulting from failed runs).
            if len(data)==1:
                print ('[Warning] Missing data for: lib=%s, model=%s, batch_size=%d' % (lib, model, batch_size))
                data = data * num_repetitions
            # Construct a DataFrame.
            df = pd.DataFrame(data)
            # Set columns and index names.
            df.columns.name = 'run characteristic'
            df.index.name = 'index'
            df = df.set_index([ 'platform', 'lib', 'model', 'num_threads', 'batch_size', 'repetition_id' ])
            # Append to the list of similarly constructed DataFrames.
            dfs.append(df)
    # Concatenate all constructed DataFrames (i.e. stack on top of each other).
    result = pd.concat(dfs)
    return result.sort_index()<jupyter_output><empty_output><jupyter_text>Plot images per second against the batch size and the number of threads<jupyter_code>def plot_trisurf(df_model, x_col, y_col, z_col, x_label=None, y_label=None, z_label=None, title=None):
    x = df_model[x_col]
    y = df_model[y_col]
    z = df_model[z_col]
    
    if x_label == None: x_label = x_col
    if y_label == None: y_label = y_col
    if z_label == None: z_label = z_col
        
    x_ticks = x.unique()
    y_ticks = y.unique()
    
    fig = plt.figure(figsize=(24, 12), dpi=600)
    ax = fig.add_subplot(111, projection='3d')
    trisurf = ax.plot_trisurf(x, y, z, cmap=cm.autumn_r, linewidth=0.2, antialiased=True, shade=True)
    ax.set_xlabel(x_label); ax.set_xticks(x_ticks); ax.set_xlim3d(x_ticks.max(), x_ticks.min())
    ax.set_ylabel(y_label); ax.set_yticks(y_ticks); ax.set_ylim3d(y_ticks.min(), y_ticks.max())
    ax.set_zlabel(z_label); ax.set_zlim3d(z.min(), z.max())
    ax.set_title(title, fontsize=20)
    fig.colorbar(trisurf, shrink=0.5, aspect=10)
    return fig<jupyter_output><empty_output><jupyter_text>Get the experimental data **NB:** Please ignore this section if you are not interested in re-running or modifying this notebook.  The Caffe experimental data was collected on the experimental platform (after installing all Caffe libraries and models of interest) as follows:```$ cd `ck find ck-caffe:script:explore-batch-size-openblas-threads`$ python explore-batch-size-openblas-threads-benchmarking.py```It can be downloaded and registered with CK e.g. as follows:```$ ck add repo:ck-caffe-openblas-threads --url=https://github.com/dividiti/ck-caffe-samsung-chromebook2.git```  Tables  All data<jupyter_code>df_all = get_experimental_results(
    repo_uoa='ck-caffe-openblas-threads ',
    tags='explore-batch-size-openblas-threads') \
    .reset_index(['platform', 'lib'], drop=True)
display_in_full(df_all)<jupyter_output><empty_output><jupyter_text>All execution time data indexed by repetitions<jupyter_code>df_time = df_all['time (ms)'].unstack(df_all.index.names[:-1])
display_in_full(df_time)<jupyter_output><empty_output><jupyter_text>Min execution time per batch<jupyter_code>df_min_time_per_batch = df_time.describe().loc['min'].unstack(level='batch_size')
display_in_full(df_min_time_per_batch)<jupyter_output><empty_output><jupyter_text>Min execution time per image<jupyter_code>batch_sizes = df_min_time_per_batch.columns.tolist()
df_min_time_per_image = df_min_time_per_batch / batch_sizes
display_in_full(df_min_time_per_image)<jupyter_output><empty_output><jupyter_text>Max images per second<jupyter_code>df_min_seconds_per_image = 1e-3 * df_min_time_per_image
df_max_images_per_second = 1 / df_min_seconds_per_image
display_in_full(df_max_images_per_second)
# A workaround to make "Delaunay triangulation calculation" to work with a single batch size:
# insert a duplicate column using the negative value of the batch size.
if df_max_images_per_second.columns.size == 1:
    batch_size = df_max_images_per_second.columns[0]
    df_max_images_per_second[-batch_size] = df_max_images_per_second.values
    display_in_full(df_max_images_per_second)
models = df_max_images_per_second.index.get_level_values('model').unique()
for model in models:
    df_model = df_max_images_per_second \
        .loc[model] \
        .unstack() \
        .reset_index() \
        .rename(columns={0 : 'time (ms)'}) \
        .dropna() \
        .sort_values(by='batch_size', ascending=False)
    fig = plot_trisurf(df_model, title=model,
                 x_col='num_threads', x_label='Number of threads',
                 y_col='batch_size', y_label='Batch size',
                 z_col='time (ms)', z_label='Images per second')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12054445750097924, 'fraction_numerical': 0.007442224833529181, 'mean_word_length': 3.659215328467153, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '86539', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3473, 'n_tokens_neox': 3369, 'n_words': 753, 'path': 'script/explore-batch-size-openblas-threads/explore-batch-size-openblas-threads-analysis.ipynb', 'repo_name': 'dividiti/ck-caffe', 'size': '10212'}