<jupyter_start><jupyter_text>Tutorial from "*Visualization of Multiscale Ring Formation in a Rotary Kiln*"**License:** MIT License**Authors:** <NAME>, <NAME>, <NAME>, <NAME>, <NAME> and <NAME>**Last Revision:** March 5, 2021  Instructions- This is a Google Colab notebook and it will provide the Python environment for this tutorial. - There are code cells and text cells. To run a code cell click on it then hold shift and press enter. Code cells are to be run in order as inputs to later cells are dependent on outputs of prior cells.- The third part of this tutorial provides instructions for loading your own operating data.  1. Import required packages<jupyter_code>import numpy as np
import pandas as pd
import datetime
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets<jupyter_output><empty_output><jupyter_text>2. Generate synthetic dataThe synthetic data is created to resemble the format of data extracted from a process historian.<jupyter_code>dti = pd.date_range("2018-07-01", periods=20000, freq="H")
t = np.arange(0,len(dti))
# Mean temperatures in deg F
y = [1410, 1379, 1360,  1327, 1232, 1188, 1127, 1090, 1077, 1041, 1021, 1004, 993,
         987, 918, 904, 882, 860, 827, 770]
y = [(z-32)*(5/9) for z in y] # Convert to Celsius

# Shell measurement position
x = np.arange(5,105,5)

df = pd.DataFrame(index = dti, columns = x)
for i in range(len(y)):
  if i <= 5:
    df[x[i]] = 45*np.sin(t*0.05)+y[i]+np.random.normal(0, 10,len(t))
  elif i <= 15:
    df[x[i]] = 30*np.sin(t*0.01)+y[i]+np.random.normal(0, 7,len(t))
  else:
    df[x[i]] = 35*np.sin(t*0.002)+y[i]+np.random.normal(0, 5,len(t))

df['PV_A'] = np.random.normal(12, 2.3, len(t))
df['PV_B'] = np.random.normal(70, 12, len(t))

descript = pd.DataFrame({5:'KST_5', 10:'KST_10', 15:'KST_15', 20:'KST_20', 25:'KST_25', 30:'KST_30', 35:'KST_35', 40:'KST_40', 45:'KST_45', 50:'KST_50', 55:'KST_55', 60:'KST_60', 
                        65:'KST_65', 70:'KST_70', 75:'KST_75', 80:'KST_80', 85:'KST_85', 90:'KST_90', 95:'KST_96', 100:'KST_100', 'PV_A':'Feed', 'PV_B':'Fuel'}, index =['descript']) 

tags = pd.DataFrame({5:'TZ410.meas', 10:'TZ411.meas', 15:'TZ412.meas', 20:'TZ413.meas', 25:'TZ414.meas', 30:'TZ415.meas', 35:'TZ416.meas', 40:'TZ417.meas', 45:'TZ418.meas', 50:'TZ419.meas', 55:'TZ420.meas', 60:'TZ421.meas', 
                        65:'TZ422.meas', 70:'TZ423.meas', 75:'TZ424.meas', 80:'TZ425.meas', 85:'TZ426.meas', 90:'TZ427.meas', 95:'TZ428.meas', 100:'TZ429.meas', 'PV_A':'TZ101.meas', 'PV_B':'TZ102.meas'}, index =['tags']) 

units = pd.DataFrame({5:'C', 10:'C', 15:'C', 20:'C', 25:'C', 30:'C', 35:'C', 40:'C', 45:'C', 50:'C', 55:'C', 60:'C', 
                        65:'C', 70:'C', 75:'C', 80:'C', 85:'C', 90:'C', 95:'C', 100:'C', 'PV_A':'T/D', 'PV_B':'m3/h'}, index =['units']) 

df = pd.concat([descript,tags,units,df[:]])<jupyter_output><empty_output><jupyter_text>Run the following cell to see the format of the synthetic data. The next section describes how to load your own data but it is important to make sure your data (after it is loaded) has the same format as the synthetic data shown below in order for the subsequent code to run.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>3. Saving and loading data. The commands below are used to save the dataframe titled df as a csv file and to load a csv file into the Python environment as a dataframe titled df. Since this tutorial will proceed with the synthetic data this code has been commented out. Find more information on using [to_csv](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) and [read_csv](https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.read_csv.htmlpandas.read_csv).To apply this visualization to your own data you could read a csv file containing your data and compare it to the format of df above. Note that extra steps may be necessary to load and save data using Google Colab. This can be avoided by running a Python environment with Jupyter Lab on your local machine.<jupyter_code>#df.to_csv(index=False)

#df = pd.read_csv('KilnData.csv')<jupyter_output><empty_output><jupyter_text>4. Interpolating the KST data<jupyter_code>d_shell = df.iloc[3:,0:-2]

d_tags = df.iloc[0:3,:]

d_shell = d_shell.T.astype('float')
index = np.arange(x.min(),x.max()+1)
df_int = pd.DataFrame(index=index)
for colname, col in d_shell.iteritems():
  df_int[colname] = np.interp(index, d_shell.index, col)

df_int = df_int.astype(float)<jupyter_output><empty_output><jupyter_text>The above code has performed piecewise linear interpolation along the KST profile in unit (e.g. 1 meter) increments. In this case x was equispaced but it does not need to be. The resulting interpolated dataframe is shown below.<jupyter_code>df_int<jupyter_output><empty_output><jupyter_text>5. Generate interactive spatiotemporal heatmap<jupyter_code>def Vis():
    y_slider = widgets.IntSlider(min=d_shell.columns[0].year,max=d_shell.columns[-1].year,value=d_shell.columns[0].year, step=1, description='year')
    m_slider = widgets.IntSlider(min=1,max=12,value=d_shell.columns[0].month, step=1, description='month')
    w_slider = widgets.IntSlider(min=12,max=len(d_shell.T),value=48, step=12, description='window size')
    d_slider = widgets.IntSlider(min=1,max=31,value=1, step=1, description='day')
    h_slider = widgets.IntSlider(min=0,max=24,value=1, step=1, description='hour')

    PV = widgets.Dropdown(options=d_tags.columns.tolist(), description = 'process variable')
    
    ui0 = widgets.VBox([y_slider,m_slider])
    ui1 = widgets.VBox([d_slider,h_slider])
    ui2 = widgets.VBox([w_slider,PV])
    ui3 = widgets.HBox([ui0,ui1,ui2])

    out = widgets.interactive_output(heatmap, {'y_slider':y_slider,'m_slider':m_slider,'d_slider':d_slider,'h_slider':h_slider,'w_slider':w_slider,'PV':PV})
    display(ui3,out)
def heatmap(y_slider, m_slider, d_slider, h_slider, w_slider, PV):
    
    DF = df_int

    # This code accounts for the manual selector for year/month/day/hour
    dt = pd.DataFrame({'year': y_slider,
                   'month': m_slider,
                   'day': d_slider,
                   'hour': h_slider}, index=[0])
    dt = pd.to_datetime(dt)
    # We find the row # of df that matches the value of dt (from the user)
    j = DF.T.index.get_loc(dt.iloc[0])

    SMALL_SIZE = 18     #Font sizes
    MEDIUM_SIZE = 20
    BIGGER_SIZE = 20
    #plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
    plt.rc('axes', titlesize=MEDIUM_SIZE)     # fontsize of the axes title
    plt.rc('axes', labelsize= MEDIUM_SIZE)    # fontsize of the x and y labels
    plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
    plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
    plt.rc('legend', fontsize=MEDIUM_SIZE)    # legend fontsize
    plt.rc('figure', titlesize=BIGGER_SIZE)  # fontsize of the figure title
    
    
    fig, ax = plt.subplots(2,figsize=(30,15))
    plt.style.use("ggplot")
    im = ax[0].imshow(DF.iloc[:,j:j+w_slider],aspect = 'auto', cmap='magma', vmin= d_shell.min().min() - 10, vmax = d_shell.max().max() + 10)
    cbar = ax[0].figure.colorbar(im, ax=ax)#[0])     # Create colorbar
    cbar.set_label(label="Temperature ($^\circ$C)", rotation=-90, va="bottom", fontsize=18)

    # THIS CODE IS FOR THE HEATMAP
    y_m = [z*0.3048 for z in DF.index]
    ax[0].tick_params(axis='both', direction='out')
    ax[0].set_xticks([0,int((w_slider)/4),int((w_slider)/2),int((w_slider)*3/4),w_slider-1])
    ax[0].set_xticklabels([DF.columns[j],DF.columns[j+int((w_slider)/4)],DF.columns[j+int((w_slider)/2)],DF.columns[j+int((w_slider)*3/4)],DF.columns[j+w_slider-1]])    
    ax[0].set_xlabel("Date")
    ax[0].set_yticks([DF.index[0], DF.index[0+int(np.round(len(df_int)/4,0))], DF.index[0+2*int(np.round(len(df_int)/4,0))], DF.index[0+3*int(np.round(len(df_int)/4,0))], DF.index[-1]]) 
    ax[0].set_yticklabels([DF.index[0], DF.index[0+int(np.round(len(df_int)/4,0))], DF.index[0+2*int(np.round(len(df_int)/4,0))], DF.index[0+3*int(np.round(len(df_int)/4,0))], DF.index[-1]])
    ax[0].set_ylabel("Distance from firing end of kiln (m)")
    
    
    ln1, = ax[1].plot(pd.Series(data=df[PV].iloc[3:].values[j:j+w_slider]), label = str(PV) + '\n' + d_tags[PV].loc['tags'])
    ax[1].autoscale(enable=True, axis='x', tight=True)
    ax[1].legend(handles=[ln1],loc="upper right")
    ax[1].set_xlabel("Date")
    ax[1].set_xticks([0,int((w_slider)/4),int((w_slider)/2),int((w_slider)*3/4),w_slider-1])
    ax[1].set_xticklabels([DF.columns[j],DF.columns[j+int((w_slider)/4)],DF.columns[j+int((w_slider)/2)],DF.columns[j+int((w_slider)*3/4)],DF.columns[j+w_slider-1]])
    ax[1].set_ylabel(str(PV) + ' (' + d_tags[PV].loc['units'] + ')')
    
    return plt.show()
Vis()<jupyter_output><empty_output><jupyter_text>6. ConclusionThere are still some minor improvements and fine-tuning that can be performed but hopefully this helps demonstrate how the interactive spatiotemporal heatmap can be created. Although we designed this tool to assist with studying kiln shell temperature profiles, these open-source tools are highly customizable and useful for many process monitoring applications. Thank you for your interest in this data visualization tutorial. If you would like to connect, or have any questions, please reach out through email or social media.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16396206533192834, 'fraction_numerical': 0.053108535300316125, 'mean_word_length': 4.847812692544671, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '539925', 'license': 'MIT', 'n_tokens_mistral': 3881, 'n_tokens_neox': 3470, 'n_words': 827, 'path': 'KilnVisualizationTutorial.ipynb', 'repo_name': 'LeeRippon/KilnVisual', 'size': '9533'}