<jupyter_start><jupyter_text>LoRa Data Analysis - ADR vs. UCB We first declare a fixed parameters.Those parameters are not changed during the experiments.Fixed communication parameters are listed below:- Code Rate: 4/5- Frequency: 866.1 MHz- Bandwidth: 125 kHzEnd nodes:- were sending different types of uplink messages.- were sending a single message each 2 minutes.- used an upper confidence bound algorithm (UCB) for communication parameters selection.Access points:- only a single access point was used- capture effect was also considered Initial declaration<jupyter_code>%matplotlib inline
import pandas as pd # import pandas
import numpy as np # import numpy
import matplotlib as mpl # import matplotlib
import matplotlib.pyplot as plt # import plotting module
import statistics
import math
import base64
from IPython.display import set_matplotlib_formats # module for svg export

# Set output format for png figures
output_format = 'png'
set_matplotlib_formats(output_format) # set export to svg file

adr_uplink_file = 'adr_uplink_messages.csv'
ucb_uplink_file = 'ucb_uplink_messages.csv'<jupyter_output><empty_output><jupyter_text>Analysis of Uplink Messages We read a csv file with uplink messages<jupyter_code>adr_uplink_messages = pd.read_csv(adr_uplink_file, delimiter=',')
ucb_uplink_messages = pd.read_csv(ucb_uplink_file, delimiter=',')<jupyter_output><empty_output><jupyter_text>Let us have a look at various columns that are present and can be evaluated.<jupyter_code>adr_uplink_messages.head()
ucb_uplink_messages.head()<jupyter_output><empty_output><jupyter_text>Remove all columns that have fixed values or there is no point in their analysis.<jupyter_code>try:
    del adr_uplink_messages['id']
    del adr_uplink_messages['msg_group_number']
    del adr_uplink_messages['is_primary']
    del adr_uplink_messages['coderate']
    del adr_uplink_messages['bandwidth']
    del adr_uplink_messages['receive_time']
except KeyError:
    print('Columns have already been removed')
try:
    del ucb_uplink_messages['id']
    del ucb_uplink_messages['msg_group_number']
    del ucb_uplink_messages['is_primary']
    del ucb_uplink_messages['coderate']
    del ucb_uplink_messages['bandwidth']
    del ucb_uplink_messages['receive_time']
except KeyError:
    print('Columns have already been removed')<jupyter_output><empty_output><jupyter_text>Payload Length<jupyter_code>adr_uplink_messages['payload_len'] = adr_uplink_messages.app_data.apply(len)
ucb_uplink_messages['payload_len'] = ucb_uplink_messages.app_data.apply(len)
adr_payload_len = round(statistics.mean(adr_uplink_messages.payload_len))
ucb_payload_len = round(statistics.mean(ucb_uplink_messages.payload_len))

print(f'Mean value of payload length for ADR is {adr_payload_len} B')
print(f'Mean value of payload length for UCB is {ucb_payload_len} B')<jupyter_output>Mean value of payload length for ADR is 47 B
Mean value of payload length for UCB is 43 B<jupyter_text>Spreading Factor<jupyter_code>sf1 = adr_uplink_messages.spf.value_counts()
sf2 = ucb_uplink_messages.spf.value_counts()

diff = abs(sf1 - sf2)
diff.fillna(0)
sf_adr = [sf1, diff]
sf_adr = pd.concat(sf_adr, axis=1, sort=False).sum(axis=1)
sf_adr.sort_index(ascending=False, inplace=True)

diff = abs(sf2 - sf1)
diff.fillna(0)
sf_ucb = [sf2, diff]
sf_ucb = pd.concat(sf_ucb, axis=1, sort=False).sum(axis=1)
sf_ucb.sort_index(ascending=False, inplace=True)
# Create a grouped bar chart, with job as the x-axis
# and gender as the variable we're grouping on so there
# are two bars per job.
fig, ax = plt.subplots(figsize=(10, 4))

# Define bar width. We need this to offset the second bar.
bar_width = 0.3
index = np.arange(len(sf_adr))

ax.bar(index, sf_adr, width=bar_width, color='green', label = 'ADR')
# Same thing, but offset the x.
ax.bar(index + bar_width, sf_ucb, width=bar_width, color='blue', label = 'UCB')

# Fix the x-axes.
ax.set_xticks(index + bar_width / 2)
ax.set_xticklabels(sf_ucb.index)

# Add legend.
ax.legend()

# Axis styling.
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['bottom'].set_color('#DDDDDD')
ax.tick_params(bottom=False, left=False)
ax.set_axisbelow(True)
ax.yaxis.grid(True, color='#EEEEEE')
ax.xaxis.grid(False)

# Add axis and chart labels.
ax.set_xlabel('Spreading Factor', labelpad=15)
ax.set_ylabel('Number of Messages', labelpad=15)
ax.set_title('Utilization of Spreading Factor', pad=15)

fig.tight_layout()

# For each bar in the chart, add a text label.
for bar in ax.patches:
  # The text annotation for each bar should be its height.
  bar_value = round(bar.get_height())
  # Format the text with commas to separate thousands. You can do
  # any type of formatting here though.
  text = f'{bar_value:,}'
  # This will give the middle of each bar on the x-axis.
  text_x = bar.get_x() + bar.get_width() / 2
  # get_y() is where the bar starts so we add the height to it.
  text_y = bar.get_y() + bar_value
  # If we want the text to be the same color as the bar, we can
  # get the color like so:
  bar_color = bar.get_facecolor()
  # If you want a consistent color, you can just set it as a constant, e.g. #222222
  ax.text(text_x, text_y, text, ha='center', va='bottom', color=bar_color, size=10)
    
fig.savefig(f'adr-ucb-sf.{output_format}', dpi=300)<jupyter_output><empty_output><jupyter_text>All nodes used the same frequency to increase a probability of collisions. We have only a single Access Point.  Analysis of End Nodes Analysis of certain aspects (active time, sleep time and collisions) of end devices.<jupyter_code>adr_unique_ens = adr_uplink_messages.node_id.nunique()
ucb_unique_ens = ucb_uplink_messages.node_id.nunique()

print(f'Number of end nodes participating for ADR is {adr_unique_ens}.')
print(f'Number of end nodes participating for UCB is {ucb_unique_ens}.')
adr_end_nodes = pd.read_csv(f'adr_end_nodes.csv', delimiter=',')
ucb_end_nodes = pd.read_csv(f'ucb_end_nodes.csv', delimiter=',')<jupyter_output><empty_output><jupyter_text>Collision Ratio<jupyter_code>adr_collisions = adr_end_nodes.collisions
ucb_collisions = ucb_end_nodes.collisions

adr_max_collisions = max(adr_end_nodes.collisions)
adr_min_collisions = min(adr_end_nodes.collisions)
ucb_max_collisions = max(ucb_end_nodes.collisions)
ucb_min_collisions = min(ucb_end_nodes.collisions)

max_collisions = max(adr_max_collisions, ucb_max_collisions)
min_collisions = min(adr_min_collisions, ucb_min_collisions)
range_collisions = max_collisions - min_collisions
buckets = 8
increment = range_collisions / buckets

print(f'Max number of collisions for ADR: {adr_max_collisions}')
print(f'Min number of collisions for ADR: {adr_min_collisions}')
print(f'Max number of collisions for UCB: {ucb_max_collisions}')
print(f'Min number of collisions for UCB: {ucb_min_collisions}')
fig, ax = plt.subplots(figsize=(10, 4))

bar_width = 0.4
index = np.arange(buckets)

bins = []

for i in range(buckets + 1):
    bins.append(round(min_collisions + i * increment))

out_adr = pd.cut(adr_collisions, bins=bins)
adr_values = out_adr.value_counts(sort=False).iloc[::-1]

out_ucb = pd.cut(ucb_collisions, bins=bins)
ucb_values = out_ucb.value_counts(sort=False).iloc[::-1]

ax.bar(index, adr_values, width=bar_width, color='green', label='ADR')
ax.bar(index + bar_width, ucb_values, width=bar_width, color='blue', label='UCB')

ax.set_xticks(index + bar_width / 2)
ax.set_xticklabels(adr_values.index, rotation=45)

ax.legend()

ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['bottom'].set_color('#DDDDDD')
ax.tick_params(bottom=False, left=False)
ax.set_axisbelow(True)
ax.yaxis.grid(True, color='#EEEEEE')
ax.xaxis.grid(False)

ax.set_xlabel('Number of Collisions', labelpad=15)
ax.set_ylabel('Number of Devices', labelpad=15)
ax.set_title('Collision Rate', pad=15)

fig.tight_layout()

for bar in ax.patches:
  bar_value = round(bar.get_height())
  text = f'{bar_value:,}'
  text_x = bar.get_x() + bar.get_width() / 2
  text_y = bar.get_y() + bar_value
  bar_color = bar.get_facecolor()
  ax.text(text_x, text_y, text, ha='center', va='bottom', color=bar_color, size=10)
    
fig.savefig(f'adr-ucb-collisions.{output_format}', dpi=300)
print(f'Mean collision number for ADR is {round(statistics.mean(adr_collisions))}')
print(f'Mean collision number for UCB is {round(statistics.mean(ucb_collisions))}')<jupyter_output>Mean collision number for ADR is 212
Mean collision number for UCB is 4<jupyter_text>Ration between active time and total nodes uptime<jupyter_code>adr_energy = (adr_end_nodes.active_time / adr_end_nodes.uptime)
adr_active_time = round(statistics.mean(adr_energy) * 100, 2)
ucb_energy = (ucb_end_nodes.active_time / ucb_end_nodes.uptime)
ucb_active_time = round(statistics.mean(ucb_energy) * 100, 2)

print(f'ADR nodes spent {adr_active_time}% of their uptime in active mode')
print(f'UCB nodes spent {ucb_active_time}% of their uptime in active mode')<jupyter_output>ADR nodes spent 23.15% of their uptime in active mode
UCB nodes spent 2.5% of their uptime in active mode<jupyter_text>Packet Delivery Ratio (PDR) Evaluation of packet delivery ratio for end nodes. Add message count from uplink data and collisions.<jupyter_code>adr_data = adr_uplink_messages.node_id.value_counts()
adr_nodes = pd.DataFrame({}, columns = ['dev_id', 'collisions', 'messages'])

collisions = []
messages = []
dev_id = []

for index,value in adr_data.items():
    dev_id.append(index)
    collision_count = adr_end_nodes.loc[adr_end_nodes.dev_id == index].collisions.values[0]
    collisions.append(collision_count)
    messages.append(value + collision_count)

adr_nodes['dev_id'] = dev_id
adr_nodes['collisions'] = collisions
adr_nodes['messages'] = messages

# Make the same for another algorithm
ucb_data = ucb_uplink_messages.node_id.value_counts()
ucb_nodes = pd.DataFrame({}, columns = ['dev_id', 'collisions', 'messages'])

collisions = []
messages = []
dev_id = []

for index,value in ucb_data.items():
    dev_id.append(index)
    collision_count = ucb_end_nodes.loc[ucb_end_nodes.dev_id == index].collisions.values[0]
    collisions.append(collision_count)
    messages.append(value + collision_count)

ucb_nodes['dev_id'] = dev_id
ucb_nodes['collisions'] = collisions
ucb_nodes['messages'] = messages
adr_nodes['pdr'] = round((1 - (adr_nodes.collisions / adr_nodes.messages))*100, 2)
adr_mean_pdr = round(statistics.mean(adr_nodes.pdr), 2)

ucb_nodes['pdr'] = round((1 - (ucb_nodes.collisions / ucb_nodes.messages))*100, 2)
ucb_mean_pdr = round(statistics.mean(ucb_nodes.pdr), 2)

print(f'Mean value of PDR for ADR is {adr_mean_pdr}%')
print(f'Mean value of PDR for UCB is {ucb_mean_pdr}%')
adr_max_pdr = max(adr_nodes.pdr)
adr_min_pdr = min(adr_nodes.pdr)
ucb_max_pdr = max(ucb_nodes.pdr)
ucb_min_pdr = min(ucb_nodes.pdr)

max_pdr = max(adr_max_pdr, ucb_max_pdr)
min_pdr = min(adr_min_pdr, ucb_min_pdr)
range_pdr = max_pdr - min_pdr
buckets = 8
increment = math.ceil(range_pdr / buckets)

print(f'Max PDR for ADR: {adr_max_pdr}%')
print(f'Min PDR for ADR: {adr_min_pdr}%')
print(f'Max PDR for UCB: {ucb_max_pdr}%')
print(f'Min PDR for UCB: {ucb_min_pdr}%')
fig, ax = plt.subplots(figsize=(10, 4))
bins = []
bar_width = 0.4
index = np.arange(buckets)

for i in range(buckets + 1):
    bins.append(round(min_pdr + i * increment))

out_adr = pd.cut(adr_nodes.pdr, bins=bins)
adr_values = out_adr.value_counts(sort=False).iloc[::-1]

out_ucb = pd.cut(ucb_nodes.pdr, bins=bins)
ucb_values = out_ucb.value_counts(sort=False).iloc[::-1]

ax.bar(index, adr_values, width=bar_width, color='green', label='ADR')
ax.bar(index + bar_width, ucb_values, width=bar_width, color='blue', label='UCB')

ax.set_xticks(index + bar_width / 2)
ax.set_xticklabels(adr_values.index, rotation=45)

ax.legend()

ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['bottom'].set_color('#DDDDDD')
ax.tick_params(bottom=False, left=False)
ax.set_axisbelow(True)
ax.yaxis.grid(True, color='#EEEEEE')
ax.xaxis.grid(False)

ax.set_xlabel('Packet Delivery Ratio [%]', labelpad=15)
ax.set_ylabel('Number of Devices', labelpad=15)
ax.set_title('Comparison of PDR', pad=15)

fig.tight_layout()

for bar in ax.patches:
  bar_value = round(bar.get_height())
  text = f'{bar_value:,}'
  text_x = bar.get_x() + bar.get_width() / 2
  text_y = bar.get_y() + bar_value
  bar_color = bar.get_facecolor()
  ax.text(text_x, text_y, text, ha='center', va='bottom', color=bar_color, size=10)

fig.savefig(f'adr-ucb-pdr.{output_format}', dpi=300)<jupyter_output><empty_output><jupyter_text>Path of Each End Node Data about position are encoded as base64. Decode base64, extract position and save the results to original data frame.<jupyter_code># Extracting X and Y coordinates from payload
adr_app_data = adr_uplink_messages.app_data.apply(base64.b64decode)
adr_app_data = adr_app_data.astype(str)
adr_app_data = adr_app_data.str.split(',')

df = pd.DataFrame({}, columns = ['node_id', 'x', 'y'])
x = []
y = []

for row in adr_app_data:
    x.append(round(float(row[1].split('\'')[0]), 2) / 1000)
    y.append(round(float(row[0].split('\'')[1]), 2) / 1000)

adr_uplink_messages['x'] = x
adr_uplink_messages['y'] = y

# Same for the second algorithm
ucb_app_data = ucb_uplink_messages.app_data.apply(base64.b64decode)
ucb_app_data = ucb_app_data.astype(str)
ucb_app_data = ucb_app_data.str.split(',')

df = pd.DataFrame({}, columns = ['node_id', 'x', 'y'])
x = []
y = []

for row in ucb_app_data:
    x.append(round(float(row[1].split('\'')[0]), 2) / 1000)
    y.append(round(float(row[0].split('\'')[1]), 2) / 1000)

ucb_uplink_messages['x'] = x
ucb_uplink_messages['y'] = y<jupyter_output><empty_output><jupyter_text>Now, we draw a path for each end node based on the received coordinates.<jupyter_code>adr_unique_ens = len(adr_uplink_messages.node_id.unique())
ucb_unique_ens = len(ucb_uplink_messages.node_id.unique())
adr_cmap = mpl.cm.summer
ucb_cmap = mpl.cm.get_cmap('PuBu')
xlim = 10
ylim = 10

fig, axis = plt.subplots(nrows=1, ncols=2, figsize=(10,5))

for i in range(0, adr_unique_ens):
    adr_data = adr_uplink_messages[adr_uplink_messages.node_id == adr_uplink_messages.node_id[i]]
    axis[0].plot(adr_data.x, adr_data.y, color=adr_cmap(i / adr_unique_ens))
    
for i in range(0, ucb_unique_ens):
    ucb_data = ucb_uplink_messages[ucb_uplink_messages.node_id == ucb_uplink_messages.node_id[i]]
    axis[1].plot(ucb_data.x, ucb_data.y, color=ucb_cmap(i / ucb_unique_ens))
    
# Add Access Point
axis[0].plot(xlim / 2, ylim / 2, '+', mew=10, ms=2, color='black')
axis[1].plot(xlim / 2, ylim / 2, '+', mew=10, ms=2, color='black')
# ax.plot(xlim / 2 + 5, ylim / 2 - 5, 'X', mew=10, ms=2, color='black')

for i in range(2):
    axis[i].set_xlim([0,xlim])
    axis[i].set_ylim([0,ylim])
    axis[i].spines['top'].set_visible(False)
    axis[i].spines['right'].set_color('#dddddd')
    axis[i].spines['left'].set_visible(False)
    axis[i].spines['bottom'].set_color('#dddddd')
    axis[i].tick_params(bottom=False, left=False)
    axis[i].set_axisbelow(True)
    axis[i].yaxis.grid(True, color='#eeeeee')
    axis[i].xaxis.grid(True, color='#eeeeee')
    axis[i].set_xlabel('X [km]', labelpad=15)
    axis[i].set_ylabel('Y [km]', labelpad=15)

axis[0].set_title('Paths of ADR Nodes', pad=15)
axis[1].set_title('Paths of UCB Nodes', pad=15)

fig.tight_layout()

fig.savefig(f'adr-ucb-paths.{output_format}', dpi=300)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13228468282619907, 'fraction_numerical': 0.014311500773594636, 'mean_word_length': 6.083561643835616, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '59851', 'license': 'MIT', 'n_tokens_mistral': 6143, 'n_tokens_neox': 5908, 'n_words': 1268, 'path': 'comparison/lora-adr-vs-ucb.ipynb', 'repo_name': 'alexandervalach/lora-notebooks', 'size': '15512'}