<jupyter_start><jupyter_text>Higher Ground Python Task  Compass Calibration Script Notes - All satellites are rotating around the equator at different latitudes - Looking for the optimal azimuth based on SatPaq signal strength in dB - Want faster than linear time search<jupyter_code>#imports
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from plotnine import ggplot, aes, geom_col, geom_histogram, geom_bar
import random
import statistics<jupyter_output><empty_output><jupyter_text>Read data into pandas dataframes for plotting<jupyter_code>df_G12 = pd.read_excel('G12_Test_RX.csv.xlsx')
print(df_G12.shape)
df_G12.head()
df_G3C = pd.read_excel('G3C_Test_RX.csv.xlsx')
print(df_G3C.shape)
df_G3C.head()<jupyter_output>(455, 5)<jupyter_text>Plot Data  Using matplotlib<jupyter_code>plt.bar(df_G12['Azimuth'], df_G12['Esn0'])
plt.title('G12')
plt.ylabel('dB')
plt.xlabel('Azimuth')
plt.bar(df_G3C['Azimuth'], df_G3C['Esn0'])
plt.title('G3C')
plt.ylabel('dB')
plt.xlabel('Azimuth')<jupyter_output><empty_output><jupyter_text>using ggplot<jupyter_code>ggplot(df_G12, aes(x="Azimuth", y = "Esn0")) + geom_bar(stat='identity', width = 1)
ggplot(df_G3C, aes(x="Azimuth", y = "Esn0")) + geom_bar(stat='identity', width = 1)<jupyter_output><empty_output><jupyter_text>Scratch work: Finding the SatPaq position based on signal strength and known location of the satellite<jupyter_code># random starting position
start = random.randrange(-90, 90, 2)
print(start)<jupyter_output>66<jupyter_text>Circle shift signal so we don't know where we are Example of Understanding: - If start = 60 we want to "shift the values by 60 degrees" or np.roll so that values measured at one angle are shifted in a list to where values measured at 60 degrees away are. - If we group readings by the number of readings by the number of readings per angle, we need to roll by (60/2)  - 60/2 because we are measuring every 2 degrees<jupyter_code>#group by readings per angle
num_readings = 5

#these loops jump by 5 each iteration but only read number of num_readings, this willbe useful for testing
GF12_signals = []
for i in range(0,len(df_G12['Esn0']), 5):
    GF12_signals.append(list(df_G12['Esn0'][i:i+num_readings]))
#print(GF12_signals)

GF3C_signals = []
for i in range(0,len(df_G3C['Esn0']), 5):
    GF3C_signals.append(list(df_G3C['Esn0'][i:i+num_readings]))
#print(GF3C_signals)


roll = (start//2)
GF12_W129 = np.array(GF12_signals)
GF3C_W95 = np.array(GF3C_signals)

#shift
GF12_W129 = np.roll(GF12_W129,roll, axis = 0)
GF3C_W95 = np.roll(GF3C_W95,roll,axis = 0)<jupyter_output><empty_output><jupyter_text>After the roll, we are lost, I am assuming, based on the given data, that the strongest signal is between 90 degrees left and 90 degrees right for a total 180 degree range of where we are facing. If we take 5 readings every time we check a position for a signal strength each check takes 2 seconds. The step size of the stepper motor is 2 degrees we have 90 total signals to check (if we calculate the averages). We can use the difference between the starting position and strongest signal position for the offset and use the offset and known longitude of the satellites to calculate a starting position. If we use a stepper motor at 2 revolutions per second we'll get 720 degrees per second so 2 degree turn would take around 0.003 seconds (most step at 1.8 degrees though right...). We can use a modified binary search to find the peak.  Method 1: Binary search(es)<jupyter_code>def find_peak(signals, jump):
    l,r = 0, len(signals)-1
    total_read_time = 0
    total_turn_time = 0
    starting = l + (r-l)//2
    while l + jump < r:
        #the first iteration of mid is the unknown starting position, afterward, it is adjusted so we need to add in the turn time
        try:
            total_turn_time += abs(mid - (l + (r - l) // 2))*0.003
        except:
            total_turn_time +=0
        mid = l + (r - l) // 2
        
        #the next if requires x readings per turn = x * 0.4 seconds 
        total_read_time += 2*(len(signals[mid]) * 0.4)
        
        #plus time per turn = jump/turn size * 0.003 seconds
        total_turn_time += (jump * 0.003)
        
        #if the direction we are pointing is less than 
        if statistics.mean(signals[mid]) < statistics.mean(signals[mid+jump]):
            l = mid + jump
        else:
            r = mid 
    angle_offset = 2*(starting - mid)
    signal_strength = statistics.mean(signals[mid])
    peak = mid
    #print(total_read_time, total_turn_time)
    total_time = total_read_time + total_turn_time
    return signal_strength, total_time, angle_offset, peak
find_peak(GF12_W129,2)<jupyter_output><empty_output><jupyter_text>Read time is very significant. Above is the total_read_time (top left), and total_turn_time (top right). Below it is signal_strength, total_time, angle_offset, peak.  Plot distribution to visualize where the search ended up vs where the actual peak was. As we can see it found the peak around index 78, where the best signal strength would be found.<jupyter_code>plt.bar([x for x in range(0,91)],[statistics.mean(x) for x in GF12_W129])
plt.title('G12')
plt.ylabel('dB')
plt.xlabel('Azimuth')<jupyter_output><empty_output><jupyter_text>Try same thing for GF3C data<jupyter_code>find_peak(GF3C_W95,5)
plt.bar([x for x in range(0,91)],[statistics.mean(x) for x in GF3C_W95])
plt.title('G12')
plt.ylabel('dB')
plt.xlabel('Azimuth')<jupyter_output><empty_output><jupyter_text>Trial Function with binary search function<jupyter_code>def search_params(signals, num_readings = 5, step_size = 2, sat_longitude = 129,shift_angle = 60):
    #random starting position offset
    true_start = 90-shift
    
    #group signals by number of readings
    grouped_signals = []
    for i in range(0,len(signals['Esn0']), 5):
        grouped_signals.append(signals['Esn0'][i:i+num_readings])
    
    #rotate/circle shift readings
    signals = np.array(signals)
    shifted_signals = np.roll(grouped_signals, (shift_angle//2), axis = 0)
    
    #search for peak
    signal_strength, time, offset, peak = find_peak(shifted_signals,step_size)
    final_angle = peak*2 - shift_angle
    
    position_estimate = sat_longitude + offset
    
    return signal_strength, time, offset, position_estimate, true_start, final_angle<jupyter_output><empty_output><jupyter_text>Compass calibration trials for G3C satellite<jupyter_code>G3C_2step_2read = pd.DataFrame(columns=['Signal Strengths','Times','Offset','Shift Angle','Position Estimate', 'True Start','Error', 'Final Angle'])
for _ in range(25):
    shift = random.randrange(-90, 90, 2)
    ss,t,o,pe,tsp, final = search_params(df_G3C,2,2,95,shift)
    err = (pe - tsp)
    G3C_2step_2read = G3C_2step_2read.append({'Signal Strengths':ss, 'Times':t, 'Position Estimate':pe,'Error':err,'Offset':o,'True Start':tsp, 'Shift Angle':shift, 'Final Angle': final}, ignore_index=True)



G3C_2step_2read<jupyter_output><empty_output><jupyter_text>Edge Cases Found when starting at 70 and 78 rotation with step size of 2<jupyter_code>G3C_2step_2read_edges = pd.DataFrame(columns=['Signal Strengths','Times','Offset','Shift Angle','Position Estimate', 'True Start','Error', 'Final Angle'])
for shift in range(60,80,2):
    ss,t,o,pe,tsp, final = search_params(df_G3C,2,2,95,shift)
    err = (pe - tsp)
    G3C_2step_2read_edges = G3C_2step_2read_edges.append({'Signal Strengths':ss, 'Times':t,'Error':err, 'Position Estimate':pe,'Offset':o,'True Start':tsp, 'Shift Angle':shift, 'Final Angle': final}, ignore_index=True)

G3C_2step_2read_edges<jupyter_output><empty_output><jupyter_text>Visaulization to show why the edge cases may exist in G3C<jupyter_code>#group by readings per angle
num_readings = 5

#these loops jump by 5 each iteration but only read number of num_readings, this willbe useful for testing
GF3C_signals = []
for i in range(0,len(df_G3C['Esn0']), 5):
    GF3C_signals.append(list(df_G3C['Esn0'][i:i+num_readings]))
#print(GF3C_signals)


roll = (70//2)
GF3C_W95 = np.array(GF3C_signals)


#shift
GF3C_W95_70 = np.roll(GF3C_W95,70//2, axis = 0)
GF3C_W95_78 = np.roll(GF3C_W95,78//2,axis = 0)
plt.bar([x for x in range(0,91)],[statistics.mean(x) for x in GF3C_W95_70])
plt.title('GF3C_W95_70')
plt.ylabel('dB')
plt.xlabel('Azimuth')
plt.bar([x for x in range(0,91)],[statistics.mean(x) for x in GF3C_W95_78])
plt.title('GF3C_W95_78')
plt.ylabel('dB')
plt.xlabel('Azimuth')
#scatter
plt.scatter(G3C_2step_2read.index,G3C_2step_2read['Position Estimate'],label='Estimate')
plt.scatter(G3C_2step_2read.index,G3C_2step_2read['True Start'], color='r',label = 'Actual')
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.legend(loc= 'upper right')
plt.show()

#bar
width = 0.8
plt.bar(G3C_2step_2read.index, G3C_2step_2read['Position Estimate'], width=width, color='b', label='Estimate')
plt.bar([i+0.25*width for i in G3C_2step_2read.index], G3C_2step_2read['True Start'], width=0.5*width, color='r', alpha=1, label = 'Actual')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.xticks(G3C_2step_2read.index+width/2., ['T{}'.format(i) for i in range(len(G3C_2step_2read['Position Estimate']))] )
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Compass calibration Trials for G12 satellite with position at 129<jupyter_code>G12_3step_5read = pd.DataFrame(columns=['Signal Strengths','Times','Offset','Shift Angle','Position Estimate', 'True Start','Error', 'Final Angle'])
for i in range(25):
    shift = random.randrange(-90, 90, 2)
    ss,t,o,pe,tsp, final = search_params(df_G12,5,3,129,shift)
    err = (pe - tsp)
    G12_3step_5read = G12_3step_5read.append({'Signal Strengths':ss, 'Times':t, 'Position Estimate':pe,'Offset':o,'True Start':tsp, 'Error':err, 'Shift Angle':shift, 'Final Angle': final}, ignore_index=True)


G12_3step_5read<jupyter_output><empty_output><jupyter_text>The differnces between estimate and true start for G12 are much worse because the the strongest angle is not at the 129 angle but instead also around 95.<jupyter_code>#scatter
plt.scatter(G12_3step_5read.index,G12_3step_5read['Position Estimate'],label='Estimate')
plt.scatter(G12_3step_5read.index,G12_3step_5read['True Start'], color='r',label = 'Actual')
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.legend(loc= 'upper right')
plt.show()

#bar
width = 0.8
plt.bar(G12_3step_5read.index, G12_3step_5read['Position Estimate'], width=width, color='b', label='Estimate')
plt.bar([i+0.25*width for i in G12_3step_5read.index], G12_3step_5read['True Start'], width=0.5*width, color='r', alpha=1, label = 'Actual')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.xticks(G3C_2step_2read.index+width/2., ['T{}'.format(i) for i in range(len(G3C_2step_2read['Position Estimate']))] )
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Compass calibration Trials for G12 satellite with position at 96<jupyter_code>G12_3step_5read = pd.DataFrame(columns=['Signal Strengths','Times','Offset','Shift Angle','Position Estimate', 'True Start','Error', 'Final Angle'])
for i in range(25):
    shift = random.randrange(-90, 90, 2)
    ss,t,o,pe,tsp, final = search_params(df_G12,5,3,96,shift)
    err = (pe - tsp)
    G12_3step_5read = G12_3step_5read.append({'Signal Strengths':ss, 'Times':t, 'Position Estimate':pe,'Offset':o,'True Start':tsp, 'Error':err, 'Shift Angle':shift, 'Final Angle': final}, ignore_index=True)


G12_3step_5read
#scatter
plt.scatter(G12_3step_5read.index,G12_3step_5read['Position Estimate'],label='Estimate')
plt.scatter(G12_3step_5read.index,G12_3step_5read['True Start'], color='r',label = 'Actual')
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.legend(loc= 'upper right')
plt.show()

#bar
width = 0.8
plt.bar(G12_3step_5read.index, G12_3step_5read['Position Estimate'], width=width, color='b', label='Estimate')
plt.bar([i+0.25*width for i in G12_3step_5read.index], G12_3step_5read['True Start'], width=0.5*width, color='r', alpha=1, label = 'Actual')
plt.title('G32 Binary Search, 2 steps, 2 Reads')
plt.xticks(G3C_2step_2read.index+width/2., ['T{}'.format(i) for i in range(len(G3C_2step_2read['Position Estimate']))] )
plt.xlabel('Trial')
plt.ylabel('Angle')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Graphing different times of different strategies Strategy adjustments- Different step size or turn of stepper motor in binary search- Different number of readings<jupyter_code>def algorithm_differences(signals_df, longitude):
    final_df = pd.DataFrame(columns=['Step Size','Number of Readings','Average Total Time','Average Error'])
    
    steps_for_search = [1,2,3,5,6,10]
    diff_num_readings = [2,3,4,5]
    for s in steps_for_search:
        for n in diff_num_readings:
            trial_df = pd.DataFrame(columns=['Signal Strengths','Times','Offset','Shift Angle','Position Estimate', 'True Start','Error', 'Final Angle'])
            #5 random shifts
            for _ in range(5):
                shift = random.randrange(-90, 90, 2)
                #print(s,n,shift)
                ss,t,o,pe,tsp, final = search_params(signals_df,n,s,longitude,shift)
                err = (pe - tsp)
                trial_df = trial_df.append({'Signal Strengths':ss, 'Times':t, 'Position Estimate':pe,'Offset':o,'True Start':tsp, 'Error':err, 'Shift Angle':shift, 'Final Angle': final}, ignore_index=True)
                #print(trial_df)
            avg_error = trial_df["Error"].mean()
            avg_time = trial_df["Times"].mean()
            final_df = final_df.append({'Step Size': s, 'Number of Readings':n, 'Average Total Time':avg_time, 'Average Error':avg_error}, ignore_index=True)
    return final_df
            
df_G3C_diff = algorithm_differences(df_G3C,95)
df_G3C_diff
plt.scatter(x = df_G3C_diff['Number of Readings'], y = df_G3C_diff['Step Size'], s = [s*10 for s in df_G3C_diff['Average Total Time']])
plt.xlabel('Number of Readings')
plt.ylabel('Size of Step')
plt.title('G3C Differences - Avg Total Time')
plt.xticks([2,3,4,5])
plt.yticks([x for x in range(11)])
plt.show()

plt.scatter(x = df_G3C_diff['Number of Readings'], y = df_G3C_diff['Step Size'], s = [s*10 for s in df_G3C_diff['Average Error']])
plt.xlabel('Number of Readings')
plt.ylabel('Size of Step')
plt.title('G3C Differences - Avg Total Error')
plt.xticks([2,3,4,5])
plt.yticks([x for x in range(11)])
plt.show()
df_G12_diff = algorithm_differences(df_G12,96)
df_G12_diff
plt.scatter(x = df_G12_diff['Number of Readings'], y = df_G12_diff['Step Size'], s = [s*10 for s in df_G3C_diff['Average Total Time']])
plt.xlabel('Number of Readings')
plt.ylabel('Size of Step')
plt.title('G12 Differences - Avg Total Time')
plt.xticks([2,3,4,5])
plt.yticks([x for x in range(11)])
plt.show()

plt.scatter(x = df_G12_diff['Number of Readings'], y = df_G12_diff['Step Size'], s = [s*10 for s in df_G3C_diff['Average Error']])
plt.xlabel('Number of Readings')
plt.ylabel('Size of Step')
plt.title('G12 Differences - Avg Error')
plt.xticks([2,3,4,5])
plt.yticks([x for x in range(11)])
plt.show()
def trivial_analysis(signals, readings):
    grouped_signals = []
    for i in range(0,len(signals['Esn0']), 5):
            grouped_signals.append(signals['Esn0'][i:i+num_readings])
            
    five_reading_averages = [float(statistics.mean(s)) for s in grouped_signals]
    actual_max = max(five_reading_averages)
    actual_max_index = five_reading_averages.index(actual_max)
    print(actual_max_index)
    print(grouped_signals[50])
    Max = 0
    for angle in range(len(grouped_signals)):
        reading = statistics.mean(grouped_signals[angle])
        if reading > Max:
            Max = reading
            final_angle = angle
    print(grouped_signals[int(final_angle)])
    return Max, actual_max, angle, actual_max,(Max - actual_max), (final_angle - actual_max_index)*2    
trivial_analysis(df_G3C, 3)<jupyter_output>50
250    8.8
251    8.8
252    8.4
253    8.8
254    8.8
Name: Esn0, dtype: float64
250    8.8
251    8.8
252    8.4
253    8.8
254    8.8
Name: Esn0, dtype: float64
{'fraction_non_alphanumeric': 0.13314010853478048, 'fraction_numerical': 0.043907252096694625, 'mean_word_length': 5.156795747911921, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '372106', 'license': 'MIT', 'n_tokens_mistral': 6390, 'n_tokens_neox': 5862, 'n_words': 1447, 'path': 'Higher Ground Project.ipynb', 'repo_name': 'JeffreyBradley772/Higher-Ground-Project', 'size': '16216'}