<jupyter_start><jupyter_text>Data ProcessingFirst we add some data to our project. In this| sample we get rid of 30% of rows, removing a column in a given row with probability 50%.<jupyter_code>p = pd.read_csv('C:/Users/Serhan/PycharmProjects/virtual-scada/Data/output_p.csv', header = None)
q = pd.read_csv('C:/Users/Serhan/PycharmProjects/virtual-scada/Data/output_q.csv', header = None)
v = pd.read_csv('C:/Users/Serhan/PycharmProjects/virtual-scada/Data/output_v.csv', header = None)
a = pd.read_csv('C:/Users/Serhan/PycharmProjects/virtual-scada/Data/output_a.csv', header = None)
vRemoved = removeRows(v, .3, colPercentage = .5)
aRemoved = removeRows(a, .3, colPercentage = .5)<jupyter_output>..\virtualscada\vs.py:66: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  dataRemoved[i][j] = None<jupyter_text>Virtual SCADAThen we fill in our missing values. Our code takes in the fixed real power, reactive power, voltage, and phase angle and fills in the missing values using linear regression trained on the supplied data.<jupyter_code>vFilled, aFilled = fillValuesLRInverse(p, q, vRemoved, aRemoved)<jupyter_output><empty_output><jupyter_text>ResultsNow we look at the results of our filling. The following code manually computes the RMSE of our fill by comparing it with the values we removed at the first step.<jupyter_code>numRows, numCols = v.shape
predictions = []
actuals = []
for i in range(numCols):
    for j in range(numRows):
        if np.isnan(vRemoved[i][j]):
            predictions.append(vFilled[i][j])
            actuals.append(v[i][j])
                
print(np.sqrt(np.nanmean((np.array(predictions)-np.array(actuals))**2)))<jupyter_output><empty_output><jupyter_text>Here is a plot of the results. The first plot is the actual voltage, the second is the voltage after being filled, and the third plot is the plot of the voltage with the missing values.<jupyter_code>sampleBus = np.random.randint(0,29)

plt.figure(figsize=(10, 11))
plt.subplot(221)
plt.plot(v[sampleBus][0:50])
plt.title('Actual Voltage')
plt.xlabel('Time Stamp [unit]')
plt.ylabel('Voltage Magnitude')
plt.subplot(222)
plt.plot(vFilled[sampleBus][0:50])
plt.title('Filled in Voltage')
plt.xlabel('Time Stamp [unit]')
plt.ylabel('Voltage Magnitude')
plt.subplot(223)
plt.plot(vRemoved[sampleBus][0:50])
plt.title('MissingVoltage')
plt.xlabel('Time Stamp [unit]')
plt.ylabel('Voltage Magnitude')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12412993039443156, 'fraction_numerical': 0.013921113689095127, 'mean_word_length': 5.435323383084577, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '246312', 'license': 'BSD-3-Clause-LBNL', 'n_tokens_mistral': 942, 'n_tokens_neox': 890, 'n_words': 232, 'path': 'virtual-scada/Notebooks/LinearInverseSample.ipynb', 'repo_name': 'slaclab/VADER-Analytics', 'size': '2586'}