<jupyter_start><jupyter_text>Simple fake dataset:<jupyter_code>beta_1, beta_2 = 7, 2
noise_scale = 0
max_high = 50
max_low = 125

n_low = np.arange(max_low)
n_high = np.arange(max_high)

mse = n_high.reshape(-1, 1)*beta_1 + n_low.reshape(1, -1)*beta_2
mse = mse + (np.random.rand(*mse.shape) - .5) * noise_scale

plt.imshow(mse, origin='lower', cmap='viridis')
plt.contour(mse, levels=8, colors='black', alpha=.25)
plt.xlabel('num low')
plt.ylabel('num high')
plt.show()<jupyter_output><empty_output><jupyter_text>A linear fit gives the following:<jupyter_code>features = np.hstack(list(reversed([x.reshape(-1, 1) for x in np.meshgrid(n_low, n_high)])))
values = mse.ravel()
reg = LinearRegression().fit(features, values)
print(reg.coef_)<jupyter_output><empty_output><jupyter_text>Which plotted shows up as follows:<jupyter_code>plt.figure()
plt.imshow(mse, origin='lower', cmap='viridis')
plt.contour(mse, levels=8, colors='black', alpha=.25)
plt.plot(n_low.ravel(), n_low.ravel()*(reg.coef_[0]/reg.coef_[1]), color='C1')  # x* beta_1/beta_2
plt.ylim([0, max_high])
plt.xlim([0, max_low])
plt.xlabel('num low')
plt.ylabel('num high')
plt.tight_layout()
plt.savefig('lin_reg_without_cost.pdf', bbox_inches='tight')
plt.show()<jupyter_output><empty_output><jupyter_text>Now what if we scale along the x-axis, i.e. `n_low`, by some cost ratio $\phi$?<jupyter_code>cost_ratio = 1/5

cost_n_low = np.arange(np.ceil(cost_ratio * max_low))

cost_values = mse[:,::5].ravel()
cost_features = np.hstack(list(reversed([x.reshape(-1, 1) for x in np.meshgrid(cost_n_low, n_high)])))
cost_reg = LinearRegression().fit(cost_features, cost_values)

plt.figure()
plt.imshow(mse[:,::5], origin='lower', cmap='viridis')
plt.contour(mse[:,::5], levels=8, colors='black', alpha=.25)
plt.plot(cost_n_low.ravel(), cost_n_low.ravel()*(cost_reg.coef_[0]/cost_reg.coef_[1]), color='C1')  # x* beta_1/beta_2
plt.ylim([0, max_high])
plt.xlim([0, max_low*cost_ratio])
plt.xlabel('cost ratio * num low')
plt.ylabel('num high')
plt.tight_layout()
plt.savefig('lin_reg_with_cost.pdf', bbox_inches='tight')
plt.show()<jupyter_output><empty_output><jupyter_text>Then the explanatory effect of the y-axis, i.e. `n_high`, will remain the same, thus not influencing the first coefficient ($\beta_1$).However, $\beta_2$ is fit on `n_low`, where the cost ratio will be included, so that value is expected to change.Using the linear regression formula, we can see that $\beta_2$ is determined as follows if the cost ratio $\phi$ is excluded:$\log_{10}(MSE) = \beta_1 n_h + \beta_2 n_l + c$$\beta_2 n_l = \log_{10}(MSE) - \beta_1 n_h - c$$\beta_2 = \dfrac{\log_{10}(MSE) - \beta_1 n_h - c}{n_l}$By including the cost ratio, no values other than $\beta_2$ should change, so we mark the cost-ratio-included value as $\beta^\prime_2$. Stating the linear regression formula with cost ratio included and solving for $\beta^\prime_2$, we see the exact same result as before, but with $\phi$ added in the nominator:$\log_{10}(MSE) = \beta_1 n_h + \beta^\prime_2 \phi n_l + c$$\beta^\prime_2 \phi n_l = \log_{10}(MSE) - \beta_1 n_h - c$$\beta^\prime_2 = \dfrac{\log_{10}(MSE) - \beta_1 n_h - c}{\phi n_l}$In other words:$\beta^\prime_2 = \dfrac{\beta_2}{\phi}$This is confirmed by the example above:<jupyter_code>print("beta_2 / cost_ratio:", reg.coef_[1] / cost_ratio)
print("beta^prime_2:       ", cost_reg.coef_[1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17347838870920318, 'fraction_numerical': 0.024992649220817408, 'mean_word_length': 5.957055214723926, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '93704', 'license': 'MIT', 'n_tokens_mistral': 1410, 'n_tokens_neox': 1330, 'n_words': 293, 'path': 'notebooks/lin-reg-fit.ipynb', 'repo_name': 'sjvrijn/multi-level-co-surrogates', 'size': '3401'}