<jupyter_start><jupyter_text>![title](model.png)<jupyter_code># The model
with pm.Model() as TL:
    
    σ = pm.Uniform('σ',lower=σy/1000,upper=σy*1000,shape=9)
    μ = pm.Normal('μ',mu=μy,sigma=σy*100,shape=9)
    ν_minus_one = pm.Exponential('ν_minus_one',lam=1/29)
    ν = pm.Deterministic('ν',ν_minus_one+1)

    # Likelihood
    [pm.StudentT(f'obs_{i}', nu=ν, mu=μ[i], sigma=σ[i], observed=y_joint[i]) for i in range(9)]
    # For ploting normality with ease     
    pm.Deterministic('log10_ν',np.log10(ν))<jupyter_output><empty_output><jupyter_text>I assume that the data is normally distributed. The model utilizes t distribution for describing noise in the data and handle the outliers. A Uniform (σ), Normal (μ) and Exponential distribution (ν_minus_one) are used for the prior distributions on parameters of the t distribution. The Uniform distribution is applied to the scale parameter, Normal distribution to the mean parameter and Exponential distribution to the normality parameter that controls the heaviness of the tails. The parameters of μ are set to be noncommittal by setting its mean (M in the graph) to the actual mean of the collected data and its standard deviation (S in the graph) to 100 times larger than the standard deviation of the data. The parameters of σ (L and H in the graph) are chosen in a way that put a broad uniform distribution directly on the scale parameter of the likelihood function. Last but not least the prior normality parameter ν is a continuous value between 1 and infinity with an equal opportunity to small values of ν (less than 30) and larger values of ν (greater than 30) since the t distribution with ν > 30  is essentially a normal distribution.<jupyter_code>pm.model_to_graphviz(TL)<jupyter_output><empty_output><jupyter_text>I draw 21000 samples from the posterior using NUTS sampler which is the default sampler in pymc3.<jupyter_code>with TL:
    trace_TL=pm.sample(21000, chains=4, cores=4, tune=1000)<jupyter_output>Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [ν_minus_one, μ, σ]
Sampling 4 chains, 0 divergences: 100%|██████████| 88000/88000 [02:56<00:00, 498.61draws/s]<jupyter_text>Let's check autocorrelation for our three parameters.<jupyter_code>pm.autocorrplot(trace_TL, var_names=['μ'])
pm.autocorrplot(trace_TL, var_names=['σ'])
pm.autocorrplot(trace_TL, var_names=['ν']);<jupyter_output><empty_output><jupyter_text>As you can see above, autocorrelation is about zero for all chains of three parameters which indicates the chains are random.<jupyter_code>pm.summary(trace_TL)<jupyter_output><empty_output><jupyter_text>The mcse_mean and mcse_sd values are 0. Hence I will have zero standard error on the posterior mean and Scale. Besides, all values of the r_hat are 1 which is a sign of a good convergence. I now plot sampled values and graphically check the convergence.<jupyter_code>warnings.simplefilter('ignore', UserWarning) # To avoid printing the annoying user warning which is most likely a bug
pm.traceplot(trace_TL);
pm.plot_posterior(trace_TL, var_names=['log10_ν'], point_estimate='mode', credible_interval=.95);<jupyter_output><empty_output><jupyter_text>The above plot indicates that the data have outliers since the 95% of its values are within [1.2, 2.2]. Ok, let's check if hypotheses in question hold. Before that, I want to make it clear why I am using the ROPE values ± 0.2 and ± 0.35. The former opts base on the belief that says a 20% difference is reasonable to make a decision whether a transform language performs better than the other one. The latter is chosen based on the standard deviation of collected data which is approximately 0.3.<jupyter_code># Define gridspec
from matplotlib import gridspec
fig = plt.figure(constrained_layout=True, figsize=(12,7), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(2, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])
ax3 = fig.add_subplot(gs[1, :3])
ax4 = fig.add_subplot(gs[1, 3:])

fig.suptitle('H1 and H1_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,3] - trace_TL['μ'][:,0],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - ATL (Comprehend)")

pm.plot_posterior( trace_TL['μ'][:,6] - trace_TL['μ'][:,0],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax2.set_title("Difference of Means")
ax2.set_xlabel("Xtend - ATL (Comprehend)")

pm.plot_posterior( trace_TL['σ'][:,3] - trace_TL['σ'][:,0],
         ax=ax3, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax3.set_title("Difference of Scales")
ax3.set_xlabel("QVT-O  - ATL (Comprehend)")

pm.plot_posterior( trace_TL['σ'][:,6] - trace_TL['σ'][:,0],
         ax=ax4, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax4.set_title("Difference of Scales")
ax4.set_xlabel("Xtend - ATL (Comprehend)");<jupyter_output><empty_output><jupyter_text>H1 and H1_0:In the top histograms, the 95% HDI falls partially within the ROPE, so I cannot make any dicision! Even though the 95% HDI of the top right histogram is nearly falls within the ROPE it is better to do not make dicision.<jupyter_code>fig = plt.figure(constrained_layout=True, figsize=(12,7), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(2, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])
ax3 = fig.add_subplot(gs[1, :3])
ax4 = fig.add_subplot(gs[1, 3:])

fig.suptitle('H2 and H2_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,4] - trace_TL['μ'][:,1],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - ATL (Change)")

pm.plot_posterior( trace_TL['μ'][:,7] - trace_TL['μ'][:,1],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax2.set_title("Difference of Means")
ax2.set_xlabel("Xtend - ATL (Change)")

pm.plot_posterior( trace_TL['σ'][:,4] - trace_TL['σ'][:,1],
         ax=ax3, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax3.set_title("Difference of Scales")
ax3.set_xlabel("QVT-O  - ATL (Change)")

pm.plot_posterior( trace_TL['σ'][:,7] - trace_TL['σ'][:,1],
         ax=ax4, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax4.set_title("Difference of Scales")
ax4.set_xlabel("Xtend - ATL (Change)");<jupyter_output><empty_output><jupyter_text>H2 and H2_0: In the top left histogram, the 95% HDI of difference between QVT_O and ATL falls within the ROPE, so I conclude that there is no difference between the performance of QVT_O and ATL in this hypothesis. Therefore I reject the hypothesis as subjects who change a transformation written in QVT-O perform better than those who change a transformation in ATL. However, I cannot reject or accept hypothesis as Xtend performs better than ATL. That is I cannot make any decision about H2 hypothesis.On the other hand, I accept the H2_0 hypothesis since the performance of QVT_O and ATL are equal. When using MCMC to estimate the HDI, the endpoints of HDI are not very precisely estimated hence I take the variation of estimation into account and assume the entire HDI falls within the ROPE in the top left histogram considering that only one of the endpoints has this condition. This assumption applies to all the decisions.<jupyter_code>fig = plt.figure(constrained_layout=True, figsize=(12,7), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(2, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])
ax3 = fig.add_subplot(gs[1, :3])
ax4 = fig.add_subplot(gs[1, 3:])

fig.suptitle('H3 and H3_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,5] - trace_TL['μ'][:,2],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - ATL (Create)")

pm.plot_posterior( trace_TL['μ'][:,8] - trace_TL['μ'][:,2],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax2.set_title("Difference of Means")
ax2.set_xlabel("Xtend - ATL (Create)")

pm.plot_posterior( trace_TL['σ'][:,5] - trace_TL['σ'][:,2],
         ax=ax3, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax3.set_title("Difference of Scales")
ax3.set_xlabel("QVT-O  - ATL (Create)")

pm.plot_posterior( trace_TL['σ'][:,8] - trace_TL['σ'][:,2],
         ax=ax4, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax4.set_title("Difference of Scales")
ax4.set_xlabel("Xtend - ATL (Create)");<jupyter_output><empty_output><jupyter_text>H3 and H3_0: In the top histograms, the 95% HDI falls partially within the ROPE, so I cannot make any dicision!<jupyter_code>fig = plt.figure(constrained_layout=True, figsize=(12,4), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(1, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])

fig.suptitle('H4 and H4_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,3] - trace_TL['μ'][:,6],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - Xtend (Comprehend)")

pm.plot_posterior( trace_TL['σ'][:,3] - trace_TL['σ'][:,6],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax2.set_title("Difference of Scales")
ax2.set_xlabel("QVT-O  - Xtend (Comprehend)");<jupyter_output><empty_output><jupyter_text>H4 and H4_0: In the right histogram, the 95% HDI falls partially within the ROPE, so I cannot make any dicision!<jupyter_code>fig = plt.figure(constrained_layout=True, figsize=(12,4), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(1, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])

fig.suptitle('H5 and H5_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,4] - trace_TL['μ'][:,7],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - Xtend (Change)")

pm.plot_posterior( trace_TL['σ'][:,4] - trace_TL['σ'][:,7],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax2.set_title("Difference of Scales")
ax2.set_xlabel("QVT-O  - Xtend (Change)");<jupyter_output><empty_output><jupyter_text>H5 and H5_0: In the right histogram, the 95% HDI falls within the ROPE, so I accept the H5 hypothesis and consequently reject the H5_0 hypothesis. In the all above histograms of the difference of Scales, the 95% HDI falls nicely within the ROPE and suggests Qvt_O or Xtend does not introduce any difference in Scale.<jupyter_code>fig = plt.figure(constrained_layout=True, figsize=(12,4), facecolor='w', edgecolor='k')
gs = fig.add_gridspec(1, 6)
ax1 = fig.add_subplot(gs[0, :3])
ax2 = fig.add_subplot(gs[0, 3:])

fig.suptitle('H6 and H6_0 hypotheses', fontsize='14')

pm.plot_posterior( trace_TL['μ'][:,5] - trace_TL['μ'][:,8],
         ax=ax1, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.2,+0.2)
    )
ax1.set_title("Difference of Means")
ax1.set_xlabel("QVT-O - Xtend (Create)")

pm.plot_posterior( trace_TL['σ'][:,5] - trace_TL['σ'][:,8],
         ax=ax2, point_estimate='mode',  round_to=4,
        credible_interval=0.95, kind='hist', ref_val=0.0, rope=(-0.35,+0.35)
    )
ax2.set_title("Difference of Scales")
ax2.set_xlabel("QVT-O  - Xtend (Create)");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14793856103476152, 'fraction_numerical': 0.043411479385610345, 'mean_word_length': 5.07911547911548, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '393209', 'license': 'MIT', 'n_tokens_mistral': 4841, 'n_tokens_neox': 4618, 'n_words': 1180, 'path': 'transformation-languages.ipynb', 'repo_name': 'amir-souri/model-transformation-languages', 'size': '12370'}