<jupyter_start><jupyter_text>Importing some utilities<jupyter_code>%cd ../..
import glob
import sys
from dmg.realism.mle import whichFitsBetter
from scripts.modelSet import datasets_supported
msetObject = datasets_supported['yakindu-github']
train_path = 'data/yakindu-github/train'
backend = 'python'<jupyter_output>/home/antolin/wakame/DMG-Python<jupyter_text>Loading Training set<jupyter_code>Gs = [msetObject.getGraphReal(f,backend) 
                for f in glob.glob(train_path + "/*")]<jupyter_output><empty_output><jupyter_text>Random EMF For each rule in RandomEMF, depending on the type of rule, we estimate its parameters. More concretely, for shapes we use the function `whichFitsBetter` that selects the best distribuntion by using maximum likeihood. For priorities in alternative rules, the procedure described in the paper is done and it is based on counting each different alternative in the set $R_{II}$.  Number of regions per statechart For the rule:```Root : Statechart ->		regions += RegionsStatechartDistribution(parameters)	; ```<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

nums = []
for G in Gs:
    nums.append(G.out_degree(0))
bins = np.arange(1, 3, 0.5)
plt.hist(nums, bins = bins, alpha=0.5, density = False)
print()
whichFitsBetter(nums)<jupyter_output>Warning: Maximum number of function evaluations has been exceeded.
Optimization terminated successfully.
         Current function value: 0.283499
         Iterations: 18
         Function evaluations: 37
Optimization terminated successfully.
         Current function value: 1.074438
         Iterations: 11
         Function evaluations: 22<jupyter_text>Number of regions per state For the rule:```	RState (Region r) : State ->		regions += RegionsStateDistribution(parameters)	;	```<jupyter_code>numberSubvertex = []
for G in Gs:
    for n in G:
        if G.nodes[n]['type'] =='State':
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'regions'):
                        cont = cont + 1
            numberSubvertex.append(cont)
            
bins = np.arange(0, 10, 1)
plt.hist(numberSubvertex, bins = bins, alpha=0.5, density = True)
best = whichFitsBetter(numberSubvertex)
print(best)
#print(r,p)
#print(best)<jupyter_output>Optimization terminated successfully.
         Current function value: 0.415573
         Iterations: 26
         Function evaluations: 50
Optimization terminated successfully.
         Current function value: 0.831163
         Iterations: 16
         Function evaluations: 33
Optimization terminated successfully.
         Current function value: 0.471621
         Iterations: 8
         Function evaluations: 16
{'dis': 'poisson', 'params': 0.1453900709219858}<jupyter_text>Type of vertices For the rule:```	alter Vertices (Region r) : Vertex ->		 RPseudoState(r)a | RRegularState(r)b	;	```<jupyter_code>import numpy as np
ps = []
for G in Gs:
    p = [0, 0]
    for n in G:
        if (G.nodes[n]['type'] == 'FinalState'):
            p[0] = p[0] + 1
        if (G.nodes[n]['type'] == 'State'):
            p[0] = p[0] + 1
        if (G.nodes[n]['type'] == 'Synchronization'):
            p[1] = p[1] + 1
        if (G.nodes[n]['type'] == 'Choice'):
            p[1] = p[1] + 1
        if (G.nodes[n]['type'] == 'Exit'):
            p[1] = p[1] + 1
        if (G.nodes[n]['type'] == 'Entry'):
            p[1] = p[1] + 1
    p = np.array(p)
    ps.append(p/np.sum(p))
ps = np.array(ps)
print(np.mean(ps, axis = 0)/np.min(np.mean(ps, axis = 0)))<jupyter_output>[1.98176941 1.        ]<jupyter_text>For the rule:```	alter RRegularState (Region r) : RegularState ->		RFinala | RState(r)b 	;	```<jupyter_code>ps = []
for G in Gs:
    p = [0, 0]
    for n in G:
        if (G.nodes[n]['type'] == 'FinalState'):
            p[0] = p[0] + 1
        if (G.nodes[n]['type'] == 'State'):
            p[1] = p[1] + 1
    p = np.array(p)
    ps.append(p/np.sum(p))
ps = np.array(ps)
print(np.mean(ps, axis = 0)/np.min(np.mean(ps, axis = 0)))<jupyter_output>[ 1.         22.93782383]<jupyter_text>For the rule:```	alter RPseudoState(Region r) : Pseudostate ->		 RTypeSynchronization(r)a  | RTypeExitb | RTypeChoice(r)c		 | if (r.vertices.filter[it instanceof Entry].size == 0 		 	&& r.vertices.size > 0 		 ) RTypeEntry(r)d	;	```We want to estimate `a`, `b`, `c` and `d`.<jupyter_code>ps = []
for G in Gs:
    p = [0, 0, 0, 0]
    for n in G:
        if (G.nodes[n]['type'] == 'Synchronization'):
            p[0] = p[0] + 1
        if (G.nodes[n]['type'] == 'Choice'):
            p[1] = p[1] + 1
        if (G.nodes[n]['type'] == 'Exit'):
            p[2] = p[2] + 1
        if (G.nodes[n]['type'] == 'Entry'):
            p[3] = p[3] + 1
    p = np.array(p)
    ps.append(p/np.sum(p))
ps = np.array(ps)
print(np.mean(ps, axis = 0))
print(np.mean(ps[:,1:], axis = 0)/np.min(np.mean(ps[:,1:], axis = 0)))<jupyter_output>[0.         0.06991342 0.01731602 0.91277056]
[ 4.0375  1.     52.7125]<jupyter_text>Transitions per state For the rule:```	RState (Region r) : State ->		outgoingTransitions += RTransition(self,r)Distribution(parameters)	;	```<jupyter_code>numberTransitions = []
for G in Gs:
    for n in G:
        if (G.nodes[n]['type'] == 'State'):
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'outgoingTransitions'):
                        cont = cont + 1
            numberTransitions.append(cont)
bins = np.arange(0, 10, 1)
plt.hist(numberTransitions, bins = bins, alpha=0.5, density = True)
print()
whichFitsBetter(numberTransitions)<jupyter_output>Warning: Maximum number of function evaluations has been exceeded.
Optimization terminated successfully.
         Current function value: 1.368860
         Iterations: 20
         Function evaluations: 41
Optimization terminated successfully.
         Current function value: 1.282858
         Iterations: 11
         Function evaluations: 22<jupyter_text>For the rule:```	RTypeSynchronization (Region r) : Synchronization ->		outgoingTransitions += RTransition(self,r)Distribution(parameters)	;	```<jupyter_code>numberTransitions = []
for G in Gs:
    for n in G:
        if (G.nodes[n]['type'] == 'Synchronization'):
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'outgoingTransitions'):
                        cont = cont + 1
            numberTransitions.append(cont)
bins = np.arange(0, 3, 0.5)
plt.hist(numberTransitions, bins = bins, alpha=0.5, density = False)
print()
whichFitsBetter(numberTransitions)<jupyter_output>zero-size array to reduction operation maximum which has no identity
Can not fit negbin
Can not fit normal
Can not fit poisson<jupyter_text>For the rule:```	RTypeChoice (Region r) : Choice->		outgoingTransitions += RTransition(self,r)Distribution(parameters)	;	```<jupyter_code>numberTransitions = []
for G in Gs:
    for n in G:
        if (G.nodes[n]['type'] == 'Choice'):
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'outgoingTransitions'):
                        cont = cont + 1
            numberTransitions.append(cont)
bins = np.arange(0, 10, 0.5)
plt.hist(numberTransitions, bins = bins, alpha=0.5, density = False)
print()
whichFitsBetter(numberTransitions)
0.42847841252506524**2<jupyter_output><empty_output><jupyter_text>Number vertex per region  Statechart For the rule:```	RegionsStatechart : Region ->		vertices += Vertices(self)Distribution(parameters)	;	```<jupyter_code>def fromStateChart(G,n):
    for m in G:
        if (G.nodes[m]['type'] == 'Statechart'):
            try:
                for e in G[m][n]:
                    if (G[m][n][e]['type'] == 'regions'):
                        return True
            except:
                return False
    return False

numberVert = []
for G in Gs:
    for n in G:
        if (G.nodes[n]['type'] == 'Region') and (fromStateChart(G,n)):
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'vertices'):
                        cont = cont + 1
            numberVert.append(cont)
bins = np.arange(0, 10, 1)
plt.hist(numberVert, bins = bins, alpha=0.5, density = True)
print()
print(whichFitsBetter(numberVert))<jupyter_output>Warning: Maximum number of function evaluations has been exceeded.
Optimization terminated successfully.
         Current function value: 1.730549
         Iterations: 23
         Function evaluations: 46
Optimization terminated successfully.
         Current function value: 1.821253
         Iterations: 12
         Function evaluations: 24
{'dis': 'uniform', 'params': (2, 7)}<jupyter_text>State For the rule:```	RegionsState : Region ->		vertices += Vertices(self)Distribution(parameters)	;	```<jupyter_code>def fromState(G,n):
    for m in G:
        if (G.nodes[m]['type'] == 'State'):
            try:
                for e in G[m][n]:
                    if (G[m][n][e]['type'] == 'regions'):
                        return True
            except:
                return False
    return False

numberVert = []
for G in Gs:
    for n in G:
        if (G.nodes[n]['type'] == 'Region') and (fromState(G,n)):
            cont = 0
            for e in G[n]:
                for e2 in G[n][e]:
                     if (G[n][e][e2]['type'] == 'vertices'):
                        cont = cont + 1
            numberVert.append(cont)
bins = np.arange(0, 10, 1)
plt.hist(numberVert, bins = bins, alpha=0.5, density = True)
print()
whichFitsBetter(numberVert)<jupyter_output>Warning: Maximum number of function evaluations has been exceeded.
Optimization terminated successfully.
         Current function value: 1.517887
         Iterations: 21
         Function evaluations: 43
Optimization terminated successfully.
         Current function value: 1.684943
         Iterations: 12
         Function evaluations: 24
{'fraction_non_alphanumeric': 0.1385872272591412, 'fraction_numerical': 0.0340739264720534, 'mean_word_length': 2.608195542774982, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '114746', 'license': 'MIT', 'n_tokens_mistral': 3627, 'n_tokens_neox': 3267, 'n_words': 882, 'path': 'notebooks/randomEMF/yakindu-github.ipynb', 'repo_name': 'Antolin1/DMG-Python', 'size': '10037'}