<jupyter_start><jupyter_text>Atividade 10  Descrição da atividade 1. Escolha algum algoritmo de REGRA DE ASSOCIAÇÃO apresentados na aula, aplique ao seu projeto.- Lembre-se de variar os parâmetros.2. Defina a metodologia experimental.- Divisão da base de dados, range de parâmetros e métricas de avaliação.3. Descreva os resultados obtidos.4. Entregue a seção 2.2, 3.4 e 4.1 do artigo.  Referências: * [Aula 10 - Regras de Associação](https://drive.google.com/file/d/16ybZx_CleXpl0u5T4GndIcefZ0EBBi_0/view)*   [Implementing Apriori algorithm in Python](https://www.geeksforgeeks.org/implementing-apriori-algorithm-in-python/)*   [Association Rule Learning with Scikit-learn](https://www.kaggle.com/sangwookchn/association-rule-learning-with-scikit-learn)*   [Introduction to Apriori Algorithm in Python](https://intellipaat.com/blog/data-science-apriori-algorithm/)* [Apriori: Association Rule Mining In-depth Explanation and Python Implementation](https://towardsdatascience.com/apriori-association-rule-mining-explanation-and-python-implementation-290b42afdfc6)* [Association Rule Mining via Apriori Algorithm in Python](https://stackabuse.com/association-rule-mining-via-apriori-algorithm-in-python/)* [Frequent Itemsets via Apriori Algorithm](http://rasbt.github.io/mlxtend/user_guide/frequent_patterns/apriori/)*   [Understand and Build FP-Growth Algorithm in Python](https://towardsdatascience.com/understand-and-build-fp-growth-algorithm-in-python-d8b989bab342)*   [Implementing Apriori and FP-growth](https://www.oreilly.com/library/view/practical-machine-learning/9781784399689/ch07s02.html)*   [FP Growth: Frequent Pattern Generation in Data Mining with Python Implementation](https://towardsdatascience.com/fp-growth-frequent-pattern-generation-in-data-mining-with-python-implementation-244e561ab1c3)  Bibliotecas Utilizadas<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori,association_rules
from mlxtend.frequent_patterns import association_rules
import itertools
import warnings
warnings.filterwarnings("ignore")
warnings.simplefilter(action='ignore', category=FutureWarning)

%matplotlib inline

from google.colab import drive
drive.mount('/content/drive', force_remount=True)<jupyter_output>Mounted at /content/drive<jupyter_text>Base A base é de dados singulares, logo foi criada uma lógica, após normalização dos dados. Os campos que apresentarem valores expressivos, maiores que 0.5, terão seus campos setados para "1" e aqueles que não atenderem a regra terão serão setados para "0".<jupyter_code>path = '/content/drive/Shareddrives/MINERAÇÃO DE DADOS ENG74291/Assign10/dadosTedioB.xlsx'
dataset = pd.read_excel(path, index_col=0)
dataset.drop(["game_name"], axis=1,inplace=True)
dataset = dataset.dropna()
dataset.reset_index(drop=True,inplace=True)

scaler = MinMaxScaler(feature_range=(0,1))
scaler.fit(dataset)
dataset2 = scaler.transform(dataset)

dataset2 = pd.DataFrame(dataset2)
dataset2.columns = dataset.columns

for i in dataset2.columns:
  for j in dataset2.index:
    if dataset2[i][j] >= 0.5:
      dataset2[i][j]= 1
    else:  
       dataset2[i][j] = 0

dataset = pd.DataFrame(dataset2)
dataset = dataset.rename_axis(columns=None)

df = dataset.loc[:,'p1':'p24']<jupyter_output><empty_output><jupyter_text>Mostrar o resutado após transformação<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>Este código é para a transformação, no nosso caso não foi necessário devido a transformação anterior. A nossa base diverge deste tipo de estudo.<jupyter_code># # Transformação desnecessária
# te = TransactionEncoder()
# te_ary = te.fit(dataset).transform(dataset)
# df = pd.DataFrame(te_ary, columns=te.columns_)
# df
df.shape
import seaborn as sns

df_tratado = pd.DataFrame(df)
df_tratado['angry'] =  dataset['angry']

filtro = df_tratado['angry'] >= 0.5
if df_tratado[filtro] is True:
      df_tratado['angry'] = 1

plt.figure(figsize=(30, 15))
sns.heatmap(df_tratado.corr(),
            annot = True,
            fmt = '.2f',
            cmap='Blues')
plt.title('Correlação entre variáveis do dataset de tédio')
plt.show()
# criar os conjuntos de dados e classes para treinamento e teste
from sklearn.model_selection import train_test_split

X_treinamento, X_teste, y_treinamento, y_teste = train_test_split(df_tratado.drop(columns=['angry']), df_tratado.angry, test_size=0.3)

from sklearn.ensemble import RandomForestClassifier
clf_RFC  = RandomForestClassifier()
clf_RFC.fit(X_treinamento, y_treinamento)

# Mostrando importância de cada feature
clf_RFC.feature_importances_

importances = pd.Series(data=clf_RFC.feature_importances_, index=df_tratado.columns[0:24])
sns.barplot(x=importances, y=importances.index, orient='h').set_title('Importância de cada feature')<jupyter_output><empty_output><jupyter_text>Gráfico, mas achei o resultado meio estranho. Qualquer coisa podemos mudar ou alterar o tipo de gráfico.<jupyter_code>p = df.copy()
for i in range(1, len(p.columns)+1):
    p["Cat"] = np.where(p[p.columns[i]] == 1, 1, 0)
    p[p.columns[i]]= i
    g =sns.scatterplot(p.index, p[p.columns[i]], hue = p.Cat, legend = False)
    g.yaxis.set_label_text('Capturas')
    g.set_yticks(np.arange(1, len(p.columns)))
    g.set_xticks(df.index)
    g.set_yticklabels(df.columns)
    plt.title("Estrutura de dados")<jupyter_output><empty_output><jupyter_text>Encontrar a frequência de itens.<jupyter_code>df.sum()<jupyter_output><empty_output><jupyter_text>Apriori Cálculo do Suporte com mínimo de 60%.<jupyter_code>apriori(df, min_support=0.6, use_colnames=True)<jupyter_output><empty_output><jupyter_text>Selecionando e filtrando os resultados.<jupyter_code>frequent_itemsets = apriori(df, min_support=0.6, use_colnames=True)
frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(lambda x: len(x))
frequent_itemsets
frequent_itemsets[ (frequent_itemsets['length'] == 2) &
                   (frequent_itemsets['support'] >= 0.7) ]
frequent_itemsets[ frequent_itemsets['itemsets'] == {'p1', 'p13'} ]
te = TransactionEncoder()
oht_ary = te.fit(df).transform(df, sparse=True)
sparse_df = pd.DataFrame.sparse.from_spmatrix(oht_ary, columns=te.columns_)
sparse_df
apriori(sparse_df, min_support=0.5, use_colnames=True)<jupyter_output><empty_output><jupyter_text>Definindo um valor limite para o valor de suporte e calculando o valor de suporte.<jupyter_code>df_new = apriori(df, min_support = 0.2, use_colnames = True)
df_new<jupyter_output><empty_output><jupyter_text>Vendo os valores de interpretação usando a função da regra de associação.<jupyter_code>df_ar = association_rules(df_new, metric = "confidence", min_threshold = 0.6)
df_ar
df_results = list(df_ar)
print(len(df_results))
df_results<jupyter_output><empty_output><jupyter_text>FP-Growth(Frequent Pattern Growth) utilizando pyfpgrowth<jupyter_code>!pip install pyfpgrowth
import pyfpgrowth<jupyter_output><empty_output><jupyter_text>Encontrar padrões frequentes em capturas que ocorrem acima do limite de suporte<jupyter_code>patterns = pyfpgrowth.find_frequent_patterns(df, 2)
patterns<jupyter_output><empty_output><jupyter_text>Use gerar regras de associação para encontrar padrões que estão associados a outro com uma certa probabilidade mínima<jupyter_code>rules = pyfpgrowth.generate_association_rules(patterns, 0.7)
rules<jupyter_output><empty_output><jupyter_text>FP-Growth utilizando o Spark  Referência * [Market Basket Analysis using PySpark](https://towardsdatascience.com/market-basket-analysis-using-pysparks-fpgrowth-55c37ebd95c0)<jupyter_code>%%capture
!sudo apt-get update --fix-missing
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz
#!wget -q https://downloads.apache.org/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz
!mv spark-3.0.0-bin-hadoop3.2.tgz sparkkk
!tar xf sparkkk
!pip install -q findspark
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.0.0-bin-hadoop3.2"
import findspark
findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()
spark = SparkSession \
.builder \
.appName('fpgrowth') \
.getOrCreate()
spark
from pyspark.sql import functions as F
from pyspark.ml.fpm import FPGrowth
import pandas
sparkdata = spark.createDataFrame(df)
capturedata = sparkdata.dropDuplicates(['p1', 'p5']).sort('p1')
capturedata = capturedata.groupBy("p1").agg(F.collect_list("p5")).sort('p1')<jupyter_output><empty_output><jupyter_text>Executando o FP-Growth minerando um conjuntos de itens frequentes usando support, lift, e confidence<jupyter_code>fpGrowth = FPGrowth(itemsCol="collect_list(p5)", minSupport=0.006, minConfidence=0.006)
model = fpGrowth.fit(capturedata)<jupyter_output><empty_output><jupyter_text>Exibir conjuntos de itens frequentes.<jupyter_code>model.freqItemsets.show()
items = model.freqItemsets<jupyter_output><empty_output><jupyter_text>Exibir regras de associação geradas.<jupyter_code>model.associationRules.show()
rules = model.associationRules<jupyter_output><empty_output><jupyter_text>Examinar os itens de entrada em relação a todas as regras de associação e resumir os consequentes como predição<jupyter_code>model.transform(capturedata).show()
transformed = model.transform(capturedata)
result_pdf = items.select("*").toPandas()
result_pdf.head()
rules_pdf = rules.select("*").toPandas()
rules_pdf.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1164021164021164, 'fraction_numerical': 0.019089117128332816, 'mean_word_length': 6.590551181102362, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 76, 'https://': 12, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '171429', 'license': 'MIT', 'n_tokens_mistral': 3677, 'n_tokens_neox': 3511, 'n_words': 674, 'path': '2020.1/Grupo 2/code/entregas/Assign10_Teste.ipynb', 'repo_name': 'prof-alexandre-maciel/mineracao-de-dados', 'size': '9639'}