<jupyter_start><jupyter_text>Lo hago a 4 dígitos<jupyter_code>df = df[df["level"]==4]<jupyter_output><empty_output><jupyter_text>necesito una matriz de documentosXvocabulario --> paisXproducto<jupyter_code>matrix = df.pivot_table(index=['rep_iso','year'],columns='SITC',values='value')
matrix = matrix.fillna(0)
matrix = matrix.to_sparse(fill_value=0)<jupyter_output><empty_output><jupyter_text>Utilizo TF-IDF para normalizar la importancia relativa<jupyter_code>tfidf = TfidfTransformer()

matrix = tfidf.fit_transform(matrix)
matrix.shape
### para hacer las pruebas de TF-IDF me quedo sólo con 30 componentes. 
# #10 components
# lda_model_10 = LatentDirichletAllocation(n_components=10) #instance
# lda_model_10.fit(matrix) #fit
# pickle.dump(lda_model_10, open("lda10.pkl", 'wb')) #save
# #20 components
# lda_model_15 = LatentDirichletAllocation(n_components=15, max_iter=75, random_state=1234, learning_method='online',batch_size=200,learning_decay=0.5) #instance
# lda_model_15.fit(matrix) #fit
# pickle.dump(lda_model_15, open("lda15.pkl", 'wb')) #save

# #20 components
lda_model_20 = LatentDirichletAllocation(n_components=20, max_iter=75, random_state=1234,batch_size=200, learning_method='batch') #instance
lda_model_20.fit(matrix) #fit
pickle.dump(lda_model_20, open("lda20.pkl", 'wb')) #save
#30 components
lda_model_30 = LatentDirichletAllocation(n_components=30, max_iter=75, random_state=1234,batch_size=200, learning_method='batch') #instance
lda_model_30.fit(matrix) #fit
pickle.dump(lda_model_30, open("lda30.pkl", 'wb')) #save
# #50 components
# lda_model_50 = LatentDirichletAllocation(n_components=50) #instance
# lda_model_50.fit(matrix) #fit
# pickle.dump(lda_model_50, open("lda50.pkl", 'wb')) #save
  
# load the model from disk
#lda_model = pickle.load(open(filename, 'rb'))
productos = df.pivot_table(index=['rep_iso','year'],columns='SITC',values='value').columns

lda_model_30.components_.shape
product_df = pd.read_excel("data/UN Comtrade Commodity Classifications.xlsx")
ClassS3 = product_df[product_df["Classification"]=='S3']
Codes = ClassS3.set_index('Code')['Description'].to_dict()
#Codes = ClassH0[['Code','Description']].to_dict(orient='index')<jupyter_output><empty_output><jupyter_text>20 componentes<jupyter_code>norm_components = lda_model_20.components_ / lda_model_20.components_.sum(axis=1)[:, np.newaxis]
for i in range(lda_model_20.components_.shape[0]):
    topic = norm_components[i]
    print("Componente %d:" % (i+1))
    top_n = [productos[j] for j in np.argsort(topic)[:-15 :-1]]
    top_n_props = topic[np.argsort(topic)][:-15 :-1]
    for j,producto in enumerate(top_n):
        try:
            print("{}-{}. Prop: {}%".format(producto, Codes[producto],+np.round(top_n_props[j]*100,3)))
        except:
            pass
    print("----------------------")<jupyter_output>Componente 1:
2861-Uranium ores and concentrates. Prop: 36.927%
2875-Zinc ores and concentrates. Prop: 19.275%
2874-Lead ores and concentrates. Prop: 9.719%
3330-Crude petroleum. Prop: 3.138%
0344-Fish fillets, frozen. Prop: 2.776%
2841-Nickel ores and concentrates. Prop: 2.66%
0430-Barley, unmilled. Prop: 2.602%
2831-Copper ores and concentrates. Prop: 1.919%
1123-Beer made from malt (including ale, stout and porter). Prop: 1.479%
2222-Soya beans. Prop: 1.01%
2876-Tin ores and concentrates. Prop: 0.945%
2877-Manganese ores and concentrates (including manganiferous iron ores and co. Prop: 0.907%
2116-Sheepskins and lambskins (not those of Astrakhan, Broadtail, Caracul, Per. Prop: 0.728%
2783-Sodium chloride, pure, and common salt (including table salt and denature. Prop: 0.702%
----------------------
Componente 2:
2841-Nickel ores and concentrates. Prop: 42.179%
1212-Tobacco, wholly or partly stemmed/stripped. Prop: 16.088%
2724-Carnallite, sylvite and other crude natural potassium sal[...]<jupyter_text>Guardo los resultados para analizarlos en R<jupyter_code>components_df = pd.DataFrame(norm_components)
components_df.columns = [int(j) for j in productos]

components_df.to_csv("results/Dist_cadenas20.csv", index=False)
result = lda_model_20.transform(matrix)
result = pd.DataFrame(result)
result.index = df.pivot_table(index=['rep_iso','year'],columns='SITC',values='value').index
result = result.reset_index()
result.to_csv("results/Dist_paises20.csv",index=False)<jupyter_output><empty_output><jupyter_text>30 componentes<jupyter_code>norm_components = lda_model_30.components_ / lda_model_30.components_.sum(axis=1)[:, np.newaxis]
for i in range(lda_model_30.components_.shape[0]):
    topic = norm_components[i]
    print("Componente %d:" % (i+1))
    top_n = [productos[j] for j in np.argsort(topic)[:-15 :-1]]
    top_n_props = topic[np.argsort(topic)][:-15 :-1]
    for j,producto in enumerate(top_n):
        try:
            print("{}-{}. Prop: {}%".format(producto, Codes[producto],+np.round(top_n_props[j]*100,3)))
        except:
            pass
    print("----------------------")<jupyter_output>Componente 1:
2861-Uranium ores and concentrates. Prop: 37.268%
2851-Aluminium ores and concentrates. Prop: 30.884%
0422-Rice, husked but not further prepared (cargo rice or brown rice). Prop: 13.696%
0344-Fish fillets, frozen. Prop: 2.384%
1123-Beer made from malt (including ale, stout and porter). Prop: 1.208%
2485-Wood of non-coniferous species (including strips and friezes for parquet. Prop: 0.921%
2484-Wood of non-coniferous species, sawn or chipped lengthwise, sliced or pee. Prop: 0.56%
2875-Zinc ores and concentrates. Prop: 0.548%
1222-Cigarettes containing tobacco. Prop: 0.494%
7863-Containers (including containers for the transport of fluids) specially d. Prop: 0.366%
0019-Live animals, n.e.s.. Prop: 0.326%
2783-Sodium chloride, pure, and common salt (including table salt and denature. Prop: 0.272%
2482-Wood of coniferous species, sawn or chipped lengthwise, sliced or peeled,. Prop: 0.247%
0355-Flours, meals and pellets of fish, fit for human consumption. Prop: 0.225%
--------[...]<jupyter_text>Guardo los resultados para analizarlos en R<jupyter_code>components_df = pd.DataFrame(norm_components)
components_df.columns = [int(j) for j in productos]

components_df.to_csv("results/Dist_cadenas30.csv", index=False)
result = lda_model_30.transform(matrix)
result = pd.DataFrame(result)
result.index = df.pivot_table(index=['rep_iso','year'],columns='SITC',values='value').index
result = result.reset_index()
result.to_csv("results/Dist_paises30.csv",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14720420142106888, 'fraction_numerical': 0.06271238801359283, 'mean_word_length': 5.661522633744856, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '453972', 'license': 'MIT', 'n_tokens_mistral': 2633, 'n_tokens_neox': 2347, 'n_words': 501, 'path': 'desagregado/LDA/models/.ipynb_checkpoints/LDA_TF-IDF-checkpoint.ipynb', 'repo_name': 'DiegoKoz/master_thesis', 'size': '6474'}