<jupyter_start><jupyter_text>- factor_analyzer网站: https://github.com/EducationalTestingService/factor_analyzer  Loading Data<jupyter_code>df= pd.read_csv("bfi.csv")
# Dropping unnecessary columns
df.drop(['Unnamed: 0', 'gender', 'education', 'age'],axis=1,inplace=True)
# Dropping missing values rows
df.dropna(inplace=True)
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Adequacy Test- 测试数据是否可以进行因子分析<jupyter_code>from factor_analyzer.factor_analyzer import calculate_bartlett_sphericity
chi_square_value,p_value=calculate_bartlett_sphericity(df)
chi_square_value, p_value
from factor_analyzer.factor_analyzer import calculate_kmo
kmo_all,kmo_model=calculate_kmo(df)
print(kmo_model)<jupyter_output>0.848539722194922<jupyter_text>Choosing the Number of Factors<jupyter_code># Create factor analysis object and perform factor analysis
fa = FactorAnalyzer(25, rotation=None)
fa.fit(df)
# Check Eigenvalues
ev, v = fa.get_eigenvalues()
ev
# Create scree plot using matplotlib
plt.scatter(range(1,df.shape[1]+1),ev)
plt.plot(range(1,df.shape[1]+1),ev)
plt.title('Scree Plot')
plt.xlabel('Factors')
plt.ylabel('Eigenvalue')
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>Performing Factor Analysis<jupyter_code>fa = FactorAnalyzer(5, rotation="varimax")
fa.fit(df)
# 25*5(变量个数*因子个数)
fa.loadings_<jupyter_output><empty_output><jupyter_text>下面图像的含义- 每一个变量都可以由5个隐变量所表示- $A1 = 0.11*F1 + 0.04*F2 + 0.023*F3 + -0.43*F4 + -0.078*F5$- $A2 = 0.13*F1 + 0.21*F2 + 0.14*F3 + 0.63*F4 + 0.062*F5$<jupyter_code># 画为可视化的矩阵
import seaborn as sns

df_cm = pd.DataFrame(fa.loadings_, index=df.columns)

plt.figure(figsize = (12,10))
sns.heatmap(df_cm, annot=True, cmap="BuPu")
# 由于有负数, 我们看绝对值
# 我们顺便查看一下sns如何插入标题, 修改index字体的大小
df_cm = pd.DataFrame(np.abs(fa.loadings_), index=df.columns)

plt.figure(figsize = (14,14))
ax = sns.heatmap(df_cm, annot=True, cmap="BuPu")
# 设置y轴的字体的大小
ax.yaxis.set_tick_params(labelsize=15) 
plt.title('Factor Analysis', fontsize='xx-large')
# Set y-axis label
plt.ylabel('Sepal Width', fontsize='xx-large')
plt.savefig('factorAnalysis.png', dpi=500)
pd.DataFrame(fa.loadings_).apply(lambda x:np.sum(x**x), axis=1)
ev, v = fa.get_eigenvalues()
ev
v
# Get variance of each factors
fa.get_factor_variance()<jupyter_output><empty_output><jupyter_text>转换为新的变量- 每组数据使用5个隐特征进行表示<jupyter_code>fa.transform(df).shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1451409339503576, 'fraction_numerical': 0.03828355069415229, 'mean_word_length': 8.474103585657371, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '21641', 'license': 'MIT', 'n_tokens_mistral': 1122, 'n_tokens_neox': 1065, 'n_words': 144, 'path': '2019_08_12/FactorAnalysis/FA.ipynb', 'repo_name': 'wmn7/PyTorch-Practice', 'size': '2377'}