<jupyter_start><jupyter_text>[View in Colaboratory](https://colab.research.google.com/github/RXV06021/basicPy/blob/master/basicPandas.ipynb)  Pandasでデータ集計をする際に最低限覚えておきたいメソッドhttps://qiita.com/kyo-bad/items/f5ddb7e4b8e7ad9103c5使用するライブラリ：  *   sklearn　datasets*   pandas*   collections　OrderedDict*   matplotlib.pyplot*   seabornhttp://codechord.com/2012/01/readme-markdown/  http://www.stdio.h.kyoto-u.ac.jp/~sakura/ipynb/usage+of+jupyter+notebook.html  前準備<jupyter_code>from sklearn import datasets
import pandas as pd
from collections import OrderedDict

iris = datasets.load_iris()

df = pd.concat([pd.DataFrame(iris.data,columns=iris.feature_names),pd.DataFrame(iris.target,columns=["species"])],axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>集計<jupyter_code>df.describe()
df["petal length (cm)"].describe()
df["species"].value_counts()<jupyter_output><empty_output><jupyter_text>いわゆるダミー変数が作成可能<jupyter_code>pd.get_dummies(df["species"]).iloc[[0,1,2,50,51,52,100,101,102]]<jupyter_output><empty_output><jupyter_text>dfのソートが可能（ascendingで昇順降順を指定）<jupyter_code>df.sort_values("sepal length (cm)",ascending=False)<jupyter_output><empty_output><jupyter_text>複数指定でソート<jupyter_code>df.sort_values(["sepal length (cm)","sepal width (cm)"],ascending=False)<jupyter_output><empty_output><jupyter_text>**groupby**  groupbyオブジェクトを再利用できるため、同じような集計を複数かけたいときはgroupbyオブジェクトを変数に格納したほうが早い<jupyter_code>df_groupby = df.groupby("species",as_index=False)
df_groupby.mean()<jupyter_output><empty_output><jupyter_text>**groupby.agg**  辞書形式で指定することで、カラムごとの個別集計が可能（ただし、一つのカラムに複数の集計を指定した場合、マルチカラムになるため、注意が必要）<jupyter_code>df_groupby.agg({"sepal length (cm)": "mean",
                "sepal width (cm)": ["mean","count"],
                "petal length (cm)": ["max","min"],
                "petal width (cm)": ["sum","var","std"]})<jupyter_output><empty_output><jupyter_text>また、上記は順不同になるため、順序を指定したい場合はOrderedDictを利用する<jupyter_code>df_groupby.agg(OrderedDict((["sepal length (cm)", "mean"],
                            ["sepal width (cm)", ["mean","count"]],
                            ["petal length (cm)", ["max","min"]],
                            ["petal width (cm)", ["sum","var","std"]])))<jupyter_output><empty_output><jupyter_text>**to_csv**  index=Falseにすると、次の読み込みが楽。  encodingを指定しないと読み込めない時がある（とくにWindows）。<jupyter_code>df.to_csv("test.csv",index=False,encoding="utf8")
pd.read_csv("test.csv")<jupyter_output><empty_output><jupyter_text>可視化<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>**箱ひげ図**<jupyter_code>sns.boxplot(data=df, x="species", y="sepal length (cm)")<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/categorical.py:454: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  box_data = remove_na(group_data)<jupyter_text>**seaborn pairplot**<jupyter_code>sns.pairplot(data=df)<jupyter_output><empty_output><jupyter_text>セグメント別に色分けすることもできる<jupyter_code>sns.pairplot(data=df, hue="species")<jupyter_output><empty_output><jupyter_text>**seaborn jointplot**  1つのカラムについて，分布を確認する方法<jupyter_code>sns.distplot(df["sepal length (cm)"], rug=True,)<jupyter_output><empty_output><jupyter_text>データフレームの前の行との差分を元のデータフレームに追加**...  df.Shift(1)**   　1行下にずらす  **...  df.fillna(0)**    　NAにゼロを代入する<jupyter_code>import pandas as pd

df = pd.DataFrame({'x': [-3954305.448, -3954305.448, -3954305.443, -3954305.444, -3954305.446, -3954305.445, -3954305.444]})
df

df['diff'] = (df['x'] - df['x'].shift(1)).fillna(0)

df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16732172936552497, 'fraction_numerical': 0.03425042111173498, 'mean_word_length': 7.585542168674698, 'pattern_counts': {'":': 4, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '238122', 'license': 'MIT', 'n_tokens_mistral': 1722, 'n_tokens_neox': 1642, 'n_words': 131, 'path': 'basicPandas.ipynb', 'repo_name': 'RXV06021/basicPy', 'size': '3562'}