<jupyter_start><jupyter_text>Creating a pandas dataframe<jupyter_code>house_prices_df = pd.read_csv('../data/house-prices/train.csv')
melb_data = pd.read_csv('../data/melbourne_housing_snapshot/melb_data.csv')<jupyter_output><empty_output><jupyter_text>Taking a look at the dataframe.<jupyter_code>house_prices_df.head()
melb_data.head()
house_prices_df.dtypes<jupyter_output><empty_output><jupyter_text>Descibing the dataframes.<jupyter_code>house_prices_df.describe()<jupyter_output><empty_output><jupyter_text>Describe a particular column<jupyter_code>house_prices_df['MSSubClass'].describe()<jupyter_output><empty_output><jupyter_text>Get the dataframe shape and columns<jupyter_code>house_prices_df.shape
house_prices_df.columns<jupyter_output><empty_output><jupyter_text>Changing the column names of the dataframes<jupyter_code>d = [[1, "a", "foo", 3.0]]
df = pd.DataFrame(data=d)
df
df.columns = ['id', 'x1', 'x2', 'x3']
df
columns_map = {"id": "Id", "SalePrice": "SalePriceDollars"}

changed_cols_df = house_prices_df.rename(index=str, 
                                         columns=columns_map)
changed_cols_df.head()<jupyter_output><empty_output><jupyter_text>Unique values in a column<jupyter_code>house_prices_df['MSSubClass'].unique()
len(house_prices_df['MSSubClass'].unique())<jupyter_output><empty_output><jupyter_text>Value Counts<jupyter_code>house_prices_df["MSSubClass"].value_counts()<jupyter_output><empty_output><jupyter_text>Order by and group by<jupyter_code>df_agg = house_prices_df.groupby(['MSSubClass'])
(df_agg
    .size()
    .sort_values(ascending=False))<jupyter_output><empty_output><jupyter_text>Filtering data<jupyter_code>house_prices_df[house_prices_df["MSSubClass"] == 20].head()<jupyter_output><empty_output><jupyter_text>Membership in dataframe<jupyter_code>house_prices_df["MSSubClass"].isin(["20","60"])
validMembership = house_prices_df[house_prices_df["MSSubClass"].isin(["20","60"])]
validMembership.shape<jupyter_output><empty_output><jupyter_text>Concatenating to the dataframe.<jupyter_code>arr = np.arange(12).reshape((3, 4))
arr
np.concatenate([arr, arr], axis=1)<jupyter_output><empty_output><jupyter_text>Handling missing data.For various reasons, many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. Such datasets however are incompatible with scikit-learn estimators which assume that all values in an array are numerical, and that all have and hold meaning. A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. However, this comes at the price of losing data which may be valuable (even though incomplete). A better strategy is to impute the missing values, i.e., to infer them from the known part of the data.<jupyter_code>house_prices_df.head()<jupyter_output><empty_output><jupyter_text>Many will recognise the fillna method<jupyter_code>melb_data['YearBuilt_new'] = melb_data['YearBuilt']
melb_data['YearBuilt_new'] = melb_data['YearBuilt_new'].fillna(1964)
melb_data.head()<jupyter_output><empty_output><jupyter_text>Another way is using the scikit-learmn imputer. The advantage of the imputer class is that the imputer class will not actually compute the matrix and you can plug it in a sklearn pipeline in case of a large matrix.<jupyter_code>features_in_focus = ['BuildingArea', 'YearBuilt']
melb_data = melb_data[features_in_focus]
melb_data.head()
from sklearn.impute import SimpleImputer

my_imputer = SimpleImputer(missing_values=np.nan,
                           strategy='mean')
imputed_melb_data = my_imputer.fit_transform(melb_data)
imputed_melb_data[0]<jupyter_output><empty_output><jupyter_text>Discretization and Binning<jupyter_code>ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]
bins = [18, 25, 35, 60, 100]
cats = pd.cut(ages, bins, right=False)
cats.codes<jupyter_output><empty_output><jupyter_text>Getting a particular data<jupyter_code>house_prices_df.loc[0, "MSSubClass"]
house_prices_df["MSSubClass"][0]<jupyter_output><empty_output><jupyter_text>Sorting<jupyter_code>house_prices_df.sort_values("MSSubClass",
                            ascending=False).head()<jupyter_output><empty_output><jupyter_text>Reshaping and Pivoting<jupyter_code>data = [["memories","book","q1",10],
        ["dreams","book","q2",20], 
        ["reflections","book","q3",30],
        ["how to build a house","book","q4",40],
        ["wonderful life","music","q1",10], 
        ["million miles","music","q2",20],
        ["run away","music","q3",30],
        ["mind and body","music","q4",40]]

df_products = pd.DataFrame(data)
df_products.columns = ["product","category","quarter","profit"]
df_products.head()
df_products.pivot_table(index="category",
                        columns=["quarter"],
                        values=["profit"],
                        aggfunc=np.sum)<jupyter_output><empty_output><jupyter_text>Merges and Joins<jupyter_code>llist = pd.DataFrame([["bob", "2015-01-13", 4],
                      ["alice", "2015-04-23",10]])
llist.columns = ["name","date","duration"]
right = pd.DataFrame([["alice", 100],
                      ["bob", 23]])
right.columns = ["name", "upload"]
df = pd.merge(llist, right,
              on=['name', 'name'])
df.head()
llist.append(right, sort=True)<jupyter_output><empty_output><jupyter_text>Working on time series data<jupyter_code>crypto_df = pd.read_csv("../data/crypto_data/LTC-USD.csv",
                        names=["time", "low", "high", "open", "close", "volume"])

FUTURE_PERIOD_PREDICT = 3
crypto_df["future"] = crypto_df["close"].shift(-FUTURE_PERIOD_PREDICT)

crypto_df.head()
crypto_df.tail()<jupyter_output><empty_output><jupyter_text>Function application, transformations and mapping<jupyter_code>def classify(current, future):
    if float(future) > float(current):
        return 1
    else:
        return 0

crypto_df["target"] = crypto_df[["close", "future"]].apply(lambda x: classify(*x), axis=1)

crypto_df.head()
house_prices_df["SalePrice"] = house_prices_df["SalePrice"].apply(float)
house_prices_df.head()<jupyter_output><empty_output><jupyter_text>How to do machine learning. An example using sklearn.<jupyter_code>bike_sharing_df = pd.read_csv("../data/bike-sharing/hour.csv")

featureCols = ["season", "yr", "mnth", "hr", 
               "holiday", "weekday", "workingday",
               "weathersit", "temp", "atemp",
               "hum", "windspeed"]
output_col = "cnt"

bike_sharing_df_X = bike_sharing_df[featureCols]
bike_sharing_df_y = bike_sharing_df[output_col]
bike_sharing_df.head()
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    bike_sharing_df_X, bike_sharing_df_y, test_size=0.33, random_state=42)

lm = LinearRegression()
model = lm.fit(X_train, y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13694452512343885, 'fraction_numerical': 0.017426662794074933, 'mean_word_length': 6.049129989764586, 'pattern_counts': {'":': 2, '<': 93, '<?xml version=': 0, '>': 94, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '414502', 'license': 'MIT', 'n_tokens_mistral': 2556, 'n_tokens_neox': 2412, 'n_words': 418, 'path': 'notebooks/common pandas transformations.ipynb', 'repo_name': 'infinite-Joy/datahack2018', 'size': '6886'}