<jupyter_start><jupyter_text>Fuel Econemy Analyse<jupyter_code>import pandas as pd
import os
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
cwd = os.getcwd()
os.listdir(cwd)
df_18 = pd.read_csv('all_alpha_18.csv')
df_18.head()
df_08 = pd.read_csv('all_alpha_08.csv')
df_08.head()
df_18.info()
df_08.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2404 entries, 0 to 2403
Data columns (total 18 columns):
Model                   2404 non-null object
Displ                   2404 non-null float64
Cyl                     2205 non-null object
Trans                   2205 non-null object
Drive                   2311 non-null object
Fuel                    2404 non-null object
Sales Area              2404 non-null object
Stnd                    2404 non-null object
Underhood ID            2404 non-null object
Veh Class               2404 non-null object
Air Pollution Score     2404 non-null object
FE Calc Appr            2205 non-null object
City MPG                2205 non-null object
Hwy MPG                 2205 non-null object
Cmb MPG                 2205 non-null object
Unadj Cmb MPG           2205 non-null float64
Greenhouse Gas Score    2205 non-null object
SmartWay                2404 non-null object
dtypes: float64(2), object(16)
memory usage: 338.1+ KB<jupyter_text>find duplicated<jupyter_code>sum(df_08.duplicated())
df_08.describe()
df_18.shape<jupyter_output><empty_output><jupyter_text>Find the uniques<jupyter_code>df_18.SmartWay.unique()
df_08.SmartWay.unique()
df_08['Sales Area'].unique()
df_18['Cert Region'].unique()
df_08['Trans'].unique()
len(df_08['Trans'].unique())
df_08.Fuel.unique()
df_18.Fuel.unique()<jupyter_output><empty_output><jupyter_text>Cleaning Column LabelsUse `all_alpha_08.csv` and `all_alpha_18.csv`  Drop Extraneous Columns<jupyter_code>df_08.head(1)<jupyter_output><empty_output><jupyter_text>删除多余的行列<jupyter_code>df_08.drop(['Stnd', 'Underhood ID', 'FE Calc Appr', 'Unadj Cmb MPG'], axis=1, inplace=True)
df_08.head(1)
df_18.drop(['Stnd', 'Stnd Description', 'Underhood ID', 'Comb CO2'], axis=1,inplace=True)
df_18.head(1)<jupyter_output><empty_output><jupyter_text>Rename Columns<jupyter_code>df_08.rename(index=str, columns={'Sales Area': 'Cert Region'}, inplace=True)
df_08.head(1)
# replace spaces with underscores and lowercase labels for 2008 dataset
df_08.rename(columns=lambda x: x.strip().lower().replace(' ', '_'), inplace=True)
df_08.head(1)
df_18.rename(columns=lambda x: x.strip().lower().replace(' ', '_'), inplace=True)
df_08.columns == df_18.columns
# make sure they're all identical like this
# 数组中所有都是True 才是True
(df_08.columns == df_18.columns).all()
df_08.to_csv('data_08.csv', index=False)
df_18.to_csv('data_18.csv', index=False)<jupyter_output><empty_output><jupyter_text>Filter, Drop Nulls, DedupeUse `data_08.csv` and `data_18.csv`<jupyter_code>df_08 = pd.read_csv('data_08.csv')
df_08.head()
df_18 = pd.read_csv('data_18.csv')
df_18.head()
df_08.shape
df_18.shape<jupyter_output><empty_output><jupyter_text>Filter by Certification Region<jupyter_code># filter datasets for rows following California standards
df_08.cert_region.unique()
df_08 = df_08.query('cert_region == "CA"')
df_08.cert_region.unique()
df_18.columns<jupyter_output><empty_output><jupyter_text>删除列<jupyter_code>df_08.drop('cert_region', axis=1, inplace=True)
df_08.head(1)
df_18.shape
df_08.shape<jupyter_output><empty_output><jupyter_text>Drop Rows with Missing Values<jupyter_code>df_08.isnull().sum()
df_18.isnull().sum()<jupyter_output><empty_output><jupyter_text>删除缺数据的行<jupyter_code>df_18.dropna(inplace=True)
df_08.dropna(inplace=True)
df_08.isnull().sum().any()
df_18.isnull().sum().any()<jupyter_output><empty_output><jupyter_text>Dedupe Data 删除重复数据<jupyter_code>sum(df_08.duplicated())
sum(df_18.duplicated())
df_08.drop_duplicates(inplace=True)
df_18.drop_duplicates(inplace=True)
sum(df_08.duplicated())
sum(df_18.duplicated())
# save progress for the next section
df_08.to_csv('data_08.csv', index=False)
df_18.to_csv('data_18.csv', index=False)<jupyter_output><empty_output><jupyter_text>Inspecting Data TypesUse the space below to explore `data_08.csv` and `data_18.csv` to answer the quiz questions below regarding datatypes.<jupyter_code>df_08.dtypes
df_18.dtypes<jupyter_output><empty_output><jupyter_text>Fixing `cyl` Data Type- 2008: extract int from string- 2018: convert float to int<jupyter_code>df_08.cyl.dtypes
df_08.cyl.value_counts()
# 提取字符串中的数字
df_08['cyl'] = df_08['cyl'].str.extract('(\d+)', expand=True).astype(int)
df_18.cyl.dtypes
# 转换数据dataframe的数据类型
df_18['cyl'] = df_18.cyl.astype(int)
df_18.cyl.dtypes
df_08.to_csv('data_08.csv', index=False)
df_18.to_csv('data_18.csv', index=False)<jupyter_output><empty_output><jupyter_text>Fixing `air_pollution_score` Data Type- 2008: convert string to float- 2018: convert int to float<jupyter_code>df_08 = pd.read_csv('data_08.csv')
df_18 = pd.read_csv('data_18.csv')
# try using Pandas to_numeric or astype function to convert the
# 2008 air_pollution_score column to float -- this won't work
df_08.air_pollution_score.head(3)
df_08.air_pollution_score.astype(float)<jupyter_output><empty_output><jupyter_text>Figuring out the issueLooks like this isn't going to be as simple as converting the datatype. According to the error above, the value at row 582 is "6/4" - let's check it out.<jupyter_code>df_08.query("air_pollution_score == '6/4'")<jupyter_output><empty_output><jupyter_text>It's not just the air pollution score!The mpg columns and greenhouse gas scores also seem to have the same problem - maybe that's why these were all saved as strings! According to [this link](http://www.fueleconomy.gov/feg/findacarhelp.shtmlairPollutionScore), which I found from the PDF documentation:    "If a vehicle can operate on more than one type of fuel, an estimate is provided for each fuel type."    Ohh.. so all vehicles with more than one fuel type, or hybrids, like the one above (it uses ethanol AND gas) will have a string that holds two values - one for each. This is a little tricky, so I'm going to show you how to do it with the 2008 dataset, and then you'll try it with the 2018 dataset.<jupyter_code>hb_08 = df_08[df_08.fuel.str.contains('/')]
hb_08<jupyter_output><empty_output><jupyter_text>Looks like this dataset only has one! The 2018 has MANY more - but don't worry - the steps I'm taking here will work for that as well!<jupyter_code>hb_18 = df_18[df_18.fuel.str.contains('/')]
hb_18<jupyter_output><empty_output><jupyter_text>We're going to take each hybrid row and split them into two new rows - one with values for the first fuel type (values before the "/"), and the other with values for the second fuel type (values after the "/"). Let's separate them with two dataframes!<jupyter_code># create two copies of the 2008 hybrids dataframe
df1 = hb_08.copy()  # data on first fuel type of each hybrid vehicle
df2 = hb_08.copy()  # data on second fuel type of each hybrid vehicle

# Each one should look like this
df1<jupyter_output><empty_output><jupyter_text>For this next part, we're going use Pandas' apply function. See the docs [here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html).<jupyter_code># columns to split by "/"
split_columns = ['fuel', 'air_pollution_score', 'city_mpg', 'hwy_mpg', 'cmb_mpg', 'greenhouse_gas_score']

# 对用函数对每一项进行处理
for c in split_columns:
    df1[c] = df1[c].apply(lambda x: x.split('/')[0])
    df2[c] = df2[c].apply(lambda x: x.split('/')[1])
df1
df2
newrow = df1.append(df2)
newrow
hb_08.index
df_08.drop(hb_08.index, inplace=True)
df_08.shape
df_08 = df_08.append(newrow)
df_08.shape
df_08[df_08.fuel.str.contains('/')]<jupyter_output><empty_output><jupyter_text>Repeat this process for the 2018 dataset<jupyter_code>df1 = hb_18.copy()
df2 = hb_18.copy()
split_columns = ['fuel', 'city_mpg', 'hwy_mpg', 'cmb_mpg']

for c in split_columns:
    df1[c] = df1[c].apply(lambda x: x.split('/')[0])
    df2[c] = df2[c].apply(lambda x: x.split('/')[1])
df1
df2.shape
newrow = df1.append(df2)
newrow.shape
print('shape before %d',df_18.shape[0])
df_18.drop(hb_18.index, inplace=True)
print('shape after %d',df_18.shape[0])
df_18 = df_18.append(newrow)
df_18.shape[0]<jupyter_output><empty_output><jupyter_text>Now we can comfortably continue the changes needed for `air_pollution_score`! Here they are again:- 2008: convert string to float- 2018: convert int to float<jupyter_code>df_08.air_pollution_score = df_08.air_pollution_score.astype(float)
df_18.air_pollution_score = df_18.air_pollution_score.astype(float)
df_08.shape
df_18.shape
df_08.drop('cert_region', axis=1 ,inplace=True)
df_08.shape
df_08.to_csv('data_08.csv', index=False)
df_18.to_csv('data_18.csv', index=False)<jupyter_output><empty_output><jupyter_text>Fix `city_mpg`, `hwy_mpg`, `cmb_mpg` datatypes    2008 and 2018: convert string to float<jupyter_code>mpg_columns = ['city_mpg', 'hwy_mpg', 'cmb_mpg']
for c in mpg_columns:
    df_18[c] = df_18[c].astype(float)
    df_08[c] = df_08[c].astype(float)<jupyter_output><empty_output><jupyter_text>Fix `greenhouse_gas_score` datatype    2008: convert from float to int<jupyter_code># convert from float to int
df_08['greenhouse_gas_score'] = df_08.greenhouse_gas_score.astype(int)
df_08.dtypes
df_18.dtypes
(df_08.dtypes == df_18.dtypes).any()
# Save your new CLEAN datasets as new files!
df_08.to_csv('clean_08.csv', index=False)
df_18.to_csv('clean_18.csv', index=False)<jupyter_output><empty_output><jupyter_text>Exploring with VisualsUse `clean_08.csv` and `clean_18.csv`<jupyter_code># load datasets

df_08 = pd.read_csv('clean_08.csv')
df_18 = pd.read_csv('clean_18.csv')<jupyter_output><empty_output><jupyter_text>Compare the distributions of greenhouse gas score in 2008 and 2018.<jupyter_code>df_08.greenhouse_gas_score.value_counts().hist();
df_18.greenhouse_gas_score.value_counts().hist();<jupyter_output><empty_output><jupyter_text>How has the distribution of combined mpg changed from 2008 to 2018?<jupyter_code>df_08.cmb_mpg.value_counts().hist()
df_18.cmb_mpg.value_counts().hist();<jupyter_output><empty_output><jupyter_text>Describe the correlation between displacement and combined mpg.<jupyter_code>df_08.plot.scatter('displ','cmb_mpg');
df_18.plot.scatter('displ','cmb_mpg');<jupyter_output><empty_output><jupyter_text>Describe the correlation between greenhouse gas score and combined mpg.<jupyter_code>df_08.plot.scatter('greenhouse_gas_score','cmb_mpg');
df_18.plot.scatter('greenhouse_gas_score','cmb_mpg');<jupyter_output><empty_output><jupyter_text>Drawing ConclusionsUse the space below to address questions on datasets `clean_08.csv` and `clean_18.csv`  Q1: Are more unique models using alternative sources of fuel? By how much?<jupyter_code>df_08.head()
df_08.fuel.unique()
df_18.fuel.unique()
# nunique numer of unique
alt_08 = df_08.query("fuel in ['CNG', 'ethanol']").model.nunique()
alt_08
alt_18 = df_18.query("fuel in ['Ethanol', 'Electricity']").model.nunique()
alt_18
plt.bar(['2008', '2018'],[alt_08, alt_18])
plt.title('Number of Unique models Using Alternative Fuels')
plt.xlabel('years')
plt.ylabel('Number of Unique Model')

plt.legend()
total_08 = df_08.model.nunique()
total_18 = df_18.model.nunique()
total_08, total_18
prop_08 = alt_08/total_08
prop_18 = alt_18/total_18
prop_08, prop_18
plt.bar(['2008', '2018'],[prop_08, prop_18])
plt.title('Number of Unique models Using Alternative Fuels')
plt.xlabel('years')
plt.ylabel('Number of Unique Model');<jupyter_output><empty_output><jupyter_text>Q2: How much have vehicle classes improved in fuel economy?<jupyter_code>veh_08 = df_08.groupby('veh_class').cmb_mpg.mean()
veh_08
veh_18 = df_18.groupby('veh_class').cmb_mpg.mean()
veh_18
inc = veh_18 - veh_08
inc
inc.dropna(inplace=True)
# 创建一个图标框
plt.subplots(figsize=(8,4))
plt.bar(inc.index, inc)
plt.title('Improvements in Fuel Economy from 2008 to 2018 by Vehicle Class')
plt.xlabel('Vehicle Class')
plt.ylabel('Increase in Average Combined MPG');<jupyter_output><empty_output><jupyter_text>Q3: What are the characteristics of SmartWay vehicles? Have they changed over time?<jupyter_code>df_08.smartway.unique()
df_18.smartway.unique()
smart_08 = df_08.query('smartway == "yes"')
smart_08.describe()
smart_18 = df_18.query("smartway in ['Yes', 'Elite']")
smart_18.describe()
width = 0.35
length = smart_08.mean().count()
location = np.arange(1,length+1)*2
plt.subplots(figsize=(11,5))
bar_08 = plt.bar(location-width/2, smart_08.mean(),width, color='r', alpha=.7);
bar_18 = plt.bar(location+width/2, smart_18.mean(),width, color='b', alpha=.7);
plt.xticks(location, smart_08.mean().index);
plt.xlabel('Characteristics')
plt.ylabel('Mean of Every Characteristics')
plt.title('The Characteristics of Smartway Vehicle')

plt.legend([bar_08, bar_18], ['2008', '2018']);<jupyter_output><empty_output><jupyter_text>Q4: What features are associated with better fuel economy?  Merging DatasetsUse Pandas Merges to create a combined dataset from `clean_08.csv` and `clean_18.csv`.<jupyter_code>df_08 = pd.read_csv('clean_08.csv')
df_18 = pd.read_csv('clean_18.csv')<jupyter_output><empty_output><jupyter_text>Create combined dataset<jupyter_code>df_08.rename(index=str, columns=lambda x:x[:10]+'_2008' ,inplace=True)
df_08.head()
df_combined = pd.merge(df_08,df_18, left_on='model_2008', right_on='model',how='inner')
df_combined.head()
df_combined.info()
(df_combined.model == df_combined.model_2008).any()
df_combined.to_csv('combined_dataset.csv', index=False)<jupyter_output><empty_output><jupyter_text>Results with Merged DatasetRemember to use your new dataset, `combined_dataset.csv`  Q5: For all of the models that were produced in 2008 that are still being produced now, how much has the mpg improved and which vehicle improved the most?<jupyter_code># load data 

df = pd.read_csv('combined_dataset.csv')
df.head()<jupyter_output><empty_output><jupyter_text>- Create a new dataframe, `model_mpg`, that contain the mean combined mpg values in 2008 and 2018 for each unique model<jupyter_code>model_mpg = pd.DataFrame({'cmb_mpg':df.groupby('model').mean()['cmb_mpg'],
                          'cmb_mpg_2008':df.groupby('model').mean()['cmb_mpg_20_2008']})

model_mpg.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 118 entries, ACURA MDX to VOLVO XC 90
Data columns (total 2 columns):
cmb_mpg         118 non-null float64
cmb_mpg_2008    118 non-null float64
dtypes: float64(2)
memory usage: 2.8+ KB<jupyter_text>- Create a new column, ` mpg_change `, with the change in mpg<jupyter_code>model_mpg['mpg_change'] = model_mpg.cmb_mpg - model_mpg.cmb_mpg_2008
model_mpg.info()
model_mpg.describe()<jupyter_output><empty_output><jupyter_text>- Find the vehicle that improved the most<jupyter_code>max_change = model_mpg.mpg_change.max()
model_mpg.query('mpg_change == {}'.format(max_change))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1281776108390905, 'fraction_numerical': 0.0533905694546918, 'mean_word_length': 6.244897959183674, 'pattern_counts': {'":': 0, '<': 153, '<?xml version=': 0, '>': 153, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '130324', 'license': 'Apache-2.0', 'n_tokens_mistral': 6116, 'n_tokens_neox': 5558, 'n_words': 1198, 'path': 'fuel econemy/.ipynb_checkpoints/fuel_econemy-checkpoint.ipynb', 'repo_name': 'AprilYoungs/analyze_basic', 'size': '14909'}