<jupyter_start><jupyter_text>Data Project: Analysis of Danish import from 1997-2019 by category  Import relevant packages<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import ipywidgets as widgets<jupyter_output><empty_output><jupyter_text>Reading the data of danish import  The dataset of interest contains nominal seasonally and non seasonally adjusted values of danish import in the period from 1997 to 2019 by import category. The dataset is downloaded from Statistics Denmark (Danmarks Statistik: https://www.statistikbanken.dk/UHV1) and originally named "UHV1". The values are reported in millions DKK.<jupyter_code>filepath = './data/Denmark_import.xlsx' 
pd.read_excel(filepath).head(25) #Printing the first 25 lines of the dataset<jupyter_output><empty_output><jupyter_text>Data cleaning  The two first rows of the dataset are empty why they are removed during the importing process. Moreover, the two left columns are removed as well. Lastly, all rows from 10 to infinity are removed as we only want to consider the development of non seasonally adjusted values. The last line of code print out the first five rows after our adjustments.<jupyter_code>imp = pd.read_excel(filepath, skiprows = 2) 
Dropped = ['Unnamed: 0', 'Unnamed: 1']
imp.drop(Dropped, axis = 1, inplace = True) 
imp.drop(imp.index[10:], inplace=True)
imp.head(5)<jupyter_output><empty_output><jupyter_text>Renaming the first column from "Unnamed: 2" to "year"<jupyter_code>imp.rename(columns ={'Unnamed: 2':'Year'}, inplace=True)
imp.head(2)<jupyter_output><empty_output><jupyter_text>Transposing the data We transpose the dataset as we want to compare categories within years. Hence we obtain a dataset with categories on the horizontal axis and years on the vertical axis.<jupyter_code>imp_t=imp.T
imp=imp_t
imp.head(5)<jupyter_output><empty_output><jupyter_text>Removing the top horizontal index  We remove the top index as transposing a pandas dataframe also transposes the index numbers. Thus the danish import categories define the variable names. Note, running this code more than once will replace the category names with the observations of 1997.<jupyter_code>imp.columns = imp.iloc[0]
imp.drop(imp.index[0], inplace=True)
imp.head(5)<jupyter_output><empty_output><jupyter_text>Reading the dataset of the danish consumer price index (CPI) The dataset of danish consumer price index (CPI) covers the period of 1980 to 2019. The dataset is downloaded from Statistics Denmark (Danmarks Statistik: https://www.statistikbanken.dk/PRIS112) and originally named "PRIS112". The CPI is indexed with year 2015 = 100<jupyter_code>filepath1 = './data/Denmark_CPI.xlsx' 
pd.read_excel(filepath1).head(25) #Printing the first 25 lines of the dataset<jupyter_output><empty_output><jupyter_text>Like we did with the danish import dataset we drop unnecessary rows and columns. Thus we drop the first two rows and the first column.<jupyter_code>cpi = pd.read_excel(filepath1, skiprows = 2) 
Dropped = ['Unnamed: 0']
cpi.drop(Dropped, axis = 1, inplace = True) 
cpi.head()<jupyter_output><empty_output><jupyter_text>Moreover, as we only have data on danish import from 1997 to 2019 we drop the years from 1980 to 1996.<jupyter_code>cpi.drop(cpi.loc[:, '1980':'1996'], inplace = True, axis = 1) 
cpi.head()<jupyter_output><empty_output><jupyter_text>In order to merge the danish import data with the CPI data we have to transpose the CPI dataset, as we need to have an uniqe key in both datasets. In our case, "year" is the identifier.<jupyter_code>cpi_t=cpi.T
cpi = cpi_t
cpi.head()<jupyter_output><empty_output><jupyter_text>We rename the "0"-column such it is called "CPI" instead as this column indicates the consumper price index data.<jupyter_code>cpi.rename(columns={0:'CPI'}, inplace=True)
cpi.index.names = ['Year']
cpi.head()<jupyter_output><empty_output><jupyter_text>Re-indexing CPI such that 1997 = 100 The CPI data is indexed with 2015 as base value. We want to change this such that 1997 becomes the new index base (1997 = 100).<jupyter_code>cpi['CPI'] = cpi['CPI'] /cpi['CPI'][0]<jupyter_output><empty_output><jupyter_text>We are now ready to merge the danish import data and the CPI data into one dataset.<jupyter_code>merged = pd.concat([imp, cpi], axis=1, sort=False)
merged<jupyter_output><empty_output><jupyter_text>The reason why we want to include the consumer price into our project is to transform the nominal prices in our import data, such it become real prices. Hence, we divide the CPI column with all the columns of the merged dataset.<jupyter_code>imp_real = merged.copy()
imp_real.drop('CPI', axis=1)
imp_real=imp_real.div(merged['CPI'], axis=0)
imp_real<jupyter_output><empty_output><jupyter_text>As we have no further use of the "CPI" column, we remove it from the dataset.<jupyter_code>imp_real.drop(['CPI'], axis = 1, inplace = True) 
imp_real<jupyter_output><empty_output><jupyter_text>Calculating yearly growth rate in percentage To inspect the evolution of the import categories the following code calculates the percentages change over the years by categories. This produce an empty row for the year 1997, which is removed from the dataset.<jupyter_code>imp_perc = imp_real.pct_change()*100
imp_perc.drop(imp_perc.index[0], inplace=True)
imp_perc.head()<jupyter_output><empty_output><jupyter_text>Descriptive statistics of yearly growth rate data In order to examine the data we print the statistics of the dataset<jupyter_code>imp_perc.describe()<jupyter_output><empty_output><jupyter_text>We note that there are 22 observations of each variable, which corresponds to the number of years, hence we have no empty cells. Moreover each category has on average positive growth rates over the years. The total growth rate of danish import from 1998 to 2019 has on average a growth rate of 2,2% and the categories which experience the highest and lowest average growth rate are "Vessels, aircraft etc." and "Manufactured Goods Classified Chiefly By Materia" with average growth rates of 16,9% and 1,2%, respectively.   Checking the types of variables<jupyter_code>imp_perc.dtypes<jupyter_output><empty_output><jupyter_text>Throughout the data analysis, pandas has not manipulated the "year" category, why a new variable called "time" is added to the dataset. We use this variable to plot the dataset.<jupyter_code>imp_perc['time'] = np.arange(1998,2020)<jupyter_output><empty_output><jupyter_text>Visualisation of the data In order to illustrate the evolution of the danish import data, the following blok of code visualizes the percentage growth of the years by category in an interactive line chart. The x-axis represents years and the y-axis represents the percentage change in growth rate. Moreover, the plot enables the user to show growth rates for each category as well as adjusting the time period for which the data is visualised.<jupyter_code>from matplotlib import cm

def _plot_timeseries(df, x, y):
    fig = plt.figure(dpi=100)
    ax = fig.add_subplot(1,1,1)
    I = (df.time >= y[0]) & (df.time <= y[1])
    series = df.loc[I,[x]]
    years = np.arange(y[0],y[1]+1)
    ax.plot(years, series, )
    ax.set_ylabel('Percentage change (%)')
    ax.set_xlabel('Years')
    plt.grid()
    plt.xticks(np.arange(y[0],y[1]+1, 2.0))
    
    return fig, ax
    
def plot_timeseries(df):
    varlist = [col for col in imp_perc.columns[:-1]]

    widgets.interact(_plot_timeseries,
    df = widgets.fixed(df),
    x = widgets.Dropdown(
        description='Variable', 
        options=varlist, 
        value='TOTAL'),
    y=widgets.IntRangeSlider(
        description="Years",
        min=1998,
        max=2019,
        value=[1998, 2019],
        continuous_update=False,)

    ); 

    
plot_timeseries(imp_perc)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08745198463508323, 'fraction_numerical': 0.02471190781049936, 'mean_word_length': 5.476782752902156, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '223653', 'license': 'MIT', 'n_tokens_mistral': 2600, 'n_tokens_neox': 2405, 'n_words': 874, 'path': 'dataproject/dataproject/Dataproejct_2020.ipynb', 'repo_name': 'lcn363/projects-2020-emil-kirstine-thomas', 'size': '7810'}