<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 3, Module 1*--- Define ML problemsYou will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your decisions.- [x] Choose your target. Which column in your tabular dataset will you predict?      There are 5 potential target columns: 'T_HDI_Mean_plus_5', 'T_HDI_Trend_plus_5', 'T_State_Fail_yr_1', 'T_State_Fail_plus_5', and 'T_State_Fail_yr_5'. T_HDI_Mean_plus_5 is the average human development index of a nation state in the next 5 years. T_HDI_Trend_plus_5 is the growth rate of the human development index of a nation state in the next 5 years. T_State_Fail_yr_1 is a state failure event next year. T_State_Fail_plus_5 indicates whether there is a state failure event in any of the next 5 years. T_State_Fail_yr_5 is a state failure event occuring in the fifth year after the present year.- [x] Is your problem regression or classification?      I intend to do both. Any predictive models involving the two Human Development Index targets will be regressions. Predictive models involving any of the 3 state failure targets will be classification.- [x] How is your target distributed?    - Classification: How many classes? Are the classes imbalanced?      There are 2 classes, 1 or 0, meaning "yes this nation experienced a state failure event in year 0" or "no this nation did not experience a state failure event in year 0". In the State_failure columns, 1 only occurs between 11% and 22% of the time. The classes are imbalanced.    - Regression: Is the target right-skewed? If so, you may want to log transform the target.      If anything, they are ever so slightly left skewed. But only slightly. One is actually bimodal.- [x] Choose your evaluation metric(s).    - Classification: Is your majority class frequency >= 50% and < 70% ? If so, you can just use accuracy if you want. Outside that range, accuracy could be misleading. What evaluation metric will you choose, in addition to or instead of accuracy?    - Regression: Will you use mean absolute error, root mean squared error, R^2, or other regression metrics?      For the classification models, I will use F1 Score. For regression, I will use both RMSE and R squared.- [x] Choose which observations you will use to train, validate, and test your model.    - Are some observations outliers? Will you exclude them?    - Will you do a random split or a time-based split?      The true test set is time-based, excluding datapoints that will be forecasts of future events, and will not be scorable until early 2025. The rest will be randomized splits, stratified by year0.- [x] Begin to clean and explore your data.- [x] Begin to choose which features, if any, to exclude. Would some features "leak" future information?      For HDI, any columns that indicate past or future HDI will be excluded. The same applies for the state failure columns.If you haven't found a dataset yet, do that today. [Review requirements for your portfolio project](https://lambdaschool.github.io/ds/unit2) and choose your dataset.Some students worry, ***what if my model isn't “good”?*** Then, [produce a detailed tribute to your wrongness. That is science!](https://twitter.com/nathanwpyle/status/1176860147223867393)<jupyter_code>import pandas as pd

df = pd.read_excel('Master_MELT.xlsx')
df
df['T_State_Fail_yr_1'].value_counts(normalize=True)
df['T_State_Fail_plus_5'].value_counts(normalize=True)
df['T_State_Fail_yr_5'].value_counts(normalize=True)
import seaborn as sns
sns.distplot(df['T_HDI_Mean_plus_5'], hist=True, kde=True, 
             bins=int(180/5), color = 'darkblue', 
             hist_kws={'edgecolor':'black'},
             kde_kws={'linewidth': 4})
sns.distplot(df['T_HDI_Trend_plus_5'], hist=True, kde=True, 
             bins=int(180/5), color = 'darkblue', 
             hist_kws={'edgecolor':'black'},
             kde_kws={'linewidth': 4})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07291405662741168, 'fraction_numerical': 0.018541718867451765, 'mean_word_length': 4.358389261744967, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '194225', 'license': 'MIT', 'n_tokens_mistral': 1165, 'n_tokens_neox': 1082, 'n_words': 523, 'path': 'module1-define-ml-problems/DAVID_DODDS_LS_DS_231_assignment.ipynb', 'repo_name': 'ddodds42/DS-Unit-2-Applied-Modeling', 'size': '3991'}