<jupyter_start><jupyter_text>The Prohack Competition.In this task we are going to predict the well being of the species.  Problem Statement.optimize the well-being across all the galaxies. Obectives- To understand what index makes the intergalactic species better off:    - Use the data and train a model to predict this index with the highest possible level of certainty.- To achieve the highest possible level of well-being through optimized allocation of additional energy:    - A  star of an unusually high energy of 50000 zillion DSML has been discovered.    - No one galaxy will consume more than 100 zillion DSML and,    - at least 10% of the total energy will be consumed by galaxies in need with existence expectancy index below 0,7. Target variable y - an index of measure of well being. Problem Type- Regression problem with a continuous target variable.  MetricRoot Mean Square Error.  The data set is contained in the following [link](https://prohack.org/media/prohack_dataset_avOqBYc.zip). Lets begin the analysis by extracting the data into our working directory.<jupyter_code>#Data Extraction
!unzip -q prohack_dataset.zip<jupyter_output><empty_output><jupyter_text>Required libraries.<jupyter_code>import pandas as pd
import numpy as np 
# Dataset.
pd.set_option('display.max_columns',None)

train = pd.read_csv('/content/train.csv')
test = pd.read_csv('/content/test.csv')<jupyter_output><empty_output><jupyter_text>Basic information about the dataset ie the shape and info about the variables.<jupyter_code>def explore_data(df,start=10,stop=20):
    data = df.iloc[start:stop]#the slice you want

    print(f'The dimensions of your data is {df.shape}')
    print(f'Number of rows:{df.shape[0]}')
    print(f'Number of columns: {df.shape[1]}')

    print('A glimpse of your Data is:-')
    return data


explore_data(train)
explore_data(test)
#checking for any duplicates in the data.
def duplicates(df):
    mask = df.duplicated()
    return df[mask].shape

duplicates(train)<jupyter_output><empty_output><jupyter_text>Seems that there are  zero duplicates in the train dataset. Lets also check the for duplicates in the test dataset.<jupyter_code>duplicates(test)<jupyter_output><empty_output><jupyter_text>Similarly, test has no duplicated rows.   Exploratory Data Analysis.  Viewing the information about the two datasets.<jupyter_code>def information(data1,data2):
    data1.info()
    data2.info()


information(train,test)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3865 entries, 0 to 3864
Data columns (total 80 columns):
 #   Column                                                                                   Non-Null Count  Dtype  
---  ------                                                                                   --------------  -----  
 0   galactic year                                                                            3865 non-null   int64  
 1   galaxy                                                                                   3865 non-null   object 
 2   existence expectancy index                                                               3864 non-null   float64
 3   existence expectancy at birth                                                            3864 non-null   float64
 4   Gross income per capita                                                                  3837 non-null   float64
 5   Income Index                                                  [...]<jupyter_text>Distinct galaxies.<jupyter_code>def galaxies(df):
    return df.galaxy.nunique()

print(f'Distinct galaxies in the train dataset: {galaxies(train)}')
print(f'Distinct galaxies in the test dataset: {galaxies(test)}')<jupyter_output>Distinct galaxies in the train dataset: 181
Distinct galaxies in the test dataset: 172<jupyter_text>Some prior statistics, also a comparison of the number of missing values.<jupyter_code>def statistics(df):
    return df.describe()

#for train set
statistics(train)
#for test set.
statistics(test)<jupyter_output><empty_output><jupyter_text>Correlation Existing within the data.<jupyter_code>import matplotlib.pyplot as plt
plt.style.use('ggplot')
import seaborn as sns<jupyter_output><empty_output><jupyter_text>AnalysisThis is a regression problem and the only two algorithms we can employ are either Decision trees or Linear Regression.In this particular task, we decide on XGBoost which involves creation of meta-models ie a composition of many other models which do a combination for a final prediction. The objective is to find some base learners that when combined will be possible to create a final prediction that is non-linear.Let's begin by trying some trees as the base learners.<jupyter_code>#required libraries.
import xgboost as xgb

#split the data to X_train,y_train,X_test
y_train = train['y']
X_train = train.drop(columns='y').set_index(['galactic year','galaxy'])
X_test = test.set_index(['galactic year','galaxy'])      #since there is no y column here.

y_train.head()
X_test.head()
X = 
y = train['y'].fillna(0)
X.head()
#y.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0782967032967033, 'fraction_numerical': 0.014521193092621664, 'mean_word_length': 2.849697885196375, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '564381', 'license': 'MIT', 'n_tokens_mistral': 1428, 'n_tokens_neox': 1337, 'n_words': 496, 'path': 'Prohack.ipynb', 'repo_name': 'Rohianon/Prohack-Challenge', 'size': '5096'}