<jupyter_start><jupyter_text>We now continue with the Data Visualizatinon part<jupyter_code>#we make a copy for the data for the visualizations of the data and encode the categorical data
# for our visualization purpose.
train_copy = train.copy()

colums_enc = ['column1', 'column2', 'column3']

for col in colums_enc:
    for i, item in enumerate(train[col].unique()):
        train_copy[col] = train_copy[col].replace(item, i)

## The above command will give a unique number to the categorical data<jupyter_output><empty_output><jupyter_text>Box Plot<jupyter_code>#Box Plot
Col1 = <Column name>.tolist()
Col2 = <Column bname>.tolist()

labels = ["<Column 1 Name", "Column 2 Name"]
plt.boxplot([Col1, Col2])
plt.xticks(list(range(1,len(labels)+1)), labels)
plt.show()<jupyter_output><empty_output><jupyter_text>Bar Chart<jupyter_code>import matplotlib.pyplot as plt

# creating the dataset

x_axis_value = list()
y_axis_value = list()
  
fig = plt.figure(figsize = (10, 5))
 
# creating the bar plot
plt.bar(x_axis_value, y_axis_value, color ='maroon', width = 0.4)
 
plt.xlabel("X - Axis Name")
plt.ylabel("Y - Axis Name")
plt.title("Title of the chart")
plt.show()<jupyter_output><empty_output><jupyter_text>Stacked Bar Chart<jupyter_code>labels = ['Label 1', 'Label 2', 'Label 3']
#The length of below values and upper values should be equal to the length of Labels
below_values = <Column Name>.to_list()
upper_values = <Column Name>.to_list()
width = 0.35       # the width of the bars: can also be len(x) sequence

fig, ax = plt.subplots()

ax.bar(labels, below_values, width, label='Below Values')
ax.bar(labels, upper_values, width, label='Upper Values')

ax.set_ylabel('Y - Axis Values')
ax.set_title('Title of the chart')

ax.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>SNS Plot<jupyter_code>sns.lineplot(x="column name", y="column name",
             hue="region", style="event",
             data=<Your Dataframe Name>)<jupyter_output><empty_output><jupyter_text>Outlier<jupyter_code>import numpy as np
 
def detect_outlier(data):
    # find q1 and q3 values
    q1, q3 = np.percentile(sorted(data), [25, 75])
 
    # compute IRQ
    iqr = q3 - q1
 
    # find lower and upper bounds
    lower_bound = q1 - (1.5 * iqr)
    upper_bound = q3 + (1.5 * iqr)
 
    outliers = [x for x in data if x <= lower_bound or x >= upper_bound]
 
    return outliers
detect_outlier((data["Column Name"]))<jupyter_output><empty_output><jupyter_text>Data Encoding and Normaliztion 1) Encoding for Categorical Features<jupyter_code>#we make copy to avoid data conflict when running the jupytertabs in non sequential order
train_data = train.copy()

## We select the colums with categorical data and we encode them
colums_enc = ['column1', 'column2', 'column3']
train_data = pd.get_dummies(train_data, columns=colums_enc)

train_data = train_data.drop(columns=['column1', 'column2', 'column3'], axis=1)

train_data.head()<jupyter_output><empty_output><jupyter_text>2. Normalization for Numeric Features<jupyter_code>#Using the Scaler function  library
num_col = ['column1', 'column2', 'column3']
scaler = StandardScaler()
scaler.fit(train_data[num_col])
train_data[num_col] = scaler.transform(train_data[num_col])<jupyter_output><empty_output><jupyter_text>Here we begin with the Model Training part<jupyter_code># Before we do the traiin  test split we remove the target column from the data
Y = train_data['target_col']
X = train_data.drop(columns=['target_col'], axis=1)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

print(X_train.shape)
print(X_test.shape)<jupyter_output><empty_output><jupyter_text>hERE WE SELECT MODEL<jupyter_code>##If the problem is regression we use the linear regression model
model =  RandomForestRegressor()
model.fit(X_train, Y_train)
#if the problem is classification we use the logistic regression model
model = LogisticRegression()
model.fit(X_train, Y_train)

## OR in classification itself we use the XGBClassifier model
model = XGBClassifier()
model.fit(X_train, Y_train)
accuracy_score(Y_test, model.predict(X_test))<jupyter_output><empty_output><jupyter_text>Feature Importance from the Dataset<jupyter_code>## <NAME>ya data nakh
#Feature importances FROM THE RANDOM FOREST 
# compute feature importance from random forest regression model
feature_imp=pd.DataFrame()
for feature,imp in zip(X_train.columns,<model fitted>.feature_importances_):
    temp=pd.DataFrame([feature,imp]).T
    feature_imp=feature_imp.append(temp)
feature_imp.columns=['feature','relative_importance']
feature_imp.sort_values(by='relative_importance',inplace=True)
feature_imp.set_index('feature',inplace=True)
feature_imp.iloc[-20:,:].plot(kind='barh',figsize=(10,8))
plt.show()<jupyter_output><empty_output><jupyter_text>MODEL Testing<jupyter_code>test = pd.read_csv("test_file_name.csv")
test_data = test.copy()
test_data_withid = test.copy()

test_data.drop[columns=['id'], axis=1, inplace=True]
test_data.isnull().sum()
#if the above shows null for numeric we if first replace it with mean or median values
test_data.loc[train['column_name'].isna(), 'column_name'] = test_data['column_name'].median()

#If if null exist  for categorical values we should just drop it as it cannot be imputed
tetest_datast = test_data.dropna()
# Check if all the null values are handled in the data
test_data.isnull.sum()
# We drop the columns that are not needed
test_data.drop[columns=['column_name'], axis=1, inplace=True]

# We replace the binary categorical colums with0 and 1
tetest_data['column_name'] = test_data['column_name'].map({'yes':1, 'no':0})<jupyter_output><empty_output><jupyter_text>Again Encode the and Normalize the Data<jupyter_code>## We select the colums with categorical data and we encode them
colums_enc = ['column1', 'column2', 'column3']
test_data = pd.get_dummies(test_data, columns=colums_enc)

test_data = test_data.drop(columns=['column1', 'column2', 'column3'], axis=1)
test_data.head()
#Using the Scaler function  library
num_col = ['column1', 'column2', 'column3']
scaler = StandardScaler()
scaler.fit(test_data[num_col])
test_data[num_col] = scaler.transform(test_data[num_col])
submission = model.predict(test_data)

submission_out = pd.DataFrame('id':test_data_withid['id'], 'target_col':submission)

submission_out.to_csv('submission_file_name.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12284820031298904, 'fraction_numerical': 0.01189358372456964, 'mean_word_length': 6.536556603773585, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '392603', 'license': 'MIT', 'n_tokens_mistral': 2285, 'n_tokens_neox': 2204, 'n_words': 556, 'path': 'ML_process.ipynb', 'repo_name': 'vraj130/ML_templates', 'size': '6391'}