<jupyter_start><jupyter_text>Introduction In the notebook 'determining_redshift', the sample was small (1500 sources) and it contained only very bright (mean 17 in magnitude in the dered_r band) sources. The new dataset contains 135000 sources with a mean of 19 in magnitudes.To note that SDSS removed their photozRF table because it does poorly with the faint red galaxies targeted in  Baryon Oscillation Spectroscopic Survey (BOSS).<jupyter_code>import os
import numpy as np
import pandas as pd
from pandas.plotting import table
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
sns.set()
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading data and some EDA<jupyter_code>filepath= "/home/veroastro/Documents/spectraTypes/data/newData/"
data = pd.read_csv(filepath+"full_sources.csv")


#Scramble the data set so that it is not ordered by spectral class (QSO, galaxy or star) 
data = data.sample(frac=1)

data.head()
data.describe()<jupyter_output><empty_output><jupyter_text>photoZ and photoErr have values set as -9999. This is the same as a Null value. Since it is not part of our determination of the redshift, the sources with null values of SDSS photoZ and photoErr wont be deleted from the dataset. Since photoZ and photoErr are needed for comparison, the dataframes photoz and photozErr are created whithout the null values. This leaves us with 129,195 sources of the 135,000<jupyter_code>photoz= data.loc[(data['photoZ'] >= 0.0)]['photoZ']
photozErr=data.loc[(data['photoZ'] >= 0.0)]['photoErr']

print('Mean photoError:', round(np.mean(photozErr), 3))
photozErr.describe()
print('Mean specError:', round(np.mean(data['zErr']), 3))<jupyter_output>Mean specError: 0.07<jupyter_text>The mean value of the spectroscopic errors is 0.07 while the mean value of the photometric errors is 0.056.<jupyter_code>data_clean = data.drop(columns=['specObjID','objID','err_u', 'err_g', 'err_r', 'err_i', 'err_z', 'zErr', 'class', 'subClass', 'photoZ','photoErr','type'])
data_clean.head()
data_clean.describe()
def hist_plots( data: pd.DataFrame,
               rows: int,
              cols: int,
              figsize: tuple):
    fig, axes = plt.subplots(rows,cols, figsize=figsize)
    for i, ax in enumerate(axes.flatten()):
        if i < len(data.columns):
            data[sorted(data.columns)[i]].plot.hist(bins=30, ax=ax)
            ax.set_title(f'{sorted(data.columns)[i]} distribution', fontsize=10)
            ax.tick_params(axis='x', labelsize=10)
            ax.tick_params(axis='y', labelsize=10)
            ax.get_yaxis().get_label().set_visible(False)
        else:
            fig.delaxes(ax=ax)
    fig.tight_layout()
#Histograms of the columns in the dataset
hist_plots(data=data_clean,
          rows=2,
          cols=4,
          figsize=(20,10))<jupyter_output><empty_output><jupyter_text>Mostly normal distributions of the bands around 18-20 mag. Most redshifts are less than 0.5.<jupyter_code>data['ur'] =  data['dered_u'] - data['dered_r']
data['gr'] =  data['dered_g'] - data['dered_r']

sns.lmplot( x="ur", y="gr", data=data, fit_reg=False, hue='class', legend=False)
 
# Move the legend to an empty part of the plot
plt.legend(loc='lower right')

plt.show()<jupyter_output><empty_output><jupyter_text>As seen by the graph above, colours (the difference in magnitude between 2 bands) gives no extra information.<jupyter_code># Creating the atributes set as X and the target redshift as y
from sklearn.preprocessing import MinMaxScaler
cols= data_clean.columns[:-1]
X = data_clean[cols]
y = data_clean['z']
scaler = MinMaxScaler()
X_t = scaler.fit_transform(X)<jupyter_output><empty_output><jupyter_text>Training models<jupyter_code># Splitting the Xand y datasets into training and test set and normalising the data
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X_t, y, 
                                                    test_size=0.3, random_state=42)
# Linear regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

LR = LinearRegression()

# Storage for error values
error_df = list()


LR = LR.fit(X_train, y_train)
y_train_pred = LR.predict(X_train)
y_test_pred = LR.predict(X_test)

error_df.append(pd.Series({'train': mean_squared_error(y_train, y_train_pred),
                           'test' : mean_squared_error(y_test,  y_test_pred)}))

error_df
# Random Forest with GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV
RF = RandomForestRegressor(random_state = 42)
param_grid = {  'bootstrap': [True],
                'max_features': ['log2'], 
                'n_estimators': range(300,400, 20)}

rf= GridSearchCV(RF,
                  param_grid=param_grid)

# Train the model on training data
rf.fit(X_train, y_train);
predictions = rf.predict(X_test)

rf.best_params_ ,rf.best_score_<jupyter_output><empty_output><jupyter_text>'bootstrap': True, 'max_features': 'log2', 'n_estimators': 121 best parameters of the random forest regressor.<jupyter_code># Print out the mean square error between the predicted redshift and the spectroscopic redshift
print('Mean Square Error RF:', round(mean_squared_error(y_test, predictions), 3))
# MSE between the SDSS photo redshift and the spectroscopic redshift
sZ= data.loc[(data['photoZ'] >= 0.0)]['z']
print('Mean Square Error SDSS:', round(mean_squared_error(sZ,photoz), 3))<jupyter_output>Mean Square Error SDSS: 0.259<jupyter_text>The test error of the linear regression model is (mse) 0.23 while we get an mse of 0.075 with randome forest classifier. To compare errors the mse from the difference between photometric redshift and spectroscopic redshift is 0.259.<jupyter_code>X_test = pd.DataFrame(X_test)
index = pd.DataFrame(index=X_test.index)
errors = index.join(data, how= 'inner')
prediction = pd.DataFrame(predictions, columns = ['pred'])
errors['pred'] = predictions
errors['pred_err'] = abs(errors['pred'] - errors['z'])

errors.head()

# plot of dered_r versus the absolute error of predicted redshift and spectroscopic redshif (pred_err)
sns.lmplot( x="dered_r", y="pred_err", data=errors, fit_reg=False, hue='class',scatter_kws={'alpha':0.25})
plt.show()<jupyter_output><empty_output><jupyter_text>There doesn't seem to be a correlation between the absolute error of predicted redshift and spectroscopic redshif (pred_err) andthe r band. There does seem to be a fraction of galaxies with high pred_err, all around magnitude 20<jupyter_code># Scatter plot of the redshift with the pred_err for the 3 classes (star, galaxies and QSO)
sns.lmplot( x="z", y="pred_err", data=errors, fit_reg=False, hue = 'class', scatter_kws={'alpha':0.25})<jupyter_output><empty_output><jupyter_text>There is a correlation of some sources with higher redshift, the higher the pred_err.<jupyter_code># histograms of the absolute errors for the predicted redshift and the photometric redshift.
plt.hist(abs(photoz-sZ), bins=50,density =True,histtype= 'step')
plt.hist(errors['pred_err'], bins=50, density = True,histtype= 'step')
labels= ["SDSS_err", "Pred_err"]
plt.legend(labels)
plt.show()
# getting data of the histogram
count, bins_count = np.histogram(errors['pred_err'], bins=100)
count2, bins_count2 = np.histogram(abs(photoz-sZ), bins=100) 
# finding the PDF of the histogram using count values
pdf = count / sum(count)
pdf_photo = count2 / sum(count2)
# using numpy np.cumsum to calculate the CDF
cdf = np.cumsum(pdf)
cdf_photo =  np.cumsum(pdf_photo)
# plotting PDF and CDF

plt.plot(bins_count[1:], cdf, label="pred_err")
plt.plot(bins_count2[1:], cdf_photo, color="red", label="photo_err")
plt.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11245965138799226, 'fraction_numerical': 0.017172369270497093, 'mean_word_length': 5.191846522781774, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '328333', 'license': 'CC0-1.0', 'n_tokens_mistral': 2705, 'n_tokens_neox': 2477, 'n_words': 739, 'path': 'script_notebooks/determining_redshift_less_bias.ipynb', 'repo_name': 'lifemannequin/MLphotometry', 'size': '7745'}