<jupyter_start><jupyter_text>Labelling Tutorial This Jupyter notebook provides a simple, semi-automated, method to produce a ground truth data set that can be used to train a neural network for use as a spectral shape classifier in the MCALF package. The following code can be adapted depending on the number of classifications that you want. [Download LabellingTutorial.ipynb](https://mcalf.macbride.me/en/stable/guide/examples/neural_network/LabellingTutorial.ipynb) Load the required packages<jupyter_code>import mcalf.models
from mcalf.utils.spec import normalise_spectrum
import requests
import numpy as np
from astropy.io import fits
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Download sample data<jupyter_code>path = 'https://raw.githubusercontent.com/ConorMacBride/mcalf/main/examples/data/ibis8542data/'

for file in ('wavelengths.txt', 'spectra.fits'):
    r = requests.get(path + file, allow_redirects=True)
    with open(file, 'wb') as f:
        f.write(r.content)<jupyter_output><empty_output><jupyter_text>Load data files<jupyter_code>wavelengths = np.loadtxt('wavelengths.txt')  # Original wavelengths

with fits.open('spectra.fits') as hdul:  # Raw spectral data
    datacube = np.asarray(hdul[0].data, dtype=np.float64)<jupyter_output><empty_output><jupyter_text>Initialise the model that will use the labelled data<jupyter_code>model = mcalf.models.IBIS8542Model(original_wavelengths=wavelengths)<jupyter_output><empty_output><jupyter_text>Select the spectra to label<jupyter_code>n_points = 50

flat_choice = np.random.choice(np.arange(datacube[0].size), n_points, replace=False)
i_points, j_points = np.unravel_index(flat_choice, datacube[0].shape)
np.save('labelled_points.npy', np.array([i_points, j_points]))
i_points, j_points = np.load('labelled_points.npy')<jupyter_output><empty_output><jupyter_text>Select the spectra to label from the data file<jupyter_code>raw_spectra = datacube[:, i_points, j_points].T<jupyter_output><empty_output><jupyter_text>Normalise each spectrum to be in range [0, 1]<jupyter_code>labelled_spectra = np.empty((len(raw_spectra), len(model.constant_wavelengths)))
for i in range(len(labelled_spectra)):
    labelled_spectra[i] = normalise_spectrum(raw_spectra[i], model=model)<jupyter_output><empty_output><jupyter_text>Script to semi-automate the classification process- Type a number 0 - 4 for assign a classification to the plotted spectrum- Type 5 to skip and move on to the next spectrum- Type `back` to move to the previous spectrum- Type `exit` to give up (keeping ones already done)The labels are present in the `labels` variable (-1 represents an unclassified spectrum)<jupyter_code>labels = np.full(len(labelled_spectra), -1, dtype=int)
i = 0
while i < len(labelled_spectra):
    
    # Show the spectrum to be classified along with description
    plt.figure(figsize=(15, 10))
    plt.plot(labelled_spectra[i])
    plt.show()
    print("i = {}".format(i))
    print("absorption --- both --- emission / skip")
    print("       0    1    2    3    4         5 ")
    
    # Ask for user's classification
    classification = input('Type [0-4]:')
    
    try:  # Must be an integer
        classification_int = int(classification)
    except ValueError:
        classification_int = -1  # Try current spectrum again
    
    if classification == 'back':
        i -= 1  # Go back to the previous spectrum
    elif classification == 'exit':
        break  # Exit the loop, saving labels that were given
    elif 0 <= classification_int <= 4:  # Valid classification
        labels[i] = int(classification)  # Assign the classification to the spectrum
        i += 1  # Move on to the next spectrum
    elif classification_int == 5:
        i += 1  # Skip and move on to the next spectrum
    else:  # Invalid integer classification
        i += 0  # Try current spectrum again<jupyter_output><empty_output><jupyter_text>Plot bar chart of classification populations<jupyter_code>unique, counts = np.unique(labels, return_counts=True)
plt.figure()
plt.bar(unique, counts)
plt.title('Number of spectra in each classification')
plt.xlabel('Classification')
plt.ylabel('N_spectra')
plt.show()<jupyter_output><empty_output><jupyter_text>Overplot the spectra of each classification<jupyter_code>for classification in unique:
    plt.figure()
    for spectrum in labelled_spectra[labels == classification]:
        plt.plot(model.constant_wavelengths, spectrum)
    plt.title('Classification {}'.format(classification))
    plt.yticks([0, 1])
    plt.show()<jupyter_output><empty_output><jupyter_text>Save the labelled spectra for use later<jupyter_code>np.save('labelled_data.npy', labelled_spectra)
np.save('labels.npy', labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10214827295703455, 'fraction_numerical': 0.009477674810446503, 'mean_word_length': 5.011392405063291, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '69340', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1547, 'n_tokens_neox': 1475, 'n_words': 398, 'path': 'docs/guide/examples/neural_network/LabellingTutorial.ipynb', 'repo_name': 'ConorMacBride/mcalf', 'size': '4748'}