<jupyter_start><jupyter_text>Analyzing videos of a particleIn this tutorial, we will train a network to extract the intensity of a particle regardless of focus. The network will be trained to predict on videos of centered particles moving in and out of focus randomly.  1 SetupImports needed for this example.<jupyter_code>import deeptrack as dt

import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>2.1 Define the particleWe want to generate a video where the position of the particle changes over time. The x-y position of the particle will be roughly centered in each frame. This is to simulate a tracked particle centered by cropping a region of interest around it for each frame. The z position, however, will follow a Brownian motion with a random diffusion constant. To allow these properties to change in the video, we define two functions that return the value of these properties for each frame.<jupyter_code>def get_position():
    
    v = 32 + np.random.randn(2) * 0.5

    return v

def get_z(previous_value, diffusion_constant, dt=1/30):
    return np.clip(previous_value + np.random.randn() * np.sqrt(diffusion_constant * dt) * 1e7, -60, 60)<jupyter_output><empty_output><jupyter_text>2.2 Set initial values of the propertiesThe properties passed to create the instance of `Sphere` are used for the first frame of the video.* `position` is initialized with the same function as that used for each subsequent frame.* `z` is initialized as a normally distributed number centered around zero.* `intensity` is constant 1. The intensity will be varied in a later step.* `radius` is a random number between 200nm and 1000nm* `position_unit` is set to pixels* `upsample` is 4, increasing the resolution of the sphere.<jupyter_code>particle_feature = dt.Sphere(
    position=get_position,
    z=lambda: np.random.randn() * 15,
    intensity=1,
    radius=lambda: 200e-9  + np.random.rand()*800e-9,
    diffusion_constant=lambda: (1 + np.random.rand() * 9) * 1e-12,
    position_unit="pixel",
    upsample=4
)<jupyter_output><empty_output><jupyter_text>2.3 Make it sequentialTo tell `particle_feature` how to change for each frame in a video, we call `Sequential` with the two functions defined in 2.1.<jupyter_code>sequential_particle = dt.Sequential(particle_feature, z=get_z, position=get_position)<jupyter_output><empty_output><jupyter_text>3.1 Define the full feature seriesWe define an optical device with some aberration. It is hard for the network to learn with heavy aberration. To mitigate this, the network will start training with very little aberration, and slowly increase the aberration as the network trains. To achieve this, we define a dummy property, `epoch`, which will return a number which increases by one each time the property is updated, up to `200`. The property `coefficient`, in turn, returns a normally distributed random number, with a standard deviation that scales with `epoch`. This way, the more times the property is updated, the more the optical device is aberrated.<jupyter_code>epoch_cap = 200
get_coefficient = lambda epoch: np.random.randn() * 0.1 * epoch / epoch_cap

spherical_aberration = dt.SphericalAberration(
    epoch=iter(range(1, epoch_cap + 1)), 
    coefficient=get_coefficient
)

horizontal_coma = dt.HorizontalComa(
    epoch=iter(range(1, epoch_cap + 1)), 
    coefficient=get_coefficient
)

vertical_coma = dt.VerticalComa(
    epoch=iter(range(1, epoch_cap + 1)), 
    coefficient=get_coefficient
)

aberrations = spherical_aberration >> horizontal_coma >> vertical_coma

optics = dt.Fluorescence(
    NA=0.7,
    magnification=10,
    resolution=1e-6,
    wavelength=633e-9,
    output_region=(0, 0, 64, 64),
    pupil=aberrations
)<jupyter_output><empty_output><jupyter_text>The on-camera captured intensity scales with the radius cubed times the property `intensity`. This means that the apparent intensity is much more dependent on the radius of the object than the intensity of each voxel. To remedy this, we define and apply a normalization feature to the `Sphere`. `normalize_sum` ensure that the sum of all pixels in the input is equal to `scale`. Thus, the apparent intensity will only depend on the property `intensity` of the sphere.<jupyter_code>normalize_sum = dt.Lambda(
    lambda scale: (
        lambda image: image / np.sum(image) * scale
    ),
    scale=100
)
def propagate_sequential_data(X, **kwargs):
    for dep in X.recurse_dependencies():
        if isinstance(dep, dt.properties.SequentialProperty):
            for key, value in kwargs.items():
                if hasattr(dep, key):
                    getattr(dep, key).set_value(value)<jupyter_output><empty_output><jupyter_text>3.2 Bring it togetherWe define the full feature series by imaging the sequential particle and the normalization feature with `optics`. `Sequence(imaged_article, sequence_length=10)`, in turn, creates a feature that resolves videos of length 10.<jupyter_code>imaged_particle = optics(sequential_particle / dt.Sum(sequential_particle) * 100) 

imaged_particle_sequence = dt.Sequence(imaged_particle, sequence_length=10)  


imaged_particle_sequence.update()
imaged_particle_sequence.plot()<jupyter_output><empty_output><jupyter_text>3.3 Vary the intensityVarying the intensity of the particle is equivalent to scaling the output image. This allows us to define an augmentation that converts a video of a particle of any intensity to any other intensity.We additionally define the method `update_properties`. This method is unique to the class `Augmentation`, and it lets us update the properties of the video in accordance with the augmentation used. Here, it updates all instances of the `intensity` property in the video by multiplying it by the `multiplier` property of the feature.<jupyter_code>class AugmentIntensity(dt.Augmentation):
    def get(self, image, multiplier, number_of_updates, **kwargs):
        return image * multiplier
    
    def update_properties(self, image, multiplier, **kwargs):
        for prop in image.properties:
            if "intensity" in prop:
                prop["intensity"] *= multiplier<jupyter_output><empty_output><jupyter_text>3.4 Augment the videosFor augmentation we will use a 8-way mirroring augmentation, followed by the aforementioned intensity augmentation. Noise is added after the augmentation to further diversify the generated dataset.<jupyter_code># Augment by flipping
augmented_particle_sequence = dt.FlipUD(dt.FlipLR(dt.FlipDiagonal(imaged_particle_sequence)))

# Augment by scaling intensity
augmented_particle_sequence = AugmentIntensity(augmented_particle_sequence, 
                                               multiplier=lambda: 1 + np.random.rand() * 9,
                                               updates_per_reload=2)

offset = dt.Add(value=lambda: np.random.rand() * 1)
augmented_particle_sequence += offset + dt.Poisson(snr=40, background=offset.value)
augmented_particle_sequence.update()
augmented_particle_sequence.plot()<jupyter_output><empty_output><jupyter_text>4.1 Define the modelThe model used will be `RNN`, which is a Recurrent Neural Network. The model acts like a convolutional neural network applied to each step of the video, followed by a LSTM layer, which merges all the information into a single output. RNNs can model complicated temporal behavours of videos of arbitrary length.<jupyter_code>model = dt.models.RNN(input_shape=(None, 64, 64, 1), 
            rnn_layers_dimensions=(32, 32),
            dense_layers_dimensions=(32, 32),
            number_of_outputs=1, 
            loss="mse")<jupyter_output><empty_output><jupyter_text>4.2 Define the label and the generatorThe label function extracts the property `intensity` from the first frame of the video. The generator is a standard generator with `ndim` set to 5 to account for it resolving videos instead of images.<jupyter_code>def get_intensity_of_particle(video):
    return video[0].get_property("intensity")

generator = dt.generators.ContinuousGenerator(
    augmented_particle_sequence, 
    get_intensity_of_particle, 
    batch_size=16,
    min_data_size=512,
    max_data_size=1024,
    ndim=5
)<jupyter_output><empty_output><jupyter_text>5. Train the modelWe train the model for 30 epochs, consisting of 100 batches eachs. This may take some time (a few minutes).Note that some versions of Tensorflow do not correctly handle videos like these correctly. If you see an error such as `GPU sync failed` try updating Tensorflow.Note that the model does not always train. If the model hasn't decreased past a loss of ~5 after 40 epochs, consider restarting (by creating a new model in cell 12)<jupyter_code>with generator:
    model.fit(generator, epochs=200)<jupyter_output><empty_output><jupyter_text>6. Validate the modelWe validate the correctness of the model by generating 200 videos and creating a scatter plot of predicted intensity vs. actual intensity.<jupyter_code>predicted_intensity = []
true_intensity = []
for _ in range(200):
    image = augmented_particle_sequence.update().resolve()
    label = get_intensity_of_particle(image)

    prediction = model.predict(np.array([image]))
    predicted_intensity.append(prediction[0])
    true_intensity.append(label)
        
plt.scatter(true_intensity, predicted_intensity)
plt.plot([1, 10], [1, 10])
plt.gca().set_aspect('equal', adjustable='box')
plt.title("Predicted intensity vs true intensity")
plt.xlabel("True intensity")
plt.ylabel("Predicted intensity")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07601174250366953, 'fraction_numerical': 0.017194380373243865, 'mean_word_length': 4.725690276110444, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '122428', 'license': 'MIT', 'n_tokens_mistral': 2839, 'n_tokens_neox': 2618, 'n_words': 1001, 'path': 'examples/tutorials/analyzing_video_tutorial.ipynb', 'repo_name': 'HarshithBachimanchi/DeepTrack-2.0', 'size': '9538'}