The rules for feature alignment were presented as a global rule: the auxiliary loss is defined with the output layer and the loss is defined with the input layer, so we have full communication between all layers. However, we can reformulate this rule with local losses, similar to target propagation rules [1]}, [2]}, [3]}: the auxiliary loss and loss are defined as the interaction between two connected layers only (or even individual neurons), as follows: for each layer from the first to the last, we activate it from the its inputs and store a second activation from a random input with the same dimension, we then extract the feature by optimizing the random input with an auxiliary loss between activation of the random output and the true input. Finally, the parameters of the chosen layer is updated by optimizing the loss between the reconstruction and true input. This technique of local training is summarized in the Algorithm 2 and illustrated at FIG. REF .
