Memory networks keep the history of rare examples, enabling them perform well even with insufficient data. To train memory networks in an unsupervised manner, a novel threshold triplet loss was introduced by the authors [1]}. In memory networks, the key-memory holds spatial information and computes cosine similarity against the input. Likewise, value-memory keep records about color information utilized by the colorization networks as a condition. Similarly, age keeps the time-stamp for each entry in the memory. The key and value memories are generated from the training data. The memory is updated by averaging the key-memory and a new query image owing to the distance between the color information of the new query image and the available top-1 value-memory element lies below the threshold. Otherwise, a new record is stored for the input image color information.
