Driven by the recent advancement of big data and Artificial Intelligence (AI), deep neural networks have presented substantial breakthroughs in a broad range of applications, spanning from speech recognition [1]} and smart home appliances to image recognition [2]}, video surveillance and robotics [3]}. Such high performance of deep networks is related to the ability of the model to obtain distinctive high-level features from raw data, owing to its extensive mathematical operations, its complex structure, the large number of parameters and hidden layers, and the significant amount of processed data. A representative example of a state-of-the-art Deep Neural Network that has demonstrated unprecedented performance in vision recognition tasks, is VGG. VGG [4]} includes more than 144 million parameters, 15 million neurons and 3 billion connections and can only be executed on powerful devices as deploying it on mobile devices presents intolerable classification latency, reaching over 16 seconds [5]}.
