In recent times, large pre-trained neural network models have been successfully used to achieve state-of-the-art performance in computer vision (CV) [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}, [18]}, [19]}, [20]}, [21]} and language processing (NLP) [22]}, [23]}, [24]}, [25]}, [26]}, [27]}, [28]}, [29]}, [30]}, [31]}, [32]}, [33]}, [34]}, [35]}. In particular, the field of NLP is seeing an exponential expansion in terms of real-life applications such as machine translation [36]}, document classification [37]}, interaction with a chatbot [38]}) but also network size as Transformer-based models seem to outperform long-standing architectures such as Long Short-Term Memory (LSTM) recurrent neural networks [39]}. However, the main issue with this marvellous kind of neural networks is that the appalling size of parameters (in the order of hundreds of billions in the case of OpenAI’s GPT models [40]}, [41]}, [42]}, [28]}) usually requires a training dataset of huge dimension such as the complete Wikipedia corpus in several languages, and a massive computer cluster to carry out the training.
An estimation of the training cost in terms of the electricity bill is in the order of almost 5 million US dollars for GPT-3, and it would take more than 350 years if a single GPU was being used. Some see this as evidence that the entire field of artificial intelligence is becoming more and more undemocratic (a small company, let alone single person, simply can’t compete with tech giants), while others acknowledge that the progression resembles what happened in science in the past decades: it was only the concerted effort of national states over three decades that yielded the discovery of the Higgs boson at the CERN Large Hadron Collider [44]}, [45]}.
