Since the breakthrough of the CNN during the 2010s, there has been fairly limited work applying AR models to the manufacturing domain in order to understand and facilitate HRC. Wang [1]} fine-tune two AlexNet[2]} models to classify three parts being used as well as three current “motions" (grasping, holding and assembling) during an engine block assembly within a controlled environment. Al-Amin et al. [3]} utilise a multi-modal sensor fusion, including image data, to classify seven actions within a controlled assembly process. Zhang et al. [4]} utilise a bi-stream CNN to extract human and object information from video frames, with a variable-length Markov model making future action predictions within a controlled assembly scenario. Xiong [5]} incorporates motion information via the use of a modest two stream network classifying actions occurring within an engine block assembly video scraped from YouTube. Jones [6]} develop a framework for understanding fine-grained assembly actions as a structure. In contrast to these works, which are all limited in approach, and are deployed within controlled environments, our work utilises fully spatio-temporal feature extraction and is applied to our novel real world assembly dataset.
