To enable the revolutionary
shift of data computation from remote cloud servers to pervasive end-devices (e.g., mobile and IoT devices),
collaborative deep learning strategies have been proposed in the literature. Indeed, two collaborative modes have been designed: The first mode, namely federated learning, consists of distributing the training tasks over multiple helpers while keeping the data locally in devices, which improves the efficiency and the speed of the learning [1]}. In the second mode, studied in this paper, the inference phase is distributed. Particularly, the trained DNN model is partitioned into segments, which are assigned to different participants. Each participant computes a segment and shares the output to the subsequent device until generating the final prediction. The distributed inference has attracted the focus of academia, particularly the mobile-cloud collaboration scenarios [2]}, [3]}. A step forward, the researchers studied the feasibility to use only the resource-constrained IoT devices to execute the entire inference at the proximity of the data source by jointly compute different segments of the neural network [4]}, [5]}. These efforts examined different DNN splitting scenarios and mainly focused on finding the optimal partitioning that reduces the dependency and data sharing between participants. However, designing a joint strategy where participants are scheduled to compute different segments of online inference requests while being constrained by their limited resources was not considered by previous works. Therefore, in order to support deep neural processing in IoT systems existing at the proximity of the data-generating device, the design of DNN distribution must be completely rethought by considering hardware and physical constraints, which will be done in our work.
