Task-Oriented and Semantics-Aware 6G Networks

Upon the arrival of emerging devices, including Extended Reality (XR) and Unmanned Aerial Vehicles (UAVs), the traditional bit-oriented communication framework is approaching Shannon's physical capacity limit and fails to guarantee the massive amount of transmission within latency requirements. By jointly exploiting the context of data and its importance to the task, an emerging communication paradigm shift to semantic level and effectiveness level is envisioned to be a key revolution in Sixth Generation (6G) networks. However, an explicit and systematic communication framework incorporating both semantic level and effectiveness level has not been proposed yet. In this article, we propose a generic task-oriented and semantics-aware (TOSA) communication framework for various tasks with diverse data types, which incorporates both semantic level information and effectiveness level performance metrics. We first analyze the unique characteristics of all data types, and summarise the semantic information, along with corresponding extraction methods. We then propose a detailed TOSA communication framework for different time critical and non-critical tasks. In the TOSA framework, we present the TOSA information, extraction methods, recovery methods, and effectiveness level performance metrics. Last but not least, we present a TOSA framework tailored for Virtual Reality (VR) data with interactive VR tasks to validate the effectiveness of the proposed TOSA communication framework.


I. INTRODUCTION
Inspired by Shannon's classic information theory [1], Weaver and Shannon proposed a more general definition of a communication system involving three different levels of problems, namely, (i) transmission of bits (the technical problem); (ii) semantic exchange of transmitted bits (the semantic problem); and (iii) effect of semantic information exchange (the effectiveness problem). The first level of communication, which is the transmission of bits, has been well studied and realized in conventional communication systems based on Shannon's bit-oriented technical framework. However, with the massive deployment of emerging devices, including Extended Reality (XR) and Unmanned Aerial Vehicles (UAVs), diverse tasks with stringent requirements pose critical challenges to traditional bit-oriented communications, which are already approaching the Shannon physical capacity limit. This imposes the Sixth Generation (6G) network towards a communication paradigm shift to semantic level and effectiveness level by exploiting the context of data and its importance to the task. Nikolaos Pappas is with the Department of Computer and Information Science, Linköping University, Sweden (email:nikolaos.pappas@liu.se).
Initial works on "semantic communications" have mainly focused on identifying the content of the traditional text and speech [2], [3], and the information freshness, i.e., AoI [4] as a semantic metric that captures the timeliness of the information. However, these cannot capture the data importance sufficiently of achieving a specific task. In [5], a joint design of information generation, transmission, and reconstruction was proposed. Although the authors explored the benefits of including the effectiveness level in [6], [7], an explicit and systematic communication framework incorporating both semantic level and effectiveness level has not been proposed yet. There is an urgent need for a unified communication framework aiming at task-oriented performances for diverse data types.
Motivated by this, in this paper, we propose a generic taskoriented and semantics-aware (TOSA) communication framework, which jointly considers the semantic level information about the data context, and effectiveness level performance metric that determines data importance, for different tasks with various data types. The main contributions of this paper are: 1) We first present unique characteristics of the traditional text, speech, image, video data types, and emerging 360 • video, sensor, haptic, and machine learning models. For each data type, we summarize the semantic information definition and extraction methods in Section II. 2) We then propose a generic TOSA communication framework for typical time critical and non-critical tasks, where semantic level and effectiveness level are jointly considered. Specifically, by exploiting the unique characteristics of different tasks, we present TOSA information, their extraction and recovery methods, and effectiveness level performance metrics to guarantee the task requirements in Section III. 3) To demonstrate the effectiveness of our proposed TOSA communication framework, we present the TOSA solution tailored for interactive Virtual Reality (VR) data with the aim to maximize the long-term quality of experience (QoE) within the VR interaction latency constraints and analyze the results in Section IV. 4) Finally, we conclude the paper in Section V.

II. SEMANTIC INFORMATION EXTRACTION
In this section, we focus on analyzing the characteristics of all data types, and summarizing the semantic information definition with corresponding extraction methods as shown in Table I.

A. Speech and Text
For a one-dimensional speech signal, the speech-to-text conversion can be first performed by speech recognition. With arXiv:2210.09372v1 [eess.SY] 17 Oct 2022

B. Image and Video
The image is a two-dimensional data type, where the image geometric structures, including edges, corners, blobs, and ridges, can be identified as typical semantic information. Although various traditional signal processing methods have been developed to extract image geometric structures such as SIFT (Scale-Invariant Feature Transform), the convolutional Neural Network (CNN) has shown stronger capability to extract complex geometric structures with its matrix kernel.
Video is a typical three-dimensional data type as the combination of two-dimensional images with an extra time dimension. Therefore, the temporal correlation between adjacent frames can be identified as important semantic information, where the static background can be ignored during transmission. To extract temporal correlation information from video, different CNN structures have been utilized.

C. 360 • Video
The 360 • rendered video is a new data type in emerging XR applications. The most important semantic information is identified as human field-of-view (FoV), which occupies around one-third of the 360 • video and only has the highest resolution requirement at the center [8]. In this case, biological information, such as retinal foveation and ballistic saccadic eye movements can be leveraged for semantic information extraction. Therefore, biological information compression methods have been utilized to extract the semantic information, where retinal foveation and ballistic saccadic eye movements are jointly considered to optimize the semantic information extraction process.

D. Haptic Data
Haptic data consists of two submodalities, which are tactile information and kinesthetic [9]. For tactile information, five major dimensions can be identified, which are friction, hardness perception, warmth conductivity, macroscopic roughness, and microscopic roughness. Kinesthetic information refers to the position/orientation of human body parts and external forces/torques applied to them. To reduce the redundant raw haptic data, Just Noticeable Difference (JND) is identified as valuable semantic information to filter the haptic signal that cannot be perceived by the human, where the Weber's law serves as an important semantic information extraction criterion.

E. Sensor and Control data
Sensors are usually deployed to monitor the physical characteristics of the environment (e.g., temperature, humidity, or traffic) in a geographical area. The acquisition of data is transformed to status updates that are transmitted through a network to the destination nodes. Then these data are processed in order to extract useful information, such as control commands or remote source reconstruction, that can be further utilized in the prediction of the evolution of the initial source. The accuracy of the reconstructed data either in control commands or to predict the evolution is directly related to the relevance or the semantic value of the data measurements. Thus, one important aspect is the generation of traffic and how it can be affected in order to filter only the most important samples so the redundant or less useful data will be eliminated to reduce potential congestion inside the network.
The AoI has also a critical role in dynamic control systems, since it was shown that non-linear AoI and VoI are paradigm shifts and they can improve the performance of such systems. Furthermore, we have seen in early studies that the semantics of information (beyond timeliness) can provide further gains, by reducing the amount of information that is generated and transmitted without degrading the performance.

F. Machine Learning Model
With the massive deployment of machine learning algorithms, machine learning-related model has been regarded as another important data type.
• Federated Learning (FL) Model: The FL framework has been considered as a promising approach to preserve data privacy, where each participating device uploads the model gradients or model weights to the server and receives the global model from the server. • Split Learning (SL) Model: Due to the limited computation capability of devices and heavy computation burden, SL has been proposed to split the neural network model between the server and devices, where the device executes the model up to the cut layer and sends the smashed data to the central server to execute the remaining layers. Then the gradient of the smashed data is transmitted back from the server to update the local model.  However, it is noted that explicit semantic information definition for machine learning model has not been proposed yet.

III. TOSA COMMUNICATION FRAMEWORK
In this section, we propose a generic TOSA communication framework incorporating both semantic level and effectiveness level, for typical time critical and non-critical tasks as shown in Fig. 1, where the TOSA information, its extraction and recovery methods, and effectiveness level performance metrics are presented in detail.

A. One-hop Task
We consider one-hop tasks with a single link transmission in this section, where each communication entity can be either human or machine as summarized in Table II. 1) Speech Recognition: In a speech recognition task, the human speech needs to be transmitted to the server, and the speech recognition task can be further divided into conversation-type task (e.g., human inquiry) and commandtype task (e.g., smart home control) depending on the speech content. The conversation-type task focuses on understanding the intent, language, and sentiment to provide human with free-flow conversations. The command-type task focuses on parsing the specific command over the transmitted speech and then controlling the target device/robot.
In the conversation-type task, the TOSA information can be keywords and emotions. The device can obtain the TOSA information by transforming the speech signal into text and extracting keywords and emotions via BERT. Then the server recovers the text via transformer decoder. In the commandtype task, the TOSA information can be the binary command, the device can directly parse the speech signal and obtain the binary command signal for transmission, where no recovery is needed at the receiver side. The effectiveness level performance metrics include F-measure, accuracy, bilingual evaluation understudy (BLEU), and perplexity, where user satisfaction should also be considered.
2) Face Detection and Road Segmentation: Face detection and road segmentation are two emerging image processing tasks [10], [11], where the captured images are required to be transmitted to the central server for processing. However, the road segmentation task in autonomous driving applications imposes stringent latency and reliability requirements due to road safety issues. This is because the vehicles need to instantaneously react to the rapidly changing environment. For the time non-critical face detection task, TOSA information can be the face feature that is extracted via CNN. After being transmitted to the central server, the regions with CNN features (R-CNN) can be applied to perform face detection. For the time critical road segmentation task, one possible solution is to identify the region of interest (ROI) features, i.e., road, as the TOSA information, and crop the images via region proposal algorithms. Then, the central server can perform image segmentation via mask R-CNN. Both tasks can be evaluated via effectiveness level performance metrics, including Intersection over Union (IoU), mean average precision (mAP), F-measure, and mean absolute error (MAE). It is noted that road segmentation in the autonomous driving application can be evaluated via pixel accuracy, and mean pixel accuracy (MPA). However, the trade-off between accuracy and latency remains to be an important challenge to solve.
3) Display in Extended Reality: Based on the Milgram & Kishino's Reality-Virtuality Continuum, the XR can be classified as Augmented Reality (AR), Mixed Reality (MR), and VR, where MR is defined as a superset of AR. Therefore, we focus on the AR and VR display tasks in the following.
In the AR display task, the central server transmits the rendered 3D model of a specific virtual object to the user. It is noted that the virtual object identification and its pose information related to the real world is the key to achieving alignment between virtual and physical objects. Therefore, the virtual object identification and pose information can be extracted as TOSA information to reduce the data size. Then, by sharing the same 3D model library, the receiver can locally reconstruct the 3D virtual object model based on the received TOSA information. To evaluate the 3D model transmission, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) can be adopted as effectiveness level performance metrics. However, how to quantify the alignment accuracy among the virtual objects and physical objects as a performance metric remains to solve.
In the VR display task, the central server transmits virtual 360 • video streaming to the user. To avoid the transmission of the whole 360 • video, the central server can predict the eye movements of the user and extract the corresponding FoV as TOSA information. Apart from the PSNR and SSIM mentioned in AR, timing accuracy and position accuracy are also important effectiveness level performance metrics to avoid cybersickness including: 1) initial delay: time difference between the start of head motion and that of the corresponding feedback; 2) settling delay: time difference between the stop of head motion and that of the corresponding feedback; 3) precision: angular positioning consistency between physical movement and visual feedback in terms of degrees; and 4) sensitivity: capability of inertial sensors to perceive subtle motions and subsequently provide feedback to users. 4) Grasping and Manipulation: Haptic communication has been incorporated by industries to perform grasping and manipulation for efficient manufacturing and profitable production rates, where the robot transmits the haptic data to the manipulator. The shape and weight of the objects to be held are measured using cutaneous feedback derived from the fingertip contact pressure and kinesthetic feedback of finger positions, which should be transmitted within stringent latency requirement to guarantee industrial operation safety.
Due to the difficulty in supporting massive haptic data with stringent latency requirement, JND can be identified as an important TOSA information to ignore the haptic signal that cannot be perceived by the manipulator. Two effectiveness level performance metrics including SNR and SSIM have been verified to be applicable to vibrotactile quality assessment. 5) Control: In networked control systems (NCS), typically, multiple sensors measure the system state of their control processes and transmit the generated data over a resource-limited shared wireless network. These data usually are en-queued and then transmitted over unreliable channels that cause excessive delays resulting in outdated or even obsolete for decisionmaking based on less reliable information. Therefore, data freshness and importance are extracted as TOSA information via AoI and value of information (VoI) to guarantee the timing requirement, respectively [12]. A typical effectiveness level performance metric that is used to minimize is the Linear Quadratic Gaussian (LQG) cost function, and usually the lower the value of the LQG function the higher the quality of control (QoC).
6) Machine Learning: In the following, we focus on taskoriented communications for two distributed ML models, which are FL and SL. a) Federated Learning: For the time non-critical tasks, such as NLP and image classification, the goal of FL is to guarantee a high learning accuracy without latency constraints. Traditional loss functions for NLP and image classification, such as mean square error (MSE), MAE, and cross-entropy, can be directly used as effectiveness level performance metrics. However, for time critical tasks, such as object recognition in self-driving cars, the goal of FL is to balance the trade-off between learning accuracy, communication latency, and computation latency. The effectiveness level performance metrics are loss functions with latency constraints. Time critical tasks bring communication challenges, and communication-efficient FL should be designed to decrease the model size to satisfy latency constraints via federated dropout, federated pruning, and model compression.
Federated dropout is a simple way to prevent the learning model from overfitting through randomly dropping neurons and is only used during the training phase, which decreases communication and computation latencies and slightly improves learning accuracy. However, during the testing phase, the extracted task-oriented information is the whole learning model and transmitted between the server and devices. The extracted task-oriented information is the model with nondropped weights. Meanwhile, federated dropout does not need model recovery.
Unlike federated dropout only temporarily removing neurons, federated pruning permanently removes neurons in either or both training and testing phases. The extracted task-oriented information in federated pruning is the pruned model. The decision of which parameters to remove is made by considering the importance of each parameter. The pruning ratio should be carefully designed to guarantee learning accuracy and extra computation latency is required to calculate the importance of parameters. Thus, how to design federated pruning methods with low computation complexity needs to be investigated. In addition, federated pruning does not need model recovery.
Model-compression schemes, such as sparsification and quantization decrease the model size. The extracted taskoriented information is the sparse or quantized model. However, these methods slightly decrease the convergence rate and achieve a modest accuracy (about 85%). Thus, how to design a model compression algorithm with high learning accuracy still needs to be investigated. In addition, compressed FL model needs to be recovered at the receiver. b) Split Learning: In SL, the smashed data and its gradient associated with the cut layer are the extracted taskoriented information and transmitted between the server and devices, where no model recovery is required. When multiple devices exist in SL, all devices interact with the edge server in a sequential manner, resulting in high training latency. For time non-critical tasks, such as NLP and image classification, the goal of SL is to achieve high learning accuracy without latency constraints and the effectiveness level performance metrics are the same as that of the FL. However, for time critical tasks, the SL cannot guarantee the requirement of low latency because of its sequential training pattern.
To adapt the SL to time critical tasks, such as real-time object tracking, splitfed learning (SFL) [13] and hybrid split and federated learning (HSFL) [14] are proposed, where they combine the primary advantages of FL and SL. The effectiveness level performance metrics of the SFL and HSFL are learning accuracy and training latency. However, SFL and HSFL assume that the model is split at the same cut layer and the server-side model is trained in a synchronous mode. Splitting at the same cut layer leads to asynchronization of device-side model training and smashed data transmission. Thus, how to select an optimal split point and deal with the asynchronization of SL remain important challenges to solve. Also, different split points can result in different smashed data. Thus, how to merge these smashed data in the server-side model should be considered.

B. Chain Task
In this section, we analyze more complicated but practical chain tasks including XR-aided teleoperation and chain of control, where multiple entities cooperate through communication links to execute the task.
1) XR-aided Teleoperation: XR-aided teleoperation aims to integrate 3D virtual objects/environment into remote robotic control, which can provide the manipulator with immersive experience and high operation efficiency [15]. To implement a closed-loop XR-aided teleoperation system, the wireless network is required to support mixed types of data traffic, which includes control and command (C&C) transmission, haptic information feedback transmission, and rendered 360 • video feedback transmission. Stringent communication requirements have been proposed to support XR-aided teleoperation use case, where over 50 Mb/s bandwidth is needed to support video transmission, and reliability over 1 − 10 −6 within millisecond latency is required to support haptic and C&C transmission.
As XR-aided teleoperation task relies on both parallel and consecutive communication links, how to guarantee the cooperation among these communication links to execute the task is of vital importance. Specifically, the parallel visual and haptic feedback transmissions should be aligned with each other when arriving at the manipulator, and consecutive C&C and feedback transmissions should be within the motion-to-photon delay constraint. Either violation of alignment in parallel links or latency constraint in consecutive links will lead to BIP and cybersickness. Therefore, both parallel alignment and consecutive latency should be quantified into effectiveness level performance metrics to guarantee the success of XR-aided teleoperation. Moreover, due to the motion-to-photon delay, the control error between the expected trajectory and actual trajectory will accumulate along with the time, which may lead to task failure. Hence, how to alleviate the accumulated error remains an important challenge to solve.
2) Chain of control: In the scenario of a swarm of (autonomous) robots where they need to perform a collaborative task (or a set of tasks) within a deadline over a wireless network, an effective communication protocol that takes into account the peculiarities of such a scenario is needed. Otherwise, the generated and transmitted data will be of very high volume that eventually will face congestion in the network causing large delays and the operated control mechanisms will not be synced causing inefficient or even dangerous operation. Consider the simple case of two robots, let's say Robot A and Robot B that are communicating through a wireless network and they are not collocated. Robot A controls remotely Robot B such that to execute a task and the outcome of that operation will be fed to Robot A for performing a second operation to send the outcome back to Robot B. All this must happen within a strict deadline. The amount of information that is generated, transmitted, processed, and sent back can be very large with the traditional information agnostic approach. On the other hand, if we take into account the semantics of information and the purpose of communication, we change the whole information chain, from its generation point until its utilization. Therefore, defining TOSA metrics for the control loop and communication between a swarm of (autonomous) robots is crucial and it will significantly reduce the amount of information leading to a more efficient operation.

IV. CASE STUDY
We validate the effectiveness of ML algorithms in optimizing time-critical TOSA communication under one typical case, namely, MEC-enabled and reconfigurable intelligent surface (RIS)-assisted terahertz (THz) VR network. The goal of wireless VR networks is to guarantee high QoE under VR interaction latency constraints. Note that the QoE guarantees the seamless, continuous, smooth, and uninterrupted experience of each VR user. However, traditional wireless VR networks transmit whole 360 • videos, which leads to low QoE and high VR interaction latency. In the simulation, we extract the FoV of VR users as TOSA information to decrease the transmission data size. To extract the FoV, viewpoints of VR users are predicted via recurrent neural network (RNN) algorithms. Based on the predicted viewpoints, the corresponding FoVs of VR video frames can be rendered and transmitted in advance. Thus, the MEC does not need to render and transmit the whole 360 • VR video frames, which can substantially decrease the VR interaction latency and improve the QoE of VR users. Fig. 2 plots the average QoE and VR interaction latency of each time slot via constrained deep reinforcement learning with the viewpoint prediction via RNN. It is observed that the proposed ML algorithms maximize the long-term QoE of VR users under the VR interaction latency threshold.

V. CONCLUSION
In this article, we propose a generic task-oriented and semantics-aware (TOSA) communication framework incorporating both semantic and effectiveness levels for various tasks with diverse data types. We first identify the the unique characteristics of all existing and new data types in 6G networks and summarize the semantic information with its extraction methods. To achieve task-oriented communications for various data types, we then present the corresponding TOSA information, their TOSA information extraction and recovery methods, and effectiveness level performance metrics for both time critical and non-critical tasks. Importantly, our results demonstrate that our proposed TOSA communication framework can be tailored for VR data to maximize the long-term QoE within VR interaction latency constraints. The paradigm shift from conventional Shannon's bit-oriented communication design towards the TOSA communication design will flourish new research on task-driven, context and importance-aware data transmission in 6G networks.