Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups

We develop data-driven models to predict when a robot should feed during social dining scenarios. Being able to eat independently with friends and family is considered one of the most memorable and important activities for people with mobility limitations. While existing robotic systems for feeding people with mobility limitations focus on solitary dining, commensality, the act of eating together, is often the practice of choice. Sharing meals with others introduces the problem of socially appropriate bite timing for a robot, i.e. the appropriate timing for the robot to feed without disrupting the social dynamics of a shared meal. Our key insight is that bite timing strategies that take into account the delicate balance of social cues can lead to seamless interactions during robot-assisted feeding in a social dining scenario. We approach this problem by collecting a Human-Human Commensality Dataset (HHCD) containing 30 groups of three people eating together. We use this dataset to analyze human-human commensality behaviors and develop bite timing prediction models in social dining scenarios. We also transfer these models to human-robot commensality scenarios. Our user studies show that prediction improves when our algorithm uses multimodal social signaling cues between diners to model bite timing. The HHCD dataset, videos of user studies, and code are available at https://emprise.cs.cornell.edu/hrcom/


Introduction
Nearly 27% of people living in the United States have a disability, and close to 24 million people aged 18 years or older need assistance with activities of daily living (ADL) [1]. Key among these activities is feeding, which is both time-consuming for the caregiver, and challenging for the care recipient (patient) to accept socially [2]. Indeed, needing help with one or more ADLs is the most cited reason for moving to assisted or institutionalized living [3,4]. Although there are several automated feeding systems on the market [5][6][7][8][9][10][11][12][13], they have lacked widespread acceptance. One of the key reasons is that all of them require manual triggering of bite timing by the user, which is challenging for users with cognitive disabilities and inconvenient in social settings. A key challenge for the realization of autonomous robotic feeding systems is therefore to infer proper bite timing [14]. While existing systems focus on solitary dining (e.g. [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32]), commensality, the act of eating together, is often the practice of choice. People like to share meals with others. The social experience of a shared meal is an important part of the overall eating experience and current robot feeding systems are not designed with that experience in mind. Transferring the challenge of inferring appropriate bite timing to a social dining setting requires not only attuning to the user's eating  We conduct a social robot-assisted feeding user study by deploying a variation of SoNNET on a robot. We refer to the User also as a Target user.
behavior but also to the complex social dynamics of the group. For example, a robot should not attempt to feed a user who is actively engaged in conversation. Motivated by a growing body of research that seeks to develop models for robots to function in group settings [33,34] we ask the seemingly simple question: How should an assistive feeding robot decide the right timing for feeding a user in ever-changing and dynamic social dining scenarios?
We developed an intelligent autonomous robot-assisted feeding system that uses multimodal sensing to feed people in dynamic social dining scenarios. We collected a novel audio-visual Human-Human Commensality Dataset (HHCD) capturing human social eating behaviors. Using this data, we then trained multimodal machine learning models to predict bite timing in human-human commensality. We explored how our models trained on human-human commensality scenarios performed in a human-robot commensality setting and evaluated them in a user study. The overall workflow is shown in Fig. 1. We made algorithmic and experimental design decisions by consulting with care recipients, caregivers, and occupational therapists. We find that bite timing prediction improves when our model accounts for social signaling among diners, and such a model is preferred over a manual trigger and a fixed-interval trigger. Our main contributions include: • A SOcial Nibbling NETwork (SoNNET) which captures the subtle inter-personal social dynamics in human-human and human-robot groups for predicting bite timing in social-dining scenarios. • Methods that can successfully transfer bite timing strategies learned from human-human commensality cues to human-robot commensality situations, which we evaluate in a user study with a robot in 10 triadic human groups. • A socially-aware robot-assisted feeding system that extends our capacity to feed people in solitary settings to groups of people sharing a meal. • An analysis of various social and functional factors that affect human feeding behaviors during human-human commensality. • A novel Human-Human Commensality Dataset (HHCD) containing multi-view RGBD video and directional audio recordings capturing 30 groups of three people sharing a meal.

Human-Robot Commensality
Eating is a complex activity that requires the sensitive coordination of several motor and sensory functions. Anyone who has fed another knows that feeding, particularly social feeding where a person is being fed in a social setting, is a delicate dance of multimodal signaling (via gaze, facial expressions, gestures, and speech, to name a few). Research on commensality, the practice of eating together, has highlighted the importance of the social nature of eating for social communion, order, health, and well-being [35]. As a consequence, digital commensality has focused on understanding the role of technology in facilitating or inhibiting the more pleasurable social aspects of dining [36]. When a person relies on assisted feeding, meals require that patient and caregiver coordinate their behavior [37]. To achieve this subtle cooperation, the people involved must be able to initiate, perceive, and interpret each other's verbal and non-verbal behavior. The main responsibility for this cooperation lies with caregivers, whose experiences, educational background, and personal beliefs may influence the course of the mealtime [38]. Our goal in this work is to understand the rhythm and timing of this dance to enable an automated feeding assistant to be thoughtful of when it should feed the user in social dining settings. We introduce the concept of Human-Robot Commensality at the intersection of commensality and robot-assisted feeding in social group settings. Our research is motivated by the key insight that bite timing strategies that take into account everchanging social signals and group dynamics can lead to a seamless human-robot collaboration in social dining scenarios. Fueled by this insight, we believe a feeding device that takes the initiative and offers bites proactively during the meal at times when a bite is likely to be desired will create a more seamless dining experience than a device that requires the user to initiate bites. Herlant [39] designed an HMM to predict bite timing in dyadic robot-assisted feeding. However, her model only considered the social cues of the user. Bhattacharjee et al. [40] found users preferred less intrusive interfaces in a social dining scenario, specifically a web interface over a voice interface. Our work aims to build non-intrusive bite timing strategies by focusing on learning when to feed a user in triadic scenarios while using implicit social features from all diners. Particularly, bite timing is important because the consequences of presenting a bite to the diner earlier than expected is poorly tolerated. This can include an interruption to conversation or to finishing chewing the prior bite. The consequences of presenting a bite later than desired can include frustration towards the robot and disruption of the natural flow of conversation during the meal. Parallels can be drawn to interruptibility research on finding the most appropriate timing to probe a user. Researchers have found that people performed best on a task if interruptions were mediated rather than timed immediately or on scheduled intervals [41,42], often mediated based on modeling contextual and social factors [43][44][45][46]. A socially-aware robot-assisted feeding system should be designed such that if needed, the user should be able to communicate these intentions via multiple different modalities such as body language, gaze, or speech. These various modalities have been found to be effective in modeling social interactions [47][48][49][50]. Capturing these natural social interactions in computational models are likely crucial to provide accurate bite timing without distracting users from the social ambiance.

Problem Formulation
The objective of the bite timing prediction problem in robot-assisted feeding with a single diner is to predict the timing of when this user will take a bite of food by capturing their signals U such as voice, body gestures, head movements or speaking status. We define the proper timing for when a robot should feed as when the user intends to take a bite of food. It takes input signals U(t 0 : t) from time t 0 to time t and learns a function F(U) to predict a Boolean y(t + h) = F (U (t 0 : t)), which indicates whether the user intends to take a bite in the time horizon h and trigger a bite transfer at time t + 1. When a person lifts their fork off the plate to eat, they intend to take a bite of food, where this time horizon h is the time it takes to transfer the food to their mouth from their plate. In this paper, we consider a social variant of the bite timing prediction problem where a user is interacting with two co-diners. Our goal is to predict the timing of a user to take a bite of food based on the social cues within the interaction. From an initial time t 0 to time t, the user receives social signals L(t 0 : t) and R(t 0 : t) from their left and right conversational co-diners, respectively. Given these external social signals and the target user's own history of signals U(t 0 : t), we aim to predict y. We note that it may not always be possible to track the same set of features for a user and their co-diners. Therefore, for some time range k = t − t 0 and feature dimensions n, m for the user and co-diners respectively, U ∈ R k×n while L, R ∈ R k×m , where n does not necessarily equal m. The function to learn is:  behavior: bite features, which include the number of bites taken so far and the time since the last bite of food b ∈ R 2 , a diner's gaze and head pose direction d ∈ R 4 , binary speaking status s ∈ {0, 1}, and face and body keypoints o ∈ R 168 from OpenPose [51]. We note that, in our case, the bite features b are computed only for the user and not the co-diners, since we do not estimate in real-time whether a co-diner is taking a bite of food. Thus, for a time interval k = t − t 0 , these features are temporally stacked to construct the input signals U ∈ R k×175 , L ∈ R k×173 , R ∈ R k×173 for the user, left co-diner, and right co-diner, respectively. Recently, convolutional neural networks (CNNs) have demonstrated significant success for multichannel time series classification from various kinds of signals [52][53][54]. Wu et al. [46] proposed PazNet: a multi-channel deep convolutional neural network which is able to handle inputs of different dimensions. PazNet is designed to predict the interruptibility of individual drivers. However, the information of different channels is not shared, and it lacks ability to capture social interactions among multiple people. We design the Social Nibbling NETwork (SoNNET), a new model architecture which follows a multi-channel pattern allowing multiple interconnected branches to interleave and fuse at different stages. We create input processing channels for each diner, then add interleaving tunnels between each convolutional module and adjacent branches. The information capturing visually-observable behaviors between the diners is allowed to flow between the frames and channels. We conjecture that our model will learn a socially-coherent structure, allowing the model to implicitly represent the diners in an embedding space. Therefore, each channel has the same structure but does not share the same weight parameters. To help capture informative features, we performed dimension-reduction after the interleaving components using max pooling layers and 1 × 1 convolutional layers. These per-diner channels are concatenated and then followed by two dense layers for classification, which decides whether the user intends to feed or not. For SoNNET, the range between t and t 0 is six seconds. The social signals in this range are used to predict a user's bite intentions. Triplet-SoNNET. For modeling the bite-timing prediction of three users with no mobility limitations, we propose Triplet-SoNNET which uses social signals from the left and right co-diners L, R and signals from the user U. Depicted in Fig. 2, Triplet-SoNNET ensures that the features from other co-diners L, R interleave into the target user's features U. Couplet-SoNNET. To run Triplet-SoNNET in a robot-assisted feeding setting, there would be a distribution shift in the kinds of signals a target user outputs. Our goal is to feed people with mobility limitations while they are engaged in social conversations. The features from someone self-feeding are inherently different from someone using a robot-assisted feeding system. In the case of body pose, a target user with C3-C5 SCI would be largely still, which is different from the training data. Our Human-Human Commensality Dataset consists of adult diners with no mobility limitations, thus applying a trained Triplet-SoNNET model to robot-assisted feeding of a user with mobility limitations would be out-of-distribution. Although our target users with C3-C5 SCI cannot move their arms to feed themselves, there is still a large spectrum of severity in mobility limitations depending on the users' conditions. From our discussions with key stakeholders, caregivers, and occupational therapists we design, Couplet-SoNNET, where we ignore most social signals from the target user by removing the last channel in Triplet-SoNNET. Therefore, the intention to feed y(t + 1) = F (U b (t 0 : t) , L (t 0 : t) , R (t 0 : t)), where U b ∈ R k×2 are the user's bite features for k = t − t 0 . The user's bite features, such as the time since the last bite and the number of bites since the onset of the feeding activity, are the only social signals from the target user. Additional discussion on this design choice can be found in App. 8.2.2.

Human-Human Commensality Dataset (HHCD)
We introduce a novel Human-Human Commensality Dataset (HHCD) of three participants with no mobility limitations eating in a social scenario. We used this dataset to develop models that predict a diner's intention to take a bite of food while taking into account subtle social cues. We deployed the trained models in a social robot-assisted feeding setting where one diner is fed by a robot. Beyond predicting bite timing, we are excited for the robot learning community to find other interesting challenges within our dataset that leverage understanding social dynamics. Data Collection Setup. We recruited 90 people among our Institution-affiliated fully-vaccinated students, faculty, and staff to eat a meal in a triadic dining scenario. Each participant was 18+ years old and took part in the study only once. The study setup is illustrated in Fig. 1 (left). There are three cameras (mutually at 120 • ) in the middle of the table, each capturing one participant, and a fourth camera capturing the whole scene. All four cameras are Intel RealSense Depth Cameras D455 [55]. The scene audio was captured by a ReSpeaker Mic Array v2.0 [56] placed in the middle of the table. The ReSpeaker microphone array has four microphones arranged at the corners of a square and estimates the direction of sound arrival. For the study setup measures, see App. 8.1.2. Participants were free to bring any kind of food and any utensil with them. They could also bring a drink (some drank from a cup, others from a bottle or both, with or without a straw) and were provided with napkins. Before the study, each participant was asked to fill in a pre-study questionnaire about their demographic background, relationship to other participants, and social dining habits. The experimenter then asked them to eat their meals and have natural conversations. At this point, the experimenter started the recording and left the room. When all three participants finished eating or after 60 minutes have passed, whichever was earlier, the experimenter stopped the recording and asked participants to fill in a post-study questionnaire about their dining experience. The specific questions asked in both pre/post-study questionnaires can be found in App. 8.1.3. The study was approved by Cornell's IRB. Data Annotation. We annotated each participant's video based on their interactions with food, drink, and napkins. In particular, we annotated food entered, food lifted, food to mouth, drink entered, drink lifted, drink to mouth, napkin entered, napkin lifted, napkin to mouth, and mouth open events. We chose these events as they are key transition points during feeding. We spent 151 hours annotating and used the ELAN annotation tool [57]. We assigned the annotation value ∈ {fork, knife, spoon, chopsticks, hand} based on the utensil performing the food-to-mouth handover. While annotating, we also noted down per-participant food types and observations of interesting behaviors. All annotation types with detailed rules are provided in App. 8.1.4. Data Statistics. There were 56 female and 34 male participants, and their ages ranged 18-38 (µ = 22, σ = 3) years. Session durations ranged 21-55 (µ = 37, σ = 9) minutes and 1 session was at breakfast, 10 at lunch, and 19 at dinner time. For additional dataset statistics, see App. 8.1.5. For a summary of all available data in the dataset and its detailed analysis, see App. 8.1. For the purposes of this work, we only consider bite features, speaking status, gaze and head pose, and body and face keypoints.

Model Evaluation on Human-Human Commensality Dataset
In this section, we evaluate Triplet-and Couplet-SoNNET against other models on the HHCD. In particular, we compare against a regularized linear SVM trained with SGD to evaluate performance of linear classifiers. We also consider a Temporal Convolution Network (TCN) [58,59], which uses causal convolutions and dilations to represent temporal data. TCNs have been found to perform better than LSTMs and GRUs on temporal anomaly detection [60] and robot food manipulation tasks [20], therefore they would provide a strong baseline to compare our models to. We also perform an  ablation study to investigate the importance of various modalities. Implementation details about baseline models, SoNNET, and training procedure can be found in App. 8.2. For training, we use 6811 food lifted annotations as positive training labels since they precede an actual bite of food and indicate an intention to eat. We use a time interval of k = t − t 0 = 6 seconds because it takes roughly 6 seconds for the robot to move from its wait position to feeding the user. Since bite actions are sparsely distributed over time, we select 2486 6-second clips as negative samples that are in the middle of two food lifted annotations. All reported models are trained with leave-one-session-out (LOSO) cross-validation to evaluate generalizability to new groups of people. Due to an issue with recording, we train over 29 sessions if speaking status features are used. The user's bite features b ∈ R 2 (time since last bite and the number of bites eaten since the start) are indicators of eating rate. To ensure this feature is not dominated by higher dimensional features, we scale the size of the input by γ. This hyperparameter γ scales b ∈ R 2 → b ∈ R 2γ . We selected γ = 100 after a grid search over the training set on the TCN and SoNNET models. Evaluation Metrics. A high recall indicates that our model can reliably feed when it should. In contrast, a high precision indicates that a model tends to be stricter in deciding when to feed. Due to our dataset imbalance, the average accuracy across 29 sessions for a model that predicts it should always feed is 71.56%. This classifier achieves perfect recall, and relatively high precision, causing the model to have a high F1 score. It is clear that given this class imbalance, a high F1 score poorly represents the capabilities of this model. To evaluate our model effectively, we consider the normalized Matthews Correlation Coefficient (nMCC) in addition to F1 score, precision, recall, and accuracy. Unlike F1 score, nMCC considers the size of the majority and minority classes, and can only produce high scores if a classifier is able to make correct predictions for a majority of both the negative and positive classes [61]. A value of 0.5 indicates random prediction, while 0 is inverse prediction and 1 is perfect prediction. Effects of Modality. We are interested in investigating features that are the most informative for designing a good bite timing predictor in social dining. We perform a feature ablation study on the Triplet-SoNNET model, as shown in Table 1. We selectively remove feature streams, such as body and face data from OpenPose, gaze and head features from RT-GENE, speaking status signals, and the user's bite features. We find that users' bite features such as the time since last bite and the number of bites are important, as accuracy drops drastically without them. Intuitively, we believe this feature is important because a user's bite features are a proxy for their level of eating rate. We also see that without body and face features, F1 and recall slightly increase. This could be due to the fact these data streams are noisy; however, as indicated by the lower accuracy and nMCC when removing OpenPose features, these features are useful. Effects of Model Type. Table 2 shows the outcomes of various model comparisons when trained using LOSO. We compare performance of Triplet-SoNNET against a linear SVM and TCN trained on all three diners. We call this TCN a Triplet-TCN. Triplet-TCN has all the diners' features concatenated per-timestep, and we compare this result to Triplet-SoNNET. We find that Triplet-SoNNet achieves higher accuracy and nMCC compared to all other models; however, it performs worse on recall and F1 score compared to Triplet-TCN. In our scenario, we want to ensure that the robot feeds when it should and does not feed when it should not. A bite prediction model that overfeeds or underfeeds is not ideal. A high nMCC balances the roles of recall and precision and better represents whether a classifier should or should not feed. Therefore, for our scenarios, Triplet-SoNNET is a more effective predictor of bite timing than other models trained on all three diners. Effects of Social Scenario. We are interested in comparing the ability of models to learn social behaviors using only two co-diners' features rather than having full observability. We compare Couplet-SoNNET to a similarly-named Couplet-TCN trained on two co-diners' features and a user's bite features. As expected, Couplet-TCN and Couplet-SoNNET perform worse than their Tripletcounterparts, with Couplet-TCN being close to random prediction with an nMCC of 0.5539 while Couplet-SoNNET has an nMCC of 0.6648. We find that Couplet-SoNNET performs better than Couplet-TCN. This result reveals Couplet-SoNNET is able to understand social signals better than a predictor that always feeds. This implies that it is possible to predict the behavior of a user using only their co-diner information, which indicates that there is social coordination in human-human commensality. These findings also suggest that social signals were captured by the interleaving structure of the SoNNET models.

Transferring from Human-Human to Human-Robot Commensality
Our objective is to develop a bite timing strategy for a robot that feeds a user in a social dining setting. We design a study where users evaluate the effect of different bite timing strategies on their overall social dining experience. To simulate robotic caregiving scenarios for people with upperextremity mobility limitations, we instructed users to not move their upper body. This study was approved by Cornell's IRB. Experimental Setup. We evaluate a learned bite timing strategy against two baseline bite timing strategies inspired by our conversations with care recipients, occupational therapists, and caregivers who told us how they know when to feed. The strategies are further depicted in Fig. 3: 1. Learned Timing. This social, fully autonomous bite timing strategy feeds based on our Couplet-SoNNET model's output. We sample this model every three seconds with the last six seconds of preprocessed features at a rate of 15 frames per second. This approach takes into account the social context. Since we want to evaluate the generalization performance, we train Couplet-SoNNET on 80% of the HHCD data and use the remaining 20% of HHCD data to select earlystopping criteria. 2. Fixed-Interval Timing. This fully autonomous bite timing strategy feeds every 44.5 seconds, which is a scaled average time a robot should take to feed a user after it has picked up a food item.
To derive this value, we first find the appropriate scaling factor between human motion from the HHCD and robot motion. We note the average time for a human from the food entered transition to food lifted transition is 9.9 seconds. The robot end-effector motion is not designed to match the human speed but rather to be perceived as safe and comfortable to a user being fed. We find the equivalent key transitions for the robot to be 5x slower than a human. Since we define the intention to take a bite as when the food is lifted, the robot should take 49.5 seconds to feed a user after picking up a food item. Given the robot takes roughly 5 seconds to move to its wait position after picking the food, the robot waits 44.5 seconds. Further details about this wait-time can be found in App. 8.3.2. 3. Mouth-Open Timing. This partially autonomous bite timing strategy feeds only when the user prompts the robot by opening their mouth. The target user is prompted each time by the robot saying "When ready, look at me and open your mouth". This approach gives the user explicit control of when the robot should feed [40]. The robot user is seated on a wheelchair mounted with a Kinova Gen3 6-DoF arm [62], which is used to feed the participant (Fig. 3, left). For discussion of our bite timing strategies, the use of voice prompting, and implementation details of the robot study, see App. 8.3.2-8.3.4. Experimental Procedure. In this study, participants are seated in a similar setup as that used for HHCD data collection in Sec. 5. All participants were asked to bring their own food, and each group chose who would be fed by the robot. We recruited 30 participants over 10 sessions. There were 16 female and 14 male participants, and their ages ranged from 19-70 (µ = 27, σ = 9) years. A single trial consists of bite acquisition, followed by one of the three bite timing strategies, then bite transfer. For bite acquisition, the robot alternates feeding the user cantaloupes and strawberries. We chose these fruits due to their high acquisition success rates [24]. We used the bite acquisition strategies and bite transfer strategies from [25,26]. All participants take a survey after each trial, which administers a forced-choice question on the participants' preferences between the previous and current conditions. Each pair of comparisons between any two conditions occurs three times, leading to ten trials. The condition orderings are counterbalanced over ten trials. Additionally, we ask participants whether they felt the robot fed them too early, on-time, or too late. The experiment questionnaire after each trial further includes questions about bite timing appropriateness, distractions due to the robot, ability to have natural conversations, ability to feel comfortable around the Top right: Preferences for bite timing strategies rated by users, two co-diners, and all three diners.
Bottom right: Level of distraction by the robot perceived by users, two co-diners, and all three diners on a Likert scale 1-5 (agreement with "I felt distracted by the robot"), for each bite timing strategy. * , * * , * * * denote statistically significant differences with p 0.05 , p 0.005 , p 0.0005 respectively.
robot, as well as system reliability and trust in the robot [64]. For details on user study questionnaires see App. 8.3.6. To avoid interruptions in social conversations due to the presence of a robot in human groups, we provide the participants with a list of questions (see App. 8.3.5), which they could optionally use to help get the conversation started at each trial, similarly to previous work [39]. Results and Discussion. As shown in Fig. 3 (top right), users and co-diners preferred the Learned strategy for bite timing as compared to Fixed-Interval or Mouth-Open Timing. This confirms that our insight to incorporate social signals in model structure (SoNNET) improves bite timing prediction. These results using Couplet-SoNNET also imply that it is possible to predict the behavior of a user using only their co-diner information, which indicates that there is social coordination in human groups even in the presence of a robot. In Fig. 3 (bottom right) we further compared the level of distraction by the robot as perceived by participants. We performed Kruskal-Wallis H-tests and Tukey HSD post-hoc tests and found that Mouth-Open Timing distracts dining participants significantly more than Learned or Fixed-Interval Timing. We believe this is because the Mouth-Open strategy prompts the user using a voice interface, which can disrupt the rhythm of conversation. Even though the participants had a clear preference for the Learned strategy when given a forced-choice, when asked to individually rate the conditions using a 5-point Likert scale, interestingly we could not find any statistically significant differences between Mouth-Open Timing and Learned Timing. This is probably because the Mouth-Open Timing strategy provides full control of bite timing to the users themselves. Note, regardless of the conditions, the users found the system comfortable, reliable, and trustworthy. Detailed analysis is given in App. 8.3.7.
Limitations. There is a risk that our results from human-robot user studies on adults with no mobility limitations may not generalize to those with people with mobility limitations. People with mobility limitations may have different preferences and cognitive workload associated with a robotic intervention. Although our target diner is not a person with such C3-C5 SCI, our model does not use their movements to infer when to feed. As transferability is a function of their behavior, our experiments demonstrate good transferability across scenarios. We expect it to similarly transfer to users with C3-C5 SCI, though it remains to be investigated in future work. We also made multiple assumptions when transferring our results from human-human to human-robot commensality scenarios. During human-human commensality, the user was self-feeding whereas in human-robot commensality the user was being fed. We also assumed that the addition of a robot into a humanhuman commensality scenario does not change the social dynamics of the diners significantly. Given these assumptions, it would be interesting to see how our models perform when trained on similar human-robot commensality scenarios. Finally, it is an open question as to how these models would perform with groups of different cultures. Social science literature on commensality studied the interplay between factors such as culture [65,66], age [67], and social context [

Questionnaires
The questions we asked the participants in the pre-study and post-study questionnaires are shown in Fig. 14 and Fig. 15 respectively.

Data Annotation Details
Using the ELAN annotation tool [57], we annotated each participant's video (excluding the scene videos) based on participant's interactions with food, drink, and napkins. We defined the following annotation types and associated sets of annotation values. The annotation value was assigned based on the type of utensil involved.
• mouth open ∈ {∅}: From the time the mouth opened due to an immediately following food-to-mouth handover until it closed. The frames where the mouth was open for other reasons were ignored. If the mouth was open even when not eating, the mouth open annotation began when the mouth started opening more due to an incoming food item and similarly, the mouth open annotation ended when the mouth closed the most the first time after eating the bite. • food to mouth ∈ {fork, knife, spoon, chopsticks, hand, ∅}: From the time the food item entered the mouth (i.e., got above teeth) until the given utensil/hand first lost contact with the mouth (or started moving away from mouth in case the utensil/hand did not touch the mouth). Subsequent actions (if any) to correct/fix an unsuccessful feeding attempt were ignored unless they involved a proper food item pick up. There was exactly one food to mouth annotation for each mouth open annotation such that the mouth open annotation always started before the food to mouth annotation but they could have ended in any order. If the food was consumed without the use of utensil/hand and the person just moved head towards the table to eat a bite, an empty annotation value was assigned. • food entered ∈ {fork, knife, spoon, chopsticks, hand}: First 400 ms after the person touched/entered the food with a utensil/hand. If there were multiple such events before the next food to mouth annotation (e.g., the person first entered the food, then rested, and later entered the food again), only the first such event was annotated. The reason was to record the first intention to eat. Events when the utensil touched/entered the food just because it was put on top of the food to free up hands were ignored, and the food entered annotation started once they touched/entered the food again. So there was exactly one food entered annotation prior to each food to mouth annotation. However, when the person used two/more kinds of utensils/hands at the same time, the food entered annotation was made for each utensil/hand independently and not each of them was followed by the food to mouth annotation of the same utensil/hand type (e.g., food entered by fork, food entered by knife, food lifted by fork, food delivered to mouth using fork but without knife). Also, when the food was grabbed by hand, there might not have been a food entered annotation prior to each food to mouth annotation (e.g., when the person kept holding their food, such as a sandwich, in their hand between bites). If two/more food entered annotations with different values overlapped, some annotations were shortened below 400 ms, as ELAN does not allow overlapping annotations within one tier. • food lifted ∈ {fork, knife, spoon, chopsticks, hand, ∅}: First 400 ms after the utensil performing the food-to-mouth handover lost contact with the rest of the food or with another utensil/hand involved in food manipulation, whichever occurred later. In case the food was grabbed by hand, the first 400 ms after the food started moving towards the mouth. If there were multiple such events before the food to mouth annotation (e.g., the person first lifted the food item a bit, then returned it back to the rest of the food to dip it in a sauce, and later lifted it again), only the last lift off event was annotated. The reason was to record only such food lift off events that immediately led to feeding. So there was exactly one food lifted annotation prior to each food to mouth annotation. However, when the person used two/more kinds of utensils/hands at the same time, the food lifted annotation was made only for the last lift off before the food to mouth annotation of the same utensil/hand type (e.g., if the food was entered by fork, lifted by fork, handed over to spoon, lifted by spoon, and finally, delivered to mouth using spoon, then the fork lift off was not annotated). If the food was consumed without the use of utensil/hand and the person just moved head towards the table to eat a bite, the annotation was made when the head started moving towards the food item and the empty annotation value was used. When candies/chocolates were consumed, the food lifted annotation was made only after the candy/chocolate was unwrapped.
• drink to mouth ∈ {cup, bottle}: From the time the cup/bottle/straw touched the mouth until it left the mouth. • drink entered ∈ {cup, bottle}: First 400 ms after the person grabbed the drink with their hand. If there were multiple such events before the drink to mouth annotation (e.g., the person first grabbed the drink, then dropped it, and later grabbed the drink again), only the first such event was annotated. The reason was to record the first intention to drink. So there was exactly one drink entered annotation prior to each drink to mouth annotation unless they kept holding the drink between two drink to mouth annotations. Also, if the person used a bottle to pour drink into a cup, the drink entered annotation was made for both: when they grabbed the bottle and when they grabbed the cup. • drink lifted ∈ {cup, bottle}: First 400 ms after the drink lost contact with the table and started moving towards the mouth (or just started moving towards the mouth in case they kept the drink in hand after the last drink to mouth annotation). If there were multiple such events before the drink to mouth annotation (e.g., the person first moved the drink towards the mouth, then stopped a bit, and later completed the move) only the last move towards the mouth was annotated. The reason was to record only such drink lift off events that immediately led to drinking. So there was exactly one drink lifted annotation prior to each drink to mouth annotation. • napkin to mouth ∈ {∅}: From the time the napkin touched the mouth until it left the mouth. • napkin entered ∈ {∅}: First 400 ms after the person grabbed the napkin with their hand.
If there were multiple such events before the napkin to mouth annotation (e.g., the person first grabbed the napkin, then dropped it, and later grabbed the napkin again), only the first such event was annotated. The reason was to record the first intention to use the napkin. So there was exactly one napkin entered annotation prior to each napkin to mouth annotation unless they kept holding the napkin between two napkin to mouth annotations. • napkin lifted ∈ {∅}: First 400 ms after the napkin lost contact with the table and started moving towards the mouth (or just started moving towards the mouth in case they kept the napkin in hand after the last napkin to mouth annotation). If there were multiple such events before the napkin to mouth annotation (e.g., the person first moved the napkin towards the mouth, then stopped a bit, and later completed the move) only the last move towards the mouth was annotated. The reason was to record only such napkin lift off events that immediately led to its use. So there was exactly one napkin lifted annotation prior to each napkin to mouth annotation. • disruption ∈ {light off, participant left}: From the time the recording became disrupted due to the light turning off or due to a participant leaving the room until the normal conditions were restored. We further defined the following additional annotation rules: • When people were just unpacking their food/drink or loading their plates from shared bowls/containers -No food/drink entered and food/drink lifted associated annotations -Reason: we are not researching the preparation phase prior to eating • When people tore their food (e.g., a piece of bread) -No additional food entered annotations when the other hand touches the food -Reason: we consider tearing the food as a part of the food manipulation that follows the most recent food entered annotation and precedes the food lifted annotation • When people licked their empty utensil/fingers/hands or foils (e.g., yogurt lid) -No food/drink entered, food/drink lifted, food/drink to mouth, and mouth open associated annotations -Reason: there is no food/drink consumed • When people smelled their food/drink -No food/drink entered, food/drink lifted, food/drink to mouth, and mouth open associated annotations -Reason: there is no food/drink consumed • When people used a napkin for anything else than cleaning their mouth (e.g., blowing/swiping their nose, cleaning their hands/eyes/utensil/table) -No napkin entered, napkin lifted, napkin to mouth associated annotations, but if the person cleaned their hands and then suddenly decided to clean their mouth, then the napkin lifted annotation was made when the napkin started to move towards mouth and also the napkin to mouth annotation was made. If the initial intention to pick up the napkin seemed to be to eventually clean the mouth, then also the napkin entered annotation was made. -No other annotations (besides the disruption annotation) during the disruption interval. New * entered and * lifted annotations had to be made after the disruption (i.e., any * entered and * lifted annotations from before the disruption occurred were forgotten). -Reason: the data from the disrupted interval are not used and the disruption is considered as a reset • When people used coffee stirrer sticks to put spread/jam on a piece of bread -All the associated events were annotated with the "knife" annotation value • When people drank soup (e.g., from a cup) -The food entered/lifted/to mouth and mouth open annotations were used with the "hand" annotation value. • When people drank from the bottle cap -All the associated events were annotated with the "cup" annotation value • When people grabbed or lifted the food/drink/napkin outside of the camera view -The start of the associated annotation was estimated but the annotation was not skipped • When people picked up and ate small food items such as crumbs -The food entered/lifted/to mouth and mouth open annotations were not skipped • When people ate a sandwich/wrap and decided to pick a small piece with fingers from the rest of the sandwich -The food entered/lifted/to mouth and mouth open annotations were not skipped • When the food entered the mouth but the person did not take a bite -The food entered/lifted/to mouth and mouth open annotations were not skipped • When there was an incomplete (* entered, * lifted, * to mouth) sequence at the beginning or end of the video -For example, the first annotation could be food lifted, mouth open or food to mouth without prior food entered. Similarly, the last annotation could be food entered or food lifted. • When the feeding failed at the mouth (e.g., even if the whole food item falls down during the food-to-mouth handover) -The food entered/lifted/to mouth and mouth open annotations were not skipped    Participant ID Time gaps between annotations. In Tab. 5, we report mean and standard deviation of duration (time gap) between two consequent annotations of both the same annotation type (e.g., from food lifted to food lifted) as well as different annotation type (e.g., from food lifted to food to mouth). We aggregate the times by annotation type and annotation value.
Eating rate during dining. Figure 7 (left) shows the eating rate (number of eating actions per minute) where one eating action corresponds to one food to mouth annotation. Since the number of eating actions might vary based on the total amount of food the diner had (and hence total number of eating actions they made), in Fig. 7 (right) we also normalize the eating rate by the total number of eating actions the diner made. As we can see in both cases the eating rate increases from the start till around the 5th minute of dining time and decreases thereafter. This confirms the eating is a non-stationary activity and needs to be accounted for when designing models of commensality. Food types. The distribution of types of food the participants ate can be found in Fig. 8. Demographic background. 82 participants were right-handed and 8 left-handed. The distribution of participants' race is shown in Fig. 9 (left).
Relationship between diners. The distributions of co-diner relationship types, durations, and frequency of eating together are provided in Fig. 10 (left top-bottom) respectively. Social dining habits. The distributions of participants' typical co-diner type, social dining frequency, and dining location are shown in Fig. 10 (right top-bottom) respectively. Dining experience. Participants' ratings of their overall meal experience, social interactions with other participants, and food are presented in Fig. 9 (right).  Replies to open-ended post-study questions. We also analyze the study participants' answers to open-ended questions in the post-study questionnaire ( Fig. 15 (right)). We observe the following patterns.
* When participants think it is appropriate to take a bite of food when they are eating with others • Talking-related rules: "When I am not speaking", "When listening to others", "When others are talking or if there is a pause in the conversation", "After sharing a long piece of speech and expecting a lot of response", "When someone else is talking and I don't think they're going to ask me anything", "It is appropriate when someone is not talking about a very serious topic you need to give your full attention to." • Eye gaze-related rules: "When the person talking is not making eye contact", ". . . when i'm not making direct eye contact with someone, . . . " • Diner physical state-related rules: "when you are hungry, it should be ok to take a bite of food." • Social interaction-related rules: ". . . when other people are taking a bite too", "It is appropiate when my bite it is at the same time when the others are putting food in their mouth. . . . ", ". . . when two other people are having a subconversation that I am less engaged in" • Time-related rules: "every 10 seconds or so, . . . ", ". . . when a lot of time has passed between your previous bite" • Several participants also replied with "whenever i want" or similarly.  Dining experience rating Figure 9: HHCD: Left: Distribution of participants' race. "Other or two/more races" includes three White-Asians, three White-Hispanics/Latinos, two Latinos, and one Asian-Hispanic. Right: Participants' ratings of their overall meal experience, social interactions with other participants, and food on a Likert scale 1-7 (Strongly disagree -Strongly agree with positive experience).
Note, the replies to the bite timing questions align with choices of modalities and features we use for bite timing prediction.
* What participants liked about the meal experience • Most participants liked food, conversation, and time spent with friends. For example, "It was super interactive and I got to know my friends better" • Research contribution: "Time with friends, spicy foods, contributing to research", "fun experience to help the robots take over the world" • The study environment: "It felt comfortable and natural and the food was yummy.", "Felt like a natural interaction", "I enjoyed the food, being able to soley talk to my friends without distraction", "food was good, after getting used to cameras conversation felt pretty natural" * What participants did not like about the meal experience • 18 participants (20%) replied "nothing" or similarly.
• Complaints about food they brought: "We didn't buy enough food.", "i ate too much and my stomach hurts", "one friend talked too much, it was a bit long, I ordered too much food and did not eat all of it.", "The pizza was slightly cold and i ordered the wrong pizza from domino pizza company." • The study room and the recording setup: "I think I would rather be in a more comfortable chair and have lower lighting", "The room was too quiet for my comfort", "It was in an enclosed room. The physical setting didn't feel natural.", "A little conscious of the camera", "It was a little odd to be monitored the whole time", "I was nervous speaking about somethings because it was recorded", "The camera directly in our faces", "Not much! Distributions of co-diner relationship types, durations, and frequency of eating together. "Other" co-diner relationship type includes two acquaintances and one boyfriend. Right top-bottom: Distributions of participants' typical co-diner type, social dining frequency, and dining location. "Other" typical co-diner type includes partner or n/a for skipped breakfasts. "Other" typical dining locations include dining hall, campus, n/a for skipped breakfast, or a friends' place for dinner.
Cameras in the middle of the table made it slightly more awkward to pass food, I guess.", "Maybe the fact that we were being watched, recorded; felt a like bit performative", "I did not like that I felt that I had to lean backwards to fit in the frame", "We definitely knew and acted like we were being recorded at times", "I think just because we were participating in the study but I didn't feel uncomfortable with the cameras or anything. So, I feel like our dinner was still authentic." • Conversation topics: "Participant 2 was talking too much about politics that were boring." • Dining duration: "The amount of time I spent could have been longer to have more of a conversation" • Eating with others: "I did not choose the food we ordered and I didn't enjoy the food very much, and I get embarrassed eating around others" • Use of mobile devices while eating: "some things that i dont like about the meal is sometimes people tend to still like using their devices, which makes it feel like they dont want to be there eating a meal with you"

Feature Extraction
We utilize several feature extraction techniques to obtain various high-level features that might indicate semantic visual and audio cues. We combine these features from each person and align target user with two co-diners for each sample event.
1. Visual features: Video clips from cameras facing diners explicitly capture dining behaviours and social interactions. We estimate people's body, hand and face skeletons using OpenPose [51] across consecutive frames. Each frame at time t contains body gesture and face representation as a 168-dimensional vector o ∈ R 168 . Gaze plays a crucial role during communications and interactions. It is a predictor of participants' interests in human-robot interactions [70]. We extract participants' gaze and head pose directions using Real-Time Eye Gaze and Blink Estimation in Natural Environments (RT-GENE) [69]. Gaze and head pose direction data points are represented by Euler angles θ and φ, and together form the feature d ∈ R 4 . 2. Audio features: Using ReSpeaker Mic Array v2.0 [56] we extract raw audio (a mixture of three diners' voices) and a sound direction channel from ROS messages. We use ROS messages as they can be naturally transferred to a robot. We align these ROS messages to video frames using nearest neighbor based on the video frame and audio message timestamps. There can be repeated audio frames aligned to video frames due to audio messages varying in speed. For each aligned audio message, we apply voice activity detection using WebRTC VAD [71] and use k-means clustering on the sound direction information to localize speakers in the scene. We combine the directional clusters and detected voice activity to create a binary vector that indicates whether a diner is speaking or not at each video frame.
We also refer to this binary feature as speaking status. 3. Temporal features: Upon analyzing eating rate in HHCD (App. 8.1.5), we notice that the participant's eating rate increases a bit at the beginning and then decreases as the dining comes to the end. Therefore, we believe that explicitly providing the model with time and bite count information can better capture the non-stationary nature of commensality. We thus generate two bite features b ∈ R 2 , which indicate the time since the last bite of food was taken and the number of bites a person has consumed during the eating session.

Design Choices for Couplet-SoNNET
We chose to restrict the features of the user in Couplet-SoNNET due to a distribution shift between human-human commensality and human-robot commensality. We chose to remove most social signals from the user because it would be more generalizable across our target population. Whether a user is talking could be relevant to predicting bite timing. During preliminary testing however, we found that modeling the user's speaking status led to the model never feeding at all if they kept talking. This makes sense as talking is highly correlated with not-feeding in HHCD. Since the user is not self-feeding, they are not incentivized to stop talking. Therefore, we believe some level of coercion is required to ensure the user is fed, which we realize is a common subtle practice when we spoke with the caregivers who feed care recipients. By removing the user's speaking status, we can ensure that feeding does occur.

Implementation Details of the SoNNET
Both Triplet-SoNNET and Couplet-SoNNEt are trained using an Adam optimizer with a learning rate of 0.0001 and a batch size of 128. To prevent overfitting, we early stop if the validation loss does not increase after 10 epochs. The number of filters at each convolutional layer can be seen in Fig. 2. We use batch normalization layers after each convolutional layer. All experiments are performed on a 64-core cluster with five NVIDIA RTX 3090s.

Implementation Details of the Baseline Models
We use the Keras TCN implementation [72] and train the Triplet-TCN and Couplet-TCN using the same hyperparameters as the SoNNET models. We set the filter size to 50, which ensures a similar number of learnable parameters as the SoNNET models. In the case of the Triplet-TCN, we simply concatenate all the features of all three participants, while for the Couplet-TCN, we use features of the co-diners and only the bite features of the User.

Study Design Rationale
We considered various experimental designs for our user study. Our experiment design is a withinsubjects repeated-measures design where the conditions are counterbalanced such that A→B and B→A occur a total of 3 times and there is only 1 bite per condition at one time. This helps mitigate the recency tendency and guarantees that within each session, there is a tie-breaker. Within one session, there are 9 forced-choice comparisons from 10 trials. Across 10 sessions we ensure that each ordered pair occurs an equal number of times. This gives us a total of 30 comparisons, 15 A→B and 15 B→A (and similarly for BC, AC). With these forced-choice questions, this study design is generally better for eliciting preference data [73] and has less recency bias [74] than alternative study designs where conditions are presented repeatedly at a time. For example, an alternative study design could have X bites of condition A, followed by X bites of condition B, and X bites of condition C with a preference question at the end. This alternative study design would exacerbate the recency effect, as the participants would have to remember what they felt several bites ago. We also considered other similar alternative study designs but settled with the one we presented as we believe this design would represent people's preferences in a sample-efficient manner with less recency biases.

Bite Timing Strategy Details
We designed our three bite timing strategies (Learned Timing, Fixed-Interval Timing, and Mouth-Open Timing) based on discussions with care recipients, occupational therapists, and caregivers. While consulting people with mobility limitations, caregivers, and occupational therapists on what features we should look at based on what movements are consistent across people with mobility limitations. Our target users (with C3-C5 SCI) cannot move their arms to feed themselves. Also, there is a huge spectrum of severity of mobility limitations depending on the users' conditions, and their movements are not consistent across these users. Therefore, the Learned Timing strategy uses Couplet-SoNNET which does not use arm gesture features but uses only the features of the co-diners to make it more generalizable across this target population. For the Mouth-Open Timing, caregivers said that they estimate the appropriate time to feed when care recipients open their mouth and provide an explicit cue. This directly inspired the design of our Mouth-Open Timing. As described in App. 8.3.1, our study design mitigates the recency effect and ensures useful comparisons between conditions. Since a new condition is presented after each bite, we use a speaker to prompt the user to open their mouth when they are ready. To decide on the wait time for the Fixed-Interval Timing, we used data from HHCD to find a userinspired wait-time. We found that a human on average takes 1.8s from lifting a food item off a plate / bowl (food lifted) to bringing it to the mouth (food to mouth). The robot's equivalent approach duration is on average around 9 seconds (taking into account the variable motion planning time). Though the robot could mechanically move at a faster speed, we chose the speed that would feel safe and comfortable to a user when they are being approached (fed) by a robot with a fork. We determined this velocity of our robot to be perceived as safe and comfortable based on [40] which explicitly did a study on what approach speeds are preferred by people with mobility limitations. This scaling factor (9s / 1.8s = 5) between robot speeds and human speeds is thus user-inspired. Once we determined this scaling factor, we use this same scaling factor to scale up the bite timing from HHCD (9.9s) to human-robot commensality (9.9s*5 -5s [for robot bite acquisition to bitetiming waiting position] = 44.5s) to make sure that the proportion of time for different phases of feeding (bite acquisition -bite timing -bite transfer) are all proportional and balanced. We would also like to note that although the average time for "food entered → food lifted" was 9.9s in HHCD, the standard deviation was 27.3s. So a wait-time of 44.5s is roughly 1 standard deviation away from the equivalent annotation in the HHCD data.

Experimental Setup Details
The experiment was set up similarly the human-human commensality dataset collection described App. 8.1.2 and is depicted in Fig. 4. For the Learned timing, RT-GENE [69] and OpenPose [51] need to process video streams from all three cameras in real-time, in addition to the robot's planning and perception stack. We thus distribute compute over two machines: a 24-core PC with an NVIDIA RTX 3060 and a 32-core PC with an NVIDIA RTX 3090. We downsample the 30 FPS video streams to 15 FPS to ensure real-time performance. As noted in the formulation of the Fixed-Interval timing strategy (Sec. 7), the robot is 5x slower during feeding as compared to a human. This means there is a distribution shift in the time since the last bite was taken on the robot compared to the training data for Couplet-SoNNET. To mitigate this distribution shift, we scale down the computed time since the last bite during the user study by a factor of 5. The robot used joint space velocity control. The robot's motion was generated from different calls to a library of planners available to our platform: • planToConfiguration(goal config): plans from current configuration to a joint space goal configuration (6 degrees-of-freedom) • planToTaskSpaceRegion(ee goal pose, constraints): plans from current configuration to a task space end-effector (EE) goal pose with some given constraints [1] • planToEEOffset(ee offset): plans such that the end-effector moves in the direction of a certain vector. It is paramount for the our robot platform to ensure safety to the user, so we familiarized participants to the four levels of safety we designed: 1. We placed a conservative collision model around the user's head. The users in our study were familiar with the general workspace of the robot (we moved the robot while they were seated on the chair as a part of the pre-study familiarization procedure). 2. The fork has a Force/Torque sensor attached to it, where if a certain threshold of force is reached (beyond acceptable safety / comfortable force thresholds), the arm stops immediately. 3. We had an observer watch the experiment while the emergency stop was ready to press in the case of unexpected behaviors. Additionally, an experimenter watched the system and was ready to stop it for safety. 4. The compliant robot arm is also set up so that the user can stop it if absolutely necessary.
We also designed the speed of the robot to be at comfortable levels.

Experimental Procedure Details
Each participant was compensated for each hour of their time and for their food expenses. All participants were instructed to bring their own food. The user who was fed by the robot only ate fruits. Each of the other two participants could choose if they also want to eat fruits during the study or the food they brought. Before starting the study, we familiarized the participants with robot-assisted feeding by showing them a trial of the Mouth-Open condition and shortened Fixed-Interval condition, along with explaining safety measures. The procedure than continues as described in Sec. 7.

Conversation Starters
List of questions that the user study participants could optionally use to help get the conversation started at each trial, similar to the past work [39].

Questionnaires
The questions we asked the participants in the pre-study questionnaire included all the questions asked during data collection (Fig. 14) and an additional question about the participant's level of hunger ( Fig. 16 (a)). The questions in the experiment questionnaire we asked after each trial are shown in Fig. 16 (b) and the final post-study questionnaire at the very end of the study is shown in Fig. 16 (c).

Additional Results
Bite timing. Besides the forced-choice assessment of bite timing strategies in terms of bite timing in Sec. 7, we also evaluate absolute ratings of "how timely" each trial was. In fact, one robot user reported that "Slight timing changes seemed more noticeable than I expected." As we can see from Fig. 11 (left), the only statistically significant differences are with respect to Fixed-Interval timing suggesting that the user as well as all three diners found this strategy feeds rather late compared to other strategy/ies. It might be interesting to further evaluate whether diners would prefer Fixed-Interval timing with a higher feeding frequency.
We also investigate whether the robot users' pre-study hunger level affected their bite timing ratings. As shown in Fig. 11 (right), we do not find any statistically significant differences with p 0.05 between the three hunger levels users reported. This could suggest that their bite timing ratings were not biased by their hunger level. However, we cannot draw any strong conclusions as the hunger level self-assessment is a very subjective metric.  Figure 11: HRCom: Left: Bite timing perceived by users, two co-diners, and all three diners on a Likert scale 1-5 (Too early -Too late), for each bite timing strategy (Fixed-Interval, Mouth-Open, and Learned timing). * , * * denote statistically significant differences with p 0.05 , p 0.005 respectively. Right: Effect of robot users' hunger level on their bite timing ratings. Our study did not find any statistically significant differences with p 0.05 .
Other factors. Besides bite timing itself, we evaluate differences between bite timing strategies for other factors: distraction by the robot (already discussed in Sec. 7), ability to have natural conversations ( Fig. 12 (top left)), ability to feel comfortable around the robot (Fig. 12 (top right)), system reliability ( Fig. 12 (bottom left)), system trustworthiness ( Fig. 12 (bottom right)), overall experience of the meal (Fig. 13 (left)), social interactions with other participants (Fig. 13 (right)). We can see that the ability to have natural conversation and feel comfortable around robot is significantly lower for Mouth-Open timing than for Learned or Fixed-Interval timing. This aligns with the finding in Sec. 7 that Mouth-Open timing distracts dining participants significantly more than Learned or Fixed-Interval timing. It is however interesting to note that co-diners, not users of the robot, felt less comfortable around the robot during Mouth-Open Timing. We speculate this is because co-diners perceive the prompt from a voice interface as an external disruption factor not related to their own eating whereas for robot users it is what makes them feed so it does not set robot users into discomfort as much. For users, co-diners as well as all three diners, we do not find any statistically significant differences between bite timing strategies in system reliability, trustworthiness, overall experience nor social interactions they had.
Replies to open-ended post-study questions. We also analyze the study participants' answers to open-ended questions from Fig. 16 (c). We observe the following patterns.

User
Two co-diners All three Perceived by users, two co-diners, and all three diners on a Likert scale 1-5 (Strongly disagree -Strongly agree), for each bite timing strategy (Fixed-Interval, Mouth-Open, and Learned timing). * , * * , * * * denote statistically significant differences with p 0.05 , p 0.005 , p 0.0005 respectively.

Social interactions
Fixed-Interval Mouth-Open Learned Figure 13: HRCom: Left: Overall meal experience Right: Social interactions perceived by users, two co-diners, and all three diners on a Likert scale 1-7 (Strongly disagree -Strongly agree), for each bite timing strategy (Fixed-Interval, Mouth-Open, and Learned timing). Our study did not find any statistically significant differences with p 0.05 . * Whether participants felt safe around the robot • Robot users: -7 users (70%) replied with "Yes, . . . " -Their main concern was safety when robot was moving around with the fork: "Yes for the most part. Sometimes it felt surprising how close it got to my face when it went to pick up food.", "Yes. At first I thought it was going to stab me in the face but it moved slow and never hurt me.", "Sort of. My primary concern was the robot's resting state. When the neutral position has the fork poised at eye level it is very concerning. Simply aiming the fork down and away from the table would make a huge difference." -Many noted that the initial familiarization with the robot helped: "Not very safe at beginning , but with trials go on, I feel more safe. If I have e stop myself I'll feel more safe.", "I was a bit nervous, but after the first few trials I felt more comfortable around the robot", "Yes! Very quickly got used to it's actions, which were very regular so easy to get used to safety-wise" • Other co-diners: -No major concerns: "Yes. It seemed to be under control nicely.", "Yes: it helped that the robot moved pretty slowly and along familiar "tracks" through the air. The e-stop was nice to have too!", "Yes. It avoided my friends and I well. It didnt seem overpowering.", "Moderately. The robot was cutting it close to the person's face while going to grab the food" -Similarly to robot users, the initial familiarization with the robot helped: "Yes, after a few trials I felt safe around the robot. But might be because it's far away from me as well", "I was a little uncomfortable initially but I started feeling safe after a few trials.", "I was little uneasy at first but then I quickly forgot about it and was comfortable", "Yes. I was a little worried at first, but wound up feeling very comfortable around the robot." These replies show that familiarizing users as well as other participants with the robot helps them to feel safer around the robot.
* When participants think it is appropriate to take a bite of food when they are eating with others • Talking-related rules: "Usually when someone else is speaking and you are not expecting a question to be asked to you", "When I am not talking, or being directly talked to" • Eye gaze-related rules: "After a few second pause in speech combined with a stationary eye position.", ". . . if it is a very serious topic, or they are making eye contact, I would probaly wait." • Diner physical state-related rules: "When you are not speaking and have the desire to take a bit." • Social interaction-related rules: ". . . when the present speaker is not saying anything very emotional, energetic, or charged. For example I would not like to take a bite when consoling a crying friend.", ". . . if it is a very serious topic, or they are making eye contact, I would probaly wait." • Time-related rules: "When there is a stop(all people stop talking) longer than 1.5s, I feel it's right time . . . " • Robot-related: "The robot shouldn't wait too long after the food is on the fork.", "Almost always. I would say worst case the biter can wait to make the move towards the robot, but it seems very appropriate for the robot to "always" be feeding and take a bite almost immediately when it's ready." These replies match the same kinds of rules we find in replies to the same question asked during human-human commensality (App. 8.1.5).
* What participants liked about the meal experience • Robot users: -Food and conversation: "Conversation with people, fruit", "It was still easy to have a natural conversation, . . . " -Robot and its behavior: "I liked that the robot did the same thing over and over, making it easy to ignore", ". . . the robot was relatively quiet. It's kind of nice to be fed and I like fruit.", "The food item is placed in a proper position, not too far or close, I have the choice to eat or not.", "The robot was generally out of the way. Once we went through a few trials, the robot was less distracting", "It was very nice not having to think about bite acquisition and delivery", ". . . enjoyed the novelty of the robot" • Other co-diners: -Food and conversation: "My food was great. People were too.", "I was able to have a natural conversation", ". . . The robot was not too intrusive and was almost a cool fourth diner."