Agreement analysis.
We evaluate the quality of our human annotations,
using the Fleiss' kappa (FK) metric [1]}, which is widely used to assess the reliability of agreement between raters when classifying items. Recall that we collect 10 annotations to classify receptacles for each object-room pair into correct, misplaced, or incompatible bins. In Figure REF , we report FK agreement per object across all room-receptacle pairs (\(269 \times 128\) ) after keeping \(8/10\)  annotations with the highest inter-human agreement.
We use the agreement ranges proposed by [2]} to interpret the FK scores. We also show agreement when combining the misplaced and implausible categories.
fig:agreementperobject demonstrates about \(90\%\)  of our collected data has fair to moderate agreement between annotators.
fig:agreementperhighlevel shows the mean agreement for high-level semantic categories. The agreement is higher for sporting, tool, and stationery categories because they go to specific places (office desks, garage, etc). The agreement is low for objects like fruits, medicines, packaged foods because people differ in where they like to keep these objects (packaged food can go in cabinets, shelves, kitchen counters, refrigerators).
Overall, these results indicate that our data defines a high-quality source of ground truth rearrangement preferences agreed upon by the majority of annotators.
