Annotations for privacy datasets are notoriously difficult to obtain due to the inherent subjectivity in human judgement. For PicAlert, participants were asked to judge whether an image was considered in the private sphere of the photographer (e.g. selfies, family, friends and home) (see Tab. REF ). Possible labels were private, public and undecidable (if no decision could be made on the image label). For VISPR, the annotations are based on 68 attributes compiled from the EU Data Protection Directive 95/46/EC [1]}, the US Privacy Act 1974 [2]}, social network platform rules [3]} and additional attributes after manual inspection of images [4]}. Private images were selected from 32 privacy attributes related to personal life, health, documents, visited locations, Internet conversations, and automobiles.
For PrivacyAlert, the images are annotated through the Mechanical Turkhttps://www.mturk.com/ crowd-sourcing platform, where the annotators were asked to classify the images into 4 classes (clearly private, private, public and clearly public). The quality of the annotations was monitored using an attention checker set that discards annotators who failed to provide the expected response. The four-class annotations are then grouped into binary labels that integrate clearly private with the private and clearly public votes with the public.
PrivacyAlert provides binary labels for each image and VISPR provides privacy attributes that are classified as private or public.
In PicAlert, 17% of the dataset contains multiple ternary annotations for each image where annotators' agreement needs to be computed. Thus, labels can be assigned depending on the desired level of privacy, for instance, labelling an image as private in case of annotation disagreement.
