1. A method comprising: storing, by a server computer, a plurality of reference audio signal fingerprints, each reference audio signal fingerprint associated with a reference audio signal of a media program; for each user in a plurality of users, receiving, by the server computer, a first audio signal fingerprint from a client device operated by a user, the first audio signal fingerprint associated with a first audio signal comprising ambient sound associated with the user and an audio signal of the media program; searching, by the server computer, the stored plurality of reference audio signal fingerprints to determine one of the stored plurality of reference audio signal fingerprints that matches the first audio signal fingerprint above a given threshold; extracting, by the server computer, an ambient sound signal comprising the ambient sound from the first audio signal fingerprint by obtaining a difference between the one of the stored plurality of reference audio signal fingerprints and the first audio signal fingerprint, the obtaining a difference comprising obtaining a frequency range of the first audio signal; determining, by the server computer, using the ambient sound signal, an emotion of the user for a segment of the media program; and aggregating, by the server computer, the emotion of each user to determine a representative emotion of the plurality of users for the segment of the media program.