1. A method for determining a semantic concept classification for a digital video clip including a temporal sequence of video frames and a corresponding audio soundtrack, comprising: determining, by a processing device, reference video codeword similarity scores for each reference video clip in a set of reference video clips, wherein the determining reference video codeword similarity scores comprises: analyzing a temporal sequence of video frames for a particular reference video clip to determine a set of reference video visual features; analyzing an audio soundtrack for the particular reference video clip to determine a set of reference video audio features; comparing the set of reference video visual features to distinct visual background codewords and distinct visual foreground codewords from audio-visual grouplets of an audio-visual dictionary, wherein the audio-visual grouplets include distinct visual background codewords representing visual background content, distinct visual foreground codewords representing visual foreground content, distinct audio background codewords representing audio background content, and distinct audio foreground codewords representing audio foreground content; and comparing the set of reference video audio features to the distinct audio background codewords and the distinct audio foreground codewords from the audio-visual grouplets of the audio-visual dictionary; determining, by the prosessing device, codeword similarity scores for the digital video clip, wherein the determining codeword similarity scores comprises: analyzing the temporal sequence of video frames in the digital video clip to determine a set of visual features; analyzing the audio soundtrack in the digital video clip to determine a set of audio features; comparing the set of visual features to the distinct visual background codewords and the distinct visual foreground codewords; and comparing the set of audio features to the distinct audio background codewords and the distinct audio foreground codewords; determining, by the prosessing device, a reference video similarity score for each reference video clip representing a similarity between the digital video clip and the respective reference video clip responsive to the audio-visual grouplets, the codeword similarity scores, and the reference video codeword similarity scores; determining, by the prosessing device, a concept classification using trained semantic classifiers responsive to the determined reference video similarity scores; storing, by the prosessing device, an indication of the concept classification in a processor-accessible memory; wherein the distinct visual background codewords and the distinct foreground codewords are separate and distinct from each other, wherein the distinct audio background codewords and the distinct audio foreground codewords are separate and distinct from each other, and wherein the distinct visual background codewords and the distinct visual foreground codewords are separate and distinct from the distinct audio background codewords and the distinct audio foreground codewords.