1. An environment recognition device comprising: a data retaining unit that retains association between a range of luminance and a specific object; a luminance obtaining unit that obtains a luminance of a target portion in a detection area of a luminance image; a specific object provisional determining unit that provisionally determines the specific object corresponding to the target portion from the luminance based on the association retained in the data retaining unit; a grouping unit that groups target portions, of which positions differences in the width and height directions are within a first predetermined range and which are provisionally determined to correspond to a same specific object, into a target object; a position information obtaining unit that obtains a relative distance of the target portion; a target object group specifying unit that sequentially detects a target object, of which position differences from a position of one of the other target objects in the width, height, and depth directions fall within a second predetermined range and which is provisionally determined to correspond to a same specific object as the one target object, thereby specifying a target object group; and a specific object determining unit that determines whether or not the target object group is the specific object according to the number of the target objects in the target object group.