1. An environment recognition device comprising: a data retaining unit that retains a range of luminance associated with a specific object; a luminance obtaining unit that obtains a luminance of a target portion in a detection area of a luminance image; a specific object provisional determining unit that provisionally determines that the target portion corresponds to the specific object when the luminance of the target portion falls within the range of luminance associated with the specific object, retained in the data retaining unit; a grouping unit that groups target portions into a first target object, each of the target portions to be grouped having positions differences in width and height directions within a first range, and being provisionally determined to correspond to the same specific object; a position information obtaining unit that obtains a relative distance in a depth direction to the first target object from a vehicle having the environment recognition device; a target object group specifying unit that sequentially detects at least one second target object, the second target object having position differences from the first target object in the width, height, and relative distance, which fall within a second range wider than the first range, and being provisionally determined to correspond to the same specific object as the first target object, thereby specifying the first target object and the second target object as a target object group; and a specific object determining unit that determines that the target object group is not the specific object, but another specific object when the number of target objects including the first and second target objects included in the target object group exceeds a threshold value.