1. An environment recognition device comprising: a luminance obtaining unit that obtains a luminance of each block formed by dividing a detection area in a luminance image; an edge deriving unit that derives an edge direction based on a direction in which an edge depending on the luminance of each block extends; a trajectory generating unit that associate the blocks based on the edge direction so as to generate an edge trajectory; a grouping unit that groups regions enclosed by the edge trajectories as a target object; and a specific object determining unit that determines the target object as a specific object, wherein the edge deriving unit derives the edge direction based on a difference between totals of the luminances in two left and right regions formed by dividing the block in a vertical direction, and a difference between totals of the luminances in two upper and lower regions formed by dividing the block in a horizontal direction, and when the absolute value of the difference between the totals of a direction is not more than a predetermined threshold value, the edge deriving unit derives the edge direction without using the difference between the totals of the direction.