1. An auto focus system comprising: a reference pattern image acquisition range setting device which sets a reference pattern image acquisition range for acquiring a reference pattern image showing an image of a target subject to be followed in a range of a photographing image to be taken by a camera; a reference pattern image storing device which acquires an image in the reference pattern image acquisition range set by the reference pattern image acquisition range setting device and stores the acquired image as the reference pattern image; a reference pattern image position detecting device which detects a position of an image matching with the reference pattern image stored by the reference pattern image storing device as against the photographing images sequentially taken by the camera; an auto focus device which changes a focusing target position based on the position detected by the reference pattern image position detecting device to have the position of the target subject followed by the focusing target position within a photographing range of the camera, and adjusts a focus of an optical system of the camera to bring the target subject into focus; a reference pattern image acquisition range position changing device which changes the position of the reference pattern image acquisition range based on the position detected by the reference pattern image position detecting device to have the position of the target subject followed by the position of the reference pattern image acquisition range set by the reference pattern image acquisition range setting device; and a reference pattern image acquisition range size changing device which changes a size of the reference pattern image acquisition range set by the reference pattern image acquisition range setting device according to the size of the image of the target subject in the photographing image of the camera; wherein the reference pattern image acquisition range size changing device changes the size of the reference pattern image acquisition range only in the case where the size of the image of the target subject in the photographing image of the camera becomes smaller than an initial size.