1. An endoscope system, comprising: an endoscope which acquires a direct-view field of view image and a side-view field of view image of an object to be observed; a light source apparatus which supplies an illuminating light for illuminating the object to be observed; a detection section which detects a change of a physical quantity with respect to movement of an insertion section provided at the endoscope, which occurs due to an operation of the endoscope; an image processing unit which generates an observed image including the direct-view field of view image and the side-view field of view image in a same screen, and outputs the observed image as a video signal; and a light adjustment area selection section which detects a brightness of the direct-view field of view image and a brightness of the side-view field of view image individually based on the video signal, selects one of field of view images of the direct-view field of view image and the side-view field of view image as a light adjustment target based on a detection result of the detection section, selects a field of view image corresponding to a moving direction in which the change of the physical quantity increases as a light adjustment target, and performs control so as to make a luminance of the field of view image selected as the light adjustment target relatively higher than a luminance of the other field of view image.