1. An endoscope system, comprising: an endoscope which acquires a direct-view field of view image and a side-view field of view image of an object to be observed in a body cavity; a light source apparatus which supplies an illuminating light for illuminating the object to be observed; a sensor section which detects a change of a physical quantity indicative of a moving direction of an insertion direction or an extraction direction of an insertion section with respect to the object to be observed in the body cavity, the insertion section being provided at the endoscope, and the change of the physical quantity occurring due to an operation of the endoscope; an image processing unit which generates an observed image including the direct-view field of view image and the side-view field of view image in a same screen, and outputs the observed image as a video signal; a detection section which detects a brightness of the direct-view field of view image and a brightness of the side-view field of view image individually based on the video signal; an insertion/extraction detection section which detects whether the moving direction of the insertion section is the insertion direction or the extraction direction based on a detection result of the sensor section; a light adjustment area selection section which selects the direct-view field of view image as a light adjustment target when the moving direction of the insertion section is the insertion direction, and selects the side-view field of view image as the light adjustment target when the moving direction of the insertion section is the extraction direction based on a detection result of the insertion/extraction detection section; and a light quantity adjusting section which performs control so as to make a luminance of the field of view image selected as the light adjustment target relatively higher than a luminance of the other field of view image.