1. An endoscope system comprising: an endoscope which has an insertion portion and acquires a front field-of-view image and a lateral field-of-view image of an object of observation; a detecting section which has a function capable of detecting whether a treatment instrument is used or not in the endoscope based on a notification signal which is outputted when the treatment instrument is inserted into the endoscope or a function capable of detecting a moving direction of the insertion portion based on an acceleration or a change of a motion amount as information about the moving direction of the insertion portion; and an image processing section which generates a lateral field-of-view image having a circular ring shape to be adjacent to an outer circumference of the front field-of-view image having a round shape as an observation image in the same screen, displays a magnified image of one field-of-view image out of the front field-of-view image and the lateral field-of-view image on a display section based on a detection result of the detecting section, displays, in a case where the one field-of-view image is the front field-of-view image, only a part of the lateral field-of-view image on an insertion direction side of the endoscope which is a part adjacent to the outer circumference of the front field-of-view image, or a compressed image of the lateral field-of-view image, and performs in a case where the one field-of-view image is the lateral field-of-view image, image compression processing of displaying a compressed image of the front field-of-view image on the display section, wherein when the detection result that the moving direction of the endoscope is an extracting direction is obtained by the detecting section, the image processing section performs processing for displaying a magnified image of the lateral field-of-view image on the display section, and instead of the adjacent part display processing and the image compression processing, generates an observation image having an entire field-of-view angle of 230° including a rear field-of-view angle of 50° with respect to an insertion axis direction of the insertion portion.