1. An electron beam system inspection or measurement apparatus comprising a detection apparatus including an electron optical system comprising an electron beam source, a deflection element for deflecting electron beams emitted from the electron beam source, and an objective lens for converging and irradiating electron beams deflected by deflection element on an inspected object, an electron beam image detection optical system for detecting a secondary electron beam image generated from the inspected object by the electron beams deflected by the electron optical system and converged and irradiated, a projection optical system for projecting a luminous flux of a repetitive light pattern on the inspected object from an oblique upper direction of the inspected object and a detection optical system for detecting the position of an optical image by focusing the luminous flux of the repetitive light pattern which was reflected on a surface of the inspected object by the luminous flux of the repetitive light pattern projected by the projection optical system, an optical height detection apparatus arranged so as to optically detect a height of the surface in an area on the inspected object based on the change of the position of an optical image composed of the luminous flux of the repetitive light pattern detected by the detection optical system, a focus controller for focusing electron beams on the inspected object in a properly-focused state by controlling a current flowed to or a voltage applied to the objective lens of the electron optical system on the basis of the height of the surface on the inspected object detected by the optical height detection apparatus, and an image processor for inspecting or measuring a pattern formed on the inspected object on the basis of the secondary electron beam image detected by the electron beam image detection optical system.