1. A head-mounted display device comprising: an image display unit including an image-light generating unit that generates image light representing an image and emits the image light and a light guide unit that guides the emitted image light to an eye of a user, the image display unit being for causing the user to visually recognize a virtual image; a control unit that is connected to the image display unit and controls image display by the image display unit, the control unit including an operation surface that controls an operation of the head-mounted display device; and a direction determining unit that determines whether the user faces the operation surface of the control unit based on movement of the head-mounted display device on a head of the user wearing the head-mounted display device exceeding a fixed amount, wherein responsive to the movement of the head-mounted display device on the head of the user exceeding a fixed amount, the control unit adjusts luminance of the image-light generating unit or adjusts the image light generated by the image-light generating unit to reduce visibility of the virtual image, and when the direction determining unit determines that the movement of the head-mounted display device on the head of the user exceeds the fixed amount, the control unit adjusts the luminance of the image-light generating unit or adjusts the image light generated by the image-light generating unit to reduce the visibility of the virtual image.