1. A method comprising using at least one hardware processor for: applying an edge detection algorithm to an image of a document, to receive a map of edges from which multiple optional contours of the document in the image are identified; dividing the image of the document into patches, wherein one or more of the patches are center patches located at the center of the image of the document; calculating similarity between one or more center patches and the remainder of the patches; removing one or more patches from the image of the document which their similarity to the one or more center patches is below a predefined threshold to form a reduced image of the document; dilating the reduced image of the document to form a masked image of the document; comparing the map of the edges to the masked image of the document to verify the identified multiple optional contours; splitting the multiple optional contours into line segments; sorting the line segments into equivalence classes of collinearity representing distinct line segments of the line segments, wherein each one of the classes of collinearity represents a distinct line segment of the distinct line segments; deriving a connectivity graph based on the equivalence classes of collinearity; identifying four-vertex polygons in said connectivity graph; evaluating each one of the identified four-vertex polygons according to one or more segmentation criterions; and segmenting the document in the image according to the most highly evaluated four-vertex polygon of the four-vertex polygons.