The diagram parser first applies Hough Transformation [1]} to extract geometry primitives (points, lines, arcs, and circles), following [2]}. Then the diagram symbols and text regions are extracted through a strong object detector RetinaNet [3]}, and the textual content is further recognized by the optical character recognition tool MathPixhttps://mathpix.com/. After obtaining the primitive set \(P\)  and symbol set \(S\) , we need to ground each symbol with its associated primitives. [2]} adapts a greedy approach where each symbol is assigned to the closest primitive without considering its validity. Instead, we formulate the grounding task as an optimization problem with the constraint of geometry relations:
\(\begin{aligned}& \min \sum _{s} dist(s_i, p_j) \times \mathbb {1}\lbrace  s_i \text{ assigns to } p_j \rbrace  \\& \text{s.t. } (s_i, p_j) \in \text{Feasibility set} ~ F,\end{aligned}\) 
