It must, in fact, have been distinctly annoying to the pair of schemers."
There were two schemers, and they must have been annoyed by something.
Suspected criminals were brought here to force them to confess.