There was something like a consensus at the Supreme Court on Monday that it was not a good idea for a judge to hear a death penalty case he had worked on as a district attorney. But the justices seemed divided over how to fashion a constitutional rule that would bar such conduct yet not require judges to recuse themselves whenever they had had a glancing encounter with the matter under review.