Experts in various fields routinely perform methodical writing tasks
to plan, organize, and report their work. From a clinician writing a
differential diagnosis for a patient, to a teacher writing a lesson
plan for students, these tasks are pervasive, requiring tomethodicallygenerate structured long-form output for a given
input. We develop a typology of methodical tasks structured in the
form of a task objective, procedure, input, and output, and
introduce DoLoMiTes, a novel benchmark with specifications for
519 such tasks elicited from hundreds of experts from across
25 fields. Our benchmark further contains specific instantiations of
methodical tasks with concrete input and output examples (1,857 in
total)  which we obtain by collecting expert revisions of up to 10
model-generated examples of each task. We use these examples to
evaluate contemporary language models highlighting that automating
methodical tasks is a challenging long-form generation problem, as
it requires performing complex inferences, while drawing upon the
given context as well as domain knowledge.
Our dataset is available athttps://dolomites-benchmark.github.io/.