Another common mitigation strategy is to split the training over
multiple computational nodes [1]}. However, this
incurs significant message passing overheads and costs for hardware
with low-latency interconnects. This strategy can also be wasteful if
the peak memory consumption is only slightly larger than that of a
single compute node.
