Advice #4: Enable doorbell batching carefully.
The time of posting each request to the NIC is dominated by Memory-Mapped IO
(MMIO) [1]}. The SoC suffers a high MMIO latency when communicating
with the host (see Figure REF (a)).
A known optimization is doorbell batching (DB) [2]}:
to send a batch of \(N\)  requests, the requester first links them together in memory,
then use one MMIO to ask the NIC to read these requests with DMA in a CPU-bypass way.
DB reduces the number of MMIOs required from \(N\)  to 1.
Thus, for RNIC ➀ and SNIC ➀, DB is always helpful and
can bring 2–30% performance improvement (see Figure REF (b)).
For the communication between host and SoC (SNIC ➂),
DB is still helpful at the SoC-side. As shown in Figure REF (b),
when sending a batch of READs to the host,
DB improves the SoC performance by 2.7–4.6\(\times \)  for batch sizes 16–80.
The huge improvement is partly due to the CPU-bypass feature of DMA,
and also because the NIC is faster in using DMA to read requests stored on SoC memory
(see §REF ).
However, DB is not always helpful at the host-side, as it is slower to
read host memory using NIC DMA (see §REF ).
For batch sizes of 16, 32, and 48, DB decreases the throughput of host-SoC communication
by 9%, 7%, and 6%, respectively.
