Throughput.
For READ and WRITE, due to less cross-PCIe penalties,
SNIC ➁ has better throughput than SNIC ➀, reaching 1.08–1.48\(\times \) 
for payloads less than 512 bytes. Interestingly, for READ, SNIC ➁ is even
observably higher than RNIC ➀ before saturating the network bandwidth.
For this undocumented results, we suspect that it is due to the closer packaging
of SoC memory and the PCIe switch.
Note that a confident analysis relies on the hardware details of Bluefield, which
unfortunately are not available to us.
For WRITE, SNIC ➁ is still lower than that of RNIC ➀.
We attribute this to two reasons.
First, SoC has fewer DRAM channels compared to the host (1 vs. 8), limiting
the concurrency of write accesses. However, READ is not affected because read accesses on DRAM are faster than write accesses [1]}, [2]}.
Second, SoC can only utilize a portion of NIC cores.
Finally, for SEND/RECV, the throughput drops by up to 64%, again due to
the wimpy SoC cores.
