I have a multipath device I'm interested in:[root@xxx dm-7]# multipath -ll mpathfmpathf (3600601609f013300227e5b5b3790e411) dm-7 DGC,VRAIDsize=3.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw|-+- policy='round-robin 0' prio=130 status=active| |- 7:0:1:5 sdl 8:176  active ready running| `- 8:0:1:5 sdx 65:112 active ready running`-+- policy='round-robin 0' prio=10 status=enabled  |- 7:0:0:5 sdf 8:80   active ready running  `- 8:0:0:5 sdr 65:16  active ready runningSo it looks like the block devices backing this path are sdf, sdr, sdl, and sdx. Just taking sdf as an example I've set it's I/O scheduler as being noop:[root@xxx dm-7]# cat /sys/block/sdf/queue/scheduler[noop] anticipatory deadline cfqThe mpathf device maps to /dev/dm-7 for the actual block device. I've just noticed that this has an I/O scheduler as well:[root@xxx dm-7]# cat /sys/block/dm-7/queue/schedulernoop anticipatory deadline [cfq]Question: which one takes precedence? The scheduler on the multipath device or on the device it ends up relaying the I/O through?I'm of course assuming that IOPs aren't scheduled twice (once for the mpath device and another for the individual block device the I/O is redirected into).
Does dm-multipath schedule I/O?
rhel;performance;io;tuning;multipath storage
Short Answer:Device mapper in kernel versions after 2.6.31 (released September 9th 2009) includes support for request-based dm targets. So far only the only request-based dm target is dm-multipath. For the targets that remain BIO (i.e everything except multipath) scheduler selection is still present but irrelevant as the DM target hands off the IOP prior to that point. For request-based targets the scheduler selection supersedes what is set on the individual block device as multipathd will be communicating the requests directly to the underlying SCSI device (/dev/sg4, /dev/sg5, etc).Additional information:User application I/O is referred to as BIO (block I/O). BIO is sent to the scheduler/elevator for request merging/ordering and then is sent as a request to the lower level devices. Historically, dm-multipath has been solely at the BIO level. This created a problem where traffic from separate BIO's would be merged by the block device (sdb, sdf, etc) resulting in some request queues being shorter/less used than other possible paths. BIO dm-multipath was also unable to have visibility on things like retry events or the like, as it was hidden by the block device (/dev/sda, /dev/sdb, etc).The sysfs object for multipath block device prior to the change (RHEL 5):[root@xxxsat01 dm-1]# cat /etc/redhat-releaseRed Hat Enterprise Linux Server release 5.10 (Tikanga)[root@xxxsat01 ~]# uname -r2.6.18-371.8.1.el5[root@xxxsat01 dm-1]# cat dev253:1[root@xxxsat01 dm-1]# lltotal 0-r--r--r-- 1 root root 4096 Jan 29 13:54 devdrwxr-xr-x 2 root root    0 Apr 29  2014 holders-r--r--r-- 1 root root 4096 Jan 29 13:54 range-r--r--r-- 1 root root 4096 Jan 29 13:54 removable-r--r--r-- 1 root root 4096 Jan 29 13:54 sizedrwxr-xr-x 2 root root    0 Jan 25 06:25 slaves-r--r--r-- 1 root root 4096 Jan 29 13:54 statlrwxrwxrwx 1 root root    0 Jan 29 13:54 subsystem -> ../../block--w------- 1 root root 4096 Jan 29 13:54 ueventPost-Change (RHEL 6):[root@xxxlin01 dm-1]# cat /etc/redhat-releaseRed Hat Enterprise Linux Server release 6.5 (Santiago)[root@xxxlin01 ~]# uname -r2.6.32-431.3.1.el6.x86_64[root@xxxlin01 dm-1]# cat dev253:1[root@xxxlin01 dm-1]# lltotal 0-r--r--r-- 1 root root 4096 Jan 29 13:58 alignment_offsetlrwxrwxrwx 1 root root    0 Jan 29 13:58 bdi -> ../../bdi/253:1-r--r--r-- 1 root root 4096 Jan 29 13:58 capability-r--r--r-- 1 root root 4096 Jan 29 13:58 dev-r--r--r-- 1 root root 4096 Jan 29 13:58 discard_alignmentdrwxr-xr-x 2 root root    0 Jan 29 13:58 dm-r--r--r-- 1 root root 4096 Jan 29 13:58 ext_rangedrwxr-xr-x 2 root root    0 Jan 29 13:58 holders-r--r--r-- 1 root root 4096 Jan 29 13:58 inflightdrwxr-xr-x 2 root root    0 Jan 29 13:58 powerdrwxr-xr-x 2 root root    0 Jan 29 13:58 queue-r--r--r-- 1 root root 4096 Jan 29 13:58 range-r--r--r-- 1 root root 4096 Jan 29 13:58 removable-r--r--r-- 1 root root 4096 Jan 29 13:58 ro-r--r--r-- 1 root root 4096 Jan 29 13:58 sizedrwxr-xr-x 2 root root    0 Jan 29 13:58 slaves-r--r--r-- 1 root root 4096 Jan 29 13:58 statlrwxrwxrwx 1 root root    0 Jan 29 13:58 subsystem -> ../../../../class/blockdrwxr-xr-x 2 root root    0 Jan 29 13:58 trace-rw-r--r-- 1 root root 4096 Jan 29 13:58 ueventSince the kernel is unaware of what individual targets do it presents the same sysfs attributes regardless of what type of device mapper device it is. Since the request is then relayed to the block-level devices, the device mapper's scheduler is never invoked and so this setting is essentially a noop with other dm targets.Further Reading:Why /sys/block/dm-0/queue/scheduler exist on my Linux system?Ottawa Linux Symposium 2007 Presentation on Request-based Device Mapper