I have a conundrum and would like to hear any pointers (been working    on this for 2 days now, and need to get this server in to production).      My google-foo has not been strong with this one.I have a multi-homed Linux Server - Centos 6.xI removed the LACP / bonding on the dual 10 Gig ports on this server.Despite trying to set up separate ip route's for each NIC, everyhing   is going out via NIC em3 (the default route for the system).  What   I'd like to have this system behaving like is for:   - em3 to only handle external (internet) traffic.   - em2 to only handle 10 Gig traffic   - em1 to handle everything else / internal traffic.The next hop on the networks are:  - em1 & em2 go to a good L2/L3, 10 Gig switch   - em3 goes to a decent L2/L3, 1 Gig switchNOTE: IP addresses have been munged a bit.em1 is an internal, 1 Gig network that has rights to see everything     on our internal network as well as the ability to get out to the    open internet via our default firewall / router.em1 is  10.10.18.21/16    gw is to be 10.10.5.1em2 is an internal, 10 Gig network that is only to see other    10 Gig devices on that network segment.em2 is  10.16.64.21/26    gw is to be 10.16.64.21em3 is an external network that allow remote connections to     authorized users.em3 is  10.48.61.80/24    gw is to be 10.48.61.1I have added entries in the /etc/iproute/rt_tables file as follows:## reserved values#255 local254 main253 default0   unspec## local##1  inr.ruhep10 em1table20 em2table30 em3tableI have also created routes and rules files.-rw-r--r--  1 root root    98 Feb 15 08:54 route-em1-rw-r--r--  1 root root    87 Feb 16 12:31 route-em2-rw-r--r--  1 root root   102 Feb 16 10:37 route-em3-rw-r--r--  1 root root    65 Feb 15 08:11 rule-em1-rw-r--r--  1 root root    68 Feb 16 12:32 rule-em2-rw-r--r--  1 root root    68 Feb 16 10:37 rule-em3[/etc/sysconfig/network-scripts]$ cat route-em110.10.18.0 dev em1 src 10.10.18.21 table em1tabledefault via 10.10.5.1 dev em1 table em1table[/etc/sysconfig/network-scripts]$ cat rule-em1from 10.10.18.21/16 table em1tableto 10.10.5.1 table em1table[/etc/sysconfig/network-scripts]$ cat route-em210.16.64.0/28 dev em2 table em2tabledefault via 10.16.64.254 dev em2 table em2table[/etc/sysconfig/network-scripts]$ cat rule-em2from 10.16.64.21/28 table em2tableto 10.16.64.254 table em2table67.134[/etc/sysconfig/network-scripts]$ cat route-em310.48.161.0 dev em3 src 10.48.161.82 table em3tabledefault via 10.48.161.1 dev em3 table em3table[/etc/sysconfig/network-scripts]$ cat rule-em3from 10.48.161.82/24 table em3tableto 10.48.161.1 table em3table[/etc/sysconfig/network-scripts]$ ip route show table em1table10.10.18.0 dev em1  scope link  src 10.10.18.21 default via 10.10.5.1 dev em1 [/etc/sysconfig/network-scripts]$ ip route show table em2table10.16.64.0/28 dev em2  scope link default via 10.16.64.254 dev em2 [/etc/sysconfig/network-scripts]$ ip route show table em3table10.48.161.0 dev em3  scope link  src 10.48.161.82 default via 10.48.161.1 dev em3 Here is some additional information:[/etc/sysconfig/network-scripts]$ route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface10.48.161.0     0.0.0.0         255.255.255.0   U     0      0        0 em310.16.64.0      0.0.0.0         255.255.192.0   U     0      0        0 em210.10.0.0       0.0.0.0         255.255.0.0     U     0      0        0 em1169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 em1169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 em2169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 em30.0.0.0         10.48.161.1     0.0.0.0         UG    0      0        0 em3[/etc/sysconfig/network-scripts]$ [/etc/sysconfig/network-scripts]$ ip route show10.48.161.0/24 dev em3  proto kernel  scope link  src 10.48.161.82 10.16.64.0/18 dev em2  proto kernel  scope link  src 10.16.64.21 10.10.0.0/16 dev em1  proto kernel  scope link  src 10.10.18.21 169.254.0.0/16 dev em1  scope link  metric 1002 169.254.0.0/16 dev em2  scope link  metric 1003 169.254.0.0/16 dev em3  scope link  metric 1004 default via 10.48.161.1 dev em3 [/etc/sysconfig/network-scripts]$ [/etc/sysconfig/network-scripts]$ ip route show to match 10.16.64.0/2810.16.64.0/18 dev em2  proto kernel  scope link  src 10.16.64.21 default via 67.134.161.1 dev em3 [/etc/sysconfig/network-scripts]$ ip route show to match 10.10.18.0/1610.10.0.0/16 dev em1  proto kernel  scope link  src 10.10.18.21 default via 67.134.161.1 dev em3 [/etc/sysconfig/network-scripts]$ ip route show to match 67.134.161.0/2410.48.161.0/24 dev em3  proto kernel  scope link  src 10.48.161.82 default via 10.48.161.1 dev em3 [/etc/sysconfig/network-scripts]$ cat ifcfg-em1DEVICE=em1#MASTER=bond0#SLAVE=yesHWADDR=c8:1f:66:f4:ce:10TYPE=EthernetUUID=bfa14e4a-66b0-4b83-93a4-094f9090aea7ONBOOT=yesNM_CONTROLLED=noBOOTPROTO=noneIPADDR=10.10.18.21PREFIX=16#GATEWAY=10.10.5.1DNS1=10.10.5.8DNS2=10.10.5.9DOMAIN=AXSDEFROUTE=YESIP4_FAILURE_FATAL=yesIP6INIT=noNAME=em1NETMASK=255.255.0.0IPV6INIT=noUSERCTL=no[/etc/sysconfig/network-scripts]$ cat ifcfg-em2DEVICE=em2#MASTER=bond0#SLAVE=yesHWADDR=c8:1f:66:f4:ce:12TYPE=EthernetUUID=c8c5e1fb-ba40-4537-89ad-f7df5de59f8bONBOOT=yesNM_CONTROLLED=noBOOTPROTO=noneIPADDR=10.16.64.21PREFIX=18#GATEWAY=10.16.64.254DNS1=10.10.5.8DNS2=10.10.5.9DOMAIN=AXS#DEFROUTE=YESIP4_FAILURE_FATAL=yesIP6INIT=noNAME=em2#IPADDR=10.16.64.21NETMASK=255.255.192.0IPV6INIT=noUSERCTL=no[/etc/sysconfig/network-scripts]$ cat ifcfg-em3DEVICE=em3HWADDR=C8:1F:66:F4:CE:14TYPE=EthernetUUID=aa9552be-0075-46b2-8eff-b7c49c8c999fONBOOT=yesNM_CONTROLLED=noBOOTPROTO=noneIPADDR=10.48.161.82PREFIX=24GATEWAY=10.48.161.1DNS1=10.10.5.8DNS2=10.10.5.9DNS3=205.171.3.65DNS4=8.8.8.8DOMAIN=axs.tvDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noAnd, when I try to ping on each interface:[/etc/sysconfig/network-scripts]$ ping -I em1 10.10.5.1PING 10.10.5.1 (10.10.5.1) from 10.10.18.21 em1: 56(84) bytes of data.--- 10.10.5.1 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 2846msrtt min/avg/max/mdev = 0.081/0.101/0.135/0.024 ms[/etc/sysconfig/network-scripts]$ ping -I em2 10.16.64.154PING 10.16.64.154 (10.16.64.154) from 10.16.64.21 em2: 56(84) bytes of data.--- 10.16.64.154 ping statistics ---5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4022mspipe 3[/etc/sysconfig/network-scripts]$ ping -I em3 www.google.com--- www.google.com ping statistics ---4 packets transmitted, 4 received, 0% packet loss, time 3366msrtt min/avg/max/mdev = 2.657/2.679/2.711/0.041 ms
Centos 6.x multi-homed routing question
linux;centos;ip;routing;route