I have a physical server running Ubuntu Trusty (14.04) at a data centre where they assigned one IP address, as well as a /28 block (16 IP addresses, 14 usable) from a different subnet. They said that the additional subnet was statically routed via the physical host, which I assume means their network will only accept requests from the host NIC's hardware address. I'm going to use this with virtualbox, so that each VM has their own IP address and works as if they were real physical servers on the net (in other words, no NAT). I further assume that the VMs need to route their traffic via the host in order for their requests to use the host's HW address.I am not an expert on setting up complex networks beyond the basic static IP or NAT, so I've read man pages, Ubuntu documentation online as well as a bunch of questions and answers on different StackExchange sites. From what I gather, I need to set up a virtual bridge between the host IP address and one address in the subnet, the latter acting as the gateway for the virtual machines. The VMs will thus have a simple setup with the gateway being in their subnet.I can access the VM from the host and vice versa, but the VMs can't access the internet beyond the host and I can't access the VMs from the internet. I'm assuming the problem is that I haven't set up the bridge properly to route the requests over it. Any hints to get me moving in the right direction would be highly appreciated.Host's /etc/network/interfacesauto loiface lo inet loopbackauto  br0iface br0 inet static  name      Bridge for VMs to connect via  address   x.x.x.229  netmask   255.255.255.192  gateway   x.x.x.193  # default route to access subnet - this was provided by the data centre:  up route add -net x.x.x.192 netmask 255.255.255.192 gw x.x.x.193 eth0  bridge_ports eth0  bridge_fd 9  bridge_hello 2  bridge_maxage 12  bridge_stp offauto  br0:0iface br0:0 inet static  name      Gateway IP for VMs in subnet y.y.y.240/28  address   y.y.y.241  netmask   255.255.255.240  broadcast y.y.y.255  network   y.y.y.240Host system info:$ route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         x.x.x.193  0.0.0.0         UG    0      0        0 br0x.x.x.192  0.0.0.0         255.255.255.192 U     0      0        0 br0y.y.y.240  0.0.0.0         255.255.255.240 U     0      0        0 br0$ ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000    link/ether c8:60:00:5e:bd:e0 brd ff:ff:ff:ff:ff:ff3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default     link/ether c8:60:00:5e:bd:e0 brd ff:ff:ff:ff:ff:ff    inet x.x.x.229/26 brd x.x.x.255 scope global br0       valid_lft forever preferred_lft forever    inet y.y.y.241/28 brd y.y.y.255 scope global br0:0       valid_lft forever preferred_lft forever$ ifconfigbr0       Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0            inet addr:x.x.x.229  Bcast:x.x.x.255  Mask:255.255.255.192          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:743907 errors:0 dropped:0 overruns:0 frame:0          TX packets:519787 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:92549304 (92.5 MB)  TX bytes:172422977 (172.4 MB)br0:0     Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0            inet addr:y.y.y.241  Bcast:y.y.y.255  Mask:255.255.255.240          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1eth0      Link encap:Ethernet  HWaddr c8:60:00:5e:bd:e0            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:743968 errors:0 dropped:0 overruns:0 frame:0          TX packets:519787 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:102968103 (102.9 MB)  TX bytes:172422977 (172.4 MB)lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host          UP LOOPBACK RUNNING  MTU:65536  Metric:1          RX packets:674890 errors:0 dropped:0 overruns:0 frame:0          TX packets:674890 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:302219962 (302.2 MB)  TX bytes:302219962 (302.2 MB)Virtual Machine's /etc/network/interfacesauto loiface lo inet loopbackauto eth0iface eth0 inet static  address y.y.y.243  netmask 255.255.255.240  gateway y.y.y.241Virtual Machine system info:$ route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         y.y.y.241  0.0.0.0         UG    0      0        0 eth0y.y.y.240  0.0.0.0         255.255.255.240 U     0      0        0 eth0$ ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000    link/ether 08:00:27:f7:36:74 brd ff:ff:ff:ff:ff:ff    inet y.y.y.243/28 brd y.y.y.255 scope global eth0       valid_lft forever preferred_lft forever$ ifconfigeth0      Link encap:Ethernet  HWaddr 08:00:27:f7:36:74            inet addr:y.y.y.243  Bcast:y.y.y.255  Mask:255.255.255.240          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:18112 errors:0 dropped:0 overruns:0 frame:0          TX packets:241 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:1123962 (1.1 MB)  TX bytes:54822 (54.8 KB)lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          UP LOOPBACK RUNNING  MTU:65536  Metric:1          RX packets:444 errors:0 dropped:0 overruns:0 frame:0          TX packets:444 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:35428 (35.4 KB)  TX bytes:35428 (35.4 KB)$ ping -c 4 -W 5 x.x.x.229PING x.x.x.229 (x.x.x.229) 56(84) bytes of data.64 bytes from x.x.x.229: icmp_seq=1 ttl=64 time=0.117 ms64 bytes from x.x.x.229: icmp_seq=2 ttl=64 time=0.106 ms64 bytes from x.x.x.229: icmp_seq=3 ttl=64 time=0.102 ms64 bytes from x.x.x.229: icmp_seq=4 ttl=64 time=0.103 ms--- x.x.x.229 ping statistics ---4 packets transmitted, 4 received, 0% packet loss, time 3000msrtt min/avg/max/mdev = 0.102/0.107/0.117/0.006 ms$ ping -c 4 -W 5 x.x.x.193PING x.x.x.193 (x.x.x.193) 56(84) bytes of data.--- x.x.x.193 ping statistics ---4 packets transmitted, 0 received, 100% packet loss, time 3007ms$ ping -c 4 -W 5 8.8.8.8PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.--- 8.8.8.8 ping statistics ---4 packets transmitted, 0 received, 100% packet loss, time 3023msIn Virtualbox, I've set up the VM's virtual network interface as a Bridged Adapter connected to br0. I also tried connecting it to eth0, but it made no discernible difference.UPDATE:Following advice in the comment, I've enabled IP forwarding:$ sysctl net.ipv4.ip_forwardnet.ipv4.ip_forward = 0$ sudo sysctl -w net.ipv4.ip_forward=1net.ipv4.ip_forward = 1I've also enabled promiscuous mode in virtualbox network settings, although I don't think this made any difference.I've noticed something interesting. When I ping something on the net, such as Google's name servers, I get a reply on the first ping, but not on the subsequent ones:$ ping 8.8.8.8PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.From y.y.y.241: icmp_seq=1 Redirect Host(New nexthop: y.y.y.y.241)64 bytes from y.y.y.241: icmp_seq=1 ttl=54 time=5.97 msFrom y.y.y.241 icmp_seq=2 Destination Host UnreachableFrom y.y.y.241 icmp_seq=3 Destination Host UnreachableFrom y.y.y.241 icmp_seq=4 Destination Host UnreachableFrom y.y.y.241 icmp_seq=5 Destination Host UnreachableFrom y.y.y.241 icmp_seq=6 Destination Host UnreachableFrom y.y.y.241 icmp_seq=7 Destination Host Unreachable^C--- 8.8.8.8 ping statistics ---9 packets transmitted, 1 received, +6 errors, 88% packet loss, time 8041msrtt min/avg/max/mdev = 5.972/5.972/5.972/0.000 ms, pipe 3When IP forwarding is off, all I get is:$ ping 8.8.8.8PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.^C--- 8.8.8.8 ping statistics ---11 packets transmitted, 0 received, 100% packet loss, time 10000msThis is what I get if I sniff the IP packets on the host when pinging 8.8.4.4 on the VM for the first time since rebooting the VM. Subsequent pings to the same IP address only gives the repeated lines on the bottom (not the first two).$ sudo tcpdump -q -c 10 -e -n host y.y.y.243tcpdump: WARNING: eth0: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes12:32:57.199202 c8:60:00:5e:bd:e0 > cc:e1:7f:07:e0:af, IPv4, length 98: y.y.y.243 > 8.8.4.4: ICMP echo request, id 1060, seq 1, length 6412:32:57.204498 cc:e1:7f:07:e0:af > c8:60:00:5e:bd:e0, IPv4, length 98: 8.8.4.4 > y.y.y.243: ICMP echo reply, id 1060, seq 1, length 6412:32:58.202555 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:32:59.201628 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:00.201652 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:01.201759 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:02.201502 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:03.201394 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:04.201706 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4612:33:05.201647 08:00:27:f7:36:74 > ff:ff:ff:ff:ff:ff, ARP, length 60: Request who-has x.x.x.193 tell y.y.y.243, length 4610 packets captured11 packets received by filter0 packets dropped by kernelWhat could possibly cause such an odd result? What else could I test that could get me closer to a solution here?UPDATE 2:I am officially baffled over this one now. I ssh'd the host's IP address within the additional subnet (y.y.y.241) from the net, just to make sure that routing to that subnet works, and it did. I then ssh'd y.y.y.243 and expected to get nothing, but to my surprise I got in! Suddenly everything worked as it should... I could ping 8.8.8.8 and anything else and ssh to any internet host. And this is after doing nothing to either host or VM.So of course I did the right thing - I broke it in the name of science. I had to find out what made it work! So, on the host:sudo sysctl -w net.ipv4.ip_forward=0Obviously I enabled it again:sudo sysctl -w net.ipv4.ip_forward=1And as expected, back to the situation above. Can't contact the VM other than from the host. On the VM, ping to any internet host has a first reply, then nothing.Now the really puzzling thing is, I then just left it for a few hours, and when I came back - and I stress that I did nothing to either host or VM - it magically worked. I do not appreciate magic. I'm a man of science and I like things to either work, or not to work, preferably the former. Do anyone have any idea what could cause such a strange delay?
Can't connect to Virtualbox VM's static IP which is in a different subnet from the hosts's IP, but statically routed to the host HW addr
ubuntu;networking;virtualbox;network interface;bridge