Please refer to the illustration.I have two machines on a LAN, only one of them has an Internet connection (through a different interface).Server 1 has a garbage hardware clock.Server 2 has a good hardware RTC. Unfortunately, the Server 1 usually has an Internet connection, while the one with the good hardware RTC does not, so depending on the Internet connection status, Server 1 is either the most accurate, or highly unreliable.It is critical both machines are kept in close synchronization with each other.Is it possible to have the system automatically fail from Scenario 1 to Scenario 2, perhaps through automated adjustments to each machine's stratum? 
Changing NTP server based on available connections
debian;time;ntp
First, you are highly unlikely to be allowed to access a Stratum 1 server, unless you're known to the Stratum 1 server administrators and they gave you explicit permission to do so. Having said that, what that server's stratum is doesn't matter for this answer, so I'll refer to that machine as the low-stratum server for clarity.You should configure server2 to get its time from the low-stratum server, not from server1. This will require that it is able (at least for NTP) to access the Internet link on server1. If server2 does not have a link to the Internet with which it would be able to access the low-stratum server, then you can allow it to do that by running the following commands on server1 (as root):iptables -A FORWARD -s server2 -p udp --dport 123 -j ACCEPTiptables -A FORWARD -s server2 -p tcp --dport 123 -j ACCEPTiptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPTiptables -t nat -A POSTROUTING -o eth0 -j MASQUERADEsysctl -w net.ipv4.ip_forward=1The first two lines allow NTP traffic from host server2 only. Note: I'm not 100% sure that TCP connections are required, but it shouldn't hurt. You might want to also add -d lowstratumserver to those two lines too, so that this machine won't be able to access any other server (just the low-stratum one). The next line allows return traffic from connections that are established. The fourth line enables network address translation. The final line enables IP packet forwarding. With these, server1 is now a (minimal) router.If server2 doesn't have server1 as its default router, you will also need to add an explicit route for the low-stratum server:ip route add lowstratumserver via server1With that, server2 will use server1 as its router for network connectivity to lowstratumserver, and its regular router for everything else.Then, configure server2 to use the low-stratum server on the Internet as its NTP source, not server1.Finally, configure server1 so that it connects to server2, not the Stratum 1 server on the Internet.Now server2 synchronizes with the low-stratum server on the Internet, not with server1; and server1 will synchronize with server2 always, doing away with the need to add a failover setup (which will be fragile at best).This will lose you a slight amount of accuracy due to the extra hop, but it's always better to synchronize the machines on your local network to a machine with a good hardware RTC, rather than to a machine with a low-quality one.