I have one service running that takes long time to complete task. After switching to docker (running as docker service in docker swarm), we experienced same strange behavior: Client sends request (HTTP/SOAP) Server takes long time to process request (20 minutes) Server sends response Client does not get response and still waitsMy first thought was NAT, because docker internally uses private IP range. So I have implemented two scripts, one acting as server and one as client. Client sends request with number X, and server waits X seconds before sending response. Fairly simple. I was able to determine, that if server waits longer then 900 seconds, its response is dropped. What me confuses, is that in meantime I have been watching NAT with conntrack: $ while true; do echo $(date) ' - ' $(conntrack -L -p tcp --dport 55500 2>/dev/null); sleep 1; doneFri Jul 7 09:42:54 UTC 2017  -  tcp 6 431049 ESTABLISHED src=10.66.122.31 dst=10.66.122.160 sport=59018 dport=55500 src=172.18.0.2 dst=10.66.122.31 sport=55500 dport=59018 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t :s0 use=1Fri Jul 7 09:42:55 UTC 2017  -  tcp 6 431048 ESTABLISHED src=10.66.122.31 dst=10.66.122.160 sport=59018 dport=55500 src=172.18.0.2 dst=10.66.122.31 sport=55500 dport=59018 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t :s0 use=1Fri Jul 7 09:42:56 UTC 2017  -  tcp 6 431047 ESTABLISHED src=10.66.122.31 dst=10.66.122.160 sport=59018 dport=55500 src=172.18.0.2 dst=10.66.122.31 sport=55500 dport=59018 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t :s0 use=1Fri Jul 7 09:42:57 UTC 2017  -  tcp 6 431046 ESTABLISHED src=10.66.122.31 dst=10.66.122.160 sport=59018 dport=55500 src=172.18.0.2 dst=10.66.122.31 sport=55500 dport=59018 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t :s0 use=1Maybe I am missing something, but shouldn't my TCP connection (based on timer in conntrack) be alive for cca. 5 days before it is dropped by NAT, right?Is this potentially bug in docker or am I missing something?Here are scripts that I used for tests:Server: #!/usr/bin/pythonimport BaseHTTPServerimport reimport timeimport logginglogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')LOG = logging.getLogger()class Handler(BaseHTTPServer.BaseHTTPRequestHandler):    def do_GET(self):        data = 'OK'        m = re.match(r'^/*(\d+)', self.path)        if m is None:            self.send_response(400, 'Bad Request')            self.end_headers()        else:            sleeptime = int(m.group(1))            LOG.info(Sleeping for %s seconds ..., sleeptime)            time.sleep(sleeptime)            LOG.info(Sending response)            self.send_response(200, 'OK')            self.send_header('Content-Type', 'text/plain')            self.send_header('Content-Length', len(data))            self.end_headers()            self.wfile.write(data)            self.wfile.close()LOG.info(Starting server at %s, 8080)server_address = ('', 8080)httpd = BaseHTTPServer.HTTPServer(server_address, Handler)httpd.serve_forever()Client:#!/usr/bin/pythonimport urllib2import argparseimport socketimport logginglogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')LOG = logging.getLogger()parser = argparse.ArgumentParser()parser.add_argument(url)opts = parser.parse_args()min_time = 1.0max_time = 7200.0test_time = min_timewhile min_time < max_time:    LOG.info(test_time=%s, min_time=%s, max_time=%s, test_time, min_time, max_time)    timeout = False    final_url = %s/%s % (opts.url, test_time)    try:        response = urllib2.urlopen(final_url, timeout=test_time + 5)        LOG.info(Got response)        response.close()    except socket.timeout:        LOG.info(Got timeout!)        timeout = True    if timeout:        max_time = test_time    else:        min_time = test_time    test_time *= 2    if test_time > max_time:        test_time = (max_time + min_time) / 2.0LOG.info(NAT expire timer is %s seconds!, max_time)
Docker & NAT expire for TCP
tcp;docker;nat