I'm debugging Hadoop DataNodes that won't start. We are using saltstack and also elasticsearch on the machines. The Hadoop DataNode error is pretty clear: java.net.BindException: Problem binding to [0.0.0.0:50020]    java.net.BindException: Address already in use;       For more details see:  http://wiki.apache.org/hadoop/BindException[...]Caused by: java.net.BindException: Address already in use[...](ExitUtil.java:terminate(124)) - Exiting with status 1lsof -i -n for port 50020 says it's already used but only as source port and not destination port:  salt-mini 1733          root   25u  IPv4  17452      0t0  TCP xx.xx.132.72:50020->xx.xx.132.20:4505 (ESTABLISHED)java      2789 elasticsearch 2127u  IPv6   9808      0t0  TCP xx.xx.132.72:50020->xx.xx.132.55:9300 (ESTABLISHED)However binding on 0.0.0.0 does not seem to work: root@host:~# nc -l 50020nc: Address already in useIs this intentional? Is binding to 0.0.0.0 disallowed when the port is already used a source port? There is nothing listening to the socket - I don't really know why it shouldn't work. Ubuntu 14.04: root@host:~# uname -aLinux host 4.2.0-19-generic #23~14.04.1-Ubuntu SMP Thu Nov 12 12:33:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
bind failure, address in use: Unable to use a TCP port for both source and destination?
hadoop
It does not matter if 50020 is a source or destination port: If it is claimed, it is claimed. I would consider it a bug to need to start a service on a particular port in the range 49152 - 65535, as these are the ephemeral ports as defined by IANA. Many Linux distributions take ports higher than 32768 as ephemeral. You can review the currently ephemeral port range with:cat /proc/sys/net/ipv4/ip_local_port_rangeAny application may just use a port from the ephemeral range, so there is no guarantee that a particular port always will be free. Better to pick an unused port between 1024 and 32767. See some intro on ephemeral ports. If you want to change the ephemeral range to cater for the Hadoop DataNode requirement, you could do so by editing /etc/sysctl.conf, and setting a line along the following:net.ipv4.ip_local_port_range=56000 65000edit: Thanks @mr.spuratic, who indirectly pointed out that with a recent enough kernel (the change was commitedd in May 2010), one can make exceptions to the range.  This is recommended, as toying with the range itself is quite a drastic change. sysctl -w net.ipv4.ip_local_reserved_ports = 50020, 50021Quoting from Documentation/networking/ip-sysctl.txtip_local_reserved_ports - list of comma separated rangesSpecify the ports which are reserved for known third-partyapplications. These ports will not be used by automatic portassignments (e.g. when calling connect() or bind() with portnumber 0). Explicit port allocation behavior is unchanged.The format used for both input and output is a comma separatedlist of ranges (e.g. 1,2-4,10-10 for ports 1, 2, 3, 4 and10). Writing to the file will clear all previously reservedports and update the current list with the one given in theinput.Note that ip_local_port_range and ip_local_reserved_portssettings are independent and both are considered by the kernelwhen determining which ports are available for automatic portassignments.You can reserve ports which are not in the currentip_local_port_range, e.g.:$ cat /proc/sys/net/ipv4/ip_local_port_range32000   60999$ cat /proc/sys/net/ipv4/ip_local_reserved_ports8080,9148although this is redundant. However such a setting is usefulif later the port range is changed to a value that willinclude the reserved ports.Default: Empty