What Linux process just tried to connect to where?
I was making some firewall changes last weekend and while watching the logs I discovered that every now and then some host would try to connect to 169.254.169.254 on port 80. This was peculiar since I don't use the IPv4 link local addresses anywhere in my network. It seemed to be happening randomly from all of my Linux hosts, both physical and virtual.
The part that makes this a bit unique is that this is a connection that will certainly fail. It trips a rule in my firewall that blocks bogons outbound on my WAN interface which means that the normal tools like netstat(1) and lsof(8) will only reveal anything if I somehow catch the process between the execution of the connect(2) system call and it failing. What I need to be able to do is log in real time what is happening, which I could do with something like strace(1) but I'd need to know the PID of the process and that is what I'm trying to find out.
So off I went looking for other things that might be helpful and stumbled upon the Linux kernel audit system. The audit system has been around for a while and lets you ask the kernel to communicate the details of syscalls as they happen. There is a filtering mechanism built in so that you don't end up dumping too much information or dramatically impacting performance and the raw data is sent to a userland program via a netlink socket. By default most distributions ship auditd, which listens on that netlink socket and dumps all the messages into a log.
Since I am looking at an attempted TCP connection the connect system call is the one I am interested in. I don't know much else about it though so it turns out a pretty simple filter rule is what I was looking for.
OK, so I'm getting closer but obviously some of the data is coming out in some packed hex format and the things I want aren't all on the same line so I need to figure out how to decode this. While searching for the format of the messages in hopes of writing a quick and dirty parser I found ausearch(8) which includes the handy -i option.
I fired up tcpdump(8) on the pflog(4) interface and waited for one of the 169.254.169.254 packets to be dropped. That let me find what I was looking for in the audit log... the culprit.
It turns out it was a puppet agent run. Now I know none of my modules try to talk to that address but puppet does a lot of things including running facter to get information about the system and the environment it is running in. I know some cloud infrastructure has standardized on that address as a location for guest agents to pick up metadata from so I suspected some default module trying to see if we are running on a cloud provider. A quick locate(1) and grep(1) later and it turns out that the built in facter ec2 module does in fact try to pull metadata from 169.254.169.254.
So in the end, the Linux audit system is our friend. There is a lot of other cool stuff in there, I ran across a post from the slack engineering team that talks about how they use the audit system and how they leverage this information to alert on and challenge user actions in real time. It is also a cautionary tale that good network hygiene is important since you never know what random things you might leak out onto the Internet (or your ISP's network) if you aren't careful.