I have a server that is infected with Ebury rootkit, that doesn't show the infected files, and the process listening for connections when it is being checked over SSH for any signs of infection, but shows them normally when checking over Salt or local session.I had servers with Ebury infection before, and was always able to see suspicious files and processes when checking over SSH, but on this variant I can't, and I am wondering why exactly.Info on suspicious files and processes:Server has an atd process listening for connections using Unix domain sockets, and when it is checked over Salt, or local session, netstat shows the the process listening.$ salt -L server cmd.run netstat -plan | grep -w atdserver:   unix  2      [ ACC ]     STREAM     LISTENING     55898  5264/atd            @/tmp/dbus-GeTBbBytfBWhen I check the server over SSH, netstat comes back clean, as it should on non-infected servers.root@server [~]# netstat -plan | grep -w atdroot@server [~]#When checking for standard files that are used by Ebury, over Salt or local session, I can see the following returned.$ salt -L server cmd.run rpm -qf /lib64/tls/libkeyutils.so.1.5 /lib64/tls/libkeyutils.so.1server:    file /lib64/tls/libkeyutils.so.1.5 is not owned by any package    file /lib64/tls/libkeyutils.so.1 is not owned by any packageWhen checking over SSH, files are reported as non-existent.root@server [~]# rpm -qf /lib64/tls/libkeyutils.so.1.5 /lib64/tls/libkeyutils.so.1error: file /lib64/tls/libkeyutils.so.1.5: No such file or directoryerror: file /lib64/tls/libkeyutils.so.1: No such file or directoryroot@server [~]#When checking disk space used by folder containing suspicous files, I get different results returned on Salt and local session, then over SSHSalt and local session:root@salt-master-vps [~]$ salt -L server cmd.run du -shc /lib64/tls/; du -shc /lib64/tls/*;server:    56K /lib64/tls/    56K total    0   /lib64/tls/libkeyutils.so.1    52K /lib64/tls/libkeyutils.so.1.5    52K totalOver SSHroot@server [~]# du -shc /lib64/tls/; du -shc /lib64/tls/*;4.0K    /lib64/tls/4.0K    totaldu: cannot access `/lib64/tls/*': No such file or directory0       totalroot@server [~]#Trying to find the file over inode number, over Salt and local:$ salt -L server cmd.run find /lib64/tls -inum 213647535server:    /lib64/tls/libkeyutils.so.1.5Over SSH:root@server [~]# find /lib64/tls -inum 213647535root@server [~]#Checking loaded libraries of various executables over Salt and local session:$ salt -L server cmd.run ldd /usr/bin/wget | grep tlsserver:        libkeyutils.so.1 => /lib64/tls/libkeyutils.so.1 (0x0000003b96200000)Check over SSH, doesn't show suspicous files loadedroot@server [~]# ldd /usr/bin/wget | grep tlsroot@server [~]#Checking PID of atd with lsof, over Salt and local session, shows listener (@/tmp/dbus-GeTBbBytfB) and suspicious file /lib64/tls/libkeyutils.so.1.5:$ salt -L server cmd.run lsof -p 5264server:    COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME    atd     5264 root  cwd    DIR                8,4     4096         2 /    atd     5264 root  rtd    DIR                8,4     4096         2 /    atd     5264 root  txt    REG                8,4    23968 232524937 /usr/sbin/atd    atd     5264 root  mem    REG                8,4   113952 213647527 /lib64/libresolv-2.12.so    atd     5264 root  mem    REG                8,4    10312 213647581 /lib64/libfreebl3.so    atd     5264 root  mem    REG                8,4    43392 213647547 /lib64/libcrypt-2.12.so    atd     5264 root  mem    REG                8,4   145864 213647601 /lib64/libaudit.so.1.0.0    atd     5264 root  mem    REG                8,4    91096 213647529 /lib64/libz.so.1.2.3    atd     5264 root  mem    REG                8,4    22536 213647519 /lib64/libdl-2.12.so    atd     5264 root  mem    REG                8,4  1928936 213647514 /lib64/libc-2.12.so    atd     5264 root  mem    REG                8,4    14584 213647675 /lib64/libpam_misc.so.0.82.0    atd     5264 root  mem    REG                8,4    55848 213647656 /lib64/libpam.so.0.82.2    atd     5264 root  mem    REG                8,4   122056 213647396 /lib64/libselinux.so.1    atd     5264 root  mem    REG                8,4    50720 213647535 /lib64/tls/libkeyutils.so.1.5    atd     5264 root  mem    REG                8,4  1967392 232527760 /usr/lib64/libcrypto.so.1.0.1e    atd     5264 root  mem    REG                8,4   157072 213647475 /lib64/ld-2.12.so    atd     5264 root    0u   CHR                1,3      0t0      4245 /dev/null    atd     5264 root    1u   CHR                1,3      0t0      4245 /dev/null    atd     5264 root    2u   CHR                1,3      0t0      4245 /dev/null    atd     5264 root    3u  unix 0xffff88042a2180c0      0t0     55898 @/tmp/dbus-GeTBbBytfBWhen checking over SSH, no suspicous file, and listener using unix socket is shown as can't identify protocolroot@server [~]# lsof -p 5264COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAMEatd     5264 root  cwd    DIR    8,4     4096         2 /atd     5264 root  rtd    DIR    8,4     4096         2 /atd     5264 root  txt    REG    8,4    23968 232524937 /usr/sbin/atdatd     5264 root  mem    REG    8,4   113952 213647527 /lib64/libresolv-2.12.soatd     5264 root  mem    REG    8,4    10312 213647581 /lib64/libfreebl3.soatd     5264 root  mem    REG    8,4    43392 213647547 /lib64/libcrypt-2.12.soatd     5264 root  mem    REG    8,4   145864 213647601 /lib64/libaudit.so.1.0.0atd     5264 root  mem    REG    8,4    91096 213647529 /lib64/libz.so.1.2.3atd     5264 root  mem    REG    8,4    22536 213647519 /lib64/libdl-2.12.soatd     5264 root  mem    REG    8,4  1928936 213647514 /lib64/libc-2.12.soatd     5264 root  mem    REG    8,4    14584 213647675 /lib64/libpam_misc.so.0.82.0atd     5264 root  mem    REG    8,4    55848 213647656 /lib64/libpam.so.0.82.2atd     5264 root  mem    REG    8,4   122056 213647396 /lib64/libselinux.so.1atd     5264 root  mem    REG    8,4  1967392 232527760 /usr/lib64/libcrypto.so.1.0.1eatd     5264 root  mem    REG    8,4   157072 213647475 /lib64/ld-2.12.soatd     5264 root    0u   CHR    1,3      0t0      4245 /dev/nullatd     5264 root    1u   CHR    1,3      0t0      4245 /dev/nullatd     5264 root    2u   CHR    1,3      0t0      4245 /dev/nullatd     5264 root    3u  sock    0,6      0t0     55898 can't identify protocolroot@server [~]#File does get shown over SSH, when I check with ldconfigroot@server [~]# /sbin/ldconfig -p | grep /lib64/tls/libkeyutils.so.1        libkeyutils.so.1 (libc6,x86-64, hwcap: 0x8000000000000000) => /lib64/tls/libkeyutils.so.1root@server [~]#I am wondering if anyone can explain, why is it that I can't see the files and processes when connected over SSH, but I can when checking on local terminal session and over Salt?Why is disk usage of the folder shown differently over SSH?Why are loaded libraries not visible over SSH?Also, did anyone else encounter this behavior?All the previous instances of Ebury infection, I encountered till now, were visible over SSH when checking processes and files.
Ebury infection, suspicious processes and files visible over Salt or local session, but not over SSH
ssh;malware