I recently started migrating some of my company's servers to Google's compute engine. Among other things I set up a 2-node elasticsearch cluster.Today I run top command on one of the nodes and I noticed a few suspicious processes: PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                     11995 elastics  20   0  424m  25m  400 S 533.0  0.1 877:26.82 gg                                                                                                         30494 elastics  20   0 23.4g  16g 116m S  0.8 56.7   2:49.61 java                                                                                                       20148 newrelic  20   0  244m 5824 2872 S  0.5  0.0   9:34.29 nrsysmond                                                                                                   42 root      20   0     0    0    0 S  0.3  0.0   0:41.16 events/7                                                                                                   4 root      20   0     0    0    0 S  0.1  0.0   0:54.41 ksoftirqd/0                                                                                                 38 root      20   0     0    0    0 S  0.1  0.0   0:22.52 events/3                                                                                                    39 root      20   0     0    0    0 S  0.1  0.0   0:20.27 events/42876 root      20   0 15152 1312  928 R  0.1  0.0   0:00.10 top7022 elastics  20   0  424m  16m  380 S  0.1  0.1   1:10.45 freeBSD1 root      20   0 19340 1312  952 S  0.0  0.0   0:02.38 init2 root      20   0     0    0    0 S  0.0  0.0   0:00.12 kthreadd3 root      RT   0     0    0    0 S  0.0  0.0   0:01.44 migration/05 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0        The processes with ids 11995, 7022 seemed suspicious and I decided to have a better look.Doing an lsof for this processes uncovered that they had established connections to either China or Hong Kong.Please find an example snippet below:[root@elastic-prd-node-01 fail2ban]# lsof -p 11995COMMAND   PID          USER   FD   TYPE   DEVICE SIZE/OFF  NODE NAMEgg      11995 elasticsearch  cwd    DIR      8,1     4096     2 /gg      11995 elasticsearch  rtd    DIR      8,1     4096     2 /gg      11995 elasticsearch  txt    REG      8,1  1415201 16117 /tmp/gg (deleted)gg      11995 elasticsearch  mem    REG      8,1 99154480  4784 /usr/lib/locale/locale-archivegg      11995 elasticsearch    0u   CHR      1,3      0t0  3881 /dev/nullgg      11995 elasticsearch    1u   CHR      1,3      0t0  3881 /dev/nullgg      11995 elasticsearch    2u   CHR      1,3      0t0  3881 /dev/nullgg      11995 elasticsearch    3u  IPv4 10615541      0t0   TCP myservername.c.myproject.internal:54431->103.206.22.224:28099 (ESTABLISHED)gg      11995 elasticsearch    4u  IPv4 10656374      0t0   UDP *:49882 gg      11995 elasticsearch    5u  IPv4 10656375      0t0   UDP *:41943 gg      11995 elasticsearch    6u  IPv4 10656376      0t0   UDP *:49792 gg      11995 elasticsearch    7u  IPv4 10656377      0t0   UDP *:32849 gg      11995 elasticsearch    8u  IPv4 10656378      0t0   UDP *:35841 gg      11995 elasticsearch    9u  IPv4 10656379      0t0   UDP *:52037 gg      11995 elasticsearch   10u  IPv4 10656380      0t0   UDP *:45405 gg      11995 elasticsearch   11u  IPv4 10656381      0t0   UDP *:55345 gg      11995 elasticsearch   12u  IPv4 10656382      0t0   UDP *:49990 gg      11995 elasticsearch   13u  IPv4 10656383      0t0   UDP *:34888 gg      11995 elasticsearch   14u  IPv4 10656384      0t0   UDP *:46642 gg      11995 elasticsearch   15u  IPv4 10656385      0t0   UDP *:38881 gg      11995 elasticsearch   16u  IPv4 10656386      0t0   UDP *:38455 gg      11995 elasticsearch   17u  IPv4 10656387      0t0   UDP *:47360 gg      11995 elasticsearch   18u  IPv4 10656388      0t0   UDP *:40310 gg      11995 elasticsearch   19u  IPv4 10656389      0t0   UDP *:44127 gg      11995 elasticsearch   20u  IPv4 10656390      0t0   UDP *:46064 gg      11995 elasticsearch   21u  IPv4 10656391      0t0   UDP *:55830 gg      11995 elasticsearch   22u  IPv4 10656392      0t0   UDP *:43110 gg      11995 elasticsearch   23u  IPv4 10656393      0t0   UDP *:58793 I also ran:[root@elastic-prd-node-01 tmp]# ls -al /proc/11995/exelrwxrwxrwx. 1 elasticsearch elasticsearch 0 May  8 12:48 /proc/17525/exe -> /tmp/ggand I noticed that in the /tmp directory there were a few suspicious files named like: gg or freeBSD or syn2016. I then tarred the whole /tmp directory, killed all the suspicious processes, removed all the suspicious files from /tmp and shutdown the servers.What I should do next?Does this look normal? Does it look like an attack?What I should do in order to prevent it happening in the future?What steps can I take in order to get to the bottom of it?p.s. I am running elasticsearch 1.6.2 on CentOS 6 and I used elasticdump to migrate my indices.
Malicious processes open connections to China
centos;security;elasticsearch
This is definitely an attack. Someone managed to a) drop a file onto your server (possibly via poorly configured apache)b) execute that fileYou should do:a) mount the /tmp partition with the noexec - optionb) reboot the serverc) run a malware scan on the complete host while offlined) if that does not help : REINSTALL from scratchTo get to the bottom of it, you can a) check the gg file's creation dateb) look into the httpd-logs at that time.So you possibly will be able to see via which script the file has been dropped. Another possibility would be a privileged ftp account with poor password. Search these logs also...