I'm having a problem on one of our production servers where the login attempts become progressively slower as time passes. After roughly 5 days it becomes so slow that some critical processes/cron entries fail to fire correctly.Server information:Hardware: Dell R720 24 GB Memory 2 x Intel Xeon E502620 v2 Processors  (24 cores total, including HyperThreading) 8 x 300GB 10K SAS DrivesOS: Red Hat Enterprise Linux 6.5I experienced issues logging in via SSH and started along a long road of investigating 'red herrings'. Eventually, I noticed that even doing the following took very long:[someuser#hostname] su -Password:When running the the 'su -' command as above, it shouldn't involve SSH in any form, as I am just trying to authenticate on the box itself, right?This has happened 3 weeks in a row on the same box and it's the first time I noticed (and tested I might add) that the local only logins are also slow.When I log in via the console, it looks as follows:hostname login: userPassword:              # I enter the password and hit [enter]A long time passes and then...[user@hostname ~] $When I log in via SSH, it looks as follows (verbosity on and doing a localhost login):[user@hostname ~]$ ssh -v root@localhostOpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010debug1: Reading configuration data /etc/ssh/ssh_configdebug1: Applying options for *debug1: Connecting to localhost [127.0.0.1] port 22.debug1: Connection established.debug1: identity file /usr/local/user/.ssh/identity type -1debug1: identity file /usr/local/user/.ssh/identity-cert type -1debug1: identity file /usr/local/user/.ssh/id_rsa type -1debug1: identity file /usr/local/user/.ssh/id_rsa-cert type -1debug1: identity file /usr/local/user/.ssh/id_dsa type -1debug1: identity file /usr/local/user/.ssh/id_dsa-cert type -1debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3debug1: match: OpenSSH_5.3 pat OpenSSH*debug1: Enabling compatibility mode for protocol 2.0debug1: Local version string SSH-2.0-OpenSSH_5.3debug1: SSH2_MSG_KEXINIT sentdebug1: SSH2_MSG_KEXINIT receiveddebug1: kex: server->client aes128-ctr hmac-md5 nonedebug1: kex: client->server aes128-ctr hmac-md5 nonedebug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sentdebug1: expecting SSH2_MSG_KEX_DH_GEX_GROUPdebug1: SSH2_MSG_KEX_DH_GEX_INIT sentdebug1: expecting SSH2_MSG_KEX_DH_GEX_REPLYThe authenticity of host 'localhost (127.0.0.1)' can't be established.RSA key fingerprint is 1d:50:5e:a3:e4:63:d6:1d:d8:2c:85:07:95:81:c8:b6.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'localhost' (RSA) to the list of known hosts.debug1: ssh_rsa_verify: signature correctdebug1: SSH2_MSG_NEWKEYS sentdebug1: expecting SSH2_MSG_NEWKEYSdebug1: SSH2_MSG_NEWKEYS receiveddebug1: SSH2_MSG_SERVICE_REQUEST sentdebug1: SSH2_MSG_SERVICE_ACCEPT receiveddebug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,passworddebug1: Next authentication method: gssapi-keyexdebug1: No valid Key exchange contextdebug1: Next authentication method: gssapi-with-micdebug1: Unspecified GSS failure.  Minor code may provide more informationCredentials cache file '/tmp/krb5cc_501' not founddebug1: Unspecified GSS failure.  Minor code may provide more informationCredentials cache file '/tmp/krb5cc_501' not founddebug1: Unspecified GSS failure.  Minor code may provide more informationdebug1: Unspecified GSS failure.  Minor code may provide more informationCredentials cache file '/tmp/krb5cc_501' not founddebug1: Next authentication method: publickeydebug1: Trying private key: /usr/local/user/.ssh/identitydebug1: Trying private key: /usr/local/user/.ssh/id_rsadebug1: Trying private key: /usr/local/user/.ssh/id_dsadebug1: Next authentication method: passwordroot@localhost's password:debug1: Authentication succeeded (password).debug1: channel 0: new [client-session]debug1: Requesting no-more-sessions@openssh.comdebug1: Entering interactive session.At this point it gets stuck, for a long while again, and eventually logs in successfully.Any pointers will be appreciated. This is driving me up the walls.The following also appears in the dmesg output. It repeats for various process names (not only 'cifsd')cifsd: page allocation failure. order:5, mode:0x20Pid: 12913, comm: cifsd Not tainted 2.6.32-431.el6.x86_64 #1Call Trace: [<ffffffff8112f9e7>] ? __alloc_pages_nodemask+0x757/0x8d0 [<ffffffff8116e482>] ? kmem_getpages+0x62/0x170 [<ffffffff8116f09a>] ? fallback_alloc+0x1ba/0x270 [<ffffffff8116eaef>] ? cache_grow+0x2cf/0x320 [<ffffffff8116ee19>] ? ____cache_alloc_node+0x99/0x160 [<ffffffff8116ffe0>] ? kmem_cache_alloc_node_trace+0x90/0x200 [<ffffffff811701fd>] ? __kmalloc_node+0x4d/0x60 [<ffffffff8144feca>] ? __alloc_skb+0x7a/0x180 [<ffffffff81450fe0>] ? skb_copy+0x40/0xb0 [<ffffffffa014f57c>] ? tg3_start_xmit+0xa8c/0xd80 [tg3] [<ffffffff81460354>] ? dev_hard_start_xmit+0x224/0x480 [<ffffffff8147bd0a>] ? sch_direct_xmit+0x15a/0x1c0 [<ffffffff81460858>] ? dev_queue_xmit+0x228/0x320 [<ffffffff8149a0d8>] ? ip_finish_output+0x148/0x310 [<ffffffff8149a358>] ? ip_output+0xb8/0xc0 [<ffffffff8105a924>] ? find_busiest_group+0x244/0x9f0 [<ffffffff81499655>] ? ip_local_out+0x25/0x30 [<ffffffff81499b30>] ? ip_queue_xmit+0x190/0x420 [<ffffffff8112ff2f>] ? free_hot_page+0x2f/0x60 [<ffffffff814aee3e>] ? tcp_transmit_skb+0x40e/0x7b0 [<ffffffff814b1380>] ? tcp_write_xmit+0x230/0xa90 [<ffffffff814b1f00>] ? __tcp_push_pending_frames+0x30/0xe0 [<ffffffff814a9663>] ? tcp_data_snd_check+0x33/0x100 [<ffffffff814ad261>] ? tcp_rcv_established+0x381/0x7f0 [<ffffffff8152873a>] ? schedule_timeout+0x19a/0x2e0 [<ffffffff814b5643>] ? tcp_v4_do_rcv+0x2e3/0x490 [<ffffffff814a130a>] ? tcp_prequeue_process+0x7a/0xa0 [<ffffffff814a4a2c>] ? tcp_recvmsg+0xacc/0xe80 [<ffffffff814c58ca>] ? inet_recvmsg+0x5a/0x90 [<ffffffff8105a625>] ? select_idle_sibling+0x95/0x150 [<ffffffff81449ab3>] ? sock_recvmsg+0x133/0x160 [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81059216>] ? enqueue_task+0x66/0x80 [<ffffffff8105571d>] ? check_preempt_curr+0x6d/0x90 [<ffffffff81065c5e>] ? try_to_wake_up+0x24e/0x3e0 [<ffffffff81065e02>] ? default_wake_function+0x12/0x20 [<ffffffff8109b2b6>] ? autoremove_wake_function+0x16/0x40 [<ffffffff81449b24>] ? kernel_recvmsg+0x44/0x60 [<ffffffffa01fd7c9>] ? cifs_readv_from_socket+0x1a9/0x260 [cifs] [<ffffffffa020b11d>] ? cifs_add_credits+0x5d/0x70 [cifs] [<ffffffffa01fd8a7>] ? cifs_read_from_socket+0x27/0x30 [cifs] [<ffffffffa01fda03>] ? cifs_demultiplex_thread+0x153/0xe10 [cifs] [<ffffffff81065e02>] ? default_wake_function+0x12/0x20 [<ffffffffa01fd8b0>] ? cifs_demultiplex_thread+0x0/0xe10 [cifs] [<ffffffff8109aef6>] ? kthread+0x96/0xa0 [<ffffffff8100c20a>] ? child_rip+0xa/0x20 [<ffffffff8109ae60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20
Authentication very slow (both with ssh and su)
rhel;performance;authentication