I'm currently looking to move our system from RHEL 5 to RHEL 6, but I've run into a snag with unexpectedly high CPU usage on the RHEL 6 machines. It appears that this may be due at least in some part to the use of select to do an interruptible sleep. Here's a simple example that shows the behaviour:#include <sys/select.h>int main(){  timeval ts;  for (unsigned int ii=0; ii<10000; ++ii) {    ts.tv_sec = 0;    ts.tv_usec = 1000;    select(0, 0, 0, 0, &ts);  }  return 0;}On a RHEL 5 machine it will stay at 0% CPU usage, but on the same hardware with RHEL 6 installed it will use about 0.5% of the CPU, so when 30 to 50 programs are running using select to perform a sleep it eats up a large amount of the CPU unnecessarily.I opened a Bugzilla and I tried running OProfile and it simply shows 100% in main for the application and just over 99% in poll_idle when looking at the kernel (I have idle=poll set in my grub options so everything can be captured).Any other ideas of what I can do to try and isolate what the cause of the higher CPU usage is?UPDATE: I found the perf tool and got the following output:# Events: 23K cycles## Overhead  Command        Shared Object                                Symbol# ........  .......  ...................  ....................................#    13.11%  test_select_sma  [kernel.kallsyms]    [k] find_busiest_group     5.88%  test_select_sma  [kernel.kallsyms]    [k] schedule     5.00%  test_select_sma  [kernel.kallsyms]    [k] system_call     3.77%  test_select_sma  [kernel.kallsyms]    [k] copy_to_user     3.39%  test_select_sma  [kernel.kallsyms]    [k] update_curr     3.22%  test_select_sma  ld-2.12.so           [.] _dl_sysinfo_int80     2.83%  test_select_sma  [kernel.kallsyms]    [k] native_sched_clock     2.72%  test_select_sma  [kernel.kallsyms]    [k] find_next_bit     2.69%  test_select_sma  [kernel.kallsyms]    [k] cpumask_next_and     2.58%  test_select_sma  [kernel.kallsyms]    [k] native_write_msr_safe     2.47%  test_select_sma  [kernel.kallsyms]    [k] sched_clock_local     2.39%  test_select_sma  [kernel.kallsyms]    [k] read_tsc     2.26%  test_select_sma  [kernel.kallsyms]    [k] do_select     2.13%  test_select_sma  [kernel.kallsyms]    [k] restore_nocheckIt appears that the higher CPU usage is from the scheduler. I also used the following bash script to kick off 100 of these simultaneously:#!/bin/bashfor i in {1..100}do  ./test_select_small &doneOn RHEL 5 the CPU usage stays close to 0%, but on RHEL 6 there's a non-trivial amount of CPU usage in both user and sys. Any ideas on how to track down the true source of this and hopefully fix it?I also tried this test on a current Arch Linux build and Ubuntu 11.10 and saw similar behaviour, so this appears to be some type of kernel issue and not just a RHEL issue.UPDATE2: I hesitate a bit to bring this up because I know that it's a huge debate, but I tried out a kernel with the BFS patches on Ubuntu 11.10 and it didn't show the same high system CPU usage (user cpu usage seemed about the same).Is there some test I can run with each of them to test if this high CPU usage is just a difference in accounting of CPU usage that is making it look artificially high? Or if actual CPU cycles are being stolen by the CFS?UPDATE3: The analysis done involving this question seems to indicate that it's something related to the scheduler, so I created a new question to discuss the results.UPDATE4: I added some more information to the other question.UPDATE5: I added some results to the other question from a simpler test that still demonstrates the issue.
Isolating cause of higher CPU usage on RHEL 6 vs RHEL 5
kernel;cpu;linux kernel;scheduling