I am running some long-running programs (for scientific purposes) on a multi-core Linux box. The processes are controlled by a small daemon which restarts the next job when one finishes (I run 3-6 at once) - but I've noticed they don't always use 100% CPU. The programs (and the daemon) are written in Python.When I run this code on Mac OS X, I can run programs for weeks and they will always use as many system resources are available, while the machine is running at a normal temperature.I've just started trying to run such things on Debian Linux (on another machine), with 6 Cores and much more RAM than the jobs need. I am running 5 such jobs at once. When I first started the jobs a few days ago, I had 5 Python processes in top, each using 100% CPU. About a day later, I checked how it was all doing and I had 3 processes at 100% CPU and two running at 50% CPU. Most recently (about 4 days in) I have 5 processes all running at 20% CPU.What could be causing this? Nothing seems to suggest that CPU usage management tools are pre-installed on Debian Wheezy, and I have not installed anything like that myself (to my knowledge), nor configured it. Also, since the limits seem to vary depending on how long the demon has been running, I'm not convinced that it could be such a system. I have checked if the machine is overheating, and it doesn't seem to be much hotter than the (cold) room it's in; the air from the fans/vents is both unobstructed and cool.The processes are still running, so I can measure anything that might be useful for debugging (length of time running, process priority, etc) in order to debug this problem. Can anyone tell me where to start, or what any possible solutions might be?UPDATE:When I try the same thing with 3 threads instead of 5, I am dropped to 33% each (after initial drops to 50%.)Is there any program or scheduling policy that limits all child processes of a single process to a total of 100%? as that's what seems to be happening.Next test is to directly run the scripts in separate screen shells, (BTW, the first script was launched from inside screen) and see if we get any slowdown. Carving the jobs up by hand like this is an OK-ish workaround, but quite irritating (and should be unnecessary.) In general, of course, this sort of problem may not be solvable this way, but since all results from each job are saved to disk rather than returned to the thread manager, I'll get away with it.UPDATE 2:Separate processes launched from different screen instances are still going at 100% CPU after 14 hours, will report back if I see any slowdown but as expected this case is unaffected by any throttling.anyone care to write (or point me at) something that explains process priority on Linux? I am wondering if my spawning processes is being marked as a lower priority (since it uses very little CPU itself), and then the child processes are inheriting this.EDIT:I've been asked about the script I'm running, and the function of forking the daemon processes.The long running script is a big calculation, which always runs at 100% CPU until it finishes, and does nothing funny about parallelisation or multiprocessing. (this is a widely tested assertion.) To clarify further - the only times I've seen these processes run at less than 100% CPU on my Mac are when overheating, or when paging/swapping. Neither of these is relevant to the Linux case.Here is the function that forks out, and then manages the long running processes:from multiprocessing import Processimport time, sys, os# An alternative entry point which runs N jobs in parallel over a list of files.# Note, since this is designed to be used as a library function, we return from the initial# function call rather than exiting.def run_over_file_list(script_path, list_of_data_files, num_processes, timeout=float('inf')):    try:        pid = os.fork()        if pid > 0:            # exit first parent            return    except OSError, e:        print >>sys.stderr, fork #1 failed: %d (%s) % (e.errno, e.strerror)        sys.exit(1)    # decouple from parent environment    os.chdir(/)    os.setsid()    os.umask(0)    # do second fork    try:        pid = os.fork()        if pid > 0:            # exit from second parent, print eventual PID before            print Daemon PID %d % pid            sys.exit(0)    except OSError, e:        print >>sys.stderr, fork #2 failed: %d (%s) % (e.errno, e.strerror)        sys.exit(1)    # OK, we're inside a manager daemon.    if os.path.isfile(status_filename):        raise Exception(a daemon is already running. failed.)    f = open(status_filename, w)    f.write(str(os.getpid()))    f.close()    jobs = [script_path] * num_processes    data_files_remaining = [f for f in list_of_data_files]    update_files_remaining_file(len(data_files_remaining))    assert num_processes <= len(data_files_remaining)    restart = False    with nostdout():        while True:            processes = []            for job in jobs:                p = Process(target=file_list_worker, args=(job, data_files_remaining.pop(0)))                p.started = time.time()                p.start()                processes.append(p)            stop = False            while True:                time.sleep(10)                ended = []                for i, p in enumerate(processes):                    if not p.is_alive():                        j = i                        ended.append((j,p))                    elif time.time() - p.started > timeout:                        p.terminate()                        j = i                        ended.append((j,p))                if not stop:                    for tup in ended:                        if not data_files_remaining:                            stop = True                            break                        i, e = tup                        new_p = Process(target=file_list_worker, args=(jobs[i], data_files_remaining.pop(0)))                        new_p.started = time.time()                        new_p.start()                        processes[i] = new_p                        # old e will be garbage collected                else:                    if len(ended) == len(processes) and not data_files_remaining:                        stop = False                        break                try:                    command = check_for_command()                    if command == stop:                        stop = True                    elif command == restart:                        stop = True                        restart = True                    elif command == kill:                        for p in processes:                            p.terminate()                        clear_command()                        os.remove(status_filename)                        exit(0)                except NoCommandError:                    pass                update_files_remaining_file(len(data_files_remaining))            clear_command()            update_files_remaining_file(len(data_files_remaining))            if not restart:                os.remove(status_filename)                break            else:                jobs = None                restart = False    # While in a fork, we should never return (will continue running the original/calling script in parallel, hilarity ensues.)    exit(0)EDIT 2:prioritySo, everything seems to run with priority 20 from whatever source; the pre-throttled processes, the post-throttled processes, the daemon manager, the processes run directly from the shell under screen.ulimit -afrom bash:core file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 127788max locked memory       (kbytes, -l) 64max memory size         (kbytes, -m) unlimitedopen files                      (-n) 1024pipe size            (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority              (-r) 0stack size              (kbytes, -s) 8192cpu time               (seconds, -t) unlimitedmax user processes              (-u) 127788virtual memory          (kbytes, -v) unlimitedfile locks                      (-x) unlimitedfrom fish:Maximum size of core files created                           (kB, -c) 0Maximum size of a processs data segment                     (kB, -d) unlimitedMaximum size of files created by the shell                   (kB, -f) unlimitedMaximum size that may be locked into memory                  (kB, -l) 64Maximum resident set size                                    (kB, -m) unlimitedMaximum number of open file descriptors                          (-n) 1024Maximum stack size                                           (kB, -s) 8192Maximum amount of cpu time in seconds                   (seconds, -t) unlimitedMaximum number of processes available to a single user           (-u) 127788Maximum amount of virtual memory available to the shell      (kB, -v) unlimitedfrom fish under screen:(exactly the same as normal fish.)Much later updateI have also noticed this bug with long-running processes run from separate shells. e.g:Instance 1: 17% (one core of 6 at 100%.)Instance 2: 8% (one core of 6 at  50%.)Instance 3: 8% (one core of 6 at  50%.)if I change the priority of instance 2 to be very high, the state becomes:Instance 1: 17% (one core of 6 at 100%.)Instance 2: 17% (one core of 6 at 100%.)Instance 3: 0% (one core of 6 at    0%.)If the priorities are all equalised again, we return to the first state.I am starting to think that this problem may be related to a specific hardware configuration or something, but I lack the tools/knowledge to debug further.
What's limiting my CPU usage in forked processes?
debian;cpu;top