I have spend 2 hours reading questions about this matter, and still there is some misunderstanding.I have this process:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND    root 1452 0.4 1.8 1397012 19308 ? Sl 04:23 3:48 ./transaction_serverThis shows it uses 19.3Mb of system resident memory (I have no swap file), around 1.8% of whole 1GB system memory. Virtual size is 1.39GB?!?. I have read that ulimit -m doesn't work. People use ulimit -v e.g. setting virtual memory for the process. Is this virtual memory is the one VSZ listed with ps? What value I should set if I want to restrict this process to use 100MB system memory at most. I have read documentation for setrlimit and this seems legit:RLIMIT_AS        This  is  the maximum size of a process' total available memory,         in bytes. If this limit is exceeded, the malloc() and mmap()         functions shall fail with errno set to [ENOMEM]. In addition,         the automatic stack growth fails with the effects outlined above.But on other versions of the documentation this RLIMIT_AS parameter sets virtual memory size. What is the truth?
How to limit application memory usage?
linux;memory
Yes, VSZ is virtual memory. As to RLIMIT_AS, where did you find the paragraph quoted above? Since setrlimit(2) is a Linux system call, I do not see how it could possibly monitor malloc(3), a library function. Instead, it can only work with brk(2), sbrk(2), and mmap(2) -- this is also what its manpage (checked of Scientific Linux) suggests. However, the total amount of memory requested via these functions is virtual memory, so RLIMIT_AS indeed limits virtual memory. (This is, again, in accordance with the setrlimit(2) manpage.)Unfortunately, you cannot limit RSS under Linux (this would be ulimit -m). You can try ulimit -d (RLIMIT_DATA), but this will ignore mmap(2), typically used for large allocations. Another possibility would be to limit virtual memory, but with such a large difference between RSS and VSZ, this might be difficult.