glibc malloc uses internally mmap and brk. "ulimit -d" is used only by brk and sbrk. Consider using "ulimit -v" which imposes a limit on a virtual memory size (not resident). It is used by mmap, brk and mremap (also stack expansion).

After executing "ulimit -d 8192" and executing the testcase in the same terminal the column DATA of process viewer like top and htop grows to ~2 GiB ("ulimit -s" shows 8192).

Did you try "ulimit -v 8192" as suggested previously. It seems to be closer to what you expect.

I'm trying to limit the program break to force glibc using mmap if the program break is out of memory while having no restrictions on the memory consumption (so limiting the virtual memory would not work here). While trying to do this I have just noticed that limiting the program break seems not to work.

Created attachment 189191 [details]
This program logs brk state.

Please run the program memlog.c (from attachment 189191 [details] above) in a process having the "ulimit -d 8192" set. It is based on a program from:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1462853
I've just added the logging possibility. Please attach the resulting mem.log.

Here is the log: http://www.file-upload.net/download-10946174/mem.log.html

The log shows that the process heap is in fact limited by "ulimit -d 8192". When malloc reaches the heap limit it creates the new memory region for the remaining memory. The new memory region is not under the process data limit since it has not been allocated via brk/sbrk. The top's DATA column includes both the heap and the additional memory region.

Thanks for the information. I think I figured out the problem: The manpage of htop descibes the DATA column as the data segment size + stack size while the manpage of top descibes it as anything except the code size. I'm reporting it then to htop and marking this ticket as invalid.

The author of htop noticed that the manpage of proc(5) shows at "/proc/[pid]/statm" that the data column is calculated with "data + stack". Isn't that then wrong too?

It works as documented. The /proc/[pid]/status:VmData excludes the stack.

To clarify: data + stack means data segment size + stack size? On executing my original testcase and limiting the data segment size and stack size with ulimit to 8 MiB I'm getting the number 529681 for the 6th column of the related statm file (which would mean that the 6th column shown in the statm file is not the data segment size + stack size).

No, the "data" is not a "data segment size".