Distro: Archlinux 5.6.10-arch1-1

The machine hangs when it reaches 100% memory usage.

To be more exact, the performance drops to 1hz speed, which means that you have to wait more than 1 hour for a bash script to be executed manually and kill all the possible processes involved that consume too much memory.
I don't have swap.
It has also already been tested with a swap file using a mounted loop, even if both fill to 100%, the machine hangs.
What gets filled is the memory used and not the buffer/cache.

Using a test code, it never returns an 'out of memory' or OOM Killer never responds:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    for (;;) {
        char *buf = (char *)malloc(65535);
        if (!buf) {
            fprintf(stderr, "memory maxed out!\n");
            usleep(1000000);
        }usleep(100);
    }
    return 0;
}

possible non-relevant information:
I use a USB pendrive with a fake HDD configured from the BIOS. It's because I don't use swap.