After some testing seems like hyper-v video driver is leaking memory.

The setup:
Host OS: Windows 10 + Hyper-V
Guest OS: a linux distro

Steps to reproduce (archlinux example):
1. Setup a minimal system (and xorg-server, xorg-server-utils, xterm)
2. Launch "top" and wait. (I waited for an hour, and have seen no changes in memory allocation)
3. Use "startx" to launch basic X11 session.
4. Launch "top" and wait. (It takes 5-10 minutes for me, for memory to be mostly occupied, about 20% is left)

Steps to reproduce (ubuntu server example):
1. Setup a minimal system (and xorg, xterm)
2. Launch "top" and wait. (I waited for an hour, and have seen no changes in memory allocation)
3. Use "startx" to launch basic X11 session.
4. Launch "top" and wait. (It takes 5-10 minutes for me, for memory to be mostly occupied, about 15% is left)

As seen from the 2 examples above, we have an issue across different distros.
Also I've set up the same 2 distros on VirtualBox VMs and had no issues with memory allocation, which point into Hyper-V direction.
As seen previously the issue is not occurring until X11 is not launched, which points into display-related direction.
Also to note, I tried to use top/htop tools to pinpoint a program, which leaks memory. But the most used memory percentage is 0.3 at the time when 80% of RAM is used, which points into non-program direction (I assumed driver).

Thanks.