Posted by WanjohiRyan 11 hours ago
1) VFIO passthrough: host binds entire GPU to guest as PCI device, which only allows one VM to use the GPU, thus you sacrifice your host display too (unless you fallback to integrated graphics on cpu etc). Strongest isolation because host kernel module driver not involved.
2) virtio-gpu: guest sees paravirtual GPU and loads virgl/venus mesa driver which serializes graphics API calls and replays them on the host driver. This allows multiple VMs to use the GPU, but performance overhead can be significant, and guests can’t practically leverage lower level primitives eg NVENC without paying price of CPU readback.
3) virtio-nvgpu (this repo): guest loads standard NVIDIA user mode driver (closed source), a fake /dev/nvidia* kernel module copies ioctl bytes + handle onto queue for host kernel mode driver to execute. This also allows multiple VMs to use a GPU, but is near native speed due to low overhead. Unfortunately the tradeoff is this project has the weakest isolation, eg every guest ioctl is forwarded to the host by default, the VMM holds read/write FDs, no seccomp/caps/allowlist. With respect to There is basically no GPU related security measures here, the exposure is the same as running multiple processes using the GPU with no VM. Only caveat is these guests can’t drive a physical display, so there is some restriction of surface area but it feels incidental rather than intentional in this case.
Anyways this is a tough problem OP, I don’t want to discourage you.
Without hardware/driver support for isolation (MIG) on consumer grade NVIDIA GPUs, it won’t be possible to solve this properly for a long time.
Also a factor is that NVIDIA has no open Mesa driver to support a native context approach (guest owns GPU command buffers, host maps them) like we have for AMD/Intel.
The dream would be put the user OS in a VM in a lab, and then be able to suspend and resume seamlessly if you need to push it to a new workstation, with locally accelerated graphics available.
I suggest putting the 'multiple guests at near-native speed' use-case in the opening paragraphs of the README.
A fair comparison would be this project vs virtio.
Venus is the only one we could directly compare to, as it is the only one that supports Nvidia GPUs. vDRM works only on AMD/Intel GPUs and has a similar performance (~98% baremetal performance) to virtio-nvgpu.
Sorry. Qwerty isn't my native keyboard.
Fucukdyshd iohhfghfhgh ggbvfg gdh ghcfjh hdhvbbghb hjjgigggfv.q gfghbdbdbd cjcjcjcnn. Dr rhrjfnnf cjcucjcjjcnr rbjjdisnxbbfb hehrbrbrb. Xjcjcjcjbd rhrjrnrbrb xhfjcjcjbde hdhdbcncbfn dhdjjfjchcnbdbdbffnndnd
Now that I have you on the hook, is there any benefit to this over virtio for a single KVM passthrough situation? I previously ran a proxmox based gaming PC setup (docs here: https://github.com/mtrudel/rabble/tree/4d9329f3dd0fb09123a8f...), and was lucky enough that the GPU passthrough part of that build 'just worked'. I'd started down a path of trying to share the GPU between VMs based on a naive 'one VM owns it at a time' setup, but never really got it off the ground.
You should check out Nestri [1], another project we are working on, that helps you do exactly that. It helps run multiple gaming sessions for you and your friends on the same GPU, without anyone meddling in the other person's session. Everyone gets to stream their game to whatever desktop or device they want. It is still a work-in-progress though.
It works really well with a ~2% performance penalty. Nvproxy by google/gvisor has been doing this for years.
You should try running it yourself and see how it goes :D
And each one would have its own Nvidia driver? How do they not interfere with each other? I mean I am by no means an expert in GPU architecture or anything similar, but I thought the problems with sharing a GPU come from the differences between GPU vs CPU architecture. On the other hand, multiple processes on a host can run on the same GPU, but that is all using the same kernel. I am just very confused.
> just using PCI passthrough
However, nvgpu is meant to be used in PCI passthrough in scenarios where you would want to run and share your GPU across multiple KVM guests, or you don't want to detach the GPU from the host, and plug it into the VM you are running... you know, that little dance you do everytime you want to game inside a VM.
Its an proxmox host with local lxc drm passtrough for monitor + udev perhiperals, then cgroup the nvidia cuda api to other stream lxcs. this way i can play on my local node and friends can play on my pc remotely without anyone hogging the gpu fully.
Here is the writeup(AI gen): https://git.sahkoinsinoorikilta.fi/joona/hyper-converged-gam...
Edit: nvproxy is mentioned as the "direct inspiration" in the readme without mention of how this is different or why it doesn't use nvproxy as a backend.