Some background context on the CUDA monopoly in Kubernetes
While the AI world is abuzz with GPU and LLM wars, in the world of kubernetes the ability to share GPUs (via the SR-IOV) still has a monopoly for the large part. Enterprise grade Kubernetes (e.g. Red Hat OpenShift) still officially supports only NVIDIA for GPU slicing across your AI/ML workloads.
Are there really emerging alternatives?
There’s now emerging constructs like Moore Thread’s MT Mesh 2.0 (allowing for the ability to leverage the Kubernetes ecosystem for flexible and efficient deployment of containerized GPU resources) — this is still closed-source and not much is available in public domain. Specifications are also aplenty — CUDA vs MUSA vs Metal vs AMD ROCm vs …
A Developer and his/her localized sandbox
In the midst of all this, as a developer, wouldn’t it be nice to test and validate GPU slicing in our developer sandbox (e.g. a MacBook Pro with its inbuilt M3 Max GPU)? Alas — that GPU aint’ an NVIDIA GPU. So no SR-IOV! But wait, there’s a way! Let’s call it G-Puppetti (a spaghetti of an idea for GPU slicing in the user-space).
A simple ioreg will reveal that the Apple Metal 3 has a beast of a GPU (good enough for simplistic SLM model inferences) with a gpu-core-count of 40!
Remember that minikube has a very nifty technique to create a nvidia-gpu-device-plugin:
minikube addons enable nvidia-gpu-device-plugin\[1\]
Wouldn’t it be fun to extend this idea to metal (for Apple) with something like this?:
minikube addons enable _metal-gpu-device-plugin_\[2\]
An Architectural View of the components
Now what would that look like architecturally? Something like this:
Figure 1.0 — A simple schematic for a kubernetes plugin (a metal-device-plugin aka our G-Puppeti)
Some may argue — “Why go through this mess? Just rent out some GPU (via E2ENetworks, runpod.io, Trainium/Inferentia/Nvidia etc.).” Fair enough. But it would be nice to have an option to do some GPU development with minikube without having to reach out to any Cloud Service Provider. Just to have some self-sufficient demos. So well, why not?
How is this different from SR-IOV?
Who’s this for?
If you’re an avid developer wanting to try out AI-on-the-Edge styled use-cases and you’re still thinking about that Jetson Orin / Jetson Nano; but want to try out things on the GPU available at your disposal (that Metal M3 Mac), be my guest and use this technique with k3s or k8s. Its about time that enterprise grade kubernetes provides simpler ways for GPU abstractions (and looks at options beyond CUDA) — if only to break the N\*word monopoly (Of course I meant NVIDIA 😉).
The nuts and bolts of it — at a high level
So what exactly is the plugin? It implements the gRPC interface defined by Kubernetes. The device plugin must conform to Kubernetes’ Device Plugin API, which is based on the gRPC-based protocol. We’re essentially implementing a gRPC service that Kubelet talks to. It is therefore nothing but a custom gRPC server. Kubernetes auto-discovers the device plugin via the Unix socket. It exposes methods like:
Register()ListAndWatch()Allocate()GetDevicePluginOptions()PreStartContainer()GetPreferredAllocation()(if supported)
It communicates over a Unix domain socket
- The plugin creates a Unix socket (
/var/lib/kubelet/device-plugins/metal-gpu.sock). - The Kubelet (Kubernetes node agent) connects to this socket using gRPC.
Kubelet calls the plugin’s gRPC Methods
- Kubelet calls
ListAndWatch()to get available devices. - When a pod requests a GPU, Kubelet calls
Allocate(). - The response tells Kubelet how to expose the Metal GPU inside the pod.
And so…why did the GPU break up with CUDA?
Because it felt too controlled and wanted some OpenCL-arity! 😆
References:
\[1\] NVIDIA device plugin for kubernetes — https://github.com/NVIDIA/k8s-device-plugin
\[2\] A POC to create a Metal (Apple) plugin for kubernetes — https://github.com/Quest1Codes/metal-device-plugin
#llm #slm #gpu #nvidia #cuda #musa #AMDRoCm #OpenCL #Quest1 #onaquest #developerfreedom #developerchoice #k8s #kubernetes #sriov #metal #metal3 #applesilicon #apple #docker #podman #openshift