...
Virtlet is a Kubernetes CRI (Container Runtime Interface) implementation for running VM-based pods on Kubernetes clusters. (CRI is what enables Kubernetes to run non-Docker flavors of containers, such as Rkt.) For the sake of simplicity of deployment, Virtlet itself runs as a DaemonSet, essentially acting as a hypervisor and making the CRI proxy (Provides the possibility of mixing docker-shim and VM based workloads on the same k8s node) available to run the actual VMs This way, it’s possible to have both Docker and non-Docker pods run on the same node.
...
| draw.io Diagram | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Components
- Virtlet manager: Implements the CRI interface for virtualization and image handling
- Libvirt: The standard instance of libvirt for KVM.
- vmwrapper: Responsible for preparing the environment for the emulator
- Emulator: Currently qemu with KVM support (with possibility of disabling KVM for nested virtualization tests)
- ...
...
When testing the QAT sriov support condition with the officer virtlet image, together with QAT device plugin. we take thie simple straightforward method that add the resource name qat.intel.com/generic advertised by the QAT device plugin to fileds spec.containers.resource.limits and spec.containers.resource.requests with value "1". It works correctly in plain kubernetes pods. But in a virtlet vm pod, we encountered the conflict caused by the configuration transformed between virtual machine and pod by virtlet. The issues is that when allocating a QAT vf device to virtlet vm pod, Kubelet will add the extended device to kubeapi.PodSandboxConfig.Devices (k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2 - v1.14). Then virtlet will incorrectly transforms all these devices to its volume devices and considers them as block disk with disk drivers bound to them later.
for _, dev := range in.Config.Devices { |
|---|
It causes the errors that too many disks, disks' reading issues, denied permission and so on after a vm pod starts. And regardless of this, I want assign QAT vf to virtlet pod by pci-passthrough. So I want add corresponding fileds into libvirt instance domain xml created by virtlet. After code analysis, virtlet is a cri implentment and in its createDomain(config *types.VMConfig) *libvirtxml.Domain (pkg/libvirttools/virtualization.go) I detect the xml file creation and find it is using the libvirtxml "github.com/libvirt/libvirt-go-xml" go module. So the whole work flow is clear now and I can fix it then.
domain := &libvirtxml.Domain{ ... |
|---|
Fix
continue