This repository has been archived by the owner on Oct 22, 2024. It is now read-only.
This repository has been archived by the owner on Oct 22, 2024. It is now read-only.
Closed
Description
Applications running inside Kata Containers cannot use PMEM in App Direct mode because they don't get access to the original filesystem.
One idea for addressing this is to:
- map the entire partition into memory
- start QEMU such that it makes that memory range available inside the virtual machine
- mount that memory range inside the virtual machine
Details to be decided, and mostly has to be handled in Kata Containers...
Activity
pohly commentedon Jun 6, 2019
To reproduce the problem inside our QEMU virtual cluster, nested virtualization is needed. We also need changes to install and use Kata Containers. I have all of that in a branch:
https://github.com/pohly/pmem-CSI/commits/nested-virtualization
okartau commentedon Jul 24, 2019
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.3-VirtIO-PMEM
kiendinh commentedon Sep 26, 2019
Kernel 5.3.1 has been equipped with ClearLinux version 31090, and I've seen that VIRTIO_PMEM is there. Can we expect to see PMEM-CSI on Kata soon?
pohly commentedon Oct 11, 2019
My next step will be to try out Kata with virtiofs support, which will be released shortly. This might support volume-passthrough with fsdax.
pohly commentedon Nov 25, 2019
kata-containers >= 1.9.0 has support for virtiofs builtin (https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-virtio-fs-with-kata.md) when using the
kata-qemu-virtiofs
runtime class (https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/k8s-1.14/kata-qemu-virtiofs-runtimeClass.yaml):However, virtio-fs turned out to be not suitable for PMEM:
I have engaged with the Kata Container folks here: kata-containers/runtime#2262
pohly commentedon Jan 14, 2020
Functional PoC in #500, now we need the corresponding changes in Kata Containers.
pohly commentedon May 8, 2020
Kata Containers will have support in 1.11.0 (currently available as -rc0). PR #500 contains an E2E tst with Kata Containers, but it's still WIP and doesn't pass.
One problem is that by default, Kata Containers only allows VMs to have as much memory as the host has DRAM. If the host than wants to add a much larger PMEM volume, Kata Containers fails with something like:
One solution is to edit
/opt/kata/share/defaults/kata-containers/configuration-qemu.toml
after kata-deploy created it and increasememory_offset
: https://github.com/kata-containers/runtime/blob/master/cli/config/configuration-qemu.toml.in#L91Alternatively, that limit can be raised individually for each pod:
https://github.com/kata-containers/documentation/blob/master/how-to/how-to-set-sandbox-config-kata.md
pohly commentedon May 18, 2020
Should work now in "devel" (PR #500), but not tested in CI. Need to test once manually, then close this issue.
pohly commentedon Jun 10, 2020
Manual testing found a regression which then was fixed. Works now.