runtime-rs: device manager for runtime-rs #5441

Tim-0731-Hzt · 2022-10-17T03:21:28Z

support device manager for runtime-rs

Fixes:#5375
Signed-off-by: Zhongtao Hu zhongtaohu.tim@linux.alibaba.com

quanweiZhou

Few comments, and some advise:

Give a picture of the design.
Push code after make check and fix the clippy, I found some advise can check by make check.

src/agent/src/device.rs

src/agent/src/mount.rs

src/runtime-rs/crates/hypervisor/src/device/block.rs

src/runtime-rs/crates/hypervisor/src/device/mod.rs

src/runtime-rs/crates/hypervisor/src/device_manager.rs

src/agent/src/mount.rs

src/agent/src/device.rs

src/runtime-rs/crates/hypervisor/src/device/block.rs

src/runtime-rs/crates/hypervisor/src/device/generic.rs

src/runtime-rs/crates/hypervisor/src/device_manager.rs

src/runtime-rs/crates/hypervisor/src/utils.rs

liubin · 2022-10-18T08:42:03Z

src/runtime-rs/crates/hypervisor/src/device/generic.rs

+pub struct GenericDevice {
+    id: String,
+    device_info: GenericConfig,
+    ref_count: u64,


What's the defferent between ref_count and attach_count.

I think just use one of them would be enough to handle the device attach number.

src/runtime-rs/crates/hypervisor/src/device_manager.rs

Tim-0731-Hzt · 2022-10-24T07:00:01Z

/test

Implements get-volume-stats trait for sandbox and add RPC calls to agent. Also added type conversions in trans.rs Fixes kata-containers#5441 Signed-off-by: Tingzhou Yuan <tzyuan15@bu.edu>

Implements resize-volume trait for sandbox and add RPC calls to agent. Also added type conversions in trans.rs and replaced `ResizeRequest` in shim-mgmt handler with the agent version to reduce redundancy. The actual handler for the resize request is also added, and will be implemented in the next commit. Fixes kata-containers#5441 Signed-off-by: Tingzhou Yuan <tzyuan15@bu.edu>

jodh-intel

Thanks @Tim-0731-Hzt - a few comments...

src/runtime-rs/crates/hypervisor/src/device_manager.rs

jodh-intel · 2022-10-28T13:02:47Z

src/runtime-rs/crates/hypervisor/src/utils.rs

+    let mut i = 0usize;
+    while i < suff_len && index >= 0 {
+        let letter: u8 = b'a' + (index % base) as u8;
+        disk_letters[i] = letter;
+        index = (index / base) - 1;
+        i += 1;
+    }
+    if index >= 0 {
+        return Err(anyhow!("Index not supported"));
+    }


This chunk looks like it could be a separate function. That would allow it to be unit-tested with various values.

jodh-intel · 2022-10-28T13:04:35Z

src/runtime-rs/crates/resource/src/volume/block_volume.rs

+                &mut GenericConfig {
+                    host_path: m.source.clone(),
+                    container_path: m.destination.clone(),
+                    dev_type: "b".to_string(),
+                    major: stat::major(fstat.st_rdev) as i64,
+                    minor: stat::minor(fstat.st_rdev) as i64,
+                    file_mode: 0,
+                    uid: 0,
+                    gid: 0,
+                    id: "".to_string(),
+                    bdf: None,
+                    driver_options: options,
+                    io_limits: None,
+                    ..Default::default()


Some of these specified values are already included in ..Default::default() so I think this (and other instances) could be simplified?

Some variables in the struct is not specified here, this pr just support the basic functionality, so we still need them.

src/runtime-rs/crates/runtimes/virt_container/src/container_manager/container.rs

jodh-intel · 2022-10-28T13:09:22Z

src/agent/src/device.rs

+    devpath: &str,
+) -> Result<()> {
+    let devname = devpath
+        .strip_prefix("/dev/")


Is devpath guaranteed to be the fully expanded (canonical) value for the path? If not, what if something like /dev/foo/bar/../../../sbin/init was specified?

Yes, it is passed as the source in the storage, and the source path is set on the runtime side, we get this by get_virt_drive_name function. So it can't be like /dev/foo/bar/../../../sbin/init.

jodh-intel · 2022-10-28T13:14:49Z

src/agent/src/device.rs

@@ -401,6 +401,48 @@ async fn get_vfio_device_name(sandbox: &Arc<Mutex<Sandbox>>, grp: IommuGroup) ->
    Ok(format!("{}/{}", SYSTEM_DEV_PATH, &uev.devname))
 }

+#[derive(Debug)]
+struct MmioBlockMatcher {
+    suffix: String,


ooi, why can't we match on the entire udev path?

Could you add some unit tests using fake Uevent events? That would help understanding and also clarify the point above.

Normally, there are two path for a device, it could be sys/block/vda, and sys/devices/ ......./virtio0/block/vda, the first one will point to the second one, so match the suffix would be enough.

Tim-0731-Hzt · 2022-10-31T05:09:24Z

/test

Tim-0731-Hzt · 2022-10-31T06:05:03Z

/test

studychao · 2023-05-22T03:37:00Z

/test

Tim-0731-Hzt · 2023-05-22T05:03:38Z

/test-s390x

snir911 · 2023-05-22T07:37:19Z

/retest-s390x

This is an initial, WiP port of FC on the rust runtime of kata containers. It is built on top of the `device_manager_dev` branch [kata-containers#5441], since block device support for the container rootfs is a prerequisite for FC sandboxes (FC does not support virtio-fs). Current issues: - network namespaces are being deleted (probably some threading mixup), hence, although there's a network device present, there's no network support (the namespace in the host is deleted, so no tap0_kata, tc etc.) - we do not kill the FC process leaving instances running even after container shutdown. This leaves running firecracker instances, keeps the devmapper snapshot active etc. - no jailer support yet Assumes FC binaries are in `$INSTALL_PREFIX/bin` Signed-off-by: George Pyrros <gpyrros@nubificus.co.uk> Signed-off-by: Anastassios Nanos <ananos@nubificus.co.uk>

Support device manager for runtime-rs, add block device handler for device manager Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

add the trait implementation for vfio device, Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

add the trait implementation for vhost-user device Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com>

As dragonball support hotplug virtio-mmio device, we should handle it in agent Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

support devmapper for block rootfs Fixes: kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

support block volume in runtime-rs Fixes: kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

support linux device in runtime-rs Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Support remove device after container stop Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

use device type to store the config information for different kind of devices Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

add virtio-blk-mmio option for dragonball Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Tim-0731-Hzt · 2023-05-22T16:59:51Z

/test

Tim-0731-Hzt · 2023-05-23T00:57:27Z

/test-fc

studychao · 2023-05-23T04:26:56Z

/test

This is an initial, WiP port of FC on the rust runtime of kata containers. It is built on top of the `device_manager_dev` branch [kata-containers#5441], since block device support for the container rootfs is a prerequisite for FC sandboxes (FC does not support virtio-fs). Current issues: - network namespaces are being deleted (probably some threading mixup), hence, although there's a network device present, there's no network support (the namespace in the host is deleted, so no tap0_kata, tc etc.) - no jailer support yet Assumes FC binaries are in `$INSTALL_PREFIX/bin` Signed-off-by: George Pyrros <gpyrros@nubificus.co.uk> Signed-off-by: Anastassios Nanos <ananos@nubificus.co.uk>

This is an initial, WiP port of FC on the rust runtime of kata containers. It is built on top of the `device_manager_dev` branch [kata-containers#5441], since block device support for the container rootfs is a prerequisite for FC sandboxes (FC does not support virtio-fs). Current issues: - We should enter the network namespace in the forked process. For now, we use the pre_exec method to run a closure just before the exec of the external hypervisor. This way, the hypervisor belongs to the new namespace, whereas the runtime does not. There's an issue with nerdctl (not present in k8s deployments) where the netns file is left dangling in the host. Useful read: https://serverfault.com/a/961592 - no jailer support yet Assumes FC binaries are in `$INSTALL_PREFIX/bin` Signed-off-by: George Pyrros <gpyrros@nubificus.co.uk> Signed-off-by: Anastassios Nanos <ananos@nubificus.co.uk>

Tim-0731-Hzt added the runtime-rs label Oct 17, 2022

katacontainersbot added the size/huge Largest and most complex task (probably needs breaking into small pieces) label Oct 17, 2022

Tim-0731-Hzt requested review from quanweiZhou, Tim-Zhang and liubin October 17, 2022 03:25

Tim-0731-Hzt force-pushed the device_manager_dev branch 4 times, most recently from 3072b84 to cfd3d10 Compare October 17, 2022 09:24

quanweiZhou reviewed Oct 17, 2022

View reviewed changes

Tim-0731-Hzt force-pushed the device_manager_dev branch 2 times, most recently from 0fffa47 to 6f504b9 Compare October 17, 2022 11:23

liubin reviewed Oct 18, 2022

View reviewed changes

Tim-0731-Hzt force-pushed the device_manager_dev branch from 6f504b9 to 2a3a513 Compare October 20, 2022 03:45

Tim-0731-Hzt mentioned this pull request Oct 20, 2022

Feat: implementation of kata-ctl direct-volume operations #5467

Merged

quanweiZhou reviewed Oct 20, 2022

View reviewed changes

src/runtime-rs/crates/hypervisor/src/device_manager.rs Outdated Show resolved Hide resolved

Tim-0731-Hzt force-pushed the device_manager_dev branch from 2a3a513 to 6fc3a9b Compare October 24, 2022 04:11

Tim-0731-Hzt requested review from liubin and quanweiZhou October 24, 2022 04:12

Tim-0731-Hzt force-pushed the device_manager_dev branch from 6fc3a9b to 0268d3e Compare October 24, 2022 05:25

Tim-0731-Hzt force-pushed the device_manager_dev branch from 0268d3e to 9dda6e6 Compare October 25, 2022 01:51

Tim-0731-Hzt mentioned this pull request Oct 26, 2022

runtime-rs: implement direct volume handler for kata-ctl #5369

Closed

jodh-intel reviewed Oct 28, 2022

View reviewed changes

Tim-0731-Hzt force-pushed the device_manager_dev branch from 9dda6e6 to fc9008e Compare October 31, 2022 02:36

lifupan added the no-forward-port-needed label May 22, 2023

Tim-0731-Hzt force-pushed the device_manager_dev branch from 97e8dcc to 8ceae41 Compare May 22, 2023 02:01

katacontainersbot added size/huge Largest and most complex task (probably needs breaking into small pieces) and removed size/tiny Smallest and simplest task labels May 22, 2023

Tim-0731-Hzt force-pushed the device_manager_dev branch from 8ceae41 to 65034e3 Compare May 22, 2023 02:53

Tim-0731-Hzt added 10 commits May 23, 2023 00:53

runtime-rs: device manager

e4c5c74

Support device manager for runtime-rs, add block device handler for device manager Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: implement trait for vfio device

cc9c915

add the trait implementation for vfio device, Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: implement trait for vhost-user device

6e273d6

add the trait implementation for vhost-user device Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com>

agent: handle hotplug virtio-mmio device

b076d46

As dragonball support hotplug virtio-mmio device, we should handle it in agent Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: support block rootfs

a8bfac9

support devmapper for block rootfs Fixes: kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: block volume

fe9ec67

support block volume in runtime-rs Fixes: kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: support linux device

f16012a

support linux device in runtime-rs Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: remove device

6800d30

Support remove device after container stop Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: add devicetype enum

f9bded4

use device type to store the config information for different kind of devices Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: add virtio-blk-mmio

4719802

add virtio-blk-mmio option for dragonball Fixes:kata-containers#5375 Signed-off-by: Zhongtao Hu <zhongtaohu.tim@linux.alibaba.com> Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Tim-0731-Hzt force-pushed the device_manager_dev branch from 65034e3 to 4719802 Compare May 22, 2023 16:59

katacontainersbot added size/tiny Smallest and simplest task and removed size/huge Largest and most complex task (probably needs breaking into small pieces) labels May 22, 2023

lifupan merged commit 1703365 into kata-containers:main May 23, 2023

Tim-0731-Hzt deleted the device_manager_dev branch June 2, 2023 03:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime-rs: device manager for runtime-rs #5441

runtime-rs: device manager for runtime-rs #5441

Tim-0731-Hzt commented Oct 17, 2022

quanweiZhou left a comment

liubin Oct 18, 2022

Tim-0731-Hzt Oct 19, 2022

Tim-0731-Hzt commented Oct 24, 2022

jodh-intel left a comment

jodh-intel Oct 28, 2022

jodh-intel Oct 28, 2022

Tim-0731-Hzt Oct 31, 2022

jodh-intel Oct 28, 2022

Tim-0731-Hzt Oct 31, 2022 •

edited

Loading

jodh-intel Oct 28, 2022

Tim-0731-Hzt Oct 31, 2022 •

edited

Loading

Tim-0731-Hzt commented Oct 31, 2022

Tim-0731-Hzt commented Oct 31, 2022

studychao commented May 22, 2023

Tim-0731-Hzt commented May 22, 2023

snir911 commented May 22, 2023

Tim-0731-Hzt commented May 22, 2023

Tim-0731-Hzt commented May 23, 2023

studychao commented May 23, 2023

runtime-rs: device manager for runtime-rs #5441

runtime-rs: device manager for runtime-rs #5441

Conversation

Tim-0731-Hzt commented Oct 17, 2022

quanweiZhou left a comment

Choose a reason for hiding this comment

liubin Oct 18, 2022

Choose a reason for hiding this comment

Tim-0731-Hzt Oct 19, 2022

Choose a reason for hiding this comment

Tim-0731-Hzt commented Oct 24, 2022

jodh-intel left a comment

Choose a reason for hiding this comment

jodh-intel Oct 28, 2022

Choose a reason for hiding this comment

jodh-intel Oct 28, 2022

Choose a reason for hiding this comment

Tim-0731-Hzt Oct 31, 2022

Choose a reason for hiding this comment

jodh-intel Oct 28, 2022

Choose a reason for hiding this comment

Tim-0731-Hzt Oct 31, 2022 • edited Loading

Choose a reason for hiding this comment

jodh-intel Oct 28, 2022

Choose a reason for hiding this comment

Tim-0731-Hzt Oct 31, 2022 • edited Loading

Choose a reason for hiding this comment

Tim-0731-Hzt commented Oct 31, 2022

Tim-0731-Hzt commented Oct 31, 2022

studychao commented May 22, 2023

Tim-0731-Hzt commented May 22, 2023

snir911 commented May 22, 2023

Tim-0731-Hzt commented May 22, 2023

Tim-0731-Hzt commented May 23, 2023

studychao commented May 23, 2023

Tim-0731-Hzt Oct 31, 2022 •

edited

Loading

Tim-0731-Hzt Oct 31, 2022 •

edited

Loading