Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak - ros2 humble #262

Closed
firesurfer opened this issue Jul 12, 2023 · 19 comments
Closed

Memory leak - ros2 humble #262

firesurfer opened this issue Jul 12, 2023 · 19 comments
Assignees
@flynneva

Comments

@firesurfer
Copy link
Contributor

Issue

usb_cam node allocates memory over time without freeing it until the systems hangs and the oom kills it.

[20524.624725] Out of memory: Killed process 267615 (usb_cam_node_ex) total-vm:31232704kB, anon-rss:27694492kB, file-rss:12320kB, shmem-rss:0kB, UID:1000 pgtables:59852kB oom_score_adj:0

System

Ubuntu 22.04
ROS2 Humble (binary install)
usb_cam: 0.6.0

v4l2-ctl -D

Driver Info:
	Driver name      : uvcvideo
	Card type        : HD Pro Webcam C920
	Bus info         : usb-0000:0c:00.3-2
	Driver version   : 5.19.17
	Capabilities     : 0x84a00001
		Video Capture
		Metadata Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : uvcvideo
	Model            : HD Pro Webcam C920
	Serial           : 
	Bus info         : usb-0000:0c:00.3-2
	Media version    : 5.19.17
	Hardware revision: 0x0000000c (12)
	Driver version   : 5.19.17
Interface Info:
	ID               : 0x03000002
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : HD Pro Webcam C920
	Function         : V4L2 I/O
	Flags            : default
	Pad 0x01000007   : 0: Sink
	  Link 0x0200001c: from remote pad 0x100000a of entity 'Processing 3' (Video Pixel Formatter): Data, Enabled, Immutable

Camera yaml:

    ros__parameters:
      video_device: "/dev/video0"
      framerate: 30.0
      io_method: "mmap"
      frame_id: "camera_1"
      pixel_format: "mjpeg2rgb"
#      color_format: "yuv422p" #Will result in less then 30fps on uncompressed topic
      image_width: 1920
      image_height: 1080
      camera_name: "logitech_c920"
      camera_info_url: "package://winder_camera/config/camera_1_info.yaml"

Further info

Camera image is displayed in rviz

Let me know if you need futher information
@flynneva
Copy link
Collaborator

@firesurfer thanks for reporting. I'll dig into this over the weekend when I have some time 👍🏼

@flynneva flynneva mentioned this issue Jul 15, 2023
Merged
@flynneva
Copy link
Collaborator

@firesurfer I've addressed the major memory leak issues with #264 (or at least a lot of them 😅 ) There are still a few smaller ones left, but I've run out of time today and need to work on some other things this weekend.

Hopefully those fixes are "good enough" for you for now - I'll leave this issue open even after #264 is merged to address the rest of the memory issues later on.

@firesurfer
Copy link
Contributor Author

@flynneva I just tried #264 on my system and I am sorry to disappoint you. It still eats memory rather rapidly. After like 10min it already used like 8 GB.

@flynneva flynneva mentioned this issue Jul 23, 2023
Merged
@flynneva
Copy link
Collaborator

@firesurfer I believe I've addressed the last memory leak related to this drivers code in #268. There are still 3 memory leaks reported by asan but those are from lower-level ROS 2 packages I believe and not something I can address here in this repo.

I've added a SANITIZE option to the CMakelists.txt file as well so others can quickly and easily address memory leaks too.

Maybe down the road I can add a CI job that uses this option to auto-detect memory leaks in PRs

@firesurfer
Copy link
Contributor Author

firesurfer commented Jul 24, 2023 •
edited
Loading

@flynneva I just tested #268
It looks in general way better but still starts consuming memory without freeing it (at least way slower than before).
The interesting thing is: I had to subscribe to the raw topic (didnt try the other ones) for this to happen.

I ran valgrind with leak-check=yes, perhaps the output can help you.

==14672== Invalid free() / delete / delete[] / realloc()
==14672==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x7B99565: av_buffer_unref (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x693C098: av_packet_unref (in /usr/lib/x86_64-linux-gnu/libavcodec.so.58.134.100)
==14672==    by 0x6703ABA: std::_Sp_counted_ptr_inplace<usb_cam::formats::MJPEG2RGB, std::allocator<usb_cam::formats::MJPEG2RGB>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6701731: usb_cam::UsbCam::~UsbCam() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6605DA3: usb_cam::UsbCamNode::~UsbCamNode() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x10C4DE: main (in /home/user/workspace/webcam_ws/install/usb_cam/lib/usb_cam/usb_cam_node_exe)
==14672==  Address 0x1b9c5000 is in a rw- mapped file /dev/video0 segment
==14672== 
==14672== 
==14672== HEAP SUMMARY:
==14672==     in use at exit: 161,055,721 bytes in 4,224 blocks
==14672==   total heap usage: 163,301 allocs, 159,078 frees, 1,892,940,843 bytes allocated
==14672== 
==14672== 16 bytes in 1 blocks are possibly lost in loss record 122 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x4013E4D: malloc (rtld-malloc.h:56)
==14672==    by 0x4013E4D: allocate_dtv_entry (dl-tls.c:684)
==14672==    by 0x4013E4D: allocate_and_init (dl-tls.c:709)
==14672==    by 0x4013E4D: tls_get_addr_tail (dl-tls.c:907)
==14672==    by 0x401826B: __tls_get_addr (tls_get_addr.S:55)
==14672==    by 0x1980A147: ???
==14672==    by 0x40021A1: call_destructors (dl-close.c:129)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x4002635: _dl_close_worker.part.0.isra.0 (dl-close.c:292)
==14672==    by 0x40032A1: _dl_close_worker (dl-close.c:150)
==14672==    by 0x40032A1: _dl_close (dl-close.c:818)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672==    by 0x4D83ED7: dlclose@@GLIBC_2.34 (dlclose.c:31)
==14672== 
==14672== 64 bytes in 1 blocks are definitely lost in loss record 225 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x6703F09: usb_cam::formats::MJPEG2RGB::MJPEG2RGB(int const&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x670489C: usb_cam::UsbCam::set_pixel_format_from_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x67014B8: usb_cam::UsbCam::configure(usb_cam::parameters_t&, usb_cam::utils::io_method_t const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x660A04D: usb_cam::UsbCamNode::init() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x660BE6A: usb_cam::UsbCamNode::UsbCamNode(rclcpp::NodeOptions const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x661399D: rclcpp_components::NodeFactoryTemplate<usb_cam::UsbCamNode>::create_node_instance(rclcpp::NodeOptions const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x10BDB4: main (in /home/user/workspace/webcam_ws/install/usb_cam/lib/usb_cam/usb_cam_node_exe)
==14672== 
==14672== 128 bytes in 2 blocks are definitely lost in loss record 360 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x400F21D: malloc (rtld-malloc.h:56)
==14672==    by 0x400F21D: resize_scopes (dl-open.c:295)
==14672==    by 0x400F21D: dl_open_worker_begin (dl-open.c:707)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400DF99: dl_open_worker (dl-open.c:782)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672==    by 0x4D84747: dlopen_implementation (dlopen.c:71)
==14672==    by 0x4D84747: dlopen@@GLIBC_2.34 (dlopen.c:81)
==14672==    by 0x4A9D8CB: rcutils_load_shared_library (in /opt/ros/humble/lib/librcutils.so)
==14672== 
==14672== 143 (64 direct, 79 indirect) bytes in 1 blocks are definitely lost in loss record 366 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x50E3A23: ??? (in /opt/ros/humble/lib/librosidl_typesupport_cpp.so)
==14672==    by 0x55FC888: ???
==14672==    by 0x55DA167: ???
==14672==    by 0x55DA366: ???
==14672==    by 0x55DBFEF: ???
==14672==    by 0x4F47849: rcl_node_init (in /opt/ros/humble/lib/librcl.so)
==14672==    by 0x496408C: rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_s const&, bool, bool) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x495BFD6: rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x495D207: rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x660B12A: usb_cam::UsbCamNode::UsbCamNode(rclcpp::NodeOptions const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x661399D: rclcpp_components::NodeFactoryTemplate<usb_cam::UsbCamNode>::create_node_instance(rclcpp::NodeOptions const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672== 
==14672== 151 (64 direct, 87 indirect) bytes in 1 blocks are definitely lost in loss record 373 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x50E3A23: ??? (in /opt/ros/humble/lib/librosidl_typesupport_cpp.so)
==14672==    by 0x55FC888: ???
==14672==    by 0x55DA167: ???
==14672==    by 0x55DA366: ???
==14672==    by 0x55DA61F: ???
==14672==    by 0x4F468B2: rcl_publisher_init (in /opt/ros/humble/lib/librcl.so)
==14672==    by 0x49B48AD: rclcpp::PublisherBase::PublisherBase(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rosidl_message_type_support_t const&, rcl_publisher_options_s const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x1B041D0B: ???
==14672==    by 0x1B0423B0: ???
==14672==    by 0x4971F4B: rclcpp::node_interfaces::NodeTopics::create_publisher(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::PublisherFactory const&, rclcpp::QoS const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x1B040C6F: ???
==14672== 
==14672== 296 (192 direct, 104 indirect) bytes in 1 blocks are definitely lost in loss record 410 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x55D20E9: ???
==14672==    by 0x55DFEE7: ???
==14672==    by 0x4F4DBF7: rcl_wait_set_init (in /opt/ros/humble/lib/librcl.so)
==14672==    by 0x4944ABD: rclcpp::Executor::Executor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x494832C: rclcpp::executors::SingleThreadedExecutor::SingleThreadedExecutor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x10BA1E: main (in /home/user/workspace/webcam_ws/install/usb_cam/lib/usb_cam/usb_cam_node_exe)
==14672== 
==14672== 327 (16 direct, 311 indirect) bytes in 1 blocks are definitely lost in loss record 411 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1AFBEE0C: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 329 (16 direct, 313 indirect) bytes in 1 blocks are definitely lost in loss record 412 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1AFBEFF1: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 345 (16 direct, 329 indirect) bytes in 1 blocks are definitely lost in loss record 416 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1B03A5AC: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 347 (16 direct, 331 indirect) bytes in 1 blocks are definitely lost in loss record 417 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1AF3E1CC: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 348 (16 direct, 332 indirect) bytes in 1 blocks are definitely lost in loss record 418 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1B03A791: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 350 (16 direct, 334 indirect) bytes in 1 blocks are definitely lost in loss record 419 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x1AF3E3B1: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 364 (16 direct, 348 indirect) bytes in 1 blocks are definitely lost in loss record 422 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x137958DC: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 367 (16 direct, 351 indirect) bytes in 1 blocks are definitely lost in loss record 423 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x13795AC1: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672==    by 0x4D841AD: _dlerror_run (dlerror.c:138)
==14672== 
==14672== 496 (16 direct, 480 indirect) bytes in 1 blocks are definitely lost in loss record 428 of 476
==14672==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x6610F79: void class_loader::impl::registerPlugin<rclcpp_components::NodeFactoryTemplate<usb_cam::UsbCamNode>, rclcpp_components::NodeFactory>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x660538A: __static_initialization_and_destruction_0(int, int) [clone .constprop.0] (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672== 
==14672== 608 bytes in 1 blocks are possibly lost in loss record 438 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x4013FF0: malloc (rtld-malloc.h:56)
==14672==    by 0x4013FF0: _dl_resize_dtv (dl-tls.c:498)
==14672==    by 0x4014C0F: _dl_update_slotinfo (dl-tls.c:812)
==14672==    by 0x4014D0B: update_get_addr (dl-tls.c:919)
==14672==    by 0x401826B: __tls_get_addr (tls_get_addr.S:55)
==14672==    by 0x1980A147: ???
==14672==    by 0x40021A1: call_destructors (dl-close.c:129)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x4002635: _dl_close_worker.part.0.isra.0 (dl-close.c:292)
==14672==    by 0x40032A1: _dl_close_worker (dl-close.c:150)
==14672==    by 0x40032A1: _dl_close (dl-close.c:818)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x4E68CF2: _dl_catch_error (dl-error-skeleton.c:227)
==14672== 
==14672== 2,616 (32 direct, 2,584 indirect) bytes in 1 blocks are definitely lost in loss record 456 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x56FFBAC: ???
==14672==    by 0x56F860D: ???
==14672==    by 0x56E03D3: ???
==14672==    by 0x56D6A7C: ???
==14672==    by 0x56E5DD1: ???
==14672==    by 0x55DFD81: ???
==14672==    by 0x4F31E05: ??? (in /opt/ros/humble/lib/librcl.so)
==14672==    by 0x49531CF: rclcpp::GuardCondition::GuardCondition(std::shared_ptr<rclcpp::Context>, rcl_guard_condition_options_s) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x4944617: rclcpp::Executor::Executor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x494832C: rclcpp::executors::SingleThreadedExecutor::SingleThreadedExecutor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x10BA1E: main (in /home/user/workspace/webcam_ws/install/usb_cam/lib/usb_cam/usb_cam_node_exe)
==14672== 
==14672== 8,264 bytes in 1 blocks are possibly lost in loss record 464 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x56FFBAC: ???
==14672==    by 0x56C5A8C: ???
==14672==    by 0x56D6A77: ???
==14672==    by 0x56E5DD1: ???
==14672==    by 0x55DFD81: ???
==14672==    by 0x4F31E05: ??? (in /opt/ros/humble/lib/librcl.so)
==14672==    by 0x49531CF: rclcpp::GuardCondition::GuardCondition(std::shared_ptr<rclcpp::Context>, rcl_guard_condition_options_s) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x4944617: rclcpp::Executor::Executor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x494832C: rclcpp::executors::SingleThreadedExecutor::SingleThreadedExecutor(rclcpp::ExecutorOptions const&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x10BA1E: main (in /home/user/workspace/webcam_ws/install/usb_cam/lib/usb_cam/usb_cam_node_exe)
==14672== 
==14672== 11,296 bytes in 1 blocks are definitely lost in loss record 466 of 476
==14672==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x181AC51B: ???
==14672==    by 0x18183989: ???
==14672==    by 0x15A49C77: ???
==14672==    by 0x15A49D7A: ???
==14672==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==14672==    by 0x4006567: call_init (dl-init.c:33)
==14672==    by 0x4006567: _dl_init (dl-init.c:117)
==14672==    by 0x4E68C84: _dl_catch_exception (dl-error-skeleton.c:182)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==14672==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==14672==    by 0x4E68C27: _dl_catch_exception (dl-error-skeleton.c:208)
==14672==    by 0x400E34D: _dl_open (dl-open.c:883)
==14672==    by 0x4D846BB: dlopen_doit (dlopen.c:56)
==14672== 
==14672== 38,848 (14,568 direct, 24,280 indirect) bytes in 607 blocks are definitely lost in loss record 472 of 476
==14672==    at 0x484DE30: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x484DF92: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x7BBC9E4: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x7BBCDC1: av_mallocz (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x7B99347: av_buffer_create (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x693B502: av_packet_from_data (in /usr/lib/x86_64-linux-gnu/libavcodec.so.58.134.100)
==14672==    by 0x6703B98: usb_cam::formats::MJPEG2RGB::convert(char const*&, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6700434: usb_cam::UsbCam::process_image(char const*, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x67007C0: usb_cam::UsbCam::read_frame() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6701852: usb_cam::UsbCam::grab_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6608A3C: usb_cam::UsbCamNode::take_and_send_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x6609117: usb_cam::UsbCamNode::update() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672== 
==14672== 3,791,587 bytes in 16 blocks are possibly lost in loss record 474 of 476
==14672==    at 0x48487A9: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x7B998C7: av_buffer_realloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x693B22D: av_new_packet (in /usr/lib/x86_64-linux-gnu/libavcodec.so.58.134.100)
==14672==    by 0x6703B84: usb_cam::formats::MJPEG2RGB::convert(char const*&, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6700434: usb_cam::UsbCam::process_image(char const*, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x67007C0: usb_cam::UsbCam::read_frame() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6701852: usb_cam::UsbCam::grab_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6608A3C: usb_cam::UsbCamNode::take_and_send_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x6609117: usb_cam::UsbCamNode::update() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x660C894: rclcpp::GenericTimer<std::_Bind<void (usb_cam::UsbCamNode::*(usb_cam::UsbCamNode*))()>, (void*)0>::execute_callback() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672==    by 0x4943650: rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) (in /opt/ros/humble/lib/librclcpp.so)
==14672==    by 0x494AF3F: rclcpp::executors::SingleThreadedExecutor::spin() (in /opt/ros/humble/lib/librclcpp.so)
==14672== 
==14672== 156,904,367 (14,592 direct, 156,889,775 indirect) bytes in 608 blocks are definitely lost in loss record 476 of 476
==14672==    at 0x484DE30: memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x484DF92: posix_memalign (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==14672==    by 0x7BBC9E4: av_malloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x7BBCDC1: av_mallocz (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x7B99347: av_buffer_create (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x7B998EB: av_buffer_realloc (in /usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100)
==14672==    by 0x693B22D: av_new_packet (in /usr/lib/x86_64-linux-gnu/libavcodec.so.58.134.100)
==14672==    by 0x6703B84: usb_cam::formats::MJPEG2RGB::convert(char const*&, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6700434: usb_cam::UsbCam::process_image(char const*, char*&, int const&) (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x67007C0: usb_cam::UsbCam::read_frame() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6701852: usb_cam::UsbCam::grab_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam.so)
==14672==    by 0x6608A3C: usb_cam::UsbCamNode::take_and_send_image() (in /home/user/workspace/webcam_ws/install/usb_cam/lib/libusb_cam_node.so)
==14672== 
==14672== LEAK SUMMARY:
==14672==    definitely lost: 41,144 bytes in 1,232 blocks
==14672==    indirectly lost: 156,920,038 bytes in 1,868 blocks
==14672==      possibly lost: 3,800,475 bytes in 19 blocks
==14672==    still reachable: 292,048 bytes in 1,084 blocks
==14672==         suppressed: 0 bytes in 0 blocks
==14672== Reachable blocks (those to which a pointer was found) are not shown.
==14672== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==14672== 
==14672== For lists of detected and suppressed errors, rerun with: -s
==14672== ERROR SUMMARY: 23 errors from 23 contexts (suppressed: 0 from 0)

@flynneva
Copy link
Collaborator

@firesurfer ah you are right! I only tested without subscribing. Let me try again with subscribing and see what else I can address 👍

@flynneva
Copy link
Collaborator

@firesurfer well I cant seem to get valgrind to work for me to show the same exact memory leaks as you, but I think from your output I'm able to fix the leaks (fingers crossed). I just pushed one more commit to #268 so give it a go and let me know what you get.

Could you share with me how you run valgrind? I've tried the following but I dont see the same memory leaks as you:

 valgrind --leak-check=full \
   --show-leak-kinds=all \
   --track-origins=yes  \
   --verbose \
   --log-file=valgrind-out.txt ./install/usb_cam/lib/usb_cam/usb_cam_node_exe

@flynneva
Copy link
Collaborator

flynneva commented Jul 25, 2023 •
edited
Loading

@firesurfer oh nevermind 😓 I was using yuyv instead of mjpeg. Ignore my stupidity 😅 I see your valgrind output now

@flynneva
Copy link
Collaborator

@firesurfer I was able to track down most of the issues in the mjpeg2rgb conversion I think. Now that I can reproduce locally though I should be able to address the rest of them over the weekend

@flynneva
Copy link
Collaborator

@firesurfer actually wasn't able to solve this just yet. Give me a few days and ill track it dow!

@firesurfer
Copy link
Contributor Author

@flynneva hope you are doing fine! Did you manage to dig further into the issue by now ?

@flynneva flynneva self-assigned this Aug 16, 2023
flynneva added a commit to flynneva/usb_cam that referenced this issue Aug 22, 2023
@flynneva
Fix some minor memorly leaks for mjpeg
da715c9 
Relates to ros-drivers#262

Signed-off-by: Evan Flynn <evanflynn.msu@gmail.com>
@flynneva
Copy link
Collaborator

@firesurfer still working on it a bit and plan to do more work over the weekend.

I'll keep you posted 👍

@flynneva
Copy link
Collaborator

@firesurfer small update here: I've narrowed the major leakages down to the av_packet_from_data call. We'll need to come up with an alternative way of initializing the ffmpeg AVPacket, or figuring out what is leaking there.

Sorry this has taken so long...I had a friends wedding last week and things have been pretty busy for me the last few weeks 😅

@firesurfer
Copy link
Contributor Author

@flynneva No worries. Thanks for maintaining this package!

I just took a quick look myself into the code and the FFmpeg documentation.

So if I understand the docs correctly (https://ffmpeg.org/doxygen/2.7/group__lavc__packet.html#ga7ca877e1f0ded89a27199b65e9a077dc) One should only pass a data pointer which has been allocated with av_malloc. Which isn't the case here if I am right.

See also this stackoverflow question:
https://stackoverflow.com/questions/63617241/how-to-load-data-to-an-ffmpeg-avbuffer

@flynneva
Copy link
Collaborator

flynneva commented Aug 31, 2023 •
edited
Loading

@firesurfer I think I got it: bd6bb91

now the only leaks that are left are outside of this package it looks like, and they dont grow over time - which is good

@firesurfer
Copy link
Contributor Author

@flynneva Sounds great. I hope I find some time this week to test it!

@flynneva
Copy link
Collaborator

flynneva commented Sep 8, 2023

@firesurfer just an fyi the update is now on the main ros2 branch, so just use that one for your tests 👍🏼

@firesurfer
Copy link
Contributor Author

@flynneva Just tested the latest ros-iron release and it seems to fix the issue!
Thanks a lot

@flynneva
Copy link
Collaborator

@firesurfer awesome thanks for confirming! I'll go ahead and close this issue then 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@flynneva flynneva

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@flynneva @firesurfer