Raspberry Pi 5 self-hosted CI #8828

lakshanthad · 2024-03-10T09:49:53Z

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Expand CI testing to include Raspberry Pi 5 on Bookworm OS.

📊 Key Changes

Added rpi5-bookworm to the operating systems (OS) tested in Continuous Integration (CI) workflows.
Included conditional steps to skip certain setup actions for rpi5-bookworm.

🎯 Purpose & Impact

Testing Expansion: By incorporating Raspberry Pi 5 running Bookworm OS into CI tests, the goal is to ensure compatibility and performance on this platform. 🥧📈
Broader Support: This update aims to widen the user base and improve the experience for developers using Raspberry Pi 5 for their projects. 🌍👨‍💻
Conditional Workflow Modifications: The introduction of conditions to skip certain actions for rpi5-bookworm helps in tailoring the setup process specifically for this platform, optimizing the workflow. ⚙️🛠️

This move could significantly impact users working on Raspberry Pi 5 by providing better-supported tools and libraries, enhancing development efficiency and product reliability on this popular platform.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancements for Raspberry Pi compatibility and model export adjustments.

📊 Key Changes

🍓 Added Raspberry Pi support in the CI workflow to include tests and checks specific for Raspberry Pi platforms.
🔄 Updated pyproject.toml to include a new dependency flatbuffers for better compatibility with TensorFlow packages.
🚀 Extended the AI model export capabilities to exclude CoreML format on Raspberry Pi and refine conditions for other platforms.
🛠️ Adjusted backend requirements to handle specific numpy version issues on Raspberry Pi and NVIDIA Jetson during ONNX Runtime inference.
🌉 Modified benchmarking logic to accommodate platform-specific constraints regarding model export formats.

🎯 Purpose & Impact

🏁 Enable Raspberry Pi users to integrate and test Ultralytics AI models, expanding the user base and use-case scenarios.
🐛 Resolve compatibility issues with TensorFlow-related packages, ensuring smoother operations and deployments.
🎨 Refine export functionality across various formats, ensuring broader model applicability and user flexibility.
🎲 Ensure accurate model benchmarking across a wider range of platforms, improving user experience and model reliability.

codecov · 2024-03-10T10:01:06Z

Codecov Report

Attention: Patch coverage is 85.71429% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 76.00%. Comparing base (c54b013) to head (4ed8db7).

❗ Current head 4ed8db7 differs from pull request most recent head ca40934. Consider uploading reports for the commit ca40934 to get more accurate results

Files	Patch %	Lines
ultralytics/nn/autobackend.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8828      +/-   ##
==========================================
- Coverage   76.00%   76.00%   -0.01%     
==========================================
  Files         121      121              
  Lines       15339    15343       +4     
==========================================
+ Hits        11658    11661       +3     
- Misses       3681     3682       +1

Flag	Coverage Δ
Benchmarks	`36.02% <85.71%> (+0.01%)`	⬆️
GPU	`37.91% <14.28%> (-0.03%)`	⬇️
Tests	`71.28% <42.85%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lakshanthad · 2024-03-10T11:44:27Z

@glenn-jocher mAP falling behind and throws out error. Do we lower it? What do you suggest?

glenn-jocher · 2024-03-10T17:30:55Z

@lakshanthad yes, perhaps some issues with NCNN, we can lower the segment benchmark floor to 0.267. You do that directly in the CI.

glenn-jocher · 2024-03-10T17:57:18Z

New fail on Pose, updating to 0.179.

The Tests failures won't be resolved by updating benchmarks though, can you take a look at these as well?

lakshanthad · 2024-03-10T19:28:07Z

@glenn-jocher Benchmarks are all good now. I will take a look at Tests failures tomorrow.

glenn-jocher · 2024-03-10T20:48:33Z

@lakshanthad got it. Yes, I lowered a couple thresholds slightly to meet the lower RPi5 values.

For the tests I fixed a CI YAML bug and they are launching correctly now but aborting mid-tests for some reason, maybe in test_cli.py

lakshanthad · 2024-03-11T21:31:06Z

@glenn-jocher let's see after this change because coreml doesnt support exports on RPi:
8dfbc54

lakshanthad · 2024-03-11T21:52:44Z

no luck it seems. I will investigate further. I am suspecting this has something to do with the number of cores on the RPi. Threading issue. The problem might be inside test_python.py

glenn-jocher · 2024-04-16T23:30:32Z

@lakshanthad some of the replacements you've made I'm not understanding at all, like this one is for replacing all ARM64 environments, it's not about Raspberry Pis and Jetson Nano's they are lumped in with every other ARM64 environment no?

glenn-jocher · 2024-04-16T23:35:41Z

@lakshanthad here's another case, it seems you're forgetting that RPi and Jetson are not the only ARM64 environments in the world.

If a Macbook M2/M3 user runs this now his code will break no? Why are there so many replacements swapping ARM64 for RPi+Jetson, these statements are not equivalent and are excluding the wider world of ARM64 devices out there, or even i.e. Linux Docker images running on aarch64 devices like Macbooks.

ultralytics/engine/exporter.py

lakshanthad · 2024-04-16T23:39:22Z

@glenn-jocher M1/M2 will not break. Those ARM64 issues such as numpy versioning only affects RPi and Jetson and was not bringing about issues when running on the macos ARM runner. The reason i made them specify to rpi and jetson is because the problems only started happening for rpi and jetson, but not for macos arm runners.

I wanted to highlight the problems only happen with rpi and jetson and that is why i remove ARM64 and added the new constants.

ultralytics/utils/benchmarks.py

ultralytics/engine/exporter.py

glenn-jocher · 2024-04-17T03:12:15Z

@lakshanthad we started getting a strange failure of is_online() recently, tried to push a fix in the most recent PR, lowered the benchmark floor also to fix a benchmark failure there.

I searched for a simple way to run on schedule but it doesn't exist, we need to treat Raspberry Pi the same way we treat the GPU job, as a separate job. I'm trying to refactor now.

lakshanthad · 2024-04-17T03:26:17Z

okay thanks a lot! @glenn-jocher. my god, did you only sleep 3hrs?

glenn-jocher · 2024-04-17T03:37:35Z

@lakshanthad apologies buddy, I had to tear up the CI changes and merge them into a single smaller job (no benchmarks, just tests).

I used my "GPU" job as a blueprint, I hope this works, but haven't had time to test.

Also I found I couldn't cancel in-progress Rpi runner actions as I think maybe it was coming from your fork, but I've put them on scheduled and workflow dispatch triggers now.

glenn-jocher · 2024-04-17T03:48:20Z

pyproject.toml

@@ -101,6 +101,7 @@ export = [
    "openvino>=2024.0.0", # OpenVINO export
    "tensorflow<=2.13.1; python_version <= '3.11'", # TF bug https://github.com/ultralytics/ultralytics/issues/5161
    "tensorflowjs>=3.9.0; python_version <= '3.11'", # TF.js export, automatically installs tensorflow
+    "flatbuffers>=23.5.26,<100", # update old 'flatbuffers' included inside tensorflow package


@lakshanthad I'm not sure if this line is causing the Dockerfile deploy errors following this PR. I may have to remove it for the time being to get 8.2 out in time.

This should not be the issue. Because this change was done a long time ago and the other CI were passing at the time @glenn-jocher. Are you sure this is causing the issue?

If this is really the issue, then we can add this for only 'aarch64' @glenn-jocher

lakshanthad · 2024-04-17T04:30:09Z

@lakshanthad apologies buddy, I had to tear up the CI changes and merge them into a single smaller job (no benchmarks, just tests).

I used my "GPU" job as a blueprint, I hope this works, but haven't had time to test.

Also I found I couldn't cancel in-progress Rpi runner actions as I think maybe it was coming from your fork, but I've put them on scheduled and workflow dispatch triggers now.

@glenn-jocher understood. it is fine. Thank you for all the changes. I think this is the only way we can implement it for now. So we will manually add the benchmarks later to this separate job?

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> Co-authored-by: UltralyticsAssistant <web@ultralytics.com> Co-authored-by: Ultralytics AI Assistant <135830346+UltralyticsAssistant@users.noreply.github.com>

lakshanthad added 2 commits March 10, 2024 01:49

Add local CI for RPi5 bookworm

f4d39be

Update ci.yaml

e082966

lakshanthad added 2 commits March 10, 2024 03:05

Update ci.yaml

9d5c368

Update ci.yaml

a140864

lakshanthad changed the title ~~Add local CI for RPi5 bookworm~~ Add self-hosted CI for Raspberry Pi 5 Bookworm Mar 10, 2024

lakshanthad added 6 commits March 10, 2024 03:44

Update benchmarks.py

8c03b88

Update benchmarks.py

977a38e

Update ci.yaml

0bcac4c

Update benchmarks.py

85181a7

Update benchmarks.py

1a5cc1c

Update benchmarks.py

9016e40

lakshanthad and others added 2 commits March 10, 2024 04:50

Update ci.yaml

0fa110a

Merge branch 'main' into main

8fb00f7

Update ci.yaml

1d0575f

glenn-jocher added 4 commits March 10, 2024 18:57

Update ci.yaml

0b8d802

Update ci.yaml

9fbc888

Update ci.yaml

a0daf45

Merge branch 'main' into main

4978d1c

glenn-jocher assigned lakshanthad Mar 10, 2024

glenn-jocher and others added 3 commits March 11, 2024 16:07

Merge branch 'main' into main

339822f

Merge branch 'ultralytics:main' into main

6c479e1

Update test_python.py

8dfbc54

glenn-jocher added 2 commits April 16, 2024 16:11

Do not install export deps for tests

51dcca6

Merge branch 'main' into main

0481668

glenn-jocher reviewed Apr 16, 2024

View reviewed changes

ultralytics/engine/exporter.py Outdated Show resolved Hide resolved

Revert all changes to exporter.py

3bec04e

glenn-jocher added 2 commits April 16, 2024 16:43

Fix coreml export test

65560f3

Revert changes to autobackend.py

0b381ff

glenn-jocher reviewed Apr 16, 2024

View reviewed changes

ultralytics/utils/benchmarks.py Outdated Show resolved Hide resolved

Revert version change.

fc3cf85

lakshanthad commented Apr 16, 2024

View reviewed changes

ultralytics/engine/exporter.py Outdated Show resolved Hide resolved

glenn-jocher added 4 commits April 16, 2024 17:05

Reduce changes to benchmarks.py

377ce17

Further simplify benchmarks.py

6b45b2d

Fix missing import

e7b0007

Merge branch 'main' into main

4ed8db7

Create new RaspberryPi job

42cb4de

glenn-jocher removed the TODO High priority items label Apr 17, 2024

glenn-jocher changed the title ~~ultralytics 8.1.48 Raspberry Pi 5 self-hosted CI~~ Raspberry Pi 5 self-hosted CI Apr 17, 2024

Update ci.yaml

ca40934

glenn-jocher merged commit b76400a into ultralytics:main Apr 17, 2024
6 of 11 checks passed

glenn-jocher reviewed Apr 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raspberry Pi 5 self-hosted CI #8828

Raspberry Pi 5 self-hosted CI #8828

lakshanthad commented Mar 10, 2024 •

edited by github-actions bot

Loading

codecov bot commented Mar 10, 2024 •

edited

Loading

lakshanthad commented Mar 10, 2024

glenn-jocher commented Mar 10, 2024 •

edited

Loading

glenn-jocher commented Mar 10, 2024

lakshanthad commented Mar 10, 2024

glenn-jocher commented Mar 10, 2024

lakshanthad commented Mar 11, 2024

lakshanthad commented Mar 11, 2024 •

edited

Loading

glenn-jocher commented Apr 16, 2024 •

edited

Loading

glenn-jocher commented Apr 16, 2024

lakshanthad commented Apr 16, 2024 •

edited

Loading

glenn-jocher commented Apr 17, 2024

lakshanthad commented Apr 17, 2024 •

edited

Loading

glenn-jocher commented Apr 17, 2024

glenn-jocher Apr 17, 2024

lakshanthad Apr 17, 2024

lakshanthad Apr 17, 2024

lakshanthad commented Apr 17, 2024 •

edited

Loading

Raspberry Pi 5 self-hosted CI #8828

Raspberry Pi 5 self-hosted CI #8828

Conversation

lakshanthad commented Mar 10, 2024 • edited by github-actions bot Loading

🛠️ PR Summary

🌟 Summary

📊 Key Changes

🎯 Purpose & Impact

🛠️ PR Summary

🌟 Summary

📊 Key Changes

🎯 Purpose & Impact

codecov bot commented Mar 10, 2024 • edited Loading

Codecov Report

lakshanthad commented Mar 10, 2024

glenn-jocher commented Mar 10, 2024 • edited Loading

glenn-jocher commented Mar 10, 2024

lakshanthad commented Mar 10, 2024

glenn-jocher commented Mar 10, 2024

lakshanthad commented Mar 11, 2024

lakshanthad commented Mar 11, 2024 • edited Loading

glenn-jocher commented Apr 16, 2024 • edited Loading

glenn-jocher commented Apr 16, 2024

lakshanthad commented Apr 16, 2024 • edited Loading

glenn-jocher commented Apr 17, 2024

lakshanthad commented Apr 17, 2024 • edited Loading

glenn-jocher commented Apr 17, 2024

glenn-jocher Apr 17, 2024

Choose a reason for hiding this comment

lakshanthad Apr 17, 2024

Choose a reason for hiding this comment

lakshanthad Apr 17, 2024

Choose a reason for hiding this comment

lakshanthad commented Apr 17, 2024 • edited Loading

lakshanthad commented Mar 10, 2024 •

edited by github-actions bot

Loading

codecov bot commented Mar 10, 2024 •

edited

Loading

glenn-jocher commented Mar 10, 2024 •

edited

Loading

lakshanthad commented Mar 11, 2024 •

edited

Loading

glenn-jocher commented Apr 16, 2024 •

edited

Loading

lakshanthad commented Apr 16, 2024 •

edited

Loading

lakshanthad commented Apr 17, 2024 •

edited

Loading

lakshanthad commented Apr 17, 2024 •

edited

Loading