Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WASI-NN] Support llama 3.2 vision (mllama) #3929

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Conversation

q82419
Copy link
Collaborator

@q82419 q82419 commented Dec 19, 2024

  • Refactor cmake to reuse the image libraries
  • Helper macro for simplify the debug and error logging
  • Refactor the graph states to unload safety
  • Refine the life cycle of projection models
  • Refine the tokenizing and llama decode
  • Add the mllama patch
  • Add the vision support with mllama

Copy link
Member

juntao commented Dec 19, 2024

Hello, I am a code review agent on flows.network. Here are my reviews of changed source code files in this PR.


cmake/Helper.cmake

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinn_chattts.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinn_ggml.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinn_ggml.h

Potential issues

N/A

Summary of changes

  • Key Changes:
  • Renamed LlavaImagePosition to ImagePosition in the Context struct.
  • Added a new function finalizeExecCtx to handle finalization of execution context.

plugins/wasi_nn/wasinn_mlx.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinn_openvino.cpp

Potential issues

  1. Incorrect Context Initialization: The initExecCtx function sets the ready state on the wrong map entry (Env.NNGraph[ContextId]) instead of Env.NNContext[ContextId].

  2. Hardcoded Input Shape: The setInput function uses a hardcoded input shape {1, 224, 224, 3} which should be dynamically derived from the model or provided as an argument.

  3. Model Compilation per Input: The setInput function compiles the model for every input set, which is inefficient and should be done once during context initialization.

Summary of changes

  • Key Changes Summary:

  • Graph Management Refactor: Replaced direct vector manipulation for graph creation and error handling with newGraph and deleteGraph methods, enhancing encapsulation.

  • Graph Identification: Used a unique graph ID (GId) instead of relying on the last index in the vector, improving safety and clarity.

  • Graph State Management: Added setReady() method calls to mark graphs and contexts as ready after successful initialization.

plugins/wasi_nn/wasinn_piper.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinn_tfl.cpp

Potential issues

N/A

Summary of changes

  • Key Changes:
  • Graph Management: Replaced direct vector operations with new methods (newGraph and deleteGraph) for better encapsulation and error handling.
  • Context Management: Introduced new methods (newContext and deleteContext) for context management, similar to graph management.
  • Error Handling: Updated error handling in both functions to ensure proper cleanup using the newly introduced delete methods.

plugins/wasi_nn/wasinn_torch.cpp

Potential issues

  1. Memory Management Issue: The torch::from_blob function in setInput does not take ownership of the data, leading to potential issues if the original buffer is deallocated before the tensor is used.

  2. Error Handling Inconsistency: The error message in initExecCtx incorrectly references Env.NNGraph[ContextId], which should be Env.NNContext[ContextId].

  3. Output Size Calculation: In getOutput, the calculation of BlobSize may overflow if the tensor dimensions are large, leading to incorrect memory access or buffer overflows.

Summary of changes

    • Graph Management: Changed graph addition and removal logic to use newGraph and deleteGraph methods, replacing direct vector manipulation.
  • Graph Identification: Replaced manual index calculation for GraphId with the return value from newGraph.
  • Context Initialization: Simplified context creation by using newContext method instead of manually appending to a vector.

plugins/wasi_nn/wasinn_whisper.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasi_nn/wasinnenv.h

Potential issues

N/A

Summary of changes

    • Introduced init and reset methods in Graph and Context classes to handle initialization and cleanup.
  • Added status management (Status enum) in Graph and Context for tracking their lifecycle states.
  • Implemented graph and context recycling mechanisms (NNGraphRecycle, NNContextRecycle) in WasiNNEnvironment to optimize resource usage.

plugins/wasi_nn/wasinnfunc.cpp

Potential issues

N/A

Summary of changes

N/A

plugins/wasmedge_image/CMakeLists.txt

Potential issues

N/A

Summary of changes

N/A

@github-actions github-actions bot added c-Plugin An issue related to WasmEdge Plugin WASI-NN https://github.com/WebAssembly/wasi-nn c-CMake An issue related to CMake mechanism or options labels Dec 19, 2024
Copy link

codecov bot commented Dec 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.28%. Comparing base (dc663c9) to head (07e4d15).
Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3929   +/-   ##
=======================================
  Coverage   80.27%   80.28%           
=======================================
  Files         278      278           
  Lines       38713    38713           
  Branches     6728     6767   +39     
=======================================
+ Hits        31078    31079    +1     
- Misses       6025     6041   +16     
+ Partials     1610     1593   -17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: YiYing He <yiying@secondstate.io>
backends.

Signed-off-by: YiYing He <yiying@secondstate.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c-CMake An issue related to CMake mechanism or options c-Plugin An issue related to WasmEdge Plugin WASI-NN https://github.com/WebAssembly/wasi-nn
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants