Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support WasmEdge as an alternative engine #1

Draft
wants to merge 33 commits into
base: master
Choose a base branch
from
Draft

feat: support WasmEdge as an alternative engine #1

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
fb5b2fe
init work: make the public API clear
xxchan Feb 26, 2023
7a30006
implement filter transform
xxchan Feb 26, 2023
685ae36
wip
xxchan Apr 9, 2023
15686cc
make the public API clear
xxchan Feb 26, 2023
1937438
move wasmtime_engine to a different mod
xxchan Apr 15, 2023
6f1604e
fix wasi feature
xxchan Apr 15, 2023
6b88555
rename wasmtime-engine -> wasmtime
xxchan Apr 15, 2023
cd038a3
Merge branch 'master' into xxchan/clena
xxchan Apr 18, 2023
86326da
reorg
xxchan Apr 18, 2023
e959a9d
Merge branch 'xxchan/clena' into xxchan/wasmedge
xxchan Apr 18, 2023
f17b8ef
support init for wasmedge/common
xxchan Apr 18, 2023
8b11252
Merge branch 'master' into xxchan/wasmedge
xxchan Apr 18, 2023
11e1751
make SmartModuleInstance common
xxchan Apr 18, 2023
abd4fb6
make create_transform common & reorg trait imp to imp
xxchan Apr 18, 2023
66bcbdd
more movement
xxchan Apr 18, 2023
0132522
Merge branch 'master' into xxchan/wasmedge
xxchan Apr 29, 2023
da7a2e2
add other tests for wasmedge
xxchan Apr 29, 2023
26d9053
support agg for common/wasmedge
xxchan Apr 29, 2023
56de150
movement
xxchan Apr 29, 2023
14c4c3c
refactor wasmtime to use the common code
xxchan Apr 29, 2023
3d8f4bf
WasmTime -> Wasmtime
xxchan Apr 29, 2023
d0bb940
move transform unit tests to common
xxchan Apr 29, 2023
7924834
rename Wasmedge -> WasmEdge
xxchan Apr 29, 2023
f2f55e8
minor tweaks
xxchan Apr 29, 2023
e6f7e2e
change features
xxchan May 15, 2023
06d579f
rm wasmedge
xxchan May 31, 2023
6ff943b
Merge branch 'master' into xxchan/wasmedge
xxchan May 31, 2023
7e559c1
remove wasmedge
xxchan May 31, 2023
7d23e1a
fmt
xxchan May 31, 2023
f5650e3
fmt with group_imports = "StdExternalCrate"
xxchan May 31, 2023
fe98b8f
try to make diff smaller
xxchan May 31, 2023
136c572
make diff smaller
xxchan May 31, 2023
72f4ed7
clippy
xxchan May 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename Wasmedge -> WasmEdge
  • Loading branch information
@xxchan
xxchan committed Apr 29, 2023
commit 792483407a026b31c8f0aca97f1d17fd70a3d313
2 changes: 1 addition & 1 deletion crates/fluvio-smartengine/src/engine/common/transforms/aggregate.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ mod test {
crate::engine::common::AggregateTransform<crate::engine::wasmtime::instance::WasmtimeFn>;
} else if #[cfg(feature = "wasmedge-engine")] {
type AggregateTransform =
crate::engine::common::AggregateTransform<crate::engine::wasmedge::instance::WasmedgeFn>;
crate::engine::common::AggregateTransform<crate::engine::wasmedge::instance::WasmEdgeFn>;
} else {
compile_error!("no engine specified");
}
Expand Down
12 changes: 6 additions & 6 deletions crates/fluvio-smartengine/src/engine/wasmedge/imp.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use super::instance::{WasmedgeContext, WasmedgeFn, WasmedgeInstance};
use super::instance::{WasmEdgeContext, WasmEdgeFn, WasmEdgeInstance};

use tracing::debug;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you reorder terms in of:

  • std
  • third party
  • fluvio dep

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's OK. Are you going to review the PR now? I will resolve such issues if general ideas LGTY.

P.S. this PR is generally complete except minor points.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you split this into PR just related to without WasmEdge? We need to discuss how to maintain WasmEdge related code.

Thanks

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's OK. But I'd like to elaborate a bit: only folder wasmedge is WasmEdge-specific code, and that's ~300 lines of simple code. To discuss how to maintain WasmEdge related code, and to get the ideas of how the common code work, it might be better to keep WasmEdge related code now. This can actually make it easier to review this PR.

I can remove the WasmEdge related code after the code review if that's not decided.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like have review by team but can't seem to add reviewers

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a PR to the main repo infinyon#3257

use wasmedge_sdk::{Executor, Module, Store};
Expand All @@ -9,8 +9,8 @@ use crate::metrics::SmartModuleChainMetrics;
use crate::engine::SmartModuleChainBuilder;
use crate::engine::common::create_transform;

type SmartModuleInit = crate::engine::common::SmartModuleInit<WasmedgeFn>;
type SmartModuleInstance = crate::engine::common::SmartModuleInstance<WasmedgeInstance, WasmedgeFn>;
type SmartModuleInit = crate::engine::common::SmartModuleInit<WasmEdgeFn>;
type SmartModuleInstance = crate::engine::common::SmartModuleInstance<WasmEdgeInstance, WasmEdgeFn>;

#[derive(Clone)]
pub struct SmartEngineImp();
Expand All @@ -28,14 +28,14 @@ pub fn initialize_imp(
) -> Result<SmartModuleChainInstanceImp> {
let executor = Executor::new(None, None).expect("Failed to create WasmEdge executor");
let mut store = Store::new().expect("Failed to create WasmEdge store");
let mut ctx = WasmedgeContext { engine: executor };
let mut ctx = WasmEdgeContext { engine: executor };

let mut instances = Vec::with_capacity(builder.smart_modules.len());
// let mut state = engine.new_state();
for (config, bytes) in builder.smart_modules {
let module = Module::from_bytes(None, bytes)?;
let version = config.version();
let mut instance = WasmedgeInstance::instantiate(
let mut instance = WasmEdgeInstance::instantiate(
&mut store,
&mut ctx.engine,
module,
Expand All @@ -59,7 +59,7 @@ pub fn initialize_imp(

/// SmartModule Chain Instance that can be executed
pub struct SmartModuleChainInstanceImp {
ctx: WasmedgeContext,
ctx: WasmEdgeContext,
instances: Vec<SmartModuleInstance>,
}

Expand Down
18 changes: 9 additions & 9 deletions crates/fluvio-smartengine/src/engine/wasmedge/instance.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,23 @@ use crate::engine::wasmedge::memory::{RecordsCallBack, RecordsMemory};
use crate::engine::common::{WasmFn, WasmInstance};
use crate::engine::error::EngineError;

pub(crate) struct WasmedgeInstance {
pub(crate) struct WasmEdgeInstance {
instance: wasmedge_sdk::Instance,
records_cb: Arc<RecordsCallBack>,
params: SmartModuleExtraParams,
version: i16,
}

pub(crate) struct WasmedgeContext {
pub(crate) struct WasmEdgeContext {
pub engine: Executor,
}

pub type WasmedgeFn = Func;
pub type WasmEdgeFn = Func;

impl WasmInstance for WasmedgeInstance {
type Context = WasmedgeContext;
impl WasmInstance for WasmEdgeInstance {
type Context = WasmEdgeContext;

type Func = WasmedgeFn;
type Func = WasmEdgeFn;

fn get_fn(&self, name: &str, _ctx: &mut Self::Context) -> Result<Option<Self::Func>> {
match self.instance.func(name) {
Expand Down Expand Up @@ -76,8 +76,8 @@ impl WasmInstance for WasmedgeInstance {
}
}

impl WasmFn for WasmedgeFn {
type Context = WasmedgeContext;
impl WasmFn for WasmEdgeFn {
type Context = WasmEdgeContext;

fn call(&self, ptr: i32, len: i32, version: i32, ctx: &mut Self::Context) -> Result<i32> {
let res = self.call(
Expand All @@ -92,7 +92,7 @@ impl WasmFn for WasmedgeFn {
}
}

impl WasmedgeInstance {
impl WasmEdgeInstance {
/// instantiate new module instance that contain context
pub(crate) fn instantiate(
store: &mut Store,
Expand Down