Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support WasmEdge as an alternative engine #1

Draft
wants to merge 33 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
fb5b2fe
init work: make the public API clear
xxchan Feb 26, 2023
7a30006
implement filter transform
xxchan Feb 26, 2023
685ae36
wip
xxchan Apr 9, 2023
15686cc
make the public API clear
xxchan Feb 26, 2023
1937438
move wasmtime_engine to a different mod
xxchan Apr 15, 2023
6f1604e
fix wasi feature
xxchan Apr 15, 2023
6b88555
rename wasmtime-engine -> wasmtime
xxchan Apr 15, 2023
cd038a3
Merge branch 'master' into xxchan/clena
xxchan Apr 18, 2023
86326da
reorg
xxchan Apr 18, 2023
e959a9d
Merge branch 'xxchan/clena' into xxchan/wasmedge
xxchan Apr 18, 2023
f17b8ef
support init for wasmedge/common
xxchan Apr 18, 2023
8b11252
Merge branch 'master' into xxchan/wasmedge
xxchan Apr 18, 2023
11e1751
make SmartModuleInstance common
xxchan Apr 18, 2023
abd4fb6
make create_transform common & reorg trait imp to imp
xxchan Apr 18, 2023
66bcbdd
more movement
xxchan Apr 18, 2023
0132522
Merge branch 'master' into xxchan/wasmedge
xxchan Apr 29, 2023
da7a2e2
add other tests for wasmedge
xxchan Apr 29, 2023
26d9053
support agg for common/wasmedge
xxchan Apr 29, 2023
56de150
movement
xxchan Apr 29, 2023
14c4c3c
refactor wasmtime to use the common code
xxchan Apr 29, 2023
3d8f4bf
WasmTime -> Wasmtime
xxchan Apr 29, 2023
d0bb940
move transform unit tests to common
xxchan Apr 29, 2023
7924834
rename Wasmedge -> WasmEdge
xxchan Apr 29, 2023
f2f55e8
minor tweaks
xxchan Apr 29, 2023
e6f7e2e
change features
xxchan May 15, 2023
06d579f
rm wasmedge
xxchan May 31, 2023
6ff943b
Merge branch 'master' into xxchan/wasmedge
xxchan May 31, 2023
7e559c1
remove wasmedge
xxchan May 31, 2023
7d23e1a
fmt
xxchan May 31, 2023
f5650e3
fmt with group_imports = "StdExternalCrate"
xxchan May 31, 2023
fe98b8f
try to make diff smaller
xxchan May 31, 2023
136c572
make diff smaller
xxchan May 31, 2023
72f4ed7
clippy
xxchan May 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fmt with group_imports = "StdExternalCrate"
  • Loading branch information
xxchan committed May 31, 2023
commit f5650e35dccf0d857c3d973873fccd6f73858585
5 changes: 3 additions & 2 deletions crates/fluvio-smartengine/src/engine/common/init.rs
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
use super::{WasmFn, WasmInstance};
use anyhow::Result;
use fluvio_smartmodule::dataplane::smartmodule::{
SmartModuleInitInput, SmartModuleInitErrorStatus, SmartModuleInitOutput,
SmartModuleInitErrorStatus, SmartModuleInitInput, SmartModuleInitOutput,
};

use super::{WasmFn, WasmInstance};

pub(crate) const INIT_FN_NAME: &str = "init";

pub(crate) struct SmartModuleInit<F: WasmFn>(F);
Expand Down
1 change: 0 additions & 1 deletion crates/fluvio-smartengine/src/engine/common/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ mod init;
mod transforms;
pub(crate) use init::*;
pub(crate) use transforms::*;

use anyhow::Result;
use fluvio_protocol::{Decoder, Encoder};
use fluvio_smartmodule::dataplane::smartmodule::{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@
mod test {

use std::convert::TryFrom;

use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::SmartModuleInitialData;
use crate::engine::common::{AGGREGATE_FN_NAME};
use crate::engine::common::AGGREGATE_FN_NAME;
use crate::engine::metrics::SmartModuleChainMetrics;
use crate::engine::{SmartEngine, SmartModuleChainBuilder};
use crate::engine::config::SmartModuleConfig;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
mod test {

use std::convert::TryFrom;

use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::engine::common::ARRAY_MAP_FN_NAME;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
mod test {

use std::convert::TryFrom;

use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::engine::common::FILTER_FN_NAME;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
mod test {

use std::convert::TryFrom;

use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::engine::common::FILTER_MAP_FN_NAME;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
mod test {

use std::convert::TryFrom;

use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::engine::common::MAP_FN_NAME;
Expand Down
10 changes: 5 additions & 5 deletions crates/fluvio-smartengine/src/engine/common/transforms/mod.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
use std::any::Any;

use anyhow::Result;
use fluvio_smartmodule::dataplane::smartmodule::{
SmartModuleInput, SmartModuleOutput, SmartModuleTransformErrorStatus,
SmartModuleAggregateInput, SmartModuleAggregateOutput,
SmartModuleAggregateInput, SmartModuleAggregateOutput, SmartModuleInput, SmartModuleOutput,
SmartModuleTransformErrorStatus,
};
use tracing::debug;

use crate::{SmartModuleInitialData, engine::error::EngineError};

use super::{WasmInstance, WasmFn};
use crate::{engine::error::EngineError, SmartModuleInitialData};
use super::{WasmFn, WasmInstance};

mod filter;
mod map;
Expand Down
4 changes: 1 addition & 3 deletions crates/fluvio-smartengine/src/engine/fixture.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
use std::{
path::{PathBuf, Path},
};
use std::path::{Path, PathBuf};

pub(crate) fn read_wasm_module(module_name: &str) -> Vec<u8> {
let spu_dir = std::env::var("CARGO_MANIFEST_DIR").expect("target");
Expand Down
2 changes: 1 addition & 1 deletion crates/fluvio-smartengine/src/engine/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,6 @@ impl SmartModuleChainInstance {
}

pub(crate) mod wasmtime;
use self::wasmtime::{SmartEngineImp, initialize_imp, SmartModuleChainInstanceImp};
use self::wasmtime::{initialize_imp, SmartEngineImp, SmartModuleChainInstanceImp};
#[cfg(test)]
use self::wasmtime::WasmFnImp;
7 changes: 3 additions & 4 deletions crates/fluvio-smartengine/src/engine/wasmtime/imp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleInput, SmartModuleOu
use crate::metrics::SmartModuleChainMetrics;
use crate::engine::SmartModuleChainBuilder;
use crate::engine::common::create_transform;

use super::instance::{WasmtimeFn, WasmtimeInstance, WasmtimeContext};
use super::instance::{WasmtimeContext, WasmtimeFn, WasmtimeInstance};
use super::state::WasmState;

type SmartModuleInit = crate::engine::common::SmartModuleInit<WasmtimeFn>;
Expand Down Expand Up @@ -141,8 +140,8 @@ mod chaining_test {
use fluvio_smartmodule::{dataplane::smartmodule::SmartModuleInput, Record};

use crate::engine::{
SmartEngine, SmartModuleChainBuilder, SmartModuleConfig, SmartModuleInitialData,
metrics::SmartModuleChainMetrics,
metrics::SmartModuleChainMetrics, SmartEngine, SmartModuleChainBuilder, SmartModuleConfig,
SmartModuleInitialData,
};

const SM_FILTER_INIT: &str = "fluvio_smartmodule_filter_init";
Expand Down
10 changes: 4 additions & 6 deletions crates/fluvio-smartengine/src/engine/wasmtime/instance.rs
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
use anyhow::Result;
use fluvio_protocol::{Encoder, Decoder};
use tracing::debug;

use std::sync::Arc;

use anyhow::Result;
use fluvio_protocol::{Decoder, Encoder};
use tracing::debug;
use fluvio_smartmodule::dataplane::smartmodule::SmartModuleExtraParams;
use wasmtime::{Instance, Module, Caller, AsContext, Extern};
use wasmtime::{AsContext, Caller, Extern, Instance, Module};

use crate::engine::{
common::{WasmFn, WasmInstance},
error::EngineError,
wasmtime::memory::RecordsMemory,
};

use super::{memory::RecordsCallBack, state::WasmState};

pub struct WasmtimeInstance {
Expand Down
3 changes: 1 addition & 2 deletions crates/fluvio-smartengine/src/engine/wasmtime/memory.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
use std::sync::Mutex;

use wasmtime::*;

use anyhow::{Result, Error, anyhow};
use anyhow::{anyhow, Error, Result};

const ALLOC_FN: &str = "alloc";
const MEMORY: &str = "memory";
Expand Down
2 changes: 1 addition & 1 deletion crates/fluvio-smartengine/src/engine/wasmtime/mod.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/// Implementations of the public API
mod imp;
pub use imp::{SmartEngineImp, initialize_imp, SmartModuleChainInstanceImp};
pub use imp::{initialize_imp, SmartEngineImp, SmartModuleChainInstanceImp};
/// Implementations of the traits in `common` for the Wasmtime engine
mod instance;
#[cfg(test)]
Expand Down
7 changes: 4 additions & 3 deletions crates/fluvio-smartengine/src/transformation.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
use std::{
collections::BTreeMap,
fmt::{self, Display},
path::PathBuf,
fs::File,
io::Read,
ops::Deref,
path::PathBuf,
};

use serde::{
Deserialize, Serialize, Deserializer,
de::{Visitor, self, SeqAccess, MapAccess},
de::{self, MapAccess, SeqAccess, Visitor},
Deserialize, Deserializer, Serialize,
};

#[derive(Debug, Default, Serialize, Deserialize, Clone, PartialEq, Eq)]
Expand Down