Protoflow

"Τὰ πάντα ῥεῖ καὶ οὐδὲν μένει" — Heraclitus

Protoflow is a Rust implementation of flow-based programming (FBP), with messages encoded as Protocol Buffers. It can be used to implement dataflow systems consisting of interconnected blocks that process messages.

🚧 We are building in public. This is presently under heavy construction.

✨ Features

Implements a flow-based programming (FBP) dataflow scheduler.
Constructs systems by connecting reusable components called blocks.
Uses Protocol Buffers messages for inter-block communication.
Currently offers a threaded runtime with an in-process transport.
Planned support for pluggable runtimes (threaded, async, etc).
Planned support for pluggable transports (in-process, socket, etc).
Includes a command-line interface (CLI) for executing Protoflow blocks.
Supports opting out of any feature using comprehensive feature flags.
Adheres to the Rust API Guidelines in its naming conventions.
100% free and unencumbered public domain software.

🛠️ Prerequisites

Rust 1.70+

⬇️ Installation

Installation via Homebrew

brew tap AsimovPlatform/tap
brew install protoflow --HEAD

Installation via Cargo

cargo install protoflow

👉 Examples

Examples for Rust

For Rust examples, see the examples directory. Good places to start are the echo_lines and count_lines examples:

cargo run --example echo_lines < CHANGES.md
cargo run --example count_lines < README.md

The `count_lines` example

use protoflow::{blocks::*, BlockResult};

pub fn main() -> BlockResult {
    System::run(|s| {
        let stdin = s.read_stdin();

        let line_decoder = s.decode_lines();
        s.connect(&stdin.output, &line_decoder.input);

        let counter = s.count::<String>();
        s.connect(&line_decoder.output, &counter.input);

        let count_encoder = s.encode_lines();
        s.connect(&counter.count, &count_encoder.input);

        let stdout = s.write_stdout();
        s.connect(&count_encoder.output, &stdout.input);
    })
}

📚 Reference

Glossary

System: A collection of blocks that are connected together. Systems are the top-level entities in a Protoflow program.
Block: An encapsulated system component that processes messages. Blocks are the autonomous units of computation in a system.
Port: A named connection point on a block that sends or receives messages. Ports are the only interfaces through which blocks communicate with each other.
Message: A unit of data that flows between blocks in a system. Messages are Protocol Buffers packets that are processed by blocks.

Blocks

The built-in blocks provided by Protoflow are listed below:

Block	Description
`Buffer`	Stores all messages it receives.
`Const`	Sends a constant value.
`Count`	Counts the number of messages it receives, while optionally passing them through.
`Decode`	Decodes messages from a byte stream.
`Delay`	Passes messages through while delaying them by a fixed or random duration.
`Drop`	Discards all messages it receives.
`Encode`	Encodes messages to a byte stream.
`EncodeHex`	Encodes a byte stream into hexadecimal form.
`EncodeJSON`	Encodes messages into JSON format.
`Hash`	Computes the cryptographic hash of a byte stream.
`Random`	Generates and sends a random value.
`ReadDir`	Reads file names from a file system directory.
`ReadEnv`	Reads the value of an environment variable.
`ReadFile`	Reads bytes from the contents of a file.
`ReadStdin`	Reads bytes from standard input (aka stdin).
`WriteFile`	Writes or appends bytes to the contents of a file.
`WriteStderr`	Writes bytes to standard error (aka stderr).
`WriteStdout`	Writes bytes to standard output (aka stdout).

`Buffer`

A block that simply stores all messages it receives.

block-beta
    columns 4
    Source space:2 Buffer
    Source-- "input" -->Buffer

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Buffer block
    class Source hidden

protoflow execute Buffer

`Const`

A block for sending a constant value.

block-beta
    columns 4
    Const space:2 Sink
    Const-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Const block
    class Sink hidden

protoflow execute Const value=Hello

`Count`

A block that counts the number of messages it receives, while optionally passing them through.

block-beta
    columns 7
    Source space:2 Count space:2 Sink
    space:7
    space:7
    space:3 Result space:3
    Source-- "input" -->Count
    Count-- "output" -->Sink
    Count-- "count" -->Result

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Count block
    class Source hidden
    class Sink hidden
    class Result hidden

protoflow execute Count

`Decode`

A block that decodes T messages from a byte stream.

block-beta
    columns 7
    Source space:2 Decode space:2 Sink
    Source-- "input" -->Decode
    Decode-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Decode block
    class Source hidden
    class Sink hidden

protoflow execute Decode encoding=text

`Delay`

A block that passes messages through while delaying them by a fixed or random duration.

block-beta
    columns 7
    Source space:2 Delay space:2 Sink
    Source-- "input" -->Delay
    Delay-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Delay block
    class Source hidden
    class Sink hidden

protoflow execute Delay fixed=2

`Drop`

A block that simply discards all messages it receives.

block-beta
    columns 4
    Source space:2 Drop
    Source-- "input" -->Drop

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Drop block
    class Source hidden

protoflow execute Drop

`Encode`

A block that encodes T messages to a byte stream.

block-beta
    columns 7
    Source space:2 Encode space:2 Sink
    Source-- "input" -->Encode
    Encode-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Encode block
    class Source hidden
    class Sink hidden

protoflow execute Encode encoding=text
protoflow execute Encode encoding=protobuf

`EncodeHex`

A block that encodes a byte stream into hexadecimal form.

block-beta
    columns 7
    Source space:2 EncodeHex space:2 Sink
    Source-- "input" -->EncodeHex
    EncodeHex-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class EncodeHex block
    class Source hidden
    class Sink hidden

protoflow execute EncodeHex

`EncodeJSON`

A block that encodes messages into JSON format.

block-beta
    columns 7
    Source space:2 EncodeJSON space:2 Sink
    Source-- "input" -->EncodeJSON
    EncodeJSON-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class EncodeJSON block
    class Source hidden
    class Sink hidden

protoflow execute EncodeJSON

`Hash`

A block that computes the cryptographic hash of a byte stream, while optionally passing it through.

block-beta
    columns 7
    Source space:2 Hash space:2 Sink
    space:7
    space:7
    space:3 Result space:3
    Source-- "input" -->Hash
    Hash-- "output" -->Sink
    Hash-- "hash" -->Result

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Hash block
    class Source hidden
    class Sink hidden
    class Result hidden

protoflow execute Hash algorithm=blake3

`Random`

A block for generating and sending a random value.

block-beta
    columns 4
    Random space:2 Sink
    Random-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class Random block
    class Sink hidden

protoflow execute Random seed=42

`ReadDir`

A block that reads file names from a file system directory.

block-beta
    columns 4
    Config space:3
    space:4
    space:4
    ReadDir space:2 Sink
    Config-- "path" -->ReadDir
    ReadDir-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class ReadDir block
    class Config hidden
    class Sink hidden

protoflow execute ReadDir path=/tmp

`ReadEnv`

A block that reads the value of an environment variable.

block-beta
    columns 4
    Config space:3
    space:4
    space:4
    ReadEnv space:2 Sink
    Config-- "name" -->ReadEnv
    ReadEnv-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class ReadEnv block
    class Config hidden
    class Sink hidden

protoflow execute ReadEnv name=TERM

`ReadFile`

A block that reads bytes from the contents of a file.

block-beta
    columns 4
    Config space:3
    space:4
    space:4
    ReadFile space:2 Sink
    Config-- "path" -->ReadFile
    ReadFile-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class ReadFile block
    class Config hidden
    class Sink hidden

protoflow execute ReadFile path=/tmp/file.txt

`ReadStdin`

A block that reads bytes from standard input (aka stdin).

block-beta
    columns 4
    ReadStdin space:2 Sink
    ReadStdin-- "output" -->Sink

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class ReadStdin block
    class Sink hidden

protoflow execute ReadStdin < input.txt

`WriteFile`

A block that writes or appends bytes to the contents of a file.

block-beta
    columns 4
    space:3 Config
    space:4
    space:4
    Source space:2 WriteFile
    Config-- "path" -->WriteFile
    Source-- "input" -->WriteFile

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class WriteFile block
    class Config hidden
    class Source hidden

protoflow execute WriteFile path=/tmp/file.txt

`WriteStderr`

A block that writes bytes to standard error (aka stderr).

block-beta
    columns 4
    Source space:2 WriteStderr
    Source-- "input" -->WriteStderr

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class WriteStderr block
    class Source hidden

protoflow execute WriteStderr < input.txt 2> output.txt

`WriteStdout`

A block that writes bytes to standard output (aka stdout).

block-beta
    columns 4
    Source space:2 WriteStdout
    Source-- "input" -->WriteStdout

    classDef block height:48px,padding:8px;
    classDef hidden visibility:none;
    class WriteStdout block
    class Source hidden

protoflow execute WriteStdout < input.txt > output.txt

👨‍💻 Development

git clone https://github.com/AsimovPlatform/protoflow.git

Name		Name	Last commit message	Last commit date
Latest commit History 232 Commits
.cargo		.cargo
.github		.github
examples		examples
lib		lib
.gitattributes		.gitattributes
.gitignore		.gitignore
AUTHORS		AUTHORS
CHANGES.md		CHANGES.md
Cargo.toml		Cargo.toml
Makefile		Makefile
README.md		README.md
UNLICENSE		UNLICENSE
VERSION		VERSION
rustfmt.toml		rustfmt.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Protoflow

✨ Features

🛠️ Prerequisites

⬇️ Installation

Installation via Homebrew

Installation via Cargo

👉 Examples

Examples for Rust

The `count_lines` example

📚 Reference

Glossary

Blocks

`Buffer`

`Const`

`Count`

`Decode`

`Delay`

`Drop`

`Encode`

`EncodeHex`

`EncodeJSON`

`Hash`

`Random`

`ReadDir`

`ReadEnv`

`ReadFile`

`ReadStdin`

`WriteFile`

`WriteStderr`

`WriteStdout`

👨‍💻 Development

About

Releases

Contributors 2

Languages

License

AsimovPlatform/protoflow

Folders and files

Latest commit

History

Repository files navigation

Protoflow

✨ Features

🛠️ Prerequisites

⬇️ Installation

Installation via Homebrew

Installation via Cargo

👉 Examples

Examples for Rust

The count_lines example

📚 Reference

Glossary

Blocks

👨‍💻 Development

About

Topics

Resources

License

Stars

Watchers

Forks

Languages

The `count_lines` example