• Graph
• Stepper
• Units (Nodes) | Functions
• Connections (Edges) | Wire Functions
• Reference Links

Hyperway is a graph based functional execution library, allowing you to connect functions arbitrarily through a unique API. Mimic a large range of programming paradigms such as procedural, parallel, or aspect-oriented-programming. Build B-trees or decision trees, circuitry or logic gates; all with a simple wiring methodology.

pip install hyperway

[!NOTICE] This project is very new and I'm still putting together better docs, tests, examples, a website, and all the bits in-between. Until then these other libraries are awesome.

Connect functions, run the chain:

import hyperway
from hyperway.tools import factory as f

# Store
g = hyperway.Graph()

# Connect
first_connection = g.add(f.add_10, f.add_20)
# More connections
many_connections = g.connect(f.add_30, f.add_1, f.add_2, f.add_3)
# Cross connect
compound_connection = g.add(many_connections[1].b, f.add_3)

# Prepare to run
stepper = g.stepper(first_connection.a, 10)

# Run a step
concurrent_row = stepper.step()

Render with graphviz:

g.write('intro-example', directory='renders', direction='LR')

This library aims to simplify graph based execution chains, allowing a developer to use functional graphs without managing the connections.

Key

The Hyperway API aims to simplify standard graph-theory terminology and allows developers to extend the base library with custom or preferred terms, such as class Vertex(Unit): pass.

Hyperway components, and their conventional siblings:

The Graph is a fancy defaultdict of tuples, used to store connections:

import hyperway

g = hyperway.Graph()

• Create
• Run
• Pluck
• Wire Function
• Self Reference

Bind functions in an execution chain. Notably it's a minimum of two:

from hyperway.edges import make_edge

c = make_edge(f.add_1, f.add_2)
# <Connection>

Hyperway

import hyperway
from hyperway.tools import factory as f

g = hyperway.Graph()

connection = g.add(f.add_1, f.add_2)
# <Connection>

Or Functional

from hyperway.edges import make_edge

c = make_edge(f.add_1, f.add_2)
# <Connection>

We can run a connection, calling the chain of two nodes. Generally a Connection isn't used outside a graph unless we're playing with it.

A standard call to a connection will run node a (the left side):

# connection = make_edge(f.add_1, f.add_2)
>>> value_part_a = connection(1) # call A-side `add_1`
2.0

Then process the second part b (providing the value from the first call):

>>> connection.process(value_part_a) # call B-side `add_2`
4.0

Alternatively use the pluck() method.

We can "pluck" a connection (like plucking a string) to run a command with any arguments:

from hyperway.tools import factory as f
from hyperway.edges import make_edge

c = make_edge(f.add_1, f.add_2)
# Run side _a_ (`add_1`) and _b_ (`add_2`) with our input value.

c.pluck(1)    # 4.0 == 1 + 1 + 2
c.pluck(10)   # 13.0 == 10 + 1 + 2

The pluck() executes both nodes and the optional wire function, in the expected order. Fundamentally a connection is self-contained and doesn't require a parent graph.

An optional wire function exists between two nodes

The Connection can have a function existing between its connected Units, allowing the alteration of the data through transit (whilst running through a connection):

from hyperway.tools import factory as f
from hyperway.edges import make_edge, wire

c = make_edge(f.add_1, f.add_2, through=wire(f.mul_2))
# <Connection(Unit(func=P_add_1.0), Unit(func=P_add_2.0), through="P_mul_2.0" name=None)>

When using the connection side A (the f.add_1 function), the wire function wire(f.mul_2) can inspect the values as they move to f.add_2.

It's important to note Hyperway is left-associative. The order of operation computes linearly:

assert c.pluck(1) == 10 # (1 + 1) * 2 + 2 == 6
assert c.pluck(10) == 24 # (10 + 1) * 2 + 2 == 24

It's easy to argue a wire function is a node, and you can implement the wire function without this connection tap.

• Wire functions are optional: node to node connections are inherently not optional)
• Removing a wire function does not remove the edge: Edges are persistent to the graph
• Wire functions may be inert (e.g. just logging); Nodes cannot be inert as they must be bound to edges.

Fundamentally a wire function exists for topological clarity and may be ignored.

make_edge can accept the wire function. It receives the concurrent values transmitting through the attached edge:

from hyperway.edges import make_edge
from hyperway.packer import argspack

import hyperway.tools as t
f = t.factory

def doubler(v, *a, **kw):
    # The wire function `through` _doubles_ the given number.
    # response with an argpack.
    return argspack(v * 2, **kw)


c = make_edge(f.add_1, f.add_2, through=doubler)
# <Connection(Unit(func=P_add_1.0), Unit(func=P_add_2.0), name=None)>

c.pluck(1)  # 6.0
c.pluck(2)  # 8.0
c.pluck(3)  # 10.0

The wire function is the reason for a two-step process when executing connections:

# Call Node A: (+ 1)
c(4)
5.0

# Call Wire + B: (double then + 2)
c.process(5.0)
12.0

Or a single pluck():

c.pluck(4)
12.0

A connection A -> B may be the same node, performing a loop or self-referencing node connection.

We can use the as_unit function, and reference the same unit on the graph:

# pre-define the graph node wrapper
u = as_unit(f.add_2)
# Build an loop edge
e = make_edge(u, u)

g = Graph()
g.add_edge(e)

# Setup the start from the unit (side A)
g.stepper(u, 1)

# Call the stepper forever.
g.step()
g.step()
...
# 3, 5, 7, 9, 11, ...

A function is our working code. We can borrow operator functions from the tools:

from hyperway.tools import factory as f
add_10 = f.add_10
# functools.partial(<built-in function add>, 10.0)
add_10(1)
# 11.0
add_10(14)
# 24.0

The Unit (or node) is a function connected to other nodes through a Connection. A Unit is a wrapper for a any python function. Everything in the graph (and in a Connection) is a Unit.

When we create a new connection, it automatically wraps the given functions as Unit types:

c = make_edge(f.mul_3, f.add_4)
c.a
# <Unit(func=P_mul_3.0)>
c.b
# <Unit(func=P_add_4.0)>

A Unit has additional methods used by the graph tools, such as the process method:

# Call our add_4 function:
>>> c.b.process(1)
5.0

A new unit is very unique. Creating a Connection with the same function for both sides a and b, will insert two new nodes:

c = make_edge(f.add_4, f.add_4)
c.pluck(4)
12.0

We can cast a function as a Unit before insertion, allowing the re-reference to existing nodes.

If you create a Unit using the same function, this will produce two unique units:

unit_a = as_unit(f.add_2)
unit_a_2 = as_unit(f.add_2)
assert unit_a != unit_a_2  # Not the same.

Attempting to recast a Unit, will return the same Unit:

unit_a = as_unit(f.add_2)
unit_a_2 = as_unit(unit_a) # unit_a is already a Unit
assert unit_a == unit_a_2  # They are the same

With this we create a linear chain of function calls, or close a loop that will run forever.

Generally when inserting functions, a new reference is created. This allows us to use the same function at different points in a chain:

a = f.add_1
b = f.add_2
c = f.add_2

# a -> b -> c | done.
connection_1 = make_edge(a, b)
_ = make_edge(b, c)
_ = make_edge(c, a)

Closing a path produces a loop. To close a path we can reuse the same Unit at both ends of our path.

To ensure a node is reused when applied, we pre-convert it to a Unit:

a = as_unit(f.add_1) # sticky reference.
b = f.add_2
c = f.add_2

# a -> b -> c -> a ... forever
connection_1 = make_edge(a, b)
_ = make_edge(b, c)
_ = make_edge(c, a)

All Connections are stored within a single Graph instance. It has been purposefully designed as a small collection of connections. We can consider the graph as a dictionary register of all associated connections.

from hyperway.graph import Graph, add
from hyperway.nodes import as_units

g = Graph()
unit_a, unit_b = as_unit(f.add_2, f.mul_2)

connection = add(g, unit_a, unit_b)

Under the hood, The graph is just a defaultdict and doesn't do much.

The Stepper run units and discovers connections through the attached Graph. It runs concurrent units and spools the next callables for the next step.

from hyperway.graph import Graph
from hyperway.tools import factory as f

g = Graph()
a_connections = g.connect(f.add_10, f.add_20, f.add_30)
b_connections = g.connect(f.add_1, f.add_2, f.add_3)
c_connection = g.add(b_connections[1].b, f.add_3)

first_connection_first_node = a_connections[0].a
stepper = g.stepper(first_connection_first_node, 10)
# <stepper.StepperC object at 0x000000000258FEB0>

concurrent_row = stepper.step()
# rows. e.g: ((<Unit(func=my_func)>, <ArgsPack(*(1,), **{})>),)

For each step() call, we yield a step. When iterating from first_connection_first_node (f.add_10), the stepper will pause half-way through our call. The next step will resolve the value and prepare the next step:

# From above:
# g.stepper(first_connection_first_node, 10)

stepper.step()
(
    # Partial edge (from add_10 to add_20), with the value "20.0" (10 add 10)
    (<edges.PartialConnection>, <ArgsPack(*(20.0,), **{})>),
)


stepper.step()
(
    # Previous step complete; input(10), add(10), then add(20)
    (<Unit(func=P_add_30.0)>, <ArgsPack(*(40.0,), **{})>),
)

We initiated a stepper at our preferred node stepper = g.stepper(first_connection_first_node, 10). Any subsequent stepper.step() calls push the stepper to the next execution step.

Each iteration returns the next thing to perform and the values from the previous unit call.

# Many (1) rows to call next.
(
    (<Unit(func=P_add_30.0)>, <ArgsPack(*(40.0,), **{})>),
)

We see one row, with f.add_30 as the next function to call.

The stepper can run once (allowing us to loop it), or we can use the built-in run_stepper function, to walk the nodes until the chain is complete

from hyperway.graph import Graph
from hyperway.tools import factory as f

from hyperway.packer import argspack
from hyperway.stepper import run_stepper


g = Graph()
connections = g.connect(f.add_10, f.add_20, f.add_30)

# run until exhausted
result = run_stepper(g, connections[0].a, argspack(10))

The value of the stepper is concurrent. When a path ends, the value is stored in the stepper.stash. When executing node steps, the result from the call is given to the next connected unit.

If two nodes call to the same destination node, this causes two calls of the next node:

           +4
i  +2  +3       print
           +5

With this layout, the print function will be called twice by the +4 and +5 node. Two calls occur:

          10
 1  3  6      print
          11

# Two results
print(10)
print(11)

This is because there are two connections to the print node, causing two calls.

Use merge nodes to action one call to a node, with two results.

1. Set merge_node=True on target node
2. Flag concat_aware=True on the stepper

g = Graph()
u = as_unit(print)
u.merge_node = True

s = g.stepper()
s.concat_aware = True

s.step()
...

When processing a print merge-node, one call is executed when events occur through multiple connections during one step:

         10
1  3  6      print
         11

print(10, 11) # resultant

A path defines the flow of a stepper through a single processing chain. A function connected to more than one function will fork the stepper and produce a result per connection.

For example a graph with a a split path will yield two results:

from hyperway import Graph, as_units
from hyperway.tools import factory as f

g = Graph()

split, join = as_units(f.add_2, f.add_2)

cs = g.connect(f.add_1, split)
g.connect(split, f.add_3, join)
g.connect(split, f.add_4, join)
g.connect(join, f.add_1)

If graphviz is installed, The graph can be rendered with graph.write():

# Continued from above
g.write('double-split', direction='LR')

Connecting nodes will grow the result count. For example creating in two exits nodes will double the result count

To model this, we can extend the above code with an extra connection: g.connect(join, f.sub_1):

# same as above
from hyperway import Graph, as_units
from hyperway.tools import factory as f

g = Graph()

split, join = as_units(f.add_2, f.add_2)

cs = g.connect(f.add_1, split)
g.connect(split, f.add_3, join)
g.connect(split, f.add_4, join)
g.connect(join, f.add_1)

# connect another function
g.connect(join, f.sub_1)

g.write('double-double-split', direction='LR')

Wire functions have been removed for clarity

from hyperway import Graph, as_unit

g = Graph()

split = as_unit(f.add_2, name='split')
join_a = as_unit(f.add_2)

# sum([1,2,3]) accepts an array - so we merge the args for the response.
join = as_unit(lambda *x: sum(x) name='sum')

cs = g.connect(f.add_1, split)
g.connect(split, f.add_3, join)
g.connect(split, f.add_4, join)
g.connect(split, f.add_5, join)

g.connect(join, f.add_1)
g.connect(join, f.sub_1)
g.connect(join, f.sub_2)
g.write('triple-split', direction='LR')

The order of precedence for operations occurs through sequential evaluation (from left to right) similar to C. Each operation is executed as it is encountered, without regard to the traditional precedence of operators.

• IBM z/OS Precedence and associativity
• lagrammar.net
• Complexity in left-associative grammar

# BODMAS computes * first
 1 + 1 * 2 + 2 == 5
10 + 1 * 2 + 2 == 14

# left-to-right computes linearly
( (1 + 1)  * 2) + 2 == 6
( (10 + 1) * 2) + 2 == 24

# example sequential evaluation
from hyperway.tools import factory as f
from hyperway.edges import make_edge, wire

c = make_edge(f.add_1, f.add_2, through=wire(f.mul_2))

assert c.pluck(1) == 10 # (1 + 1) * 2 + 2
assert c.pluck(10) == 24 # (10 + 1) * 2 + 2

• Graph: The Graph is a thin and dumb dictionary, maintaining a list of connections per node.
• Node: The Node is also very terse, fundamentally acting as a thin wrapper around the user given function, and exposes a few methods for on-graph executions.
• Edges: Edges or Connections are the primary focus of this version, where a single Connection is bound to two nodes, and maintains a wire-function.
• Stepper: The Stepper performs much of the work for interacting with Nodes on a graph through Edges.

We push Node to Node Connections into the Graph dictionary. The Connection knows A, B, and potentially a Wire function.

When running the graph we use a Stepper to processes each node step during iteration, collecting results of each call, and the next executions to perform.

We walk through the graph using a Stepper. Upon a step we call any rows of waiting callables. This may be the users first input and will yield next callers and the result.

The Stepper should call each next caller with the given result. Each caller will return next callers and a result for the Stepper to call again.

In each iteration the callable resolves one or more connections. If no connections return for a node, The execution chain is considered complete.

The Graph is purposefully terse. Its build to be as minimal as possible for the task. In the raw solution the Graph is a defaultdict(tuple) with a few additional functions for node acquisition.

The graph maintains a list of ID to Connection set.

{
  ID: (
        Connection(to=ID2),
    ),
  ID2: (
        Connection(to=ID),
    )
}

A Connection bind two functions and an optional wire function.

A -> [W] -> B

When executing the connection, input starts through A, and returns through B. If the wire function exists it may alter the value before B receives its input values.

A Unit represents a thing on the graph, bound to other units through connections.

def callable_func(value):
    return value * 3

as_unit(callable_func)

A unit is one reference

unit = as_unit(callable_func)
unit2 = as_unit(callable_func)

assert unit != unit2

The argspack simplifies the movement of arguments and keyword arguments for a function.

we can wrap the result as a pack, always ensuring its unpackable when required.

akw = argswrap(100)
akw.a
(100, )

akw = argswrap(foo=1)
akw.kw
{ 'foo': 1 }

Although many existing libraries cater to graph theory, they often require a deep understanding of complex terminology and concepts. As an engineer without formal training in graph theory, these libraries are challenging. Hyperway is the result of few years of research aimed at developing a simplified functional graph-based execution library with a minimal set of core features.

I'm slowly updating it to include the more advanced future features, such as hyper-edges and connection-decisions, bridging the gap between academic graph theory and practical application, providing developers with a low-level runtime that facilitates functional execution without the need for specialized knowledge.

The core components of Hyperway were developed through more than 40 rewrites and refinements, leading to a robust framework that supports both procedural and parallel functionality with the same homogeneous interface.

Lessons Learned

Each restart was a lesson in what not-to-do. I have fundamentally 25 restart/rewrites. Roughly three of them I'd consider "complete" enough to function within a range of distinct cases, such as "electronic circuit", "logic gates", "function chains" and a bunch of other variants.

1. API/Libraries Servers/Websites/Render-loops - bin-em First version was an all-singing socketed executor. Which was crap to maintain.
2. Let's try plugging-up Devices and pipes It works, But then I also have plugin functions, on method, mixins and all sorts of 'library' bits pushing around objects. It's too limited.
3. The same thing; but with a "system" running a super-loop of futures Sure it works, but now we have an asyncio execution machine, with a bunch of mixins, structures classes, and a specific run function. Entities are too classy, with a large unwanted method stack - more simpler is required

• https://graphclasses.org/
• https://houseofgraphs.org/
• https://en.wikipedia.org/wiki/Cyber%E2%80%93physical_system
• https://en.wikipedia.org/wiki/Signal-flow_graph
• https://resources.wolframcloud.com/FunctionRepository/resources/HypergraphPlot
• https://www.lancaster.ac.uk/stor-i-student-sites/katie-howgate/2021/04/29/hypergraphs-not-just-a-cool-name/

• (TensorFlow: Introduction to Graph and tf.function)[https://www.tensorflow.org/guide/intro_to_graphs]
• (Wiki: Extract, transform, load)[https://en.wikipedia.org/wiki/Extract,_transform,_load]
• https://docs.dgl.ai/en/2.0.x/notebooks/sparse/hgnn.html | https://github.com/iMoonLab/DeepHypergraph/blob/main/README.md
• https://distill.pub/2021/gnn-intro/
• https://renzoangles.net/gdm/

• (Graphtik)[https://graphtik.readthedocs.io/en/latest/]
• (iGraph)[https://python.igraph.org/en/stable/index.html#]
• (graph-tool)[https://graph-tool.skewed.de/]
• (pyDot)[https://github.com/pydot/pydot]
• (FreExGraph)[https://github.com/FreeYourSoul/FreExGraph]
• (NetworkX)[https://networkx.org/documentation/latest/index.html]

• https://graphviz.org/