Skip to content

Latest commit

 

History

History

rstream-query

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

rstream-query

npm version npm downloads Twitter Follow

This project is part of the @thi.ng/umbrella monorepo.

About

@thi.ng/rstream based triple store & reactive query engine with declarative query specs related to Datalog / SPARQL. Inserted triples / facts are broadcast to multiple indexing streams and any query subscriptions attached to them. This enables push-based, auto-updating query results, which are changing each time upstream transformations & filters have been triggered.

Triples are 3-tuples of [subject, predicate, object]. Unlike with traditional RDF triple stores, any JS data types can be used as subject, predicate or object (though support for such must be explicitly enabled & this feature is currently WIP).

Current features

  • Dynamic & declarative dataflow graph construction via high-level data specs and/or functions
  • Entirely based on stream abstractions provided by @thi.ng/rstream
  • All data transformations done using dynamically composed tranducers
  • Query optimizations
  • Extensive re-use of existing sub-query results (via subscriptions)
  • Interim result de-duplication / dataflow gates
  • Push-based, auto-updating query results

Status

STABLE - used in production

Search or submit any issues for this package

This project is currently still in early development and intended as a continuation of the Clojure based thi.ng/fabric, this time built on the streaming primitives provided by @thi.ng/rstream.

Installation

yarn add @thi.ng/rstream-query

ES module import:

<script type="module" src="https://cdn.skypack.dev/@thi.ng/rstream-query"></script>

Skypack documentation

For Node.js REPL:

# with flag only for < v16
node --experimental-repl-await

> const rstreamQuery = await import("@thi.ng/rstream-query");

Package sizes (gzipped, pre-treeshake): ESM: 2.83 KB

Dependencies

Usage examples

Several demos in this repo's /examples directory are using this package.

A selection:

Screenshot Description Live demo Source
Triple store query results & sortable table Demo Source

API

Generated API docs

import { TripleStore, asTriples } from "@thi.ng/rstream-query";
import { trace } from "@thi.ng/rstream";

// create store with initial set of triples / facts
const store = new TripleStore([
    ["london", "type", "city"],
    ["london", "part-of", "uk"],
    ["portland", "type", "city"],
    ["portland", "partOf", "oregon"],
    ["portland", "partOf", "usa"],
    ["oregon", "type", "state"],
    ["usa", "type", "country"],
    ["uk", "type", "country"],
]);

// alternatively, convert an object into a sequence of triples
const store = new TripleStore(asTriples({
    london: {
        type: "city",
        partOf: "uk"
    },
    portland: {
        type: "city",
        partOf: ["oregon", "usa"]
    },
    oregon: { type: "state" },
    uk: { type: "country" },
    usa: { type: "country" },
});

// compile the below query spec into a dataflow graph
// pattern items prefixed w/ "?" are query variables

// this query matches the following relationships
// using all currently known triples in the store
// when matching triples are added or removed, the query
// result updates automatically...

// currently only "where" and bounded "path" sub-queries are possible
// in the near future, more query types will be supported
// (e.g. optional relationships, pre/post filters etc.)
store.addQueryFromSpec({
    q: [
        {
            // all "where" subqueries are joined (logical AND)
            where: [
                // match any subject of type "city"
                ["?city", "type", "city"],
                // match each ?city var's "part-of" relationships (if any)
                ["?city", "partOf", "?country"],
                // matched ?country var must have type = "country"
                ["?country", "type", "country"]
            ]
        }
    ],
    // `bind` is an (optional) query post-processor and
    // allows injection of new variables into the result set
    // here we create a new var "answer" whose values are derived from
    // the other two query vars
    bind: {
        answer: (res) => `${res.city} is located in ${res.country}`
    },
    // another post-processing step, only keeps "answer" var in results
    select: ["answer"]
})
.subscribe(trace("results"))
// results Set {
//   { answer: 'london is located in uk' },
//   { answer: 'portland is located in usa' } }

// helper fn to insert new city relationship to the store
const addCity = (name, country) =>
    store.into([
        [name, "type", "city"],
        [name, "partOf", country],
        [country, "type", "country"],
    ]);

addCity("berlin", "germany");
// results Set {
//     { answer: 'london is located in uk' },
//     { answer: 'portland is located in usa' },
//     { answer: 'berlin is located in germany' } }

addCity("paris", "france");
// results Set {
//     { answer: 'london is located in uk' },
//     { answer: 'portland is located in usa' },
//     { answer: 'berlin is located in germany' },
//     { answer: 'paris is located in france' } }

After setting up the above query and its internal transformations, the generated dataflow topology then looks as follows:

graphviz output

  • The blue nodes are TripleStore-internal index stream sources, emitting changes when new triples are added
  • The left set of red nodes are the sub-queries of the above where clause, responsible for joining the individual (S)ubject, (P)redicate and (O)bject sub-queries.
  • The results of these are then further joined (right red node) & transformed to produce the final solution set and post-process it

Btw. The diagram has been generated using @thi.ng/rstream-dot and can be recreated by calling store.toDot() (for the above example)

The source code for the above example is here

(Many) more features forthcoming...

Authors

Karsten Schmidt

If this project contributes to an academic publication, please cite it as:

@misc{thing-rstream-query,
  title = "@thi.ng/rstream-query",
  author = "Karsten Schmidt",
  note = "https://thi.ng/rstream-query",
  year = 2018
}

License

© 2018 - 2021 Karsten Schmidt // Apache Software License 2.0