Turbo-encabulate more documentation pages (#4863)

Just another of those docs PRs that help streamline our docs.
tenzir · Dec 16, 2024 · 4a6dd99 · 4a6dd99
2 parents 4bf7bf5 + ec59b1d
commit 4a6dd99
Show file tree

Hide file tree

Showing 62 changed files with 545 additions and 561 deletions.
diff --git a/web/docs/cribl-vs-tenzir-dataflow.excalidraw.svg b/web/docs/cribl-vs-tenzir-dataflow.excalidraw.svg
diff --git a/web/docs/cribl-vs-tenzir-pipelines.excalidraw.svg b/web/docs/cribl-vs-tenzir-pipelines.excalidraw.svg
diff --git a/web/docs/database-architecture.excalidraw.svg b/web/docs/database-architecture.excalidraw.svg
diff --git a/web/docs/faqs.md → web/docs/faqs/README.md b/web/docs/faqs.md → web/docs/faqs/README.md
@@ -205,7 +205,7 @@ what level it should be.
 
 2. **Format**. If your *X* is a wire format, either text-based like JSON or
    binary like PCAP, then look for `read_*` and `write_*`
-   [operators](tql2/operators.md)
+   [operators](tql2/operators.md).
 
 3. **Fluent Bit**. Tenzir ships with all of Fluent Bit's
    [inputs][fluentbit-inputs] and [outputs][fluentbit-outputs], since the Fluent

diff --git a/web/docs/open-vs-closed-source.svg → web/docs/faqs/open-vs-closed-source.svg b/web/docs/open-vs-closed-source.svg → web/docs/faqs/open-vs-closed-source.svg
diff --git a/web/docs/platform-connections.svg → web/docs/faqs/platform-connections.svg b/web/docs/platform-connections.svg → web/docs/faqs/platform-connections.svg
diff --git a/web/docs/glossary.md b/web/docs/glossary.md
@@ -72,6 +72,13 @@ enrichment.
 - [Manage](./operators/context.md) a context
 - [Enrich](./operators/enrich.md) with a context
 
+## Destination
+
+An pipeline ending with an [output](#output) operator preceded by a
+[`subscribe`](tql2/operators/subscribe.md) input operator.
+
+- Learn more about [pipelines](pipelines/README.md)
+
 ## Format
 
 Translates between bytes and events.
@@ -91,6 +98,12 @@ Tenzir has *sparse* indexes. Sparse indexes live in memory and point to
 
 - [Configure the catalog](./installation/tune-performance/README.md#configure-the-catalog)
 
+## Input
+
+An [operator](#operator) that only producing data, without consuming anything.
+
+- Learn more about [pipelines](pipelines/README.md)
+
 ## Integration
 
 A set of pipelines to integrate with a third-party product.
@@ -117,7 +130,7 @@ side effect that acquires bytes. Use a loader implicitly with the
 [`from`](tql2/operators/from.md) operator or explicitly with the `load_*`
 operators.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
 
 ## Node
 
@@ -149,10 +162,16 @@ contains packages that map data sources to OCSF.
 
 The building block of a [pipeline](#pipeline).
 
-An operator is a [source](#source), [transformation](#transformation), or
-[sink](#sink).
+An operator is a [input](#input), [transformation](#transformation), or
+[output](#output).
 
-- See all available [operators](./operators.md)
+- See all available [operators](./tql2/operators.md)
+
+## Output
+
+An [operator](#operator) consuming data, without producing anything.
+
+- Learn more about [pipelines](pipelines/README.md)
 
 ## PaC
 
@@ -180,7 +199,7 @@ A parser is the dual to a [printer](#printer). Use a parser in the
 can use the [`parse`](./operators/parse.md) operator to parse a single field
 with a parser.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
 - See [all formats](./formats.md)
 
 ## Partition
@@ -202,7 +221,7 @@ formats are [Parquet](https://parquet.apache.org) or
 
 Combines a set of [operators](#operator) into a dataflow graph.
 
-- Understand [how pipelines work](./pipelines.md)
+- Understand [how pipelines work](pipelines/README.md)
 - Understand the [pipeline language](./language.md)
 - [Run a pipeline](./usage/run-pipelines/README.md)
 
@@ -218,7 +237,7 @@ A [format](#format) that translates events into bytes.
 A printer is the dual to a [parser](#parser). Use a parser in the
 [`to`](./operators/to.md) or [`write`](./operators/write.md) operators.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
 - See [all formats](./formats.md)
 
 ## Saver
@@ -230,25 +249,20 @@ a side effect that emits bytes. Use a saver implicitly with the
 [`to`](tql2/operators/to.md) operator or explicitly with the `save_*`
 operators.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
 
 ## Schema
 
 A top-level record type of an event.
 
 - [Show available schemas](./usage/show-available-schemas.md)
 
-## Sink
-
-An [operator](#operator) consuming input, without producing any output.
-
-- Learn more about [pipelines](./pipelines.md)
-
 ## Source
 
-An [operator](#operator) producing output, without consuming any input.
+An pipeline starting with an [input](#input) operator followed by a
+[`publish`](tql2/operators/publish.md) output operator.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
 
 ## TQL
 
@@ -262,4 +276,4 @@ TQL is the language in which users write [pipelines](#pipeline).
 
 An [operator](#operator) consuming both input and producing output.
 
-- Learn more about [pipelines](./pipelines.md)
+- Learn more about [pipelines](pipelines/README.md)
diff --git a/web/docs/implicit-vs-explicit-networking.excalidraw.svg b/web/docs/implicit-vs-explicit-networking.excalidraw.svg
diff --git a/web/docs/integrations.md b/web/docs/integrations.md
@@ -4,8 +4,8 @@ An *integration* is a set of example pipelines to interface with an external
 product. The objective of most integrations is to establish dataflow from and/or
 to Tenzir using pipelines to implement common use cases.
 
-Please read our [FAQ entry](faqs.md#do-you-have-an-integration-for-x) on how to
-proceed if you cannot find the integration that you are looking for.
+Please read our [FAQ entry](faqs/README.md#do-you-have-an-integration-for-x) on
+how to proceed if you cannot find the integration that you are looking for.
 
 import DocCardList from '@theme/DocCardList';
 

diff --git a/web/docs/integrations/ftp/README.md b/web/docs/integrations/ftp/README.md
@@ -11,7 +11,7 @@ connection. This can be tricky for some firewalls and may require special
 attention.
 
 :::tip URL Support
-The URL schemes `ftp://` and `ftps://` dispatche to
+The URL schemes `ftp://` and `ftps://` dispatch to
 [`load_ftp`](../../tql2/operators/load_ftp.md) and
 [`save_ftp`](../../tql2/operators/save_ftp.md) for seamless URL-style use via
 [`from`](../../tql2/operators/from.md) and [`to`](../../tql2/operators/to.md).

diff --git a/web/docs/integrations/http/README.md b/web/docs/integrations/http/README.md
@@ -23,7 +23,8 @@ The URL schemes `http://` and `https://` dispatch to
 
 Since the majority of HTTP activity uses JSON-encoded request/response
 bodies, you do not need to provide a separate pipeline argument with
-`read_json`/`write_json`.
+[`read_json`](../../tql2/operators/read_json.md) and
+[`write_ndjson`](../../tql2/operators/write_ndjson.md).
 :::
 
 ## Examples
@@ -37,7 +38,7 @@ from "http://example.com:8888/api", method="GET", params={query: "tenzir"}
 ### Perform a POST request with JSON body
 
 ```tql
-from "http://example.com:8888/api", method="POST", body={query: "tenzir"}
+from "http://example.com:8888/api", method="POST", data={query: "tenzir"}
 ```
 
 ### Call a webhook API with pipeline data

diff --git a/web/docs/language.md b/web/docs/language.md
@@ -1,7 +1,7 @@
 # Language
 
 The **Tenzir Query Language (TQL)** centers around one core principle: dataflow
-[pipelines](pipelines.md) that consist of a sequence of operators, of which some
+pipelines that consist of a sequence of operators, of which some
 evaluate [expressions](language/expressions.md).
 
 import DocCardList from '@theme/DocCardList';

diff --git a/web/docs/multi-schema-example.excalidraw.svg b/web/docs/multi-schema-example.excalidraw.svg
diff --git a/web/docs/operator-pieces.excalidraw.svg b/web/docs/operator-pieces.excalidraw.svg
diff --git a/web/docs/operator-types.excalidraw.svg b/web/docs/operator-types.excalidraw.svg
diff --git a/web/docs/operators.md b/web/docs/operators.md
@@ -1,24 +1,13 @@
 # Operators
 
-An *operator* is the execution building block of a [pipeline](pipelines.md).
+An *operator* is the execution building block of a pipeline.
 
 There exist three primary types:
 
 1. **Source**: produces data
 2. **Sink**: consumes data
 3. **Transformation**: produces and consumes data
 
-Every operator have an *input type* and *output type*:
-
-![Operator Types](operator-types.excalidraw.svg)
-
-Operators can be *polymorphic* in that they can have more than a single input
-and output type. For example, `head` accepts both `bytes` and `events`,
-filtering either the first N bytes or events.
-
-Additionally, [user-defined operators](language/user-defined-operators.md) add
-an alias for a pipeline to make it usable as an operator.
-
 import DocCardList from '@theme/DocCardList';
 
 <DocCardList />
diff --git a/web/docs/packages.md b/web/docs/packages.md
@@ -4,7 +4,8 @@ A **package** is a set of pipelines and contexts that are thematically related
 and deployed together as a single unit.
 
 :::tip Use packages
-Start using packages by [installing one](installation/install-a-package.md).
+Get instant gratification by [installing a
+package](installation/install-a-package.md).
 :::
 
 ## Anatomy of a Package

diff --git a/web/docs/pipeline-chaining.excalidraw.svg b/web/docs/pipeline-chaining.excalidraw.svg
diff --git a/web/docs/pipeline-structure.excalidraw.svg b/web/docs/pipeline-structure.excalidraw.svg