• About the project
• Architecture
  • Ingress
  • Interpretation
  • Indexing
• Structure
• How to build the project
• Codestyle and tools recommendations

Pipelines for data processing and indexing of biodiversity data

Status: WIP

Vision: Consistent data processing pipelines (parsing, interpretation and quality flagging) for use in GBIF, the Living Atlases project and beyond. Built to scale from laptop to GBIF volumes. Deployable on JVM, Spark, Google Cloud, .

The project provides vanilla JVM-based parsing and interpretation libraries, and pipelines for indexing into SOLR and ElasticSearch, built using Apache Beam.

Apache Beam provides a high level abstraction framework ideal for this purpose with the ability to deploy across target environments (e.g. Spark, local JVM) and with many built in connectors for e.g. HBase, SOLR, ElasticSearch etc.

Ingress is from Darwin Core Archive (zip files of one or more "CSV"s) or ABCD Archives (compressed XML) only[1]. During ingress data is converted from it's native format and stored as Avro files containing Darwin Core compliant data.

This is depicted below:

Avro is chosen as a storage and interchange format exclusively in this project because a) it is splittable with each split compressed independently, b) it holds the data schema with the data, c) is well supported in the Hadoop ecosystem (e.g. Hive, Spark) and many other tools (e.g. Google cloud) d) is very robust in serialization and e) reduces boiler plate code thanks to schema to code generators. Darwin Core Archives and JSON for example do not exhibit all these traits.

[1] Other protocols (e.g. DiGIR, TAPIR) are supported by GBIF but are converted by crawling code upstream of this project.

During interpretation the verbatim data is parsed, normalised and tested against quality control procedures.

To enable extensibility data is interpreted into separate avro files where a separate file per category of information is used. Many interpretations such as date / time formating is common to all deployments, but not all. For example, in the GBIF.org deployment the scientific identification is normalised to the GBIF backbone taxonomy and stored in /interpreted/taxonomy/interpreted*.avro which might not be applicable to everyone. Separating categories allows for extensibility for custom deployments in a reasonably modular fashion.

Interpretation is depicted below:

Note that all pipelines are designed and tested to run with the DirectRunner and the SparkRunner at a minimum. This allows the decision to be taken at runtime to e.g. opt to interpret a small dataset in the local JVM without needing to use cluster resources for small tasks.

It is a design decision to ensure that all the underlying parsers are as reusable as possible for other projects with careful consideration to not bring in dependencies such as Beam or Hadoop.

Initial implementations will be available for both SOLR and for ElasticSearch to allow for evaluation of both at GBIF. During indexing the categories of interpreted information of use are merged and loaded into the search indexes:

Note that GBIF target 10,000 records/sec per node indexing speed (i.e. 100,000 records/sec on current production cluster). This will allow simplified disaster recovery and rapid deployment and of new features.

The project is structured as:

• .buildSrc - Tools for building the project
• docs - Documents related to the project
• examples - Examples of using project API and base classes
  • transform - Transform example demonstrates how to create Apache Beam pipeline, create the new transformation and use it together with GBIF transforms and core classes
  • metrics - The example demonstrates how to create and send Apache Beam SparkRunner metrics to ELK and use the result for Kibana dashboards
• pipelines - Main pipelines module
  • common - Only static string variables
  • beam-common - Classes and API for using with Apache Beam
  • ingest-gbif - Main GBIF pipelines for ingestion biodiversity data
  • ingest-gbif-standalone - Independent GBIF pipelines for ingestion biodiversity data
  • ingest-transforms - Transformations for ingestion biodiversity data
• sdks - Main module contains common classes, such as data models, data format iterpretations, parsers, web services clients ant etc.
  • core - Main API classes, such as data interpretations, converters, DwCA reader and etc
  • models - Data models represented in Avro binary format, generated from Avro schemas
  • parsers - Data parsers and converters, mainly for internal usage inside of interpretations
• tools - Module for different independent tools
  • archives-converters - Converters from DwCA/DWC 1.0/DWC 1.4/ABCD 1.2/ABCD 2.06 to *.avro format
  • elasticsearch-tools - Tool for creating/deleting/swapping Elasticsearch indexes
  • pipelines-maven-plugin - Maven plugin adds new annotations and interface to avro generated classes

The project uses Apache Maven tool for building. Project contains maven wrapper and script for Linux and MacOS systems, you just can run build.sh script:

./build.sh

or

source build.sh

Please read Apache Maven how-to.

• Use Intellij IDEA COMMUNITY (or better)
• The project uses Project Lombok, please install Lombok plugin for Intellij IDEA.
• The simplest way to have uniform code style is to use the Google Java Format plugin.
• Because project uses Error-prone you may have issues during building process from IDEA, to avoid issues please install Error-prone compiler integration plugin and allows to build project using error-prone java compiler to catch common Java mistakes at compile-time. To use the compiler, go to File | Settings | Compiler | Java Compiler and select Javac with error-prone in Use compiler box.
• Add a custom parameter to avoid debug problem To use the compiler, go to File | Settings | Compiler | Java Compiler | Additional command line parameters and add -Xep:ParameterName:OFF